Scrim UI

Output Comparison UI

Two answers, side by side, with the model names hidden until a winner is picked — because a visible label is the thing being measured.

feedbackcomparisonab-testeval
npx shadcn@latest add https://scrimui.dev/r/output-comparison.json
Agent promptClaude Code · Cursor · any agent
Add the Output Comparison UI component from the Scrim UI registry to this project and use it in the configuration described below.

Output Comparison UI — Two answers, side by side, with the model names hidden until a winner is picked — because a visible label is the thing being measured.

## 1. Install

```bash
npx shadcn@latest add https://scrimui.dev/r/output-comparison.json
```

This writes a single file to `components/ui/output-comparison.tsx`. It is plain React + Tailwind with no runtime dependencies — no Radix, no CVA, nothing to add to package.json. If this project does not use the shadcn CLI, copy the source from https://scrimui.dev/components/output-comparison to the same path by hand; nothing in the file depends on shadcn.

## 2. Use it

```tsx
import { OutputComparison } from "@/components/ui/output-comparison";

// Randomise the slots per comparison and record which model landed in each.
// Left wins more than right, reliably, and no component can fix that for you.
const [a, b] = shuffle([sonnet, gpt]);

<OutputComparison
  prompt={prompt}
  a={a}
  b={b}
  onChoose={(choice, winnerId) => recordPreference(pairId, choice, winnerId)}
/>
```

## 3. The configuration that matters

This is the component's "Blind" state — Names hidden. Show them and you stop measuring the answers and start measuring what the reader already believes about the models.

Every prop is at its default value. Keep the call site minimal — do not write out default values.

Reference: https://scrimui.dev/components/output-comparison

Follows the props below — change a control and the prompt changes with it, so an agent reproduces that configuration instead of the defaults.

Explain why my RAG answers cite the right document but the wrong paragraph.

Ahidden until you choose

Almost always the offsets, not the retrieval. The chunk that matched is correct; what is lost is where it sat in the source. If chunks are stored as detached strings, the only thing left to cite with is the document id, so every citation from that document resolves to the same place. Store start and end offsets with each chunk and assert the slice invariant in development.

Bhidden until you choose

This is a common issue with RAG pipelines. There are several possible causes: 1. Chunking strategy — your chunks may be too large or too small. 2. Embedding quality — consider a different embedding model. 3. Retrieval parameters — try adjusting top-k. 4. Re-ranking — a cross-encoder often improves precision.

Randomise which model is A on every comparison

Presets

Props

Component source

Single-file React + Tailwind component. No dependencies — drop it into any project with Tailwind configured.

output-comparison.tsx
"use client";

import * as React from "react";

/**
 * Two answers, side by side, pick the better one.
 *
 * The interface is trivial and the measurement is fragile, which is the
 * opposite of how it looks. Three biases will eat the data before anyone
 * reads it:
 *
 * **Label bias.** Show the model names and you are no longer measuring the
 * answers. People know which model they expect to be better and they are
 * right often enough that the belief survives contact with the evidence. So
 * the names are hidden until a choice is made, and revealed straight after —
 * concealing them permanently just makes the tool feel like it is hiding
 * something.
 *
 * **Position bias.** Left wins more than right, reliably, for no reason
 * anyone likes. The fix is not in this component: **randomise which candidate
 * you pass as `a` on every comparison**, and record which model actually sat
 * in each slot. A component cannot do that for you — it only ever sees one
 * pair — so it says so here rather than pretending the problem is solved.
 *
 * **Length bias.** Longer answers win, and a pane that grows with its content
 * advertises which one is longer before a word is read. Both panes are the
 * same height and scroll independently, which does not remove the bias but
 * stops the layout from amplifying it.
 *
 * And `tie` is a first-class answer. Forcing a winner out of two
 * indistinguishable outputs manufactures signal, and manufactured signal is
 * worse than a smaller sample — it is confidently wrong in whichever
 * direction the position bias points.
 */

export type Candidate = {
  id: string;
  /** Revealed only after a choice. */
  model: string;
  text: string;
};

export type Choice = "a" | "b" | "tie";

export type OutputComparisonProps = {
  prompt?: string;
  a: Candidate;
  b: Candidate;
  /** The recorded choice, if any. */
  choice?: Choice;
  /** Force the labels visible. Defaults to "once a choice exists". */
  revealed?: boolean;
  onChoose?: (choice: Choice, winnerId: string | undefined) => void;
  className?: string;
};

/* ------------------------------------------------------------------ */
/* Icons                                                               */
/* ------------------------------------------------------------------ */

function CheckIcon(props: React.SVGProps<SVGSVGElement>) {
  return (
    <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2.5" strokeLinecap="round" strokeLinejoin="round" width="12" height="12" {...props}>
      <path d="M20 6 9 17l-5-5" />
    </svg>
  );
}

/* ------------------------------------------------------------------ */
/* Pane                                                                */
/* ------------------------------------------------------------------ */

function Pane({
  slot,
  candidate,
  revealed,
  won,
  decided,
}: {
  slot: "A" | "B";
  candidate: Candidate;
  revealed: boolean;
  won: boolean;
  decided: boolean;
}) {
  return (
    <div
      className={`flex min-w-0 flex-col rounded-xl border transition-colors ${
        won
          ? "border-emerald-300 bg-emerald-50/40 dark:border-emerald-800 dark:bg-emerald-900/10"
          : decided
            ? "border-zinc-200 opacity-60 dark:border-zinc-800"
            : "border-zinc-200 dark:border-zinc-800"
      }`}
    >
      <div className="flex items-center gap-2 border-b border-zinc-100 px-3 py-2 dark:border-zinc-800">
        <span className="flex h-5 w-5 shrink-0 items-center justify-center rounded-md bg-zinc-100 text-[10px] font-semibold text-zinc-600 dark:bg-zinc-800 dark:text-zinc-300">
          {slot}
        </span>
        <span className="min-w-0 truncate font-mono text-[11px] text-zinc-500 dark:text-zinc-400">
          {revealed ? candidate.model : "hidden until you choose"}
        </span>
        {won && (
          <span className="ml-auto inline-flex shrink-0 items-center gap-1 text-[11px] font-medium text-emerald-700 dark:text-emerald-400">
            <CheckIcon width="11" height="11" />
            Preferred
          </span>
        )}
      </div>
      {/* Fixed height, independent scroll. A pane that grows with its content
          announces which answer is longer before either has been read. */}
      <div className="h-52 overflow-y-auto px-3 py-2.5 text-[13px] leading-6 text-zinc-700 dark:text-zinc-200">
        <p className="whitespace-pre-wrap">{candidate.text}</p>
      </div>
    </div>
  );
}

/* ------------------------------------------------------------------ */
/* OutputComparison                                                    */
/* ------------------------------------------------------------------ */

export function OutputComparison({
  prompt,
  a,
  b,
  choice,
  revealed,
  onChoose,
  className = "",
}: OutputComparisonProps) {
  const decided = choice !== undefined;
  const showLabels = revealed ?? decided;

  const buttons: { value: Choice; label: string }[] = [
    { value: "a", label: "A is better" },
    { value: "tie", label: "Tie" },
    { value: "b", label: "B is better" },
  ];

  return (
    <div className={className}>
      {prompt && (
        <p className="mb-3 rounded-lg bg-zinc-100 px-3 py-2 text-[13px] leading-6 text-zinc-700 dark:bg-zinc-800/60 dark:text-zinc-200">
          {prompt}
        </p>
      )}

      <div className="grid gap-3 sm:grid-cols-2">
        <Pane slot="A" candidate={a} revealed={showLabels} won={choice === "a"} decided={decided} />
        <Pane slot="B" candidate={b} revealed={showLabels} won={choice === "b"} decided={decided} />
      </div>

      <div className="mt-3 flex flex-wrap items-center gap-2">
        {buttons.map((btn) => {
          const active = choice === btn.value;
          return (
            <button
              key={btn.value}
              type="button"
              aria-pressed={active}
              onClick={() =>
                onChoose?.(btn.value, btn.value === "a" ? a.id : btn.value === "b" ? b.id : undefined)
              }
              className={`inline-flex h-8 items-center gap-1.5 rounded-lg border px-3.5 text-xs font-medium transition-colors ${
                active
                  ? "border-zinc-900 bg-zinc-900 text-white dark:border-zinc-100 dark:bg-zinc-100 dark:text-zinc-900"
                  : "border-zinc-200 text-zinc-600 hover:bg-zinc-100 dark:border-zinc-700 dark:text-zinc-300 dark:hover:bg-zinc-800"
              }`}
            >
              {active && <CheckIcon width="11" height="11" />}
              {btn.label}
            </button>
          );
        })}

        {!decided && (
          <span className="ml-auto text-[11px] text-zinc-400 dark:text-zinc-500">
            Randomise which model is A on every comparison
          </span>
        )}
        {choice === "tie" && (
          <span className="ml-auto text-[11px] text-zinc-500 dark:text-zinc-400">
            Recorded as a tie — better than a coin flip you cannot see later
          </span>
        )}
      </div>
    </div>
  );
}

When to use it

  • Randomise which model you pass as `a` on every comparison, and record which one sat in each slot. Left wins more than right, reliably, and this is the only place that can be fixed.
  • Keep the names hidden until a choice exists. A visible label is the belief being measured, not the answer.
  • Reveal them straight after. Permanent concealment reads as the tool having something to hide, and costs you the reviewer's trust for nothing.
  • Record ties. Two indistinguishable answers are a real finding, and the alternative is a coin flip you cannot identify later.
  • Store the prompt, both outputs and both model versions with the judgement — a preference with no idea what produced it cannot be re-analysed when the next version ships.
  • Give both panes the same fixed height. It does not remove length bias, but it stops the layout from announcing which answer is longer before either has been read.

What breaks in production

  • Showing the model names. The single most effective way to collect a few hundred judgements that measure reputation rather than output.
  • Fixing the slots — one model always on the left. Position bias then rides along inside every result and is impossible to subtract afterwards.
  • Removing the tie button to force a signal. Manufactured signal is confidently wrong in whichever direction the position bias points.
  • Letting the panes size to their content. The longer answer is visible as a taller box from across the room, and long answers already win more than they should.
  • Recording only the winner's model name. Without the pair and the slot assignment, the judgement cannot be audited or re-used.

Related Components

Response Rating

Thumbs up and down that ask the one follow-up question worth asking — with a state for feedback that has been sent and cannot be taken back.

Inline Correction

Fix the answer where it is wrong, in place — the edit becomes training data, so the component keeps the original alongside it rather than overwriting it.

Eval Results

Pass rates per test case across two runs, with the regressions surfaced first and a sample size honest enough to say when a delta means nothing.

Edit Diff View

A streamed AI edit as hunks you accept or reject one by one — id-keyed decisions, word-level marks, and buttons that stay disabled while a hunk is still arriving.