Building This Site: How We Keep a Diff Honest

A diff — the kind you'd see in a code review — marks what changed with a plus or a minus, and nothing else. It doesn't editorialize about which side is "right." That restraint is exactly why this site borrowed the visual language of a diff for its comparison tables, and it's also the hardest part of the whole design to actually get right, because the pull toward declaring a winner is constant and the visual format makes it tempting to give in.

Why a diff, specifically, as the design language

Every comparison this site publishes — one model against another, one service tier against another, a subscription plan against metered usage — has the same underlying shape: two things, several shared attributes, and a reader trying to see where they diverge. A diff is the most honest visual metaphor for that shape this site could find, because a diff's plus and minus marks aren't a verdict, they're a description of difference. Marking "this side is cheaper here" is a statement of fact about one row; it says nothing about whether that row should decide the reader's choice.

The mistake this design almost shipped with

Early in building the cross-provider cost comparison — the tool most likely to get this wrong, since it's directly comparing money against money — the left and right sides of the comparison had their favorable-side logic inverted. The tool was correctly computing both totals, but marking the more expensive side as the "cheaper" one in the visual gutter. Every number displayed was accurate; the framing around the numbers was backwards. This is precisely the kind of bug that's invisible to a purely numeric test — the totals were right, so a test checking totals would have passed — and it was only caught by looking at an actual screenshot of the rendered page and noticing the colors didn't match the numbers.

Why that bug is now a permanent lesson in how this site builds comparisons

The fix wasn't just correcting the inverted logic — it was extracting the comparison logic into its own tested module, separate from the display component, so the "which side is cheaper" determination has its own unit tests independent of whatever renders it. A rendering bug and a logic bug look identical to a user staring at a wrong-looking page; separating them makes each one independently checkable, and makes a future change to the display less likely to silently break the underlying comparison logic again.

The discipline of never declaring a universal winner

Beyond the specific bug, there's a standing design rule this site holds itself to across every comparison it publishes: a tool prices a specific workload, at specific settings, and reports which side is cheaper for that specific input — never "which provider is better," a claim this site has no basis to make and would be lying by omission to assert. Change the token counts, the model, the service tier, and the answer can flip. A comparison tool that hid that volatility behind a single fixed verdict would be more satisfying to read and less honest to trust.

Why this is harder than it sounds to maintain

It would be easy, and probably better for a casual reader's immediate satisfaction, to add a bold headline verdict above every comparison table — "X is cheaper for most users" — and let the detailed table serve as supporting evidence underneath. This site deliberately doesn't do that, because "most users" is exactly the kind of claim this site has no genuine basis to make; it would require a model of typical usage patterns this site doesn't have and isn't in a position to assert. The discipline is resisting a design that would read better in exchange for a design that stays honest about what it actually knows.

What "keeping a diff honest" means in practice, day to day

Every new comparison feature added to this site gets checked against the same standard the original bug revealed the hard way: does the visual signal match the underlying computed result, checked by an actual screenshot, not just a passing test? Is the comparison logic testable independent of its display? And does the page's language anywhere imply a verdict this site doesn't actually have grounds to make? Those three questions, asked on every new page before it ships, are the concrete version of a principle that would otherwise stay an abstract nice-sounding idea.

Why this matters beyond aesthetics

A reader trusting this site's comparisons is trusting that the plus and minus marks mean what they claim to mean. The day that trust turns out to be misplaced — a comparison that quietly favored one side regardless of the actual numbers — is the day this site stops being useful for the one thing it exists to do. Catching that kind of bug before it ships, and building the habits that make it less likely to recur, is not a footnote to this site's actual work. It is this site's actual work.

Why a screenshot caught what a passing test suite didn't

It's worth sitting with the specific way this bug was found, because it's a useful lesson about the limits of automated testing generally. A unit test checking "does this function return the correct total for these inputs" would have passed on the buggy version — the totals really were correct. What was wrong was a separate piece of logic, deciding which side to visually mark as favorable, and no test had been written to check that decision independently, because it hadn't occurred to anyone that the two could diverge. Only looking at an actual rendered screenshot — the thing a real user would actually see — surfaced the mismatch between correct numbers and an incorrect visual signal layered on top of them.

Building verification for the thing users actually experience

The practical change this produced wasn't just fixing the one bug — it was recognizing that "the underlying computation is correct" and "the page a user sees is correct" are two different claims requiring two different kinds of verification, and a codebase can satisfy the first without satisfying the second. This site now treats a rendered screenshot check as a standing part of shipping any new comparison feature, not an occasional nice-to-have reserved for major releases — specifically because the bug that taught this lesson would have shipped to production invisibly without one.

The uncomfortable part of telling this story publicly

It would be simpler to describe this site's comparison tools as having been correct from the start and leave the inverted-logic bug out of the account entirely — nobody would know to ask. Telling it anyway is a deliberate choice, for the same reason the corrections log elsewhere on this site exists: a site that only ever describes its own processes as having worked perfectly the first time isn't actually more trustworthy than one that shows its mistakes, it's just less honest about the difference.