Tools fact-check methodology

The whole system rests on the loop, not on promises:

1. 1

   Tools ship with their fact-check

   Every published tool that ships modeled data or makes verifiable claims gets a per-tool report at /tools/<slug>/fact-check.

2. 2

   Users verify by clicking sources

   Every claim has at least one URL. A user who wants to know 'is this threshold still correct?' clicks through to the primary source and sees the answer directly — not as our claim, as the source's claim.

3. 3

   Users who find issues tell us

   Every report ends with a contact link: if a modeled threshold doesn't match the statute, or a citation has changed since the verification date, that's a single click to flag. One careful user catches what we missed.

4. 4

   We re-check what's flagged

   And update the tool's data + methodology + report, advancing the verification date.

It is: a reproducible, dated, source-anchored check that each verifiable claim in a tool's UI, methodology, and modeled-data layer holds up against authoritative sources at the time of verification.

It isn't: UX review, design critique, agreement with the tool's scope choices, opinion review, or endorsement of recommendations. Editorial judgment, UX quality, and factual accuracy are separate concerns; the fact-check addresses only the third.

Categories. Some apply to every tool; some only to certain domains. Each verifiable claim is categorized under one of these. Modeled-data is the tool-specific category that doesn't exist on the article side.

Tools that compute results carry modeled-data thresholds: California's daily-OT trigger at 8 hours, federal weekly OT at 40 hours, the 30-day waiting-time-penalty cap. These flow into every user's calculation. A wrong constant ships wrong output to every user — orders of magnitude worse than a wrong sentence in an article.

The modeled-data verification step is unique to tools and load-bearing:

• Each constant maps to a statute clause. California doubleTimeThresholdHours: 12 maps to Cal. Lab. Code §510(a) “any work in excess of 12 hours in one day shall be compensated at the rate of no less than twice the regular rate of pay.” The mapping is recorded explicitly.
• The statute is verified character-by-character. The constant matches if the statute requires double-time strictly above 12, not ≥ 12. Sign / threshold / direction errors are the bug class this catches.
• Cross-verification preferred for high-stakes modeled data. Tier 1 primary statute text plus a second independent source (Cornell LII, Justia, agency interpretive guidance) for any threshold that flows into dollar amounts.

Unique to tools. The methodology page describes the rule in prose; the data layer implements it as a constant. They must say the same thing.

For each modeled-data constant, the fact-check confirms the prose's description and the constant's value implement the same rule. Methodology says “California double-time after 12 hours per workday”; the data layer has doubleTimeThresholdHours: 12; the per-day code path uses Math.max(0, hours - threshold). The prose says “after 12”; the code does “strictly above 12” (since hours - 12 > 0 requires hours > 12). That matches. ✓

Mismatches between prose and data are flagged as ⚠ Partial — the user reads one and the calculator does the other. The summary at the top of each report names whether the data-prose mapping passes for all constants or where it diverges.

These appear in tools but aren't amenable to source verification. They're skipped entirely (not even listed as “Unverifiable” — that status is for items that shouldbe verifiable but aren't).

• UX choices. Default state, default rounding mode, default input scenarios — these are design decisions, not factual claims.
• Methodology framing. “The §778.211 trap most calculators miss” is editorial framing of an underlying factual claim (which is verified).
• Hypotheticals. Worked examples are illustrations, not first-party claims.
• First-party product claims. “Clockspot tracks hours correctly” is a self-claim, not independently verifiable.

For high-stakes claims — modeled-data thresholds, major dollar amounts, court holdings, statutory definitions — we look for multiple independent sources. Independent means the sources don't all trace back to the same underlying report. Two news outlets citing the same press release is one source, not two.

Modeled-data thresholds specifically benefit from cross-verification: a statute text from the legislature's site plus the same text from Cornell LII confirms the wording wasn't misread. When only one independent source is available, that's noted in the report.

For each claim:

1. Read in context. Capture the full meaning. A claim taken out of context can be technically true but misleading.
2. Strip to the verifiable fact. What concrete, sourceable thing am I checking?
3. For modeled data: identify the statute clause. The constant matches if the statute's threshold and direction (strictly above, at-or-above, per-day, per-week) match the implementation.
4. Find the Tier 1 source. Drop to Tier 2 only if Tier 1 isn't available; Tier 3 as last resort.
5. Cross-check with a second independent source for high-stakes claims, especially modeled-data thresholds.
6. Confirm currency. Has the statute been amended? Has the case been overruled? Has the indexed annual value rolled forward?
7. Verify data-prose alignment. The methodology page's description of the rule and the data layer's constant implement the same rule.
8. Record the result. Claim text, primary source URL, secondary source URL (if used), archive URL (if needed), status, verification date.

The trust signal is the artifact itself. These principles uphold the artifact even when they cost us (showing a ✗ Issue hurts more than hiding it; we show it anyway).

1. Mark every unverifiable claim. Don't quietly omit unverifiable items — list them with ⓘ status.
2. Show every source URL. The reader should be able to click and verify.
3. Explain conflicts. When sources disagree, show both and describe how we picked (or marked Partial).
4. Date everything. Per-claim, per-tool, per-methodology.
5. Mark issues. ✗ Issue means the tool needs revision. Don't ship a fact-check that papers over real problems.
6. Cite primary where possible. When Tier 2/3 is used, note why Tier 1 wasn't available.
7. Reproducibility. A second person following this methodology on the same tool should produce a report with the same counts, the same source citations, and the same status assignments.
8. Data-prose alignment is visible. The methodology page describes the rule; the data layer implements it; the fact-check confirms both say the same thing. Mismatches are flagged, not silently reconciled.

The principles above are easier to see by contrast. The fact-checking patterns below are common, look credible, and don't actually verify anything. They're what this methodology is designed to prevent — by construction, not discipline.

• Rubber-stamping. A check that always returns ✓ Verified. If a process can't produce ✗ Issue / 🕐 Outdated outcomes, it's not a check — it's a stamp.
• Source-laundering. Citing a blog that cites a press release that cites a study. The citation chain has to terminate at the primary source (the issuing body, court of record, peer-reviewed paper). Tier 2 / Tier 3 are explicit fallbacks, not the default.
• Undated claims. 'Verified' without a date is a permanent claim about a temporary fact. Statutes amend, cases overturn, data supersedes. Every claim's verification is dated to a specific day.
• Silent omission. Pretending an unverifiable claim doesn't exist. Reports list every claim — including the ones we couldn't verify, with ⓘ Unverifiable status. Honesty about scope is the trust mechanic.
• Modeled data without statute mapping. A tool that ships a 40-hour threshold without naming the statute clause it implements. The constant becomes opaque — there's no way to verify it's right or to spot when the underlying rule changes.
• Methodology-by-vibes. 'We do extensive research' isn't a methodology — it's marketing. A methodology is a written, reproducible process that produces the same report from the same input regardless of who runs it.

In scope: verifiable factual claims in our tools — modeled-data thresholds, statutes, regulations, named cases, dollar amounts, statistical aggregates, standards, attributions. Anchored to clickable sources, marked with a status, dated to a specific verification day.

Out of scope:

• Professional advice. Tools state how the law / standard / data reads and compute against it; the fact-check confirms the reading is accurate. Applying the calculator's output to a specific reader's situation is the reader's call — or the call of a professional the reader retains.
• First-party claims. Any claim a publisher makes about its own tool, service, or organization isn't independently verifiable from third-party sources. Tools either omit such claims, mark them clearly as first-party, or link to demonstrable behavior the reader can verify directly.
• Cadence guarantees. Reports get dated; readers decide whether the date is fresh enough for their use.
• UX quality / design / accessibility. The tool's UI is reviewed separately (quality-check); the fact-check addresses only the factual layer.