vi

blog

A hard source gate for agent-written posts: check:sources blocks the build

When an agent writes posts, numbers that were right at writing time still drift or get invented. check:sources fails prebuild when a post has no .sources.yml sidecar or has data-shaped numbers it never declared. How it runs, and 4 gaps it misses.

Published
Claude CodeAI AgentBuild gateFact-checking
Diagram of 3 connected blocks: a post.md file with number tags, a check:sources block filled with the accent color in the middle that also takes a sidecar file from below, and a build block on the right joined by a dashed line

The source gate in this repo is 1 Node script that runs in prebuild. A post in content/blog/ with no <slug>.sources.yml file, or with a data-shaped number it never declared, fails the build, and the deploy does not run. The gate forces a declaration for every number; it does not check that the declared source is right. The end of this post lists where it misses, tested with a probe file.

Why posts written by an agent need a hard source gate

The site's content is written together with an agent, and an agent goes wrong in 2 ways. It invents: a plausible-sounding sentence, like a PageSpeed post that once said Chrome's docs state outright that the scoring did not change, when no doc says that. Or it drifts: a number right at writing time that goes wrong later, such as a repo's open-issue count moving from 301 to 311.

A sidecar does not stop wrong writing. It makes wrong writing checkable: 6 months later you still know where each number came from and which ones need re-measuring. The base rule is that a source the user supplies is the only basis for a fact, with 2 exceptions: anything verifiable by a command run on the spot, and the author's own measurements. If a claim cannot be grounded, cut it; if cutting breaks the argument, ask.

The sidecar declares sources: 3 tiers, 3 forms of from

Each post has 1 .sources.yml file next to it. A shortened example from the post on trimming AGENTS.md:

yaml
checked: '2026-11-17'

sources:
  - id: repo
    tier: local
    for: mọi số đo và mô tả commit dời bẫy khỏi AGENTS.md

claims:
  - text: 'AGENTS.md trước khi cắt 23.551 byte, sau khi cắt 17.282 byte'
    numbers: ['23.551', '17.282']
    from: 'cmd: git show b89dd27^:AGENTS.md | wc -c ; git show b89dd27:AGENTS.md | wc -c'

tier takes primary (the first-party page for the thing being discussed), web (a secondary source you found yourself) or local (my machine, my repo). Every tier except local requires a url. A claim's from is the id of a source, or local, or cmd: <command>. The cmd: form is the strongest because the next person can re-run it.

A number that will drift (stars, prices, supported versions) gets volatile: true. npm run check:freshness and the weekly cron report a claim that has gone more than 90 days without a re-check.

The gate blocks the build in prebuild

json
"prebuild": "npm run check:sources && node scripts/check-i18n.js && ..."

npm run build runs prebuild automatically before next build. The deploy workflow runs npm run build:static, which calls npm run build, and the publish job has needs: build, so a red gate means nothing gets published. The script exits with code 1 and prints each error. This is the output on a probe file, with paths shortened (the messages are in Vietnamese: không khai trong sidecar means "not declared in the sidecar"):

text
check-sources: nguồn khai thiếu

  gate-probe.sources.yml: claims[1].from = "nguon-khong-co" không khớp source id nào, cũng không phải cmd:/local
  gate-probe.md:7: số "17%" không khai trong sidecar
  gate-probe.md:7: số "123 MB" không khai trong sidecar

10 lỗi. Sửa sidecar hoặc bỏ số khỏi bài.

There are 3 kinds of error: the post has no sidecar, the sidecar breaks the schema (missing checked, unknown tier, a from that matches no id, leftover placeholder text from the scaffold), and undeclared numbers. The sidecar scaffold from new-post.js blog contains TODO on purpose, so an unfilled scaffold cannot slip into a build. The check is that the whole sidecar may not contain that word anywhere: this post's own sidecar was blocked on its first run because a claim describing the mechanism above spelled out TODO, and I had to describe it in a roundabout way. An English post uses the sidecar of its Vietnamese version through translationOf. The script normalizes commas to dots, so 21,066 in the English version matches 21.066 already declared in the Vietnamese one.

List the numbers to declare with --list

bash
node .claude/skills/blog-post/scripts/check-sources.js --list content/blog/<slug>.md

This is real output from when I wrote the AGENTS.md post, before its sidecar existed (số chưa khai means "numbers not declared"):

text
cat-agents-md-duoi-20k-ky-tu.md  7 số chưa khai
  21.066 · 15.562 · 0,1 · 23.551 · 17.282 · 10.782 · 12.444

The workflow: write the post from the sources you were given, run --list, write the sidecar grouping numbers by claim (1 claim can hold many numbers), and re-run without the flag until it is clean. A number that cannot be traced to a source comes out of the post; do not invent a from.

Which numbers the gate catches: a probe file with every shape

The script only catches data-shaped numbers: decimals or versions (2,5, 13.3.0), thousands separators (23.551), numbers with a unit (%, ms, MB, kB, GB, KB, USD, K, B), and dates (7/5/2026, 2/2026). Structural numbers like "3 places" or "5 releases" are just phrasing, so they are skipped. Frontmatter, code fences and inline code are skipped too. The probe file, shortened (the filler words are Vietnamese, only the numbers matter):

markdown
---
layout: BlogPost
title: 'Probe 99% title'
---

Số trần 17282 và 258 dòng. Chữ thường 95k và 20k ký tự.
Có đơn vị: 17% và 200ms và 123 MB và 95K. Thập phân 2,5 và 13.3.0.
Phân cách nghìn 23.551 và ngày 7/5/2026 và 2/2026.
Trong code `4311 bytes 88%` thì bỏ qua.

| cột | số |
| --- | --- |
| a | 12,5 |

--list on this file prints 10 numbers: 17%, 200ms, 123 MB, 95K, 2,5, 13.3.0, 23.551, 7/5/2026, 2/2026 and 12,5 from the table cell. None of the other 5 is caught: 17282, 258, 95k, 20k and 99% in the title.

4 gaps check:sources does not catch

1. Bare integers, lowercase k, frontmatter. 17282 and 258 match no pattern. The units K and B only match uppercase, so 95K is caught and 95k slips through. A percentage in title or description is not scanned because frontmatter is skipped, even though it is the first thing a reader sees.

2. Declared by number string, not by claim. The script collects every number declared in the sidecar into 1 set, then compares each number in the prose against that set. I tried assigning 200ms to a claim unrelated to the sentence with 200ms in the post: that line was not reported. A number sitting inside a claim's text also counts as declared.

3. The source itself is not checked. from only has to match a source id, start with cmd:, or be local. The command is not run and the URL is not opened. I declared 17% on a claim with from: 'cmd: echo khong-lien-quan': the error for 17% disappeared, although that command prints no such number. The gate does catch a from pointing at an id that does not exist (it is in the output above).

4. Qualitative claims. "Doc X says Y", a correct number attributed to the wrong source, a description that drifts from what an issue says: there is no number to compare, so the script skips them. For the "doc X says Y" kind, the only defense is to open X and find the exact sentence Y; if it is not there, write it as your own argument and say so.

What is still open for the gate

Gap 1 can be fixed by scanning frontmatter and catching lowercase k, but structural numbers like "5 releases" would get caught too, so each loosening of the pattern buys more false positives. Gap 2 needs numbers tied to claims sentence by sentence, and that is the biggest piece of work. Gaps 3 and 4 belong to the human re-reading: the gate cannot replace opening the source and comparing, it only leaves a trail for that step.

Related posts

all posts →