Use cases that drove the shape: a politics correspondent reading every
mention of a bill across both chambers in a sitting week; an investigations
reporter tracking who lobbied on a specific rule the morning after the
quarterly LD-2 batch drops; a tech reporter scanning a hearing’s
witness statements before the gavel falls; a Washington editor following
a single member’s floor speeches across a year.
Vanilla JavaScript on GitHub Pages, mostly. The US sits between UK and AU on
the data spectrum, so the architecture is hybrid, and that’s an honest
description rather than a fudge. The daily-news layer is UK-shaped: bills,
votes, members and Congressional Record metadata come from
api.congress.gov; the Federal Register
from federalregister.gov;
campaign finance from api.open.fec.gov.
All CORS-friendly, all free, all called from a small Cloudflare Worker that
injects API keys and smooths burst traffic.
The accountability layer is AU-shaped, because the source resists. Lobbying
Disclosure filings live at lda.gov — REST
API exists, but no CORS and token auth, so a nightly GitHub Action harvests
filings into static JSON shards and the browser searches those client-side.
Same pipeline pattern as House AU’s Hansard and Estimates harvesters,
different parser. Quarterly cadence: LD-2 filings drop ~20 days after each
quarter ends, so four times a year the shard set updates in a flood and the
rest of the time the harvester is mostly idle.
Built with Claude Code, internal newsroom tool, currently hosted on my
personal GitHub. US federal government works are public domain —
materially cleaner than UK’s Open Parliament Licence and miles cleaner
than AU’s CC-BY-NC-ND. The expectation is it moves to the Guardian
stack behind authentication once it’s doing useful work.
It doesn’t mirror UK House’s Committees and Transcripts surface.
The official US equivalent — GovInfo’s CHRG collection —
publishes the finalised hearing record six to eighteen months after the
hearing, which is useless for deadline journalism. The fresh witness
statements live on per-committee sites with no aggregator and inconsistent
URL patterns. Building a dedicated Hearings tab would be the same mistake AU
made with Committee Transcripts before cutting it on purpose. A modest
hearings feed inside Search may earn its place; a tab won’t.
It doesn’t do FEC as a primary tab. The data’s strong but the
market’s saturated — OpenSecrets, FollowTheMoney and the FEC’s
own search all do FEC, and reskinning what they already do unlocks nothing.
FEC sits as a cross-reference layer on top of Lobbying instead.
It doesn’t do Federal Register as a primary tab either. federalregister.gov’s
own UI is genuinely good. Same logic — cross-reference, not headline.
Also not in here: state-level coverage (federal first; widen the scope only
if the narrow version actually gets used), C-SPAN transcripts (no public API,
ToS forbids commercial redistribution, journalism use is grey-zone),
Senate roll-call votes (currently blocked by Akamai for scripted clients
— waiting for api.congress.gov to ship them rather than fighting the
WAF), and any analogue of Senate Estimates (Congress doesn’t have one;
Lobbying is the better quarterly-flood surface anyway).
Diary
July 2026
Playwright harvester — through the Akamai wall
The Senate’s lda.senate.gov API sunsets 31 July
(extended from an initial 30 June). The successor at
lda.gov is fronted by Akamai and refuses every
plain-HTTP tactic — browser User-Agent, full Chrome client
hints, TLS-fingerprint tricks, cookie warmup, all still 403 on
the API path. Real Chrome loads the landing page fine, then
Akamai locks the API to non-browser callers specifically.
Harvester rewrite: drive a real Chromium via Playwright. One
browser session per run, warm up with page.goto
to the API index, then use in-page fetch() with
Accept: application/json for each paginated URL.
The browser’s network stack passes Akamai; the API
returns clean JSON. Setup on a new machine is
pip install playwright plus
playwright install chromium — ~90MB of
headless Chromium in the local cache, nothing in the repo.
Same command shape as before — python3
tools/harvest_lda.py --year 2026 --quarter 3 —
just a bit slower per quarter (roughly 15 minutes for a
12,000-filing quarter vs the old 80 minutes on legacy
anonymous). Q2 2026 caught with 11,672 filings via the new
pipeline. Existing shards untouched. Tool keeps updating
each quarter, same cadence as before, from the day the
sunset actually lands.
July 2026
Q2 2026 safety-net harvest before the sunset
Q2 filings were due 20 July; the legacy host sunsets 31 July.
Harvested Q2 against lda.senate.gov mid-morning
on 20 July while the harvester still worked, before starting
the Playwright rewrite (see above). 11,125 filings that
morning; another 500-odd landed by evening as late-filers
submitted. Re-harvested through Playwright once the new
pipeline was working, replacing the shard with a fresher
count.
May 2026
Deep Dive folded into Search — one surface, chart on top
Shipped Deep Dive as its own page first; pulled it back almost
immediately. With the speaker leaderboards UK Deep Dive earns
its keep on out of reach (GovInfo doesn’t return speaker
IDs, so getting them needs a 10K-fetch granule harvest), a
standalone Deep Dive page with just a chart and a results
list was effectively Search-with-a-graph-on-top. Two surfaces,
one job between them, ambiguous mental model.
Folded the chart and stat block into Search
itself. After every search, the page now renders the result
list (fast: one query) and, in parallel, fans out monthly
counts across the date range to draw a bar chart on top
(slower: 12 to 60 queries depending on the range, but the
Worker’s edge cache makes repeats instant). Bars are
filter shortcuts — click a month and the result list
scopes to that month in place. One search, one URL, one
mental model.
X-axis labels switched to year-only at each year’s first
month, after a first version showed truncated “MA”
and “NO” labels because month-name strings
didn’t fit narrow columns. Speaker leaderboards still
on the table whenever a CREC harvester lands.
May 2026
Q1 2026 backfill — full quarter loaded overnight
The 1,500-filing sample shard was always a placeholder. Full
Q1 2026 ran overnight (~80 min anonymous, no API token), came
back with 20,971 filings and 44,501 activities — about 4.6 MB
gzipped on disk. The Lobbying tab now reads the full quarter:
search any registrant, any client, any lobbyist, any term in
the issue-description field across the entire batch. The
headline issue-codes shape sanity-checks against the kind of
quarter Q1 2026 was: Budget/Appropriations on top with 5,574
activities, then Healthcare 3,343, Tax 2,901, Defense 2,871,
Trade 2,109, Energy 1,765. Roughly what a budget-window
quarter ought to look like; if the top of the list were
something exotic that’d be the story.
The hardened-harvester change committed before the run paid
for itself once: the log shows zero failed retries through the
full 80 minutes, but the broader retry envelope (URLError,
OSError, TimeoutError, 6 attempts each, 60-second timeouts)
would have caught any blip. Worth keeping the same shape for
historical-quarter backfills when those land.
May 2026
Lobbying tab polish — de-shouted names, snippet’d descriptions, styled dropdowns
Three small fixes after a first walk-around:
LDA registrant and client names arrive in shouty all-caps
(“AIR FORCE SERGEANTS ASSOCIATION”) — same de-shout
heuristic as CREC titles now applies, with LLP/LLC/INC added to
the acronym list so law and lobbying firm suffixes stay
uppercase. Long issue descriptions were rendering in full,
which got monstrous for the firms that treat the
“specific issues” box as a memo — now snippet-windowed
to ~360 characters around the matched term, same shape as the
CREC search hits. And the Quarter / Issue Area dropdowns were
landing in browser-default chrome; they now wear the same
paper / rule / mono-chevron treatment as the rest of the
design.
May 2026
Lobbying lands — the killer surface, sample-shard first
The promised killer surface ships. Quarterly LD-2 filings live
on a separate Lobbying tab with full-text
search of the registrant, client, lobbyist names and (the gold
seam) the free-text issue-description field that lobbyists use
to describe what they actually lobbied on. Filter by Senate
issue code (BUD, DEF, HCR, TRA, TAX et al., dropdown populated
from whatever’s in the loaded shard).
Pipeline shape: pure-stdlib Python harvester pulls from the
Senate’s public LDA
API, strips each filing to the editorially-useful subset
(~600 bytes per filing), writes a gzipped JSON shard per
quarter to the repo. Browser fetches the shard, decompresses
via the native DecompressionStream, searches
client-side. Same plumbing pattern as House AU’s Hansard
shards, just over a much smaller dataset.
One honest caveat: today’s shipped shard is the first
1,500 filings of Q1 2026 — about 7% of the quarter —
because the API caps page size at 25 silently and the anonymous
15-req/min rate limit makes a full quarter an ~80-minute pull.
Enough to verify every UI path against real data; the full
backfill is a one-line script run when there’s a longer
window or a registered API token, written for next session.
The lda.gov successor host is the bigger watch-out.
Right now its Akamai front door is mis-configured and serves
403 to every scripted client. The harvester points at
lda.senate.gov (which works fine) and the legacy
host sunsets 30 June 2026. If lda.gov isn’t
fixed by then, a one-line BASE_URL swap won’t
be enough and the project breaks the morning of the sunset.
Calendar reminder set.
May 2026
Snippets — Search becomes triage-able
The thing that made title-only Search frustrating was CREC’s
recurrent generic titles — “Climate Change” over
and over, no way to tell which speech is which without clicking
through. Snippets fix it. After the search returns, each visible
row kicks off a granule HTML fetch through the Worker (bounded
concurrency 4, edge-cached for 5 minutes), strips to text, and
pulls a ~260-character window centred on the matched term with
a gold-underline highlight. Title-only results stay readable
while snippets fade in over the first second or so; rows that
fail to fetch a snippet just stay title-only.
Smaller-but-related polish from a review pass: default sort is
now most-recent-first rather than relevance-by-score (the
jumbled date order was harder to scan); the favicon’s
Old Glory Red is back as the literal full-stop next to the
navy accent dot, so the heading reads as a small flag flourish
rather than a typo; CREC’s shouty all-caps titles get
de-shouted but acronyms (CIA, FBI, EPA, POTUS et al.) stay
uppercase via a small whitelist; and parenthetical contexts
like “(Executive Session)” ghost in muted ink so
the title proper carries the weight.
May 2026
Search v0 — first real queries land
Search now talks to GovInfo through the Worker and renders ranked
results: title, date, collection tag, click-through to the full
document on govinfo.gov. Default collection is the Congressional
Record (CREC); Bills, Hearings, the Federal Register and Committee
Reports are pill toggles you can layer on. Date-range filter runs
last month / last year (default) / last 5 years / custom. The
search state lives in the URL, so a result page is shareable and
the back button works.
One honest gap to flag: GovInfo’s search returns titles
only, not the surrounding sentence. CREC titles are also shouty
all-caps (“CLIMATE CHANGE”) so there’s a small
de-shouting pass on the way in. Snippets-around-the-match comes
next, via lazy-fetching the granule HTML for visible results
with bounded concurrency. Until then, click-through is one
tap away.
May 2026
Cloudflare Worker — load-bearing, not optional
Started this commit thinking the Worker was overkill for v0 and
we could call the APIs direct from the browser. Pushed back on my
own plan, got told to skip it, then verified actual CORS headers
and discovered api.govinfo.gov sends none. Without a
proxy, full-text search of the Congressional Record from a static
page is impossible. So the Worker is back — same shape as
House UK’s, ~50 lines, with three differences for US: it
handles POST (GovInfo Search is POST), it injects the
api.data.gov key server-side, and the allow-list is the four US
hosts we’ll touch. Edge-caches GET responses for five
minutes. Lesson: verify CORS before deciding the architecture,
not after.
May 2026
Day one — static shell, brand swap, scaffold notice
Lifted the static shell, type stack, palette and component vocabulary
straight from House AU. Same Source Serif 4 / Inter / JetBrains Mono,
same cream paper, same near-black ink, same restrained-accent
discipline. Swapped the AU eucalypt green (#00843D) for Federal navy
(#1d2f5e), updated the region tag to US in the same
mono-caps treatment, and dropped in the favicon, touch icon and
OpenGraph card from the brand kit that already lived in
regions/us/ in the AU repo. Total brand-swap cost: a
single CSS-variable override and four file copies.
The homepage is honest about where it sits. Search is visible but
inert; on submit it tells you the wiring lands next and points at this
page for the plan. No coming-soon stubs, no broken nav links, no
placeholder pages with “under construction” banners. Just a
real surface that says what it is.
May 2026
Feasibility memo — agreed the shape before lifting a finger
Spent a session researching what the US data landscape actually offers
in 2026 (not 2020), pushed back on a few assumptions in the brief, and
settled on three surfaces matching AU’s discipline: Search, Deep
Dive, Lobbying. Killer surface is Lobbying Disclosure —
quarterly LD-2 floods, genuinely US-distinctive, free-text issue
fields that no existing tool searches well. Hearings becomes a feed,
not a tab. FEC and Federal Register become cross-references, not
tabs. ProPublica’s Congress API and OpenSecrets’ API are
both dead now — both got struck off the plan. The full memo and
the argument behind each call lives in the project notes.
The single most important constraint to carry into the build:
lda.senate.gov sunsets on 30 June 2026, with the successor at
lda.gov live now. Any harvester I write points at lda.gov from day
one. The legacy host is borrowed time and would be throwaway code.