Self-Writing Web Scraper
The output schema is the contract; an LLM writes the per-site extraction logic at runtime and repairs it when a site changes. A structural fingerprint keeps the model — and the cost — out of the steady state.
Self-writing extraction pipeline, left to right: a URL and a target schema go in. The scraper fetches the page and snapshots the raw HTML after a robots.txt check; JS-heavy pages fall back to a headless render. The DOM is compressed to a signal-rich view of JSON-LD and repeating-row exemplars, then hashed into a structural fingerprint and looked up in a spec cache. On a cache hit the known-good spec is reused and the LLM is skipped at zero cost; on a miss or drift, Claude writes an ExtractionSpec via structured outputs, reading the fixed JobPosting schema — the contract — at runtime. The generated spec is validated against the schema with Pydantic row and null-rate floors; validation errors are fed back to the model in a single-model self-heal loop. A valid spec, or a reused cached one, is interpreted deterministically with css, xpath, and regex selectors, producing schema-valid JSON and a reproducible run record.
The problem
Web data looks structured right up until you try to depend on it. Every site encodes the same job posting differently — a different DOM, different class names, JSON-LD on one page and a hand-rolled table on the next — so a scraper that works today breaks silently the moment a site ships a redesign. The usual answers each fail in their own way. Hand-written selectors per site are accurate but brittle, and maintaining them is a job that never ends. A single LLM call per page is flexible but slow, costly, and non-deterministic — you can’t replay it, and you’re paying the model to re-derive the same layout on every run.
The decision
Split the work in two: a fixed output schema that is the contract, and a disposable extraction spec the model writes once per layout. The schema — a Pydantic model — never changes; it’s the thing downstream code and tests are allowed to depend on. Everything site-specific is pushed into a generated spec — css, xpath, JSON-LD, and regex selectors plus whitelisted transforms — that’s treated as a cache artifact, not code: written on first contact, reused while the page’s structure holds, regenerated only when it drifts. The only non-deterministic step is codegen; everything downstream is pure and replayable. That single boundary is what buys testability, offline replay, and a cost curve that collapses to zero on sites you’ve already seen.
What I built
One end-to-end slice from a URL to schema-validated JSON. A polite fetch — robots.txt honored, raw HTML snapshotted for replay — with an automatic headless-render fallback for JS-shelled pages. A compression pass that strips noise and surfaces the signal: JSON-LD blocks and exemplars of the repeating rows. Then the cost lever: a structural fingerprint, content-insensitive but structure-sensitive, hashes the page and looks it up in a small SQLite cache. On a hit the cached spec is reused and the model never runs; on a miss or drift Claude writes the spec via structured outputs, and when the first attempt fails to validate, the self-heal loop feeds the errors back and regenerates on the same model. Interpretation is deterministic, and the validator enforces row-count floors and null-rate ceilings — so a run that quietly returns garbage fails loudly instead of passing.
Closing the loop
Because every run persists its page snapshot and its generated spec, any run is fully reproducible offline — the default test suite exercises the deterministic spine with no network, while a separate live test proves the model can actually write a working spec against a fixture. New targets are a schema and a registry entry; the rest of the pipeline is schema-agnostic. The next real step is a sandboxed code-execution path for the long tail selectors can’t reach — deferred on purpose until the deterministic spine is trustworthy.
# the shape downstream code + tests depend on # the LLM writes the selectors; this never moves class JobPosting(BaseModel): title: str company: str location: str | None = None salary_min: int | None = None salary_max: int | None = None url: HttpUrl posted_at: datetime | None = None