Self-Writing Web Scraper
The output schema is the contract; an LLM writes the per-site extraction logic at runtime and repairs it when a site changes. A structural fingerprint keeps the model — and the cost — out of the steady state.
Python · Playwright · Claude API · Pydantic
Read case studySelf-writing extraction pipeline, left to right: a URL and a target schema go in. The scraper fetches the page and snapshots the raw HTML after a robots.txt check; JS-heavy pages fall back to a headless render. The DOM is compressed to a signal-rich view of JSON-LD and repeating-row exemplars, then hashed into a structural fingerprint and looked up in a spec cache. On a cache hit the known-good spec is reused and the LLM is skipped at zero cost; on a miss or drift, Claude writes an ExtractionSpec via structured outputs, reading the fixed JobPosting schema — the contract — at runtime. The generated spec is validated against the schema with Pydantic row and null-rate floors; validation errors are fed back to the model in a single-model self-heal loop. A valid spec, or a reused cached one, is interpreted deterministically with css, xpath, and regex selectors, producing schema-valid JSON and a reproducible run record.