Build log

I built the whole thing on stream.

A search engine over 10,000 NASA documents, from an empty folder to a working system. 94 hours across 15 parts, with no editing and no skipping the bits where it broke.

Open the playlist ↗

94 hrs: of footage
15: parts
10,000: documents indexed
180/min: pages, by the end

Part 17h 8m

Rocket schematics and more: starting the build

What we're building and why it's hard. Scoping 10,000 NASA documents and designing the pipeline from nothing.

Part 25h 45m

Table extraction, formulas and the vision pipeline

Getting tables out of scanned pages with their structure intact, and the first working vision model pass.

Part 311h 5m

Formula extraction and visual anchoring

The numbered-box trick: drawing markers onto a page so the model can be told which equation to read.

Part 44h 48m

Building agents for 60 years of rocket science

Moving from plain search to an agent that investigates a question.

Part 511h 55m

Agents that think like engineers

Sub-agents and tools. When to send an agent deep on one thing versus wide across many.

Part 64h 27m

Agent tools and the interface

Designing the tools an agent gets, and the UI on top of a search loop.

Part 74h 2m

Processing 10K documents on one H100

GPU infrastructure, the scaling maths, and getting the services up.

Part 811h 55m

vLLM optimisation, OCR and Docling

Serving a 32B model and sharing one card between everything that needs it.

Part 95h 38m

The rewrite: 28 days down to 2.5

Same models, same hardware, an order of magnitude faster. All in how the work was scheduled.

Part 103h 41m

Parallel processing and load balancing

Eight readers behind a balancer, with Celery and Redis holding the queue.

Part 114h 0m

Pulling the pipeline together

Getting every piece running as one production job.

Part 126h 21m

Hitting 180 pages a minute

Finding the ceiling and then holding it across the full corpus.

Part 133h 8m

Acronyms and domain terminology

Isp, TWR, LOX, MMH: decades of unexplained notation, and building a thesaurus for it.

Part 143h 19m

Filtering 18,000 NASA terms down to 5,500

Making semantic search actually work on domain language.

Part 157h 0m

End to end, and evaluating it

10,000 documents indexed, an agent reasoning across fifty years of research, and how we checked it was any good.