What I just tagged
I just tagged v1.5.0 of The-Stein-Files. It is a searchable static site of Epstein-related public documents. Every source is a U.S. federal release or a clearly official court release. The site is at https://kleinpanic.github.io/The-Stein-Files/ and the code is at https://github.com/kleinpanic/The-Stein-Files.
This is an archive tool. This post is about how it is built. I make no claims about what is in the documents.
How it works
It is Python and Node. Raw files are stored exactly as released under data/raw. Derived text and the index live under data/derived.
- make ingest downloads releases politely, 1.5 requests a second by default, with backoff
- make extract pulls text with pdftotext and Tesseract OCR and builds a sharded index
- make build generates the static site
Scheduled GitHub Actions runs commit new data and deploy to Pages. Some DOJ pages are age gated. For those, make auth-doj captures a cookie jar with Playwright.
What is in v1.5.0
- OCR on 279 image-only PDFs
- a stats page at /stats.html
- better classification, plus detection of evidence photos and redactions
- reclassify_catalog.py to redo the catalog
- a mirror mode for PDFs
- mobile CSS
What is not done
The work before v1.5.0 never got tags.
To run it yourself, use make setup, then make dev. You need Python 3.11 or newer and Node 20 or newer.
P.S Thanks for reading. Consider subbing to the RSS feed.
Klein Panic