Pre-registered backtest that killed its own product idea

I ingested 13 years of Latvian public-procurement awards (432,015 awards), ran a temporally strict backtest against kill criteria written in advance, and stopped the product when the idea failed them.

Problem

A 12-phase build spec proposed selling "companies likely to enter your category" leads. Before building anything, the question was whether a company's award history in adjacent CPV categories predicts entry into a new category any better than the CPV subscriptions suppliers already get for free.

Constraints

• Kill criteria were fixed before looking at results.
• Temporal strictness: an as_of cutoff is enforced in SQL with an assertion, plus a leakage test.
• The data is notice-level only, so per-company bid counts and win rates cannot be built.
• Two source channels with different date semantics: eForms JSON (2023+) and archive XML (2013-2025).
• Market arithmetic done up front: a few hundred possible buyers at ~30-80 EUR/month.

What I built

 public archive (procurement authority)
      |  downloader: 5,281 files, 1.5 GB
      v
 parsers   eForms JSON (2023+) | archive XML (2013-2025)
      |    publication_date re-derived from source filenames  <- after a leak was found
      v
 SQLite    432,015 awards, 28,136 winning companies
      |    as_of cutoff asserted in SQL + leakage test
      v
 +-------------------+------------------------+
 | baselines         | hypothesis             |
 |  popularity floor |  adjacent_cpv          |
 |  parent_division  |                        |
 +-------------------+------------------------+
      |
      v  sampled entry events
 metrics   Hit@5 / Hit@10 / Hit@20, MRR, median rank
      v
 paired bootstrap CIs  ->  go / no-go in FINDINGS

• Replaced a 12-phase build plan with a one-week validation plan and pre-registered its criteria.
• Found and fixed two date-leakage paths before trusting any number.
• Added a missing effect-size criterion when a significant-but-useless lift came back, and wrote down that the criteria themselves needed repair.
• Accepted the negative result and filed the project as dead.

Outcome

Stack

Redacted screenshot