Data Catalog · how it is made
Public data, checked weekly, never redistributed.
The catalog is a map, not a mirror. Every recipe pulls from the source itself; the pages carry ids, judgement and traps, not copies of anyone's numbers. A bot that says who it is sweeps each endpoint on Sunday and writes the result on the card.
What a card says, and where it comes from
- When to use it / not for. The desk's judgement, written from using the source in the daily papers. It is an opinion about fit, never about markets.
- How to read it. Units, revisions and the traps we have fallen into: index versus rate, seasonal versus not, fiscal years, quoting direction.
- How to use it. Three recipes per source. Python recipes run under pandas; recipe 1 is re-run every week where the source allows a scripted pull. Excel and Sheets versions use the same URL where the source offers CSV.
- Series → question map. The ids we actually use, tagged with a concept (policy rate, CPI, 10-year yield…) and a region. The compare view computes from these tags.
- Status. ● OK · ◐ slow (answers, but over eight seconds) or degraded (answers, but the shape changed, a hit limit hit, or recipe 1 failed) · ✕ down (failed two weeks running) · ○ unverified (a key we do not hold, or a site that answers browsers but not declared bots — we say which).
How the sweep behaves
- Identified: FinancialLiteratureBot/1.0 (+https://financialliterature.com/data/about/; [email protected]). One request per source per week, fifteen-second timeout, plus one run of recipe 1 where marked.
- Sites that refuse declared bots are recorded as unverified, not down. We do not disguise the bot as a browser to get past them.
- A state change writes an entry to the changelog and goes out in the Sunday email.
Licences, sponsors, affiliates
- Each card states the source's terms and whether its data may be redistributed. Where the answer is no (Cboe, Stooq, the vendor APIs, the academic libraries), the recipe still works — you pull for yourself; we do not repost the file.
- Sponsored rows, if any, are marked in the Source column and on the card. The catalog's order, filters and verdicts never depend on it. There are none at launch.
- Where a source's paid tier is linked through an affiliate programme, the card says so. None at launch.
Notebooks
Three walkthrough notebooks accompany the curated comparisons. They are published as members-only posts on the paper — a free sign-up opens them; nothing is sold.
Corrections
A wrong id, a moved endpoint, a trap we missed: write to [email protected]. Fixed within a day and logged in the changelog. Educational only — we explain, we never advise.