Minimum Viable Data Set

The founding document names nine data sources as one undifferentiated list, with no cost, no licence, no necessity ranking and no alternative attached to any of them. This page proposes the smallest set that produces something a customer would pay for, says what the expensive sources actually buy on top of it, and argues that the cheapest useful thing the firm can do this week is start archiving raw bytes.

The problem in two sentences

architecture/data-pipeline.md lists nine declared sources; every row has a Status column and nothing else — no cost, no licence, no alternative, no priority. The pipeline has never been built partly because the first source on that list, taken literally, is either unaffordable or unlicensable, so scoping against the full list produces a project that cannot start.

That is the whole of it. There is no crisis here — nothing is deployed, there are no users and no revenue, so nothing is currently going wrong. The cost is that the pipeline stays at zero rows while the source question stays open, and one specific asset (history) can only be accumulated in real time — which is what the collection section below is about.

What is settled and what is not

Everything on this page is Planned or Not built. No source is integrated, no credential exists in any repository, and a grep across all five repos for gdelt|openstreetmap|overpass|opensky|worldbank|comtrade|acled|noaa|era5|copernicus|landsat returns hits only inside demostatics-web/vendor/ MIME-type tables. The tier that would run ingestion, demostatics-backend, holds a two-line README.md.

The proposed minimum viable set

A defensible product exists entirely on free or near-free sources that permit redistribution. This is the set.

Every cost and licence cell below needs confirming before code is written

None of these figures came out of a file in this project — they are public-knowledge positions carried over from the gap findings, and they are the kind of thing providers change. Read each provider’s current terms page yourself, save a dated copy of it, and put the result in the source registry described at the bottom of this page. Where a row says Confirm, treat it as an open question, not a fact.

Source	What it gives	Cost	Licence / redistribution	Declared layer covered
GDELT	Global news events, actors, themes and tone; new files roughly every 15 minutes; machine-readable, already extracted	Free download; you pay egress and storage	Described as free and openly redistributable — Confirm the current terms, and note it distributes metadata and extracted features, not full article text	Worldwide local news; geopolitical structure
Copernicus Sentinel-1/2	Radar (cloud-independent) and ~10 m optical imagery, revisit on the order of days	Data free; processing and storage are the real cost	Copernicus full, free and open policy, attribution expected — Confirm	Ecological conditions
USGS Landsat	30 m multispectral imagery with a very long archive (Landsat 1 launched 1972)	Free	US government work, generally unrestricted — Confirm	Ecological conditions
OpenStreetMap / Overpass	Road, rail, port, airport and infrastructure geometry, worldwide	Free (run your own Overpass instance or use a public one within its rate limits)	ODbL — share-alike and attribution. The one row here with a real condition; whether share-alike reaches a derived commercial database is a question for counsel	Traffic and infrastructure
OpenSky Network	ADS-B flight positions, live and historical	Free tier for research and non-commercial use	Commercial use generally needs a separate agreement — Confirm before it touches a paid product	Traffic and infrastructure
Open AIS feeds	Vessel positions and movement	Free to cheap; coverage is coastal and partial, not global	Terms vary per feed; each national or community feed needs its own check — Confirm each	Traffic and infrastructure
ERA5 / NOAA	Weather and climate reanalysis, hourly, decades deep	Free	Copernicus/NOAA open terms with attribution — Confirm	Ecological conditions
World Bank, IMF, UN Comtrade, FAO, EIA, Our World in Data	Structured national indicators with real history — economic, trade, agricultural, energy	Free APIs; Comtrade has volume limits on the free tier and a paid premium tier	Mostly permissive open-data licences with attribution (World Bank and OWID are the clearest) — Confirm per publisher, they are not uniform	Organizational and geopolitical structures
ACLED	Coded conflict, protest and political-violence events, back to the 1990s for some regions	Free for non-commercial and academic use; commercial use is a paid licence	The one row in this set with a real price for Demostatics’ actual purpose — get a quote early, it is a line item not a rounding error	Organizational and geopolitical structures
Central bank FX (TCMB, ECB, others)	Official reference rates, daily, long series	Free	ECB reference rates are redistributable with attribution — Confirm; other central banks vary	The declared “Foreign exchange rates” source

What this set does and does not cover

Five layers are declared: organizational and geopolitical structures, worldwide local news, traffic and infrastructure, ecological conditions, and meta-descriptive listings. The set above covers the first four — every layer that is actually ingested. The fifth is not a source layer at all: meta-descriptive listings are defined in the founding document as records generated from the other four, so they are an output of the processing stage, not something anyone can buy.

The three things you genuinely give up by choosing this set:

• Live road congestion. OSM gives geometry, not traffic state. Nothing free replaces real-time congestion at global scale. If a customer needs it, that is the moment to price a mapping contract — see the next section.
• Social media. The founding list names it; there is no cheap substitute since the 2023 repricings. Drop it from v1 and say so rather than leaving it on a slide.
• Sub-metre imagery. Sentinel is 10 m and Landsat is 30 m. Counting individual cars in a car park needs commercial tasking. Nothing in the product description requires it today.

Substituting the declared nine

Declared source	Free-set substitute	Note
Satellite observation	Sentinel-1/2, Landsat	Covers land-use and ecological change, which is the only genuine satellite use case in the description
Mapping and traffic APIs	OSM/Overpass for geometry; ADS-B and AIS for movement	Live road congestion has no free substitute
Foreign exchange rates	TCMB, ECB and peer central banks	Direct replacement, better provenance than a reseller
ISO datasets	Free reference data for codes (country, currency); ISO standards documents themselves are individually paid and not redistributable	Clarify what the founding document meant here — the two readings differ by a lot
Journal articles	Crossref, OpenAlex, arXiv, DOAJ metadata and open-access full text — Confirm each	Free metadata is abundant; full text is not, and publisher terms bite
News networks	GDELT, plus per-outlet RSS	Direct scraping of news sites is the licence minefield; GDELT is the way around it
Social media platforms	None at v1	X API was roughly $5,000/mo at the Pro tier (ESTIMATE, from the cost findings, needs a current quote)
Open source libraries	This entire set is that	Already satisfied
Big web data	Common Crawl	Free, monthly crawls, deep archive — Confirm terms

The history point, which is the one that actually sells

Several of these ship decades of history you can load on day one: Landsat back to the 1970s, ERA5 back to the mid-twentieth century, World Bank and IMF series back decades, Comtrade back decades, ACLED back to the 1990s, GDELT back years. That matters more than anything else in this table, because the named buyer segments — investors, analysts, risk professionals — buy on one question above all others: give me the history so I can backtest it.

A product built on these sources can answer that question in its first week. A product built on live API calls to a mapping provider cannot answer it for years, because the provider sells you the present and forbids you from keeping it.

What the expensive sources actually buy you

Honestly, and with the licence problem stated separately from the price problem.

Source	What it adds over the free set	Cost	Licence risk
Google Maps Platform	Live traffic and congestion, routing with real-world travel times, high-quality places data	Metered per call. ESTIMATE from the cost findings: sampling 1,000 corridors every 15 minutes is ~2.9M Routes calls/month, on the order of $14,000/mo at roughly $5 per 1,000 requests. “Global road networks” is trivially 10x that	High. Terms have historically restricted caching Content beyond a limited window, prohibited creating derived datasets, and prohibited redistribution to third parties
Yandex Maps	Comparable routing and traffic, strongest in its home region	Paid commercial licence, unpriced here	High, and not only legal. Brings a Russian counterparty into the supply chain — payment-rail friction and sanctions screening for a firm that later wants EU or US clients
Baidu Maps	The only realistic mapping coverage inside China	Paid, unpriced here	High. Production use generally requires a Chinese entity and an ICP filing, which a Turkish company does not have
Commercial satellite tasking (Planet, Maxar, Airbus, ICEYE, Satellogic)	Sub-metre resolution, tasked revisit, guaranteed acquisition over a chosen area	Five to six figures per year minimum for a broad-area subscription (ESTIMATE — no vendor has been contacted)	Moderate. Imagery licences are negotiable but typically restrict redistribution of pixels; derived analytics are usually permitted, which needs writing into the contract

The licensing collision

This is not the same problem as “the sources are unpriced”, and it is the more dangerous of the two. Demostatics’ stated model is: ingest via API, refine, store in query-ready form, and sell access to the result, including bulk CSV/JSON export and an API tier for institutional partners. Storing and reselling derived data is a different permission from calling an API, and the three named mapping providers are the exact category of vendor that withholds it.

Check the terms before writing an integration, not after

The direction of the problem is clear enough to act on: Google Maps terms have historically restricted caching and derived datasets and prohibited redistribution; Baidu generally needs a Chinese entity and ICP filing; Yandex adds a Russian counterparty to the supply chain. All three may be unusable for the stated purpose at any price.

This is not a legal conclusion and nothing here is legal advice. It is a list for counsel: read each provider’s current terms against “store refined output and resell it, including bulk export”, and get an answer in writing before an adapter is built. Jurisdiction for the firm itself is still undecided, which affects the Yandex question in particular.

The failure mode if this is skipped: an ingestion pipeline gets built, a customer exports the data, and the firm is redistributing licensed Content in breach of terms — with a paying customer holding the evidence. For a firm that also intends to hold a financial licence, that is materially worse than it would be for an ordinary SaaS company.

The satellite claim

The founding document rests the entire product on satellite observation — “grounded in a unified layer of satellite-enabled global communication” — and then names no operator, no constellation, no vendor, no resolution, no revisit interval and no cost. Demostatics.pdf page 3 carries the claim above a figure captioned Fig 1.1 General Satellite Communication. That figure is a generic textbook illustration of satellite eavesdropping: it labels a transmitter, a “Legitimate receiver” and an “Eavesropper” [sic]. It is not architecture, and it has nothing to do with what the platform does.

Read the specific capabilities the document claims, and none of the first three come from satellite imagery at all:

Claimed capability	Where it actually comes from
Flight paths	ADS-B — OpenSky free for research; FlightAware or Cirium commercially
Maritime routes	AIS — open and national coastal feeds free or cheap; Kpler, Windward, Spire are the paid tier
Road networks	OpenStreetMap, and the mapping APIs already listed separately
Weather shifts	ERA5 and NOAA, free
Land-use and ecological change	Genuinely satellite — and served by free Sentinel-2 and Landsat

So the expensive part of “satellite” — commercial high-resolution tasking, where the five- and six-figure annual contracts live — is probably doing no work the product requires today. Say that plainly, drop it from the plan, and add it later if and when a customer pays for something that needs it. Doing so removes what would otherwise be the largest single line in cost of goods without weakening a single stated capability. See Costs and Unit Economics.

The credibility side of this

The counterweight is the pitch, not the engineering. “Satellite” in a deck sets an expectation the company cannot currently meet, and a technical buyer or investor who asks “which constellation, what revisit, what resolution” gets nothing back. The fix is one honest sentence — free public earth-observation programmes, Sentinel-2 at 10 m and Landsat at 30 m, with commercial tasking available if a customer’s use case requires it — which is both true and more impressive than a vague claim that collapses under one question.

Start collecting now

This is the point of the page. Every other decision here can wait; this one cannot, because a day not collected is gone.

Nothing is being collected or archived anywhere today, from any source. There is no object store, no dated snapshot, no crawl log; the only scheduled task in the entire product is sanctum:prune-expired in demostatics-web/routes/console.php. Meanwhile the schema gap (no numeric time series, no history table) is a migration that can be written any time — it loses nothing by waiting. History is different in kind. Satellite passes, news volume, vessel positions and sentiment for a date you did not collect are gone; some can be back-filled at cost from archives, and some cannot be reconstructed at all.

An archiver requires none of the hard decisions. It does not need the worker tier, the taxonomy, the provenance schema, the contract reconciliation or a single migration, because it writes raw files, not rows.

The minimal version

A scheduled fetch, a dated object key, no parsing.

# crontab — one line per source, cadence per the registry
*/15 * * * * /opt/demostatics/collect.sh gdelt
0    * * * * /opt/demostatics/collect.sh opensky
0    6 * * * /opt/demostatics/collect.sh worldbank-indicators

#!/usr/bin/env bash
# collect.sh — fetch once, store the bytes, never parse.
set -euo pipefail

src="$1"
url="$(jq -r --arg s "$src" '.[$s].url' /opt/demostatics/sources.json)"
now="$(date -u +%Y%m%dT%H%M%SZ)"
key="raw/${src}/$(date -u +%Y/%m/%d)/${now}"

curl -fsSL --max-time 300 -D "/tmp/${src}.headers" "$url" | gzip -c > "/tmp/${src}.gz"

aws s3 cp "/tmp/${src}.gz"      "s3://demostatics-raw/${key}.gz"
aws s3 cp "/tmp/${src}.headers" "s3://demostatics-raw/${key}.headers"

# one append-only manifest line per fetch — this is the seed of the source registry
printf '{"source":"%s","url":"%s","fetched_at":"%s","key":"%s","sha256":"%s"}\n' \
  "$src" "$url" "$now" "$key" "$(sha256sum "/tmp/${src}.gz" | cut -d' ' -f1)" \
  >> /opt/demostatics/manifest.jsonl

Five rules that make the archive worth having later:

1. Write once. Never overwrite a key, never delete, never edit in place. Turn on bucket versioning, and object lock if the provider offers it.
2. Store the response headers and the request URL next to the body. That pair is your provenance record, and it costs nothing at fetch time and cannot be recreated afterwards.
3. Do not parse. Parsing is a decision you will change three times. The archive is the thing you cannot redo, so keep the two apart — you can re-derive every downstream table from the archive, forever.
4. Archive metadata, not pixels, for imagery. Copernicus and USGS keep the scenes for free and permanently. Archive the scene index and the identifiers you selected; pulling the raster later is a re-download, not a loss.
5. Record the licence that applied at fetch time, per source, in the manifest. Terms change, and you need to know what governed the bytes you actually hold.

ESTIMATE, with the reasoning shown: at this scope — GDELT files, indicator series, ADS-B and AIS samples, imagery metadata only — the archiver is a small always-on VM plus object storage at roughly $0.02/GB/month, which lands under $100/month at the start and grows slowly. That is the cheapest thing in the whole plan and the only one whose value is strictly decreasing in how long you wait. The wider cost picture is on Costs and Unit Economics.

Archiving is not a licence

Holding a copy of something is itself a permission question. ACLED’s free tier is non-commercial; several feeds restrict retention or redistribution. Collecting early is cheap and right, but collect only what the terms allow you to hold, and record which terms those were. If a source’s terms are unclear, that is a reason to ask counsel, not a reason to skip the archive for everything else.

A source registry

Nothing in any repository holds source metadata. There is no sources.md, no registry table, no adapter interface, no provider abstraction and no second source named for any layer — and the tier that would own one, demostatics-backend, contains a two-line README.md. Not built.

Once it exists, it is where licence terms, update cadence and cost per source live. Minimum fields:

Field	Why it has to be there
id, name, layer	Which of the five declared layers this feeds
endpoint, format, auth	What to call and with what
cadence	Drives the schedule, and makes a silently stopped feed detectable
history_start	The single most commercially relevant fact about a source
licence, redistribution, commercial_use	Whether you may store it, resell it, and export it to a customer
attribution_string	The text that must appear in every distributed artefact, including exports
cost, cost_model, rate_limit	Cost of goods per source, and the ceiling that stops a runaway bill
second_source	The fallback for this layer, at least one of which should be free
terms_url, terms_reviewed_at	Terms change; a date tells you when to look again
status	Shipped / Partial / Planned / Not built, per source

Two practical notes on where to start:

• The first row should be a source that already ships. The dr5hn countries-states-cities dataset is the only external dataset actually in the product, it is redistributed unauthenticated through the public geo cascade in routes/api.php, and its licence is recorded nowhere, it carries no attribution in any distributed artefact, and no version or checksum is pinned. Every reports.country_id and database_items.country_id is a bare integer pointing into it. That is the registry’s problem in miniature, live today.
• Two sources per layer, one of them free. One adapter interface per layer, written before the first integration, costs almost nothing; retrofitting it after a provider reprices costs a rewrite. Google raised Maps Platform prices roughly 10-14x in 2018 with a few months’ notice; X and Reddit closed free API access in 2023. Your gross margin should not be set by a supplier who is not negotiating with you.

What this page does not solve

Free sources do not give you a product. GDELT gives away structured worldwide news events at no charge, which means “we aggregate worldwide news” is not itself something anyone will pay for — the refinement layer has to be the product, and it has to be demonstrably better than free. That raises two questions this page deliberately does not answer: how you know your output is correct (Ground Truth and Model Risk) and what the refined result costs to produce (Costs and Unit Economics).

The sequence in which any of this happens is on What To Do Next; the full list of findings behind this page is in the Gap Register; the stage-by-stage pipeline these sources would feed is Data Pipeline.