Data Pipeline

The founding specification describes a four-stage pipeline that turns unstructured world data into standardized, query-ready statistics. This page states what each stage is meant to do, then shows what the only implemented server actually stores.

No stage of this pipeline is implemented

Ingestion, Processing, Structuring and Storage are all Planned. No repository contains code for any of them. The tier that would run them — see the worker tier — is Not built: demostatics-backend holds one two-line README.md and nothing else. Everything downstream of that gap is described here as a plan, not as behaviour you can observe.

  +------------+   +------------+   +--------------+   +-----------+
  | Ingestion  |-->| Processing |-->| Structuring  |-->|  Storage  |
  |  Planned   |   |  Planned   |   |   Planned    |   |  Planned  |
  +------------+   +------------+   +--------------+   +-----------+
                                                             |
                                                             | (no producer)
                                                             v
                                            +--------------------------------+
                                            | demostatics-web tables         |
                                            | reports · database_items       |
                                            | schema Shipped, rows seeded    |
                                            +--------------------------------+

The four documented stages

Stage	What it is meant to do	Status
Ingestion	Acquire unstructured and semi-structured data in real time	Planned
Processing	Clean, classify and contextualize using ML models and LLMs	Planned
Structuring	Organize data into meta-descriptive dimensions	Planned
Storage	Store refined data in accessible, query-ready formats	Planned

Ingestion

Planned. The first stage acquires unstructured and semi-structured data in real time from the declared sources below. Nothing in any repository opens a connection to an external data provider. There is no scheduler, no fetcher, no queue consumer and no credential for any source.

The display tier has one external read of its own — the world geographic store described in demostatics-web/README.md — but that is a static reference dataset loaded by an operator, not an ingestion path.

Processing

Planned. The second stage cleans, classifies and contextualizes the ingested material using machine learning models and large language models, which the founding specification treats as specialized worker nodes rather than as a separate service.

No model is named, no inference runtime is chosen, and no classification code exists. The only classification artifact anywhere in the codebase is a list of prompt sentences that nothing reads — see the vocabulary section below.

Structuring

Planned. The third stage organizes processed output into meta-descriptive dimensions so that results from unrelated sources become comparable. The founding specification fixes those dimensions at five; the desktop client’s contract replaces them with a server-defined catalog. That conflict is unresolved and is covered below.

Storage

Planned. The fourth stage stores refined data in accessible, query-ready formats.

What exists today is the landing surface for that stage, not the stage itself: two tables in demostatics-web whose schema was written to receive worker output, and which are filled only by seeders. Their exact shape is documented further down.

Declared sources

These are the sources the founding specification names. None is integrated, and no code in any repository references any of them.

Source	As described	Status
Satellite observation	Imagery, coordinates and movement data	Planned
Mapping and traffic APIs	Named as Google Maps, Yandex and Baidu	Planned
Foreign exchange rates	Currency rate feeds	Planned
ISO datasets	Standards-body reference data	Planned
Journal articles	Fields such as Ecology, Law and System Dynamics	Planned
News networks	Worldwide local and national news	Planned
Social media platforms	Public platform content	Planned
Open source libraries	Existing public code and datasets	Planned
Big web data	Open web data in general	Planned

The source list has no cost or licensing story

Google Maps, Yandex and Baidu are paid commercial APIs. News networks and social media platforms are licence- and terms-constrained, and most restrict bulk collection and redistribution outright. No founding document estimates a cost, names a vendor, cites a data agreement, or takes a compliance position on any of them. Treat every row above as an unpriced dependency.

The five meta-descriptive dimensions

The founding specification normalizes every processed output against five analytical dimensions:

Dimension	Intent
Time	Duration or time consumed
Capital	Monetary or capital value
Workforce	Human labour involved
Material–energy	Physical material and energy
Information	Informational content

These five are the only part of the model with a visible trace in code, and it is an indirect one. demostatics-web/database/seeders/CategorySeeder.php seeds a category named meta descriptive listings whose five subcategories are workforce, time consumption, material, information and capital value — the same five dimensions, flattened into the ordinary category table rather than implemented as a dimensional model.

Superseded

The desktop client’s ds-core::taxonomy explicitly replaces this fixed five-dimension model with a server-defined category catalog. demostatics-pc_application/docs/API_CONTRACT.md states it directly: the taxonomy “replaces the earlier fixed five-dimension model — the catalog can grow with no client change”.

The roadmap’s advice is to pick the taxonomy and model geography as a facet of it, because building both is how the two clients end up unable to render the same catalogue. Nothing has been decided. See The Two Contracts.

Provenance is now a pipeline requirement

Demostatics is an investment company as well as a data business — see Business Model. The same pipeline is meant to feed paying subscribers, the firm’s advisory calls, its managed portfolios and its own trading book, which makes the platform both a product and internal infrastructure. That changes what a stored figure has to carry.

If an advisory call or a trade is made on the strength of a Demostatics number, that number has to be reconstructible years later: what it was at the moment of the decision, where it came from, how it was derived, and whether it was revised afterwards. That is a record-keeping obligation, not a display feature. What the obligation is exactly, and under which regulator, is a question for counsel — see Regulatory Posture, and note that nothing there or here is legal advice.

The schema has none of it. Verified against demostatics-web/database/migrations/: reports carries one nullable published_at, database_items carries one nullable measured_at, and neither carries a source column of any kind.

Requirement	Why	Exists today
Source attribution per observation	Answers “where did this number come from” without asking a person	Not built — no source column on either table
Method, and method version	A revised model produces a different number from the same input, so you must know which one ran	Not built — no method column, and no method to name yet
measured_at and ingested_at	The lag between reality and the firm’s knowledge of it is itself a fact that was acted on	Partial — database_items has measured_at; reports has published_at; neither has an ingest time
Append-only revision history	A corrected value must not overwrite the value someone acted on	Not built — rows are written in place, and nothing retains a prior value
Point-in-time reconstruction	”What did we know at 14:02 on the day of the trade”	Not built — one row per item, no series and no as-of query

Both tables do call $table->timestamps(), and it is worth being exact about why that is not provenance. created_at records when the row was written — by a seeder, today — not when an observation was acquired from a source. updated_at is overwritten in place, so it marks that a row changed without preserving what it changed from. Row bookkeeping is not a revision history, and neither column says anything about origin or method.

The cost of this is asymmetric, and Phase 2 is the hinge

metrics, metric_observations, metric_rollups and units do not exist yet. Adding source, method, method version, ingested_at and an append-only revision chain to tables that have not been created costs a column set on a migration nobody has written.

After those tables exist and start taking worker output, the same change means backfilling every observation ever published or acted on — and for the ones acted on, a backfill cannot recover what was never recorded. Provenance added later is provenance you assert rather than provenance you hold.

What storage looks like today

Verified against the migrations in demostatics-web/database/migrations/. Both tables are Shipped as schema. Their producer is Not built.

reports

Created by 2024_12_31_234146_create_reports_table.php. Its own comment calls it “the display-tier contract for a worker-produced report row”.

Column	Type	Notes
id	bigint	Primary key
title	string	Required
summary	text	Required
region_id	unsigned bigint	Nullable
subregion_id	unsigned bigint	Nullable
country_id	unsigned bigint	Nullable
state_id	unsigned bigint	Nullable
city_id	unsigned bigint	Nullable
category_id	unsigned bigint	Nullable
subcategory_id	unsigned bigint	Nullable
published_at	timestamp	Nullable, indexed

All seven dimension ids are nullable so a report can be global or scoped to any level. There are composite indexes on (region_id, country_id) and (category_id, subcategory_id). None of the seven is a foreign key — the geographic ids point into the external world store, which is not on a constrainable relationship.

database_items

Created by 2026_07_17_130000_create_database_items_table.php. Same seven dimension ids, plus a value.

Column	Type	Notes
id	bigint	Primary key
title	string	Required
value	string	Nullable VARCHAR
summary	text	Nullable
region_id … subcategory_id	unsigned bigint	The same seven nullable dimension ids
measured_at	timestamp	Nullable, indexed

One difference from the brief worth stating

database_items does not carry published_at in addition to measured_at — it carries measured_at instead. A report has a publication date; a database item has a capture time, which is what makes “latest” well defined. summary is also nullable here and required on reports.

value is a display string, and there is no history

database_items.value is a VARCHAR. It is never cast, parsed, aggregated or compared as a number:

• App\Models\DatabaseItem declares only 'measured_at' => 'datetime' in its casts().
• App\Http\Resources\DatabaseItemResource emits value unchanged, and says why in its own docblock: the column “holds formatted measurements ("12.4%", "1,204 km²") whose units are part of the value, so casting to a number here would silently destroy them”.
• resources/views/database.blade.php echoes $item->value straight into the page.

The seeded values are literal strings such as 48,210 MW, 62.4 USD/MWh and 38 AQI.

There is no history table. Each row carries one measured_at, so a value has a timestamp but not a series. Nothing in the schema can express the same metric at two points in time.

Rows are written only by seeders

ReportSeeder.php writes 12 demo reports; DatabaseItemSeeder.php writes 7 demo values. Both call updateOrCreate keyed on title, and both say in their own comments that real rows “come from the worker tier in production”. Beyond those two seeders and the test suite (which constructs rows directly and through DatabaseItemFactory), no code path in demostatics-web inserts into either table. routes/api.php exposes reads only.

NULLs-last ordering

Both tables sort on a nullable timestamp, and database engines disagree about where NULLs belong on a descending sort — PostgreSQL places them first. All four controllers therefore force NULLs last with a portable CASE before the real ordering:

$query
    ->orderByRaw('CASE WHEN measured_at IS NULL THEN 1 ELSE 0 END')
    ->orderByDesc('measured_at');

Verified in all four places that read these tables:

File	Column
demostatics-web/app/Http/Controllers/ReportsController.php	published_at
demostatics-web/app/Http/Controllers/Api/V1/ReportController.php	published_at
demostatics-web/app/Http/Controllers/DatabaseController.php	measured_at
demostatics-web/app/Http/Controllers/Api/V1/DatabaseItemController.php	measured_at

The date filters follow the same discipline: they compare the bare column against a datetime range rather than wrapping it in whereDate(), which would be non-sargable and skip the index on every driver. Shipped, and covered by tests/Feature/PortabilityTest.php.

The classification vocabulary that exists

Three seeded artifacts make up the whole of the platform’s real classification vocabulary. None of them was produced by a pipeline.

categories and subcategories

Shipped. demostatics-web/database/seeders/CategorySeeder.php creates 5 categories and 15 subcategories. These are the values the Reports and Database filter dropdowns offer.

Category	Subcategories
local news	N/S/BC
organizational sectors	Primary, Secondary, Tertiary
meta descriptive listings	workforce, time consumption, material, information, capital value
transportation traffic	L-Transport, A-Transport, M-Transport
ecological conditions	Estimated Resources, Estimated Environment Conditions, Estimated Entities

The subcategory abbreviations are not expanded anywhere in the source. L/A/M-Transport plausibly means land, air and maritime, and N/S/BC is unexplained; neither reading is confirmed by any file.

questions

Not built as a capability, despite existing as data. The questions table has exactly two meaningful columns — id and text — and QuestionSeeder.php writes 66 classification prompts into it, grouped by comment into seven topics:

Topic	Prompts
Sociology	10
Economy	10
Politics	10
Ecology	10
Security	10
Demographics	10
Transportation	6
Total	66

They are yes/no prompts of the form “Does the content explore social dynamics or societal interactions?” — plainly written to be asked of a document by a classifier.

Both claims were checked. The count is 66. A search for Question across app/, routes/, resources/ and database/ returns only the model class itself, the seeder, and the seeder’s registration in DatabaseSeeder.php. No controller, route, view, job or test reads the table. The vocabulary for the Processing stage exists; the stage that would consume it does not.

What Phase 2 replaces this with

The roadmap in demostatics-mobile_application/docs/roadmap.md schedules the replacement as Phase 2 — Numeric time series, from source, estimated at 4–6 weeks and described as “the technical unlock for the whole product”. Status: Planned. Nothing has started.

Element	What it is	Status
metrics	Metric definitions	Planned
metric_observations	Numeric observations with timestamps — the history that does not exist today	Planned
metric_rollups	Pre-aggregated series, computed by the worker and written in, not derived by Laravel on a schedule	Planned
units	Unit vocabulary, so a number no longer has to carry its unit inside a display string	Planned
Ingest endpoint	Behind a scoped ingest:write ability with an Idempotency-Key	Planned
database_items as a projection	Rebuilt over the new tables so GET /api/v1/database-items stays byte-identical for existing clients	Planned
Provenance on every observation	Source, method, method version and ingested_at alongside measured_at	Planned
Append-only revisions	A correction writes a new row instead of overwriting the one that was acted on	Planned
As-of queries	Point-in-time reconstruction of what a series held on a given date	Planned

The last three rows are additions from the founder decision. The roadmap originally scheduled provenance for Phase 4, and only as something to draw — “catalogue → detail (sparkline, chart, provenance) → table”. That placement is superseded: provenance is a property of the record, so it has to be written at Phase 2 by whatever produces the observation, and Phase 4 then displays something that is actually there. Both this site’s Roadmap and the source file it mirrors, demostatics-mobile_application/docs/roadmap.md, now place it in Phase 2.

The ingest:write ability already exists as a string constant in demostatics-web/app/Support/TokenAbility.php and is deliberately withheld from any token a person signs in with. It is granted to nothing and required by no route today.

Convert the seeded rows by hand

The roadmap is explicit that the seven seeded database_items rows are to be converted manually, and that no regex parser should be shipped to recover values from strings like 48,210 MW. The worker already holds those values as f64; parsing them back out of a formatted string is reconstructing information the producer never needed to lose.

Rollups belong to the worker for the same reason threshold evaluation does. Roadmap §4a draws the line: request-scoped arithmetic belongs in Laravel, continuous processing over the dataset belongs in the worker, and putting the second in Laravel “would quietly turn the display tier into a compute tier”.

Next

• Worker Tier — the tier that would run all four stages, and why it does not exist
• Business Model — why the pipeline is now internal infrastructure as well as a product
• Regulatory Posture — the record-keeping obligation provenance answers to
• Roadmap — where Phase 2 sits against everything else
• The Two Contracts — the taxonomy-versus-dimensions conflict in full
• System Map — how the gap sits in the platform as a whole