What Data Lineage Means for Trading Research

By alyc

Editorial review status: a named human reviewer has not yet been assigned.

Data lineage is the record that connects a trading research result to the data, transformations, settings, software, and people that produced it. If the result is challenged six weeks later, the lineage should let someone reconstruct what happened without guessing.

A result without lineage may still look precise. It can show returns, drawdowns, fills, or confidence scores. Those numbers do not reveal whether the input data changed, a fee assumption was omitted, a timestamp boundary moved, or a model version was replaced. Lineage makes those dependencies visible.

What data lineage records

The W3C PROV family defines provenance as information about the entities, activities, and agents involved in producing data or another thing. In trading research, a dataset or result can be an entity. Cleaning, joining, filtering, or running a backtest can be an activity. A researcher, service, or software process can be an agent.

This model separates the thing being studied from the work that changed it and the party responsible for that work. That separation matters when two results share a title but were produced from different records.

A minimum lineage packet

A useful lineage packet can be small. It needs enough detail to identify the exact research path and expose the assumptions most likely to change the result. The following seven records form a practical baseline.

1. Source identity. Record the provider, dataset name, instrument identifiers, fields used, retrieval time, and a stable version or content digest when one exists.

2. Time boundary. Record the first and last observation, timezone, market calendar, bar interval or event resolution, and the rule used for late or revised data.

3. Transformation path. Record every filter, join, adjustment, resampling step, missing-value rule, and feature calculation in the order applied.

4. Research settings. Record parameters, random seeds, fee assumptions, slippage assumptions, starting capital, position rules, and any benchmark definition.

5. Software identity. Record the code revision, dependency lock, model identifier, prompt or instruction version, and execution environment that produced the result.

6. Output identity. Give the result a stable identifier and digest. Link charts, tables, narrative summaries, and machine-readable outputs to the same run.

7. Review record. Record who reviewed the result, what was checked, which issues remain open, and whether a later run superseded it.

Worked example: a fee change creates a new result

Consider an invented research run, R-104. Its source is a versioned price file, its decision cutoff is 2026-08-31T20:00:00Z, and its code revision is recorded with the dependency lockfile. The run buys 100 units at $50 and sells them at $51. Gross profit is $100; two $1 commissions leave $98 before any other modeled costs.

A reviewer then finds that the assumed commission should have been $3 per order. Keep R-104 and create R-105 with the corrected fee parameter: $100 − $6 = $94. Mark R-104 superseded and link R-105 to the same price file plus the revised fee record. A chart labeled only “profit: $94” cannot explain the $4 difference; this small lineage packet can.

These invented identifiers and numbers illustrate the entity–activity relationships in W3C PROV. They are not a Mattheus performance result. Preserve the complete fee and fill model in a real run.

How W3C PROV maps to a research run

PROV-O includes relationships such as used, wasGeneratedBy, wasDerivedFrom, and wasAttributedTo. A cleaned dataset can be derived from a raw dataset. A backtest activity can use the cleaned dataset and generate a result. The result can be attributed to the researcher or system responsible for the run.

A team does not need to publish RDF or adopt every PROV term to gain value from this structure. The useful part is the graph of responsibility and derivation. Each result points backward to the inputs and work that produced it.

Lineage is different from a transcript

A chat transcript records messages in sequence. It usually does not prove which dataset version reached a tool, which parameters were accepted, which code ran, or which output replaced an earlier one. A transcript can help explain intent. It is weak as the sole record of execution.

The context record described in Why Trading Agents Need Context Across Runs carries goals, evidence, corrections, and unresolved questions between sessions. Lineage adds a stricter claim. It links a specific output to the exact inputs and operations that produced it.

What lineage can and cannot prove

Lineage can show how a result was produced. It cannot show that the strategy is sound, that the data is complete, or that the result will hold in live conditions. A perfectly documented test can still contain a poor hypothesis, a biased sample, or an unrealistic market assumption.

The NIST AI Risk Management Framework Core calls for teams to document system limits, data collection and selection, testing details, human oversight, and the context used to interpret outputs. Those records support review, but they do not replace domain judgment.

Paper testing remains useful because it separates research from real capital. The paper-first workflow should still keep lineage. Otherwise, a promising result may be difficult to reproduce or compare with a later run.

Review lineage before reusing a result

Before a prior result guides new work, check whether its sources still resolve, its time boundary matches the question, its transformations are known, and its settings can be recovered. Confirm that the output has not been superseded. Record any disagreement instead of overwriting it.

Guardrails control what may happen after research. Lineage supports an earlier decision. It helps a reviewer decide whether the evidence is fit to reach that boundary at all.

The next useful step is simple. Attach a stable source identity and transformation record to one research run, then test whether another person can reconstruct the result without asking the original operator. Any missing answer belongs in the lineage packet.