Versioned datasets: making exports and analyses reproducible

A research export is reproducible when you can regenerate exactly the same file later and prove it is the same. That takes three things: a versioned definition of what goes into the dataset, a data cut-off that fixes up to when data is included, and a content hash that changes if a single value changes. Record those three with every export and every analysis, and the question "which data did this result use?" has a precise answer months later, even after measurements and clinical records have been corrected. This article explains each piece and how EnaResearch handles them.

Why "export the data again" is not enoughCopy Anchor Link

In a study that collects data for months, the dataset is never finished:

• New measurements keep arriving, including late ones from phones that were offline.
• Clinical records get corrected: a discharge date was wrong, a complication was entered in error.
• Someone changes the selection: one metric added, a participant subset narrowed, a resolution switched from hourly to daily.

If a dataset is simply "whatever the database says today", the same analysis run twice can give two different results, and nobody can say which version a manuscript, a report to a funder or an answer to an ethics committee was based on.

Piece 1: a versioned dataset definitionCopy Anchor Link

A dataset definition states what goes into the dataset: which participants (by status or by a list of pseudonyms), which metrics, which time window and which resolution (raw, minute, hour or day). The window can be calendar dates or anchored to an event, for example 7 days before to 30 days after hospital admission, with day 0 being the day of the anchor event.

In EnaResearch, every edit that changes the content of a definition creates an immutable new version; changing only the title or description does not. Every export is tied to a version, the Versions panel shows who changed which field and when, and you can export from an earlier version without touching the current one. That makes "we added heart rate variability in March" a recorded fact rather than a memory.

Piece 2: a data cut-offCopy Anchor Link

The cut-off is the point in time up to which data is included. An export reads every record as it was at the cut-off. A measurement correction or a clinical record correction made after the cut-off does not change a file regenerated with the old cut-off.

This is what lets you keep improving data quality without breaking earlier results. Correct the discharge date today; last month's export, regenerated with last month's cut-off, still produces last month's file.

Piece 3: a content hash and a manifestCopy Anchor Link

Every EnaResearch export comes with a manifest that records:

• the scope and filter parameters,
• the definition version and the definition hash,
• the transformation version,
• the data cut-off,
• the SHA-256 content hash,
• whether the data is pseudonymized,
• a quality summary.

The same definition version and the same cut-off produce the same SHA-256 content hash. To check that two files are the same dataset, compare the hashes; to regenerate a file, use the version and cut-off from its manifest. Export files are deleted after 30 days, but the definition and the manifest remain, and the file can be regenerated from them.

What goes into a measurement datasetCopy Anchor Link

A measurement dataset is a single ZIP file with measurements in long format (participant, metric, time, source), a participant list and the manifest. If clinical records are selected, events.csv is added. Every export includes quality.csv with completeness and quality flag counts per participant and metric, and raw rows carry their flags. When the same metric comes from several sources, the sources are not merged: a source column keeps them apart, so your analysis decides how to handle them.

Participants appear only under a study pseudonym such as P001; sex and age at the start of participation are the only participant attributes. Names, person identifiers and dates of birth never enter an export. Keep in mind that pseudonymized data is still personal data, and sharing it remains subject to your organization's and your protocol's rules.

Variables and analyses follow the same ruleCopy Anchor Link

Variables, such as length of stay or the mean of a metric in a window around an event, are versioned too: every change to a formula creates a new immutable version, and a variable dataset pins the versions it uses. A value that cannot be computed is never written as zero or an estimate; the cell stays empty and missing.csv records the reason. Variable datasets come as variables.csv, missing.csv and a codebook.csv data dictionary, optionally with an Excel file, a labeled SPSS (.sav) file and an R script that reads the data.

Statistical analyses go one step further. The analysis plan (test, variables, significance level and hypothesis) is approved before any result exists, and approval freezes the dataset version and cut-off. Tests are computed deterministically with established open-source libraries such as SciPy and statsmodels, and the input checksum and library versions are stored with each result. Approved plans never change.

What to write in your methods sectionCopy Anchor Link

When you report a result built on an EnaResearch export, record:

1. the dataset definition version,
2. the data cut-off,
3. the SHA-256 content hash from the manifest,
4. the variable versions (for variable datasets),
5. the statistical library versions stored with the analysis.

With these five lines, anyone with access to the study can regenerate the same file and check the hash, and a reviewer's question about a number can be answered from the record instead of from memory. Generation and every download are written to the audit trail, so you can also show who produced and who downloaded a file.

Read more on the research datasets and exports and variables and statistics pages.