From a written test plan to patients, with evidence.
You declare the situations your software must handle. The engine constructs a population that contains them, checks independently which ones it achieved, and records what is in the data, what is not, and what nobody tested.
The seven steps
- 1
Declare
The situations to test are written as a specification: each scenario, how many patients need it, and the checkable conditions that define it.
You getA versioned test plan
- 2
Plan
Before building anything, the engine works out how many patients go to each scenario, to each combination of scenarios, and to background fill.
You getA preview of the run
- 3
Construct
Patients are built to satisfy the scenarios: demographics, encounters, conditions, lab results, medications and allergies, linked with dates in a valid order. Values come from stated distributions, and every random choice is recorded so the run can be repeated exactly.
You getA synthetic population
- 4
Verify
Separate code checks each scenario against the finished population. An automated test forbids it from importing the construction code. Each scenario is reported as met, short or not verifiable (when a clinical fact it depends on is not yet established).
You getA coverage report
- 5
Measure
Structure, internal consistency, clinical rule checks, subgroups and privacy are measured. Each measurement states its method, its assumptions and what it cannot detect. There is no combined score.
You getAn evaluation report
- 6
Evidence
Everything is written into a Data Passport and a manifest that records every input.
You getA Passport and a reproducible record
- 7
Use
The population is exported as FHIR R4: one bundle for the population, and one file per patient as a test file. It can be loaded into a FHIR server you name.
You getTest data your system can load
Also built in
These work in the engine today, alongside the seven steps.
- Data-quality modes
- Build a population that should pass validation, one made only of deliberate defects, or both.
- Declared missing data
- Remove values at a stated rate: completely at random, dependent on another value, or missing together with another field.
- Timeline view
- See any patient’s record as snapshots over time.
- Built versus by chance
- For every scenario, how many patients were built for it and how many match it by chance.
Gap-fill: complete the test suite you already have
Available
- Read your existing FHIR test files.
- Measure which of your situations they contain: covered, short, or not readable because they use medical codes we do not map yet (then the count is a minimum).
- Build only the missing patients.
- Merge them with your files, which are copied unchanged.
- Verify the merged suite, reading it back from disk the way your tools will.
You get a complete suite and a Test Passport.
Failure to regression
Beta Controlled changes and failure minimisation, applied to your system.
- Load each test patient into a FHIR server.
- Ask your system, over HTTP.
- Judge each answer: pass, fail, or unresolved when no answer could be judged.
- Pinpoint a failure by testing which change to the record causes it, and reduce it to the smallest set.
- Save it as a regression case that rebuilds exactly.
- Rerun after your fix: fixed, reproduced, or changed.
What you receive
- FHIR R4 data
- Six resource types in a population bundle, checked against the FHIR R4 specification.
- Test files
- One per patient, ready for your test suite.
- Coverage report
- Every situation marked met, short or not verifiable, with how many patients were built for it and how many occurred by chance.
- Data Passport
- What is in the data, what is missing, and what was not tested. Its “Not tested” section is never empty.
- Reproducible manifest
- Every input recorded; rebuilding from it gives the identical dataset.
- Test Passport (New; with gap-fill, and when we test your system)
- Every record sent, every answer and how it was judged, failures pinpointed, and the regression case that reproduces each.
- Identity and intended use
- Scope and limits
- Fitness statement: validated for / not validated for / not tested
- Coverage report
- Downstream utility
- Statistical fidelity
- Structural validity
- Clinical consistency
- Privacy measurements
- Not tested (never empty, cannot be switched off)
- Manifest (every input to the run)
Sentences every Passport carries, word for word
They show how we write about what the data does and does not establish.
This assessment measures coverage of a declared specification and the properties listed. It does not establish that the data is anonymous, that its use is lawful, or that any system tested against it is clinically safe.
This rule set is incomplete and covers only the named domain. A dataset can satisfy every rule applied here and still be clinically wrong in ways these rules do not check.
Not tested: attacks using auxiliary external datasets; attacks with access to generator parameters; disclosure through attributes not declared as quasi-identifiers; longitudinal structure beyond the tested columns; any risk arising after the data leaves the environment where it was assessed.
Drug classes: 91 entries verified against an official source (WHO ATC/DDD Index), not reviewed by a clinician.
Scenario coverage from a real run is on the scenario library page.
Bring the list of situations your software must handle.
We will turn it into a test specification and show you what a pilot would deliver. Already have FHIR test data? Send a sample and we will show you what it is missing. Or email us at [email protected].