Invinite at the Finnish Health Data Hackathon 2026: Data as Software in practice

The core idea behind Data as Software is simple: data engineering doesn’t have to be different from software engineering. The same practices that make code maintainable — types, tests, code review, CI/CD — apply directly to data transformations. It’s not about replacing SQL or abandoning existing tools, but about wrapping data work in the engineering discipline that software teams take for granted.

We’ve already used the method successfully in lakehouse projects with Finnish wellbeing services counties, where it has helped us get reliable pipelines into production. That’s why we wanted to try the same approach for building a validation pipeline for synthetic data.

Why synthetic data quality matters

1. AI applications need representative distributions. If test data doesn’t reflect reality, you can’t validate AI-generated insights.
2. Research and clinical analytics need trustworthy cohorts. Synthetic data must preserve clinically meaningful patterns — prevalence, comorbidities, lab value ranges, care pathways — otherwise research findings and feasibility analyses become misleading.
3. Performance testing requires realistic patterns. Data distribution affects query performance, partitioning strategies and access patterns.
4. Compliance testing needs edge cases. Synthetic data must include the boundary conditions that trigger validation logic.

It is easy to assume synthetic data quality is mostly about generating realistic-looking records, but research shows the hard part is usually proving that the data is both safe and useful. In healthcare, synthetic datasets must balance privacy and realism, and the literature highlights anonymisation and data fidelity as two fundamental concerns. Quality evaluation is described as its own step in the synthetic data lifecycle, typically covering fidelity, privacy and downstream utility. In other words, synthetic data generation is not complete until it can be validated, and validation cannot rely on a single score or a visual inspection.

This is where a Data as Software approach becomes a natural fit: turn validation into typed, testable code and enforce it systematically.

Architecture: the medallion pattern

We follow the medallion architecture (bronze → silver → gold), but with an important design decision: only bronze and gold persist to disk.

Silver runs in memory because:

• Refactoring without migrations. Intermediate schemas probably change during development.
• No intermediate data exposure. Only raw and final outputs are visible.
• Simpler debugging. Persist silver tables only when needed.

Code-first data quality

The core innovation is expressing data validation as typed Python code. We define typed models that are more than documentation: the DataFrame schema is validated against the type definitions at runtime.

Validation rules are declarative. Failed validations don’t drop rows. They populate a validation_errors column, preserving data lineage and enabling aggregate quality reporting.

Testing transformations like software

With transformations expressed as functions, we can test them with pytest like any other code. Tests execute in milliseconds because Polars lazy evaluation only materialises data when needed.

This means:

• Code review catches logic errors. Validation rules are reviewable Python, not buried in SQL.
• CI/CD validates changes. Automated checks run on every commit.
• Refactoring is safe. Type checkers and tests catch regressions.

Visual validation

For properties that are hard to express as unit tests, such as distribution shapes, we use marimo notebooks. The gold layer aggregates data into metrics that can be visualised.

Why this matters for healthcare

Healthcare software development faces a data paradox: you need realistic test data to build quality software, but real patient data requires strict access controls. Synthetic data bridges this gap, but only if it’s actually representative.

The method provides:

1. Auditable quality. Every validation rule is version-controlled and testable.
2. Reproducible pipelines. The same code runs anywhere: laptop, CI, cloud.
3. Iterative improvement. Fix a validation gap, add a test, ship confidently.

Explore the code

The full source is available on GitHub: inviniteopen/fhir-hackathon. The stack includes uv, Polars, DuckDB, marimo and pytest, all chosen for developer experience and performance.

Conclusion

The hackathon gave us a focused opportunity to demonstrate that data engineering doesn’t have to be different from software engineering. The same practices that make code maintainable — types, tests, code review, CI/CD — apply directly to data transformations.

Data as Software isn’t about replacing SQL or abandoning existing tools. It’s about wrapping data work in the engineering discipline that software teams take for granted.