Services · How do we get a data foundation we can trust?
When you get two different figures for the same thing, the fault is in the method.
Data as Software is a method in which data production is done like software. Transformations are in version control, every change is tested automatically, and deployment is controlled. Tests and quality rules run every time, before a figure ends up in a report.
Five questions that show where you stand
Put these to your own team:
- How much work does it take to add a new source, model or column?
- Where is a particular model, who made it, and when?
- How is it built, and can it be changed?
- How long does a change take?
- Has the change been tested? How do we know it breaks nothing and that the data is correct?
If the answer is slow, uncertain or “we would have to ask whoever built it”, your data production depends on individual people.
What the method involves
Data production is moved to software engineering methods. Logic is written as small functions that can be tested. All code is in version control, and changes are reviewed. Testing and deployment run automatically through three environments: development, test and production. Data contracts state what each data structure means, who owns it and which quality rules apply to it.
Layers
Bronze, Silver and Gold are not enough on their own. We typically divide the Silver layer into three parts: sources (S1), concepts (S2) and domains (S3). A concept such as patient, visit or diagnosis is defined in one place. When the definition changes, the change is made once, and it flows through to every data product.
Tools
Python, PySpark and Git. Otherwise we use established, mature open source projects. Tools are chosen to suit your environment. The code is written so that it can run in your own environment, in a private cloud and in the public cloud.
| Part | What it means |
|---|---|
| Version control | All transformation logic is in Git. Every change is reviewed before merging (pull request), and it can be traced and reverted. |
| Four test levels | Unit tests, integration tests, data contract validation and acceptance tests. |
| Automated testing and deployment (CI/CD) | Every change is checked automatically: code style, tests, data contracts and model integrity. |
| Data contracts | An agreement between producer and consumer on structure and meaning. The contract travels with the code from development to production. |
| Quality assurance | Quality rules run in the pipeline every time. Data that breaks a rule is stopped or diverted before it reaches reports. |
| Ownership | Every data product has a named owner who is responsible for changes and for the lifecycle. |
| Documentation | Data models, decisions and scope limits are written at the same time as the code. |
| Audit trail | Version history and test results show what was changed, when, and with whose approval. |
What Data as Software is not
It is not just version control for SQL, a choice of tool, or automated deployment without ownership. Nor does it take care of data protection and regulation automatically, and it does not replace an understanding of the domain or architecture.
Why this matters in a regulated environment
In a regulated environment you have to be able to show how data was produced, who made the changes and with what logic. For personal data, the GDPR's accountability principle requires it. If the software is a medical device, the device regulations (MDR and IVDR) also require an audit trail. The EU AI Act's requirements for high-risk systems start to apply on 2 December 2027. When data production is code in version control, the answer is there.
Evidence: Western Uusimaa Wellbeing Services County
Western Uusimaa Wellbeing Services County moved its data production to this method. There were four changes: from SQL to PySpark and Python, from long pipelines to small transformation functions, automated testing and deployment, and data contracts. After that, new functionality reached production every week.
AI builds on this
The same structure is a precondition for AI. When data is defined in code and its meaning is explained in data contracts, a language model gets an unambiguous description of what each data structure means.
How to buy this
As a project or a continuous allocation. Our team works with your own data engineers, and documentation and workflows are written together. That is why the method stays with you.