Services · How do we get a data foundation we can trust?
A data platform that also holds up under the choices you have not yet made.
You live with a platform decision for a long time. In that time, regulation, prices and suppliers will change. We design and build the data platform so that you know what you depend on and can replace a part without rebuilding the whole platform.
When this is relevant
- Your platform estate is ageing or becoming expensive.
- Data is scattered across environments that are developed and maintained in different ways.
- You are worried about being locked in to a single supplier.
- Research, operational use and analytics should be served from the same foundation.
- Critical data has to be available during a disruption too.
- A platform procurement is coming up, and the requirements need to be written.
Public cloud, your own environment, or both
There are two approaches, and we do not assume either in advance. The public cloud is often the right answer: Western Uusimaa Wellbeing Services County's data production runs on Azure and Databricks. Sometimes raw data has to stay in your own environment. What matters is that you can move, in a controlled way, in the direction you choose yourselves.
A platform can be built so that your own environment processes the raw data and only pseudonymised data moves to the public cloud. The cloud's computing power can then be used for analytics and AI even though the raw data goes nowhere. The same structure can be implemented in your own data centre, in a private cloud and in the public cloud, and the part in your own environment can be fully isolated from the network if needed.
In both cases, the data, the data models and the key workflows are designed to be portable. A single component can then be replaced without rebuilding the whole platform.
What a production-ready platform requires
A platform is finished only when it can withstand critical use. Before that, we check ten things:
- identity and access management
- audit trail and logs: who did what, and when
- data protection controls
- separate environments for development, testing and production
- controlled release, and rollback to the previous version
- monitoring and alerts
- recovery plans
- automated enforcement of usage policies
- maintainable ways of ingesting and transforming data
- a repeatable way of connecting new data sources
Technical approach
Environments
Azure is a typical environment for us, but not the only one. In lakehouse implementations, Databricks is one typical option. The compute engine is chosen to suit the use case, and we use established, mature open source projects.
Tools
Infrastructure is described as code. Changes go through Git version control and automated testing and deployment (CI/CD). In our own work, a change going to production requires a review and the architect's approval.
Regulation
When raw data stays in your own environment, the processing of personal data can be limited and demonstrated. Pseudonymised data is still personal data for whoever holds the key, so pseudonymisation alone does not settle what may be taken to the cloud. The structure does not replace data protection work, but it makes that work verifiable.
Technology choices are made on the basis of purpose, regulation, continuity and governability. We are not tied to one platform or tool.
Before procurement
If a tender is coming up, we start with a Definition. It sets out what to buy, in what parts, and what stays under your control.
Evidence
We have designed and built data platforms for environments where data protection is critical and the platform is developed over years. In our public case study, the platform was the customer's own. On top of Western Uusimaa's lakehouse we built, together with the county's data engineers, the method for data production.
How to buy this
As a project or a continuous allocation in which our team fills roles in your organisation. Platform work can also begin with a Definition, which is a limited-scope project.