Book review: Data Management at Scale (Piethein Strengholt)

 This blog is a review of the book "Data Management at Scale (See also at bol.com)

Data Management is a hot topic nowadays and this book does a fantastic job at adding value to this topic. It is a must read and one of the few technical books I finished reading in a weekend.

The book gives a fantastic overview on how to implement a Data Mesh data architecture. The Data Mesh concept is explained by Martin Fowler here. The book is a good mix between conceptual and implementation architecture level. It gives a lot of examples of how this architecture at scale can work, for both small and big companies. It is practical and I used it to implement it at one of my customers.

The book describes an architecture in which the focus is on the DIAL (Data- and Integration Access Layer). 

On a high level the book covers the following topics:

• The key principles for data management at scale
  - Domain-Driven Design 
  - Domain Data Stores
  - Meta data management
• Ready Data Store
  The concept of serving data to your consumers as fit-for-purpose. Using the CQRS design pattern to split operational data store from read-only data store. This is a nice concept, however sometimes difficult to implement when using SaaS solutions. Those SaaS applications do not expose the database but only exposes APIs which are most of the time hard to use.
• API Architecture
  The most remarkable notion is that a Common Data Model (CDM) is not advised. Each (functional) domain is responsible for its data and its domain model. I also experience within my own work that a CDM is hard to accomplish and very time consuming, Furthermore the domain itself knows everything about the data, so it is best that they define and are responsible for the data and data definitions as well.
• Streaming Architecture
  Because data must be available as soon as possible Event Driven Architecture are becoming popular (again). This also has to do with the arising of Microservices,  Serverless Architectures and complex event processing, in which events are the basis of computation.
• Meta Data
  Meta data is also key to have a central data catalog that makes it possible for consumers to find the data they need, and how to get this data.
• Data governance and security is also covered
• Master Data Management

Eye-openers and remarks from my side:

1. No more use of a Common Data Model but the use of Domain Models
   
   
2. Fit-for-purpose APIs for consumers. There can be a lot of different data consumers with different requirements. I think this can be a challenge to serve all those consumers well.
   
   
3. The exposing of data will be the responsibility of each domain.
   What you see within organisations is that Integration is a different subunit. I also would like to have cross functional teams delivering data as a product and delivering value to the customers that way. Data Engineers will need to know more about the domain they are working for. But still there are data engineers that just wants to build data pipelines and discuss with fellow engineers.
   
   
4. Logical Warehouse
   The data will stay at the domain data stores and the report applications will use these stores to generate reports. The data is decentralized and this works better, because the domain knows everything about the data. DWH management is slow and this can be the next step in exposing data must faster. Also the harmonization and rationalisation of (master)data must sit at the domain site and not within a BI team. Because this team will always need and get this information from the domain itself. This introduces extra communication structures, and this will always slow down.
   
   
5. More responsabilities to the teams/domains
   This is a good thing! They know or should know everything about their data, for which they are responsable.

Happy reading ! And if you have some questions about my experience, please leave a comment.