Implementation
We typically form a “data platform team” squad, responsible for the infrastructure setup (e.g., Kubernetes, Snowflake), the creation of the required tooling (operators, drivers), and supporting other squads to create data products efficiently and in line with governance.
We introduce or harness a data catalog so squads can register their products. This enables other stakeholders to find and integrate data products, document usage, and comply with governance requirements, including lineage.
Product or service owners oversee the creation of data products. For instance, when a customer places an order (a business event), the data product “order history” provides order data in line with the product specification, which requires strict constraints, consistent scheme versioning, and reproducible data output. Such rules are critical to enable a semantic presentation and self-service re-use of a data product. The squads cover the development of the data product, which includes the transformation and storage of the data, the provision of an API, and the registration in the data catalog. The data products provide data for further processing by other products and services.
Data Marts are consumers of (multiple) data products that directly enable self-service analytics. Unlike data products that are – in comparison – rather raw, data marts sit in data platforms like Snowflake and are ready to use with reporting and analytics tools such as Tableau; they ensure the appropriate user experience required for business user self-service access. Data marts are built by the squads that created the business service and data product. For multiple data products, the “nearest” squad is typically in charge (e.g., the orders squad covers a data mart offering order histories enriched with CRM data). This maintains scalability and domain expertise when building data marts.
