Simply relying on technology is not enough to create a Data Mesh. It is crucial to establish an organizational framework that connects data ownership with business areas. Consider the three main methods for structuring your data teams.
As companies grow and expand, their data management strategies need to shift from centralized bottlenecks to decentralized networks.
One unified data engineering team is accountable for processing, converting, and delivering all data throughout the organization.
Pro: Easy to enforce governance and standardize tooling.
Con: Causes a major bottleneck as the central team lacks domain expertise, resulting in delays and inconsistent data alignment.
A central platform team, known as the Hub, constructs the self-serve infrastructure and standards, while embedded data teams within business units, referred to as Spokes, develop the actual data products.
Pro: Combining speed with standardization, domain teams efficiently utilize centralized tools.
Con: Managing funding and boundaries between Hub and Spokes can present political complexities.
A decentralized model where cross-functional domain teams have full ownership of their data products, infrastructure, and pipelines, communicating solely through standardized ports.
Pro: Infinite scalability. No central bottlenecks. Maximum domain autonomy.
Con: Significant engineering expertise is needed. Potential for redundant work and isolated systems.
The ideal operating model varies based on your organization's size, engineering maturity, and the complexity of your business domains - there is no one-size-fits-all solution.
majority of organizations that have embraced the Data Product mindset are currently thriving in a Hub and Spoke The model allows domains to function as product teams independently, with oversight from a centralized platform team to ensure DaaP standards are upheld (Discoverability, Security, etc.).
Determine the current bottlenecks in your organization and plan the shift towards a decentralized data structure driven by domains.