This is the second of a multi-part series on data & analytics operating models.
In the first article of this series, I introduced a high-level operating framework designed to align data and analytics resources within an enterprise. This framework aims to eliminate bottlenecks, foster self-service without creating silos, and establish a data-driven culture.
The model consists of three core components: enterprise data teams (blue), domain-based development teams (purple), and business domains (red) (See figure 1.). Among these, the "purple team" plays a pivotal role in the success of a data and analytics program. Yet, it is often the most overlooked or underfunded aspect of the framework.
The goal of purple teams is to bridge the enterprise's need for standardization, consistency, and scalability with the business unit's need for agility, customization, and responsiveness. Striking this balance is complex and resource-intensive, leading many organizations to default to either centralized (blue) or decentralized (red) approaches. This oscillation between extremes often prevents companies from realizing the benefits of a hybrid model, which can deliver both alignment and flexibility when implemented effectively.
Data leaders who embrace a hybrid or “purple” approach to data development get the best results. Why? A hybrid environment immerses developers in a business domain where they learn its people, processes, and data, and consequently, build solutions faster and better. At the same time, they learn enterprise methods, standards, and tools that enable them to work smarter and more efficiently and deliver results that conform with enterprise data standards.
In other words, a hybrid approach enables developers to work locally where they can have the most impact but develop uniformly to ensure data consistency and enterprise alignment. In this way, purple teams combine the best of centralized, IT-led approaches and decentralized, business-led approaches without the downsides of either.
A hybrid approach enables developers to work locally where they can have the most impact but develop uniformly to ensure data consistency and enterprise alignment.
Knowledge Flows
So, how do you build a hybrid development team that delivers the best of both worlds without any downsides? How do you build a purple team that delivers effective data solutions quickly and affordably without creating data silos, data bottlenecks, or conflicting data sets? There are many options, which we’ll examine in detail in our next article. But the major ones are:
Tiger teams that are permanently assigned to individual business units and backed up by a shared services center.
Analytics centers of excellence in which data analysts and scientists are assigned to individual business units even though they might reside centrally.
Data domain teams (a la data mesh) that adhere to enterprise standards and governance policies and use enterprise data tools and platforms.
Embedded data analysts who are hired, evaluated, and managed by a director or manager of analytics at the enterprise level.
The key to making these models work is managing the knowledge flows between blue, purple, and red teams. Data leaders need to manage these flows to ensure the success of a hybrid or federated model.
Top-Down Flows: Blue to Purple
Other than sourcing data, one of the most important roles of an enterprise data team—and one that few data leaders allocate staff time to support—is to transfer knowledge, standards, and best practices to the hybrid development team. Unless the enterprise data team shares knowledge and communicates standards, the purple teams will work sub-optimally and eventually create fragmented data that frustrate business executives to no end.
Oversight. To avoid this fate, the heads of enterprise data team departments—namely, data engineering, data architecture, and data analytics—need to oversee hybrid teams. This usually entails 1:1 meetings with individual developers, daily and weekly standup meetings with individual teams, and quarterly retreats with all teams where they can establish and review goals, learn new techniques, and share experiences and knowledge.
The enterprise data team also needs to manage data analysts and scientists, whether they are centralized in a center of excellence or embedded in business units. For example, a large urban hospital has two full-time managers who help hire, evaluate, coach, and manage the organization’s 35+ data analysts, who work for individual departments.
Train, coach, and support. The enterprise data team also needs to assign experts from each technical department to disseminate knowledge and best practices to members of purple teams. The experts should offer both formal training on the development methodology and tools and informal coaching and support through office hours, open labs, webinars, in-house events, and lunch-and-learn sessions.
Establish standards. It’s critical that the technical department heads establish standards and document them. For example, it’s important to establish a standard way to organize development teams, engage with business users, and develop and deploy code. Many companies choose Scrum or Kanban and borrow heavily from dev/ops practices. Other standards might define how to submit project requests, how to manage business logic, how to implement data quality rules, and how to propose new policies or procedures.
The enterprise data team needs to leave plenty of room for teams to adjust the methodology to meet unique requirements. They should also solicit best practices from all purple teams, compile them into a body of knowledge, and provide regular opportunities for individuals and teams to share their experiences and lessons learned.
Templates. The best way to enforce standards and best practices is to bake them into development tools via templates, wizards, and default options. Here, the developer learns by doing. Tools that auto-handle slowly changing dimensions or different join types eliminate the need for analysts to write hundreds of lines of code, simplifying complex tasks. This reduces the likelihood of mistakes and rework and enables organizations to assign less experienced people to the purple teams, saving money. (See “Data Architecture as a Service: Liberation for Data Users).
The best way to teach and enforce standards and best practices is to bake them into development tools via templates, wizards, and default options.
Data platform. Finally, the enterprise data team needs to establish a common toolset that simplifies and accelerates building and managing data solutions. At a minimum, the platform should include a cloud database for storing data; a data/ops tool for orchestrating code, pipelines, and tests; a data access management tool for governing data access; and a data observability tool for detecting data compliance and errors. The enterprise data team integrates these tools into a common data platform that abstracts the complexities of building data solutions. With a common data platform, purple teams can focus on applying their domain knowledge to a business solution rather than mastering data infrastructure, enabling them to work faster and more effectively.
What’s not listed as part of the common data platform are tools for building data applications, namely data transformation and data preparation tools and BI and analytics tools. Although it would be ideal if all purple teams used the same tools here, many domain teams that originate in the business already have a preferred tool and will fight to keep it. This is acceptable if they leverage the rest of the standard data platform. (See “What Can DataOps Do for You? Ask Roche”.)
Bottom-Up Flows – Red to Purple
Business needs. Knowledge also flows from the business to the hybrid development team. The most obvious flow is the definition of business needs that form the basis of data-driven solutions. In most cases, the purple team facilitates this information exchange through formal requirements meetings and workshops. However, the best requirements are intuited by analysts or developers who permanently reside in the business. Because they sit with the business and know its people, processes, issues, problems, and opportunities at a deep level, they can proactively define solutions before the business even asks for them.
Governance. Red teams are also responsible for data governance. Although business leaders would like to outsource governance to technical teams, only they can make decisions about the meaning and implications of data. For instance, businesspeople need to define metrics and dimensions, specify rules for data quality, define data privacy and security policies, and decide who can access what data.
The enterprise team can facilitate data governance by creating and staffing a program office that trains businesspeople how to govern data, facilitates governance decisions, documents results, and administers data governance tools and metadata. But the business must own the content, definitions, and policies and curate the data. They must take responsibility for these tasks, not technical people. Getting businesspeople to own data governance tasks can be a major bottleneck to development initiatives, which can’t proceed unless there is consensus about key data elements and data policies.
Summary
Hybrid development teams are vital to the success of any data and analytics initiative. Data leaders must dedicate significant effort to building and supporting these teams, ensuring that knowledge flows seamlessly between enterprise data teams, domain-based development teams, and business domains. Without thoughtful organizational design and sustained investment, the program risks falling back into either centralized, IT-led models or decentralized, business-led models—both of which can perpetuate silos and operational inefficiencies.
This article was originally published on Eckerson Group.
