At Oakland, we believe that data is not just a useful tool but a critical element in every aspect of business, from quality management to operational improvements. Whether you’re in the early stages of digital transformation or already reaping the rewards of data-informed decision-making, leveraging data effectively is essential for staying relevant and successful within your industry.

In this guide, we’ll delve into why data is so important for business, and how you can harness it to unlock its true potential.

Measuring Success with Business Data

One of the most fundamental reasons businesses should care about data is its role in measuring performance. At its core, data allows businesses to track key performance indicators (KPIs) and identify areas for improvement. As the old saying goes, “You can’t improve what you don’t measure.” This is particularly relevant in a world where businesses need to stay agile and adapt quickly to the ever-changing market around them.

Andy Crossley, Director at Oakland, tells us more about the continuous need to measure business data.

“Gathering accurate business data types, you can assess where your company stands, benchmark against competitors, and map out pathways for growth. As we’ve seen with several of our clients, like Yorkshire Water,, modern data tools can now deliver faster, more reliable insights to inform strategic direction.”

Using these tools, you can constantly refine and improve your processes. At Oakland, we help organisations integrate their data to measure improvements and align efforts with broader strategic objectives. To support this, we offer tailored services focused on strategy, analytics and business intelligence, and platform development.

Linking Data to Strategy

Data isn’t valuable in isolation, it’s valuable when linked to a business strategy. Too often, companies gather data for its own sake without clearly aligning it with their goals. By focusing on data that directly informs your strategic direction, you can ensure your efforts contribute meaningfully to the organisation’s mission.

A well-constructed data strategy is essential. It provides a roadmap for how data will be collected, analysed, and used to deliver business value. Whether you’re looking to improve customer experiences, drive operational efficiencies, or uncover new market opportunities, your data strategy should be tied directly to business goals and outcomes. 

To get started on developing your data strategy, check out our guide on how to write your data strategy and learn more about crafting a data-product focused strategy.

Data-Driven Decision Making

By now everyone knows, and we’ve said it before data plays a crucial role in decision-making. In the past, many decisions were based on intuition or experience. While this is still a valid approach, data enables companies to make faster, more informed decisions with less room for error.

The value of data in business is amplified when decision-makers can quickly access accurate information. Businesses that harness data effectively can reduce guesswork and take decisive action that drives results. Modern data science tools, like data warehouses and analytics platforms, can help speed up this process by organising vast amounts of information and making it actionable.

If you are looking to leverage the power of advanced analytics and artificial intelligence, Oakland offers AI services to help businesses transform raw data into actionable insights.

When discussing how data becomes business value, it’s important to consider how you’re collecting, managing, and using that data. Companies that excel are often those that understand their processes deeply and can translate insights into actions. Without well-defined processes and data governance, efforts to extract business value from data can falter. To learn more about ensuring proper governance, explore our governance services.

The Importance of Data Quality

Another key factor is data quality. Measuring the business value of data quality is crucial for ensuring that decisions are based on accurate, reliable information. Poor data quality can lead to costly mistakes, inefficiencies, and missed opportunities.

Oakland works with organisations to assess and improve data quality as part of a broader commitment to continuous improvement. A key part of this is ensuring that data is properly governed, standardised, and accessible to those who need it. Companies that invest in improving their data quality often see significant returns, from better decision-making to enhanced operational efficiency. 

For guidance on this process, take a look at our guide on mastering successful data projects.

Staying Competitive in a Data-Driven World

In today’s fast-paced business environment, staying ahead of the game means making the most of your data. It’s not just about collecting it—it’s about putting it to work. Data is the foundation of innovation, growth, and the ability to adapt to market changes. The fact is, if you’re not using your data effectively, you’re leaving opportunities on the table.

The truth is, businesses that don’t embrace a data-driven approach risk becoming obsolete. Everyone else is using data to streamline operations, uncover wastage, and risk, deliver personalised experiences, and uncover new growth opportunities. Staying competitive means you need to make your data work for you; unfortunately, hoarding it and hoping for the best isn’t going to cut it, particularly in the new world of Generative AI.

For more insights into how to build a robust data strategy that supports your organisation’s goals, explore our blog on the purpose of a company’s data strategy.

Our Verdict? Data as the Key to Continuous Improvement

Here’s the thing about data: it’s not just numbers on a screen—it’s your best friend for continuous improvement. You can’t fix what you don’t understand. To improve, you need to know where you’re starting from, and that’s where data comes in. Once you’ve got your baseline, you can make changes, track what’s working, and keep improving.

At Oakland, helping businesses do this is what we’re all about. We started out 40 years ago in quality and continuous improvement and we wrote the book on SPC (literally) so there is no-one that knows more about driving business improvement whether it’s building a strategy, setting up data governance, creating powerful platforms, or diving into advanced analytics, we make sure your data efforts align with your big-picture goals. It’s not just about collecting data—it’s about making it work for you.

If you’re ready to start treating data as the strategic asset it is, you’ll be unlocking real business value and setting your organisation up for long-term success. Curious how we can help? Check out our services page and let’s start the conversation!

The post Why Is Data Important for Business? appeared first on Oakland.

For organisations to harness AI’s potential, robust data governance must be in place. Here at Oakland, like many data management consultancies we’ve been exploring AI-driven transformations, the differences between data governance and AI governance, and how businesses can overcome common challenges to ensure AI success. 

The Critical Role of Data in AI Success 

AI, at its core, is a data product. Whether in machine learning models, large language models (LLMs), or generative AI applications, the principle of “garbage in, garbage out” remains true. If AI systems are trained on poor-quality data, the outcomes will be flawed, potentially leading to misinformed decision-making, regulatory compliance risks, and reputational damage. 

Key data challenges impacting AI initiatives are: 

1. Data Quality Issues – AI models depend on accurate, complete, and consistent data. Missing values, duplication, and outdated records compromise AI outputs. 
2. Data Ownership Ambiguities – A lack of clear data ownership makes accountability difficult and slows decision-making. 
3. Understanding Data Landscapes – Without a well-documented data architecture, it is challenging to trace data lineage, assess data integrity, and ensure compliance. 
4. Bridging the Gap Between Vision and Execution – Business leaders often have ambitious AI visions, but data professionals struggle to translate these into feasible implementations due to underlying data constraints. 
5. Unclear Business Value – Lack of alignment to wider business goals and challenges to determine where AI efforts will deliver most value. 

Additionally, there is often a disconnect between executives pushing for AI adoption and the data teams responsible for managing the infrastructure. Many assume that AI will work out of the box, failing to recognise the need for strong governance principles that allow AI initiatives to scale successfully. Without this, projects often result in frustration, misalignment, and ultimately abandonment. 

Data Governance as the Foundation of AI Success 

This is where working with a data management consultancy like Oakland can play a crucial role in addressing these challenges. We provide organisations with structured methodologies to manage data effectively, ensuring data is accurate, reliable, and accessible for AI applications. 

Key Elements of Data Governance for AI 

1. Data Lineage and Provenance – Organisations must track where data originates, how it has been transformed, and who is responsible for it. 
2. Data Quality Management – Implementing data validation, cleansing, and monitoring processes ensures high-quality input for AI models. 
3. Ownership and Accountability – Establishing clear roles for data stewardship fosters responsibility and trust. 
4. Privacy and Security Controls – Protecting sensitive data and ensuring AI models comply with regulations like GDPR and the EU AI Act is critical. 
5. Feedback Mechanisms – Continuous monitoring and improvement of data assets and AI outcomes create an adaptive governance framework. 

Organisations that embrace these principles avoid the common pitfalls of AI projects, such as reliance on poor-quality datasets, uncertainty around data sources, and misalignment between business strategy and AI deployment. 

Data Governance vs. AI Governance: What’s the Difference? 

While data governance focuses on managing data assets within an organisation, AI governance extends beyond data to include: 

• Model Lifecycle Management – Ensuring AI models are built, tested, deployed, and monitored according to best practices. 
• Regulatory Compliance – Adhering to AI-specific regulations such as the EU AI Act and industry guidelines. 
• Ethical Considerations – Addressing fairness, bias, and transparency in AI decision-making. 
• Risk Management – Identifying and mitigating risks associated with AI models, including unintended consequences and security threats. 

Ultimately, AI governance builds on data governance. Organisations with a strong data governance framework can more easily scale AI governance practices, ensuring that AI models operate with integrity and compliance. 

As organisations navigate this landscape, they must also consider how AI itself can assist in governance efforts. AI-driven tools can help automate data lineage mapping, anomaly detection, and compliance monitoring, making governance more scalable and efficient. 

The Business Case for Investing in AI and Data Governance 

To secure executive buy-in for AI and data governance initiatives, organisations should: 

• Align with Business Objectives – AI initiatives should support core strategic goals, such as improving customer experience, increasing operational efficiency, or enabling new revenue streams. 
• Prioritise Use Cases – Focusing on specific, high-impact AI applications rather than boiling the ocean ensures measurable results and quick wins. 
• Leverage Existing Capabilities – Organisations with mature data governance practices can extend them to AI governance, optimising resource utilisation. 
• Demonstrate ROI – Quantifying cost savings, risk reduction, and performance improvements strengthens the case for investment. 

How Data Governance and AI Governance Enhance Business Performance 

Improved Decision-Making 

With well-governed data, organisations can confidently rely on AI-driven insights to make strategic business decisions, reducing uncertainty and improving outcomes. 

Increased Efficiency and Scalability 

AI-powered automation can eliminate repetitive tasks, but only if the underlying data is trustworthy. Data governance ensures that automation is built on a solid foundation, allowing businesses to scale AI initiatives seamlessly. 

Regulatory Compliance and Risk Mitigation 

AI regulations are rapidly evolving, and organisations that fail to comply face fines and reputational damage. Data governance ensures that AI models adhere to legal and ethical standards, reducing regulatory risks. 

Competitive Advantage 

Organisations that integrate AI governance with strong data management practices differentiate themselves by delivering AI solutions that are accurate, reliable, and compliant, gaining an edge over competitors. 

Implementing a Structured Data Governance and AI Governance Framework 

A structured, standardised approach to governance is key to AI success. Best practices include: 

1. Building a Cross-Functional AI Governance Team – Engage data stewards, compliance officers, IT leaders, and business stakeholders. 
2. Developing a Clear Data Strategy – Define data ownership, quality standards, and governance policies. 
3. Leveraging AI for Data Governance – Use AI to automate data lineage mapping, quality assessments, and compliance monitoring. 
4. Establishing Trust and Transparency – Ensure AI decisions are explainable, auditable, and aligned with ethical standards. 
5. Continuously Refining Governance Practices – AI and data governance should evolve with changing business needs and regulatory landscapes. 

Organisations that view AI as a data-driven capability rather than just a technological innovation position themselves for long-term success. Data governance provides the critical foundation needed for AI initiatives to thrive, ensuring high-quality data, clear accountability, and compliance with regulations. By aligning data governance and AI governance, businesses can unlock AI’s full potential while mitigating risks, ultimately driving better decision-making, efficiency, and competitive advantage. 

For enterprises looking to embark on or refine their AI journey, investing in data governance is not optional it’s a necessity. 

“AI, Generative AI or Agentic AI holds great promise. However, many organisations currently feel held back by the necessity to validate their data thoroughly, unclear understanding of AI risks, and the potential for unforeseen issues, like bias and data security. Either making them hesitant to fully commit, i.e. death by POC, or unrealised sustainable value when deployed. 

These are unchartered waters for most. The good news is you can choose the pace you want to go. Navigating AI governance is like steering a ship through a storm; robust data governance is your compass, guiding you to safer, clearer waters”.

Zareene Choudhury – Oakland Data Governance Lead

Want to find out more?

If you’d like to learn more about Oakland’s data governance consulting services, then get in touch by emailing hello@weareoakland.com If you’d like to learn more about what a suitable data governance framework looks like, click on the link.

The post Data Governance in the Age of AI appeared first on Oakland.

But what does a TOM entail? How can it help complex, enterprise-level organisations achieve their goals? And, if your specific focus is an organisational subset, like Data & Analytics, how can a specific functional strategy, like data strategy, help define the roadmap to provide localised impactful results whilst also aligning with the wider Operational Model of the wider business? 

Understanding how to implement a TOM is essential if you want to drive meaningful organisational change and ensure different functional areas, like Data, share capabilities and consistently set and align to common principles. This guide explores the fundamentals, their importance, applications, and examples. We will also delve into specialised frameworks like the Data Target Operating Model and the role of capabilities, like Data Governance, in shaping a model.

Our data strategy experts can help you understand your current operating model, pains and problems within it, and help you generate the vision and strategy that will inform the design of your Target Operating Model and make it a success. Get in touch and see how we can help.

A target operating model is a blueprint that defines the desired future or target state of your organisation’s operations. It serves as a guide for how your business will work to achieve its strategic objectives. By detailing the interplay of strategy, people, processes, technology, and data, a target operating model provides clarity on what needs to change to align day-to-day activities with long-term goals. It should be an essential tool in engaging with leaders within the business to get buy-in for business transformation initiatives. Your target operating model supports the development of roadmaps that help create a culture of continuous improvement delivering your corporate strategy more efficiently.

You’ll often hear Architects talk at length about People, Process, Technology, and Data as key components or domains; but at Oakland, we feel that talking about them in such a siloed way can somewhat miss the point in terms of unlocking true business value. You can very rarely solve real problems by focusing on them independently and the reality of modern business is that technology over-indexes as the focus of investment and delivery.

Our point of view is that for TOM, the value is not in the domains themselves, but where they overlap. You can unlock competitive advantage from Enterprise Architecture to:

• ENGAGE customer and colleagues with better process experiences.
• EMPOWER your people to deliver through better tools and systems.
• SCALE your business through data-driven processes, underpinned by carefully chosen technologies
• AUTOMATE key processes, enabled by process controls and data insights.

So, what should your target operating model focus on? 

Well, it should outline how your organisation’s structure and the interplay between people, process, technology, and data can support its strategy; how processes can be streamlined for efficiency; and how technology will enable those processes. It also addresses how your team is organised and empowered to work more effectively. With a well-defined model, you can ensure your operations are aligned with your objectives and optimised for scalability and resilience.

If you think about your critical business processes, they revolve around creating and managing data around objects and entities, like Assets, Products, Employees, and Customers.  These physical, and sometimes abstract, entities exist as logical activities linked together to generate organisational value – we call these connected up activities “Value Chains” and they deliver the purpose of your business. A Target Operating Model seeks to understand the composition and organisation of people, processes, technology and data to maximise efficiency and reduce disconnects within a function and between your business functions.

A robust strategy that delivers your organisational vision is essential in creating and implementing a Target Operating Model, particularly if it is intended to inform a wider digital transformation of your business. Learn more about our data strategy consulting below.

What Is a Data Target Operating Model?

It’s exactly what it sounds like – a TOM, but specifically targeted at a more ring-fenced and focused understanding of delivering your Data Strategy. Therefore, as a subset, a Data TOM is a vital component of and must be aligned with your broader organisational TOM, even if it is designed in isolation from it. 

Data is perhaps your business’s most valuable asset. If you think about your critical business processes, they revolve around creating and managing critical data about objects and entities, like Assets, Products, Employees, and Customers. 

These physical, and sometimes abstract, entities exist as logical relationships linked together through process to generate organisational value – we call these connected up activities “Value Chains”. Value chains, and the groups of co-ordinated processes generating value are, in turn, supported by capabilities that consider a range of components, including people skills and training, culture, roles and relationship definitions in addition to the more standard process, technology and data considerations.

A Data TOM specifically focuses on how data is managed, governed, and leveraged to support the achievement of wider business goals. For example, there is little benefit in generating a whole suite of sales insight data unless that insight is provided in a way that can be used by the Sales function. Unlike traditional operating models that concentrate on your organisation’s overall functions, a Data TOM concerns the frameworks and systems that support data-driven decision-making. Data & Analytics is a core value-chain enabling activity that supports successful decision-making in other parts of the business.

Importantly, activities within these value chains may fall within the accountability sphere of people in other business functions – all of whom likely have their own “operating models” – think of operational platforms that are often the primary source of data your Data Organisation operates on. This is why a Data TOM is a vital component of your broader target operating model and why conscious effort must be put in to align with the “bigger picture”. It’s difficult to achieve operational excellence when everyone is doing their own thing in an uncoordinated way.

A well-designed Data TOM aligns data initiatives with your organisation’s strategic objectives, ensuring that data is both effectively collected, stored, and utilised. It includes robust governance frameworks that establish policies, roles, and responsibilities for managing data securely. Moreover, it incorporates the right technologies and tools to enable seamless data integration, storage, and analysis. It also acts as a feedback loop, providing insight to the business about issues in the way that data is defined, collected and managed in source platforms, which creates all sorts of issues well before the Data function touches it. 

Lastly, a Data TOM also considers the cultural aspects of a data-driven organisation, identifying the skills and mindsets required to embed data as a core asset and supporting better ownership and management of data as a holistic strategic imperative.

For further insight into crafting a data strategy that integrates seamlessly with your operating model, read our guides below.

What Are Data Governance Target Operating Models?

Like most things, Operating Models can be layered and complicated and it really depends on the language your business wants to align to. Some may call it a Data Governance Operating Model, which underpins your organisation’s ability to manage data effectively.  Others may call it a Data Governance Data Capability definition – as a component of the Data Target Operating Model. Po-ta-to, Po-ta-to.

Either way, this a prescriptive (for your organisation) model that establishes the policies, processes, and technologies necessary to ensure that data is accurate, secure, and accessible, and well managed consistently in all areas of your business. Again, Data reaches beyond the boundaries other functions may recognise to ensure everyone manages the life-blood of the business (data) in a consistent way, and largely to reduce the effort of wrestling the data into a valuable resource for data-driven decision making. 

Governance frameworks define who is owns and responsible for what types and categories of data, ensuring clear accountability and established methods of raising quality issues and ensuring ongoing improvement of data over time. They also set out policies for data business and technical, definition, usage, privacy, and compliance: critical if you operate in a regulated industry.

In addition to governance frameworks, A Data Governance Operating Model considers which are the right technologies to enable effective monitoring and control. Tools for data cataloguing, lineage tracking, and compliance reporting play a vital role in maintaining data quality. 

However, technology alone is not enough – your organisation must also foster a culture where data is seen as an essential-for-life asset. This cultural shift and shared accountability is often the most politically challenging aspect of implementing Data Governance but is crucial for long-term success.

For actionable insights into developing governance frameworks, explore our data governance services below.

What Does a Target Operating Model Comprise of?

A Target Operating Model should benefit operations across your entire organisation. Typically, it should describe each capability in terms of a consistent set of components and co-ordinate with each other to generate collective value outcomes:

• Strategy & Purpose: what is the vision and mission and what’s the driving force and intended outcomes and how it will evolve positively over time
• People: Humans and hierarchies are complex, so it should come as no surprise that there are many aspects to consider including:
  • Roles & Responsibilities – who is responsible for doing what and when, and how events and issues are escalated and resolved.
  • Leadership & Talent – how is control exerted across the organisation and ensuring succession is actively considered and planned for in colleague development.
  • Structures – how capabilities are brought together to produce effective functions, completing specific activity.
  • Culture & Ways of Working – the creation of a methodologies for co-ordinating activity within teams, and to ensure teams can interact with each other collaboratively. Governance is central in providing standardised “rules of engagement”.
• Processes: The workflows upon which your organisation operates.
• Technology: The technological infrastructure that augments with the workforce, provides efficiency through automation, and can easily be scaled as the business grows. Consistent principles help to provide better alignment between tools.
• Data: Data and its management is central to understanding of all other components, considering this is the key resource generated and managed in workflows, supports customer outcomes and the business’s understanding of how well all the components are operating with each other.
• Customer experience: How the above impacts the customer experience.

Why Are Target Operating Models Important?

The importance of a target operating model lies in its ability to translate your vision for your organisation into a successful plan of action. It creates alignment between your strategic goals and operational execution, ensuring that every function and process contributes to and drives forward success. 

By providing clarity, TOM lets you identify inefficiencies, eliminate redundancies, and optimise workflows. And if you’re preparing for growth or transformation, a target operating model acts as a target state that can be compared against the current state to identify capability development areas that can be prioritised into a roadmap of continuous change initiatives outlining how to scale your operations effectively and safely.

Going further, Target Operating Models also help mitigate risks by establishing clear roles and responsibilities, ownership of processes and data, governance structures – essential in today’s complex regulatory environment. They also help your company become more data-driven, integrating data and performance metrics into daily business discussions, facilitating better decision-making by integrating analytics into everyday operations. 

For business leaders, a TOM is not merely a tool for operational efficiency – it is a foundation for sustainable growth, innovation and continuous improvement.

“Target operating models are important because they provide a tangible description of your future organisation that can help underpin stakeholder discussions and navigate organisational politics, supports identification of key next steps and their priority, and provide a high-level technical blueprint enabling design and delivery functions.”

Alex Guy – Group Enterprise Architect

What Are Some Examples of Target Operating Models?

To understand how a target operating model works in practice, consider the example of a large public sector organisation undergoing transformation. 

The organisation, reliant on legacy systems and suffering from poor knowledge management, wants to implement new systems and behaviours to use this data. The aim is to improve its service levels for the public, become more efficient, and use its data for continual improvement, providing lasting improvements.

In developing its target operating model, the organisation identifies its strategic objectives, such as reducing operational costs by 20%, while improving customer satisfaction scores by 30%. It analyses its current operations to work out what is a barrier to these improvements – legacy systems, siloed data, gaps in knowledge management, or staff unfamiliarity with modern systems.

A plan is then created to address each of these challenges, including:

• A data strategy to support the organisation’s overall digital transformation.
• A cloud-based knowledge management platform utilising generative AI. 
• New data governance processes. 
• Staff training programmes and forums. 
• The creation of a central team to oversee the improvements and ensure implementation.

The organisation creates a roadmap to make the changes and sets about implementing them. It charts progress, holds itself accountable, and iterates upon the target operating model once it is live, focusing on new challenges.

How to Build a Target Operating Model

Developing a target operating model is a structured process.

1. Strategy: It begins with defining your strategic intent. By clearly articulating the business mission and vision, the outcomes that need to be true, you provide a baseline of what the target operating model will ultimately need to achieve. Strategy sets the foundation for aligning operational action with overarching business goals. 
2. Analysis: The next step involves assessing the business’s current state operating model to identify pain points, gaps, and inefficiencies. This diagnostic phase provides a baseline for designing the organisation’s future state. It’s not all bad, you should also identify what is good and needs to be retained as you enter business transformation mode.
3. Recommendations: Once the desired current and future states are defined, you can begin to map out gaps between the two states and understand better the scale of change required across the business or its functions. Options can be developed to inform critical decisions, and prioritisation can be driven by insight rather than best-guessing or gut feel.
4. Roadmap: Creating a detailed roadmap is essential once gaps, dependencies, and priorities are understood.  Roadmaps help with broader engagement across the business, can help sign-post change to areas of the business change will be co-ordinated with, and helps prioritise initiatives and set realistic timelines. 
5. Implementation: This involves executing planned changes while monitoring progress to ensure the target operating model delivers its intended outcomes. The process doesn’t end with implementation; you need to continuously evaluate and refine your model to adapt to evolving market conditions.

Explore our case studies, such as Yorkshire Water’s transformation journey, to see how this process has worked for leading organisations.

Common Challenges in Target Operating Modelling

Despite its benefits, developing and implementing a Target Operating Model is not without challenges. 

One common obstacle we often come across is resistance to change, as your employees and stakeholders may be hesitant to adopt new ways of working. Cultural inertia can be a significant barrier, particularly in organisations with deeply entrenched processes.

Organisational politics, prioritisation of funding, and the individual plans of the stakeholders you need support from can be extremely difficult to navigate, especially if it involves gaining financial support and borrowing resource, making delivery of their agenda more challenging. The worst case is obviously where your agenda is moving in a completely different direction – which is why co-ordinating and dovetailing functional operating models into the Master Organisational TOM is important.

Another challenge is aligning multiple business units and functions, which can be complex. Coordinating efforts across departments requires strong leadership, clear communication, and the ability to spot the right compromises at the right time. It’s important to ensure that there is an identified “ultimate arbiter” that can fairly engage with and help resolve disputes and blockages quickly. Resource constraints, such as budget, time, or expertise, can also impede progress, as can “progress” that results in constraints for other stakeholders or functions who are actually responsible for impacted processes, platforms, people and data. 

Finally, fragmented data systems can create silos, making it difficult to achieve the level of integration required for a successful target operating model. Data held by functional areas is not inherently bad – it’s the idea behind concepts like Mesh and Fabric.  However, things can start to fall apart quite rapidly without consistent standards, policies and principles.  

How Can Data Strategy Consulting Improve Target Operating Models for Large Enterprise Organisations?

Implementing target operating models can be troublesome for large organisations. Thankfully, data strategy consulting can significantly enhance the process by providing the necessary inputs about organisational issues and, key objectives, gaps in ability, future focus and an understanding of valuable capabilities that can be leveraged.  Data Strategy consulting provides expertise in methodology, frameworks, and tools to effectively integrate data-driven decision-making to clearly articulate vision, mission, and objectives, that ultimately inform the operating model. Here’s how you can improve TOM development and execution with data strategy.

Aligning Data Strategy with Business Objectives

Data strategy consulting ensures that data initiatives are closely aligned with the organisation’s strategic goals. This is where data consultancies like Oakland come in. We can help identify how data can enable your target operating model by supporting key priorities like customer insights, operational efficiencies, and new sources of revenue, and dovetailing capability developments into your business without upsetting the wider organisational operating model. This alignment ensures the model is not just operationally sound and practical to implement, but also designed to leverage data as a strategic asset.

For example, our work with RAW Charging helped them chart a vision, strategy and roadmap for data to transform their business.

Optimising Data Management 

Effective data management is foundational for a successful target operating model. A data consultancy can bring best practices for data collection, storage, integration, and analysis, ensuring data is accessible, secure, and reliable. This helps eliminate silos, reduce inefficiencies, and simplify operations across your business, or better support more complex and fragmented organisations where consistent standards and principles are needed for more federated, independent functions. Learn how we helped UK Power Networks gain all these benefits and more.

Enhancing Data Governance 

Data governance is critical to ensuring that the data used in the model is accurate, compliant, and trustworthy – particularly in heavily regulated industries like finance or healthcare. 

Our experts can be invaluable here, designing and implementing governance frameworks that define clear roles, responsibilities, and policies for data ownership and management. This includes establishing processes for data quality monitoring, security protocols, and regulatory compliance, all of which are essential. 

Learn how we set data use standards for the Information Commissioner’s Office.

Choosing the Right Technology

Choosing the right tools and technologies to support the model is a complex task, especially if you work in a large organisation with diverse needs.

Consultants evaluate and recommend the right technologies for your business, such as cloud platforms, analytics tools, and machine learning solutions, making sure they align with your goals.

Enabling Data-Driven Decision-Making

Data strategy experts ensure that data is positioned to enable better decision-making within the target operating model. By developing analytics capabilities, data consultants can help you derive useful insights from your data to optimise operations, forecast trends, and identify opportunities.

In a manufacturing enterprise, for example, predictive analytics powered by a well-designed data strategy can optimise supply chain processes, reducing costs and improving efficiency as part of the model.

Creating a Data Culture

Data experts can be invaluable in upskilling your teams and fostering a data-driven culture within the wider organisation. This ensures that employees at all levels understand how to use data to achieve their target operating model objectives. 

Training programs, workshops, and change management initiatives can all also be tailored to embed data literacy and drive the adoption of data-focused practices.

Supporting Scalability and Future Growth

A data strategy designed with scalability in mind ensures that the target operating model can evolve alongside the organisation. Our team can help you assess future requirements, such as handling larger datasets, integrating advanced technologies, or expanding into new markets, and build these considerations into the model’s framework.

Measuring and Optimising Performance

Finally, data strategy consulting integrates performance measurement into the target operating model, using key performance indicators (KPIs) and metrics to track progress and identify areas for improvement. Our experts provide ongoing support to refine data strategies and operational practices, ensuring the model continues to deliver value over time.

We Can Help Your Target Operating Model Become Reality

A Target Operating Model is a powerful tool if your organisation wants to align its operations with strategic objectives. Whether it’s a comprehensive model or a specialised one that targets just your data and technology, these frameworks let you optimise performance, enhance scalability, and drive innovation. For leaders and decision-makers, investing in a well-designed model is not just an operational necessity – it’s a strategic imperative to optimise the value delivery from business investment.

Our expertise in data strategy and data governance can help your organisation craft and implement an effective target operating model tailored to its unique needs. Visit our case studies to learn how we’ve helped leading enterprises achieve transformational success. 

Ready to take the next step? Contact our team today to start building your Target Operating Model.

The post Target Operating Model: Delivering your Business Strategy appeared first on Oakland.

• The UK AI Market is worth more than £16.6B and is projected to grow to over £700B by 2035 
• 28% of businesses have built their own AI solutions or are using existing AI tools like CoPilot or ChatGPT 
• In the Autumn Budget in November 2024, the UK government laid out plans for an AI Opportunities Action Plan, which should provide a ‘roadmap to capture the opportunities of AI to enhance growth and productivity. 

Are you in the AI game? 

There’s no question that ‘AI-driven’ business is here to stay… much like ‘data-driven’ (insert your own marketing line here). But Data Governance Consulting at Oakland is very much rooted in the operational and practical side of real-world business. 

We prefer to consider it as “AI-enabled: A business that gains real value from their AI solutions’’. 

Back to our earlier point, 2025 AI is all about Agentic AIs: But is Agentic AI the future or just another AI technology? (although we have been developing intelligent agent solutions way before they were renamed Agentic AI, and all the major tech providers got on board)  

The beauty of AI Agents is that they are so much more than chatbots. They enable organisations to deliver focused value and target specific problems and opportunities in the data ecosystem. This is why honing down on use cases and relating them to the pains of business users is crucial in creating lasting AI solutions. 

Read the small print! 

So yes, AI promises to improve decision-making and deliver real value in a game-changing way. But… make sure to pause and read the small print on this promise, in particular: 

‘Your data quality and your AI governance capabilities will directly impact the value you extract from your investments in AI’ 

A study from RAND concludes that 80% of AI projects will likely fail. This prognosis is not dissimilar to Gartner’s, which predicts that 30% of GenAI projects will be abandoned after POC.  A key reason mentioned in both studies is the lack of high-quality data. 

What is high-quality data? (A holy grail..? Of course not!) 

• First and foremost, it’s important to know what data you have, where it exists, and who in the business has accountability for it. You don’t need to boil the ocean. Start with critical data domains related to your AI use cases.
• It knows what benchmarks and rules need to be applied to your data to classify it as good or bad: it is unlikely that you need 100% completeness for some of your data sets – understand what business purpose the data caters for, and this helps you define your benchmarks. 
• Having a view of your data eco-system is not just about having an architecture map that details the technical integrations. Have a view of the business processes and information requirements that underpin this architecture. Highlight where the risks are, e.g., because of duplication, manual interventions, or siloed processes. 
• Having mechanisms to monitor, log, and address data issues as they get flagged. This allows the business to mitigate risk and avoid confusion when resolving issues quickly but also allows it to be more strategic in improving business capabilities. 
• Documenting your critical data dependencies and how your data flows cross-functionally, i.e., your data lineage. This not only helps to make informed decisions on what protocols are needed to keep your data secure and address issues but also allows you to nail down requirements when scaling your data capabilities quickly. 
• If you are in a heavily regulated industry, then data security and regulatory compliance will require more robust data governance unless you want to risk £££ in fines and loss of your reputation. 
• People, People, People: Do not forget that ultimately, it is your business users who will make or break any data initiative. Training, engagement, data mindset, and change management are all key contributors to creating and sustaining good data. 

AI Governance vs Data Governance

Governing AI solutions is indeed wider than just governing your AI data. Trust is perhaps the greatest challenge to business adoption of AI, which is why it is just as important to govern the technical side of things, for example, how the models and algorithms are developed, trained, and deployed, as well as the architecture they rely on. 

AI governance faces a few challenges, including: 

• Explainability: A lot of AI models can feel like mysterious black boxes – we know something is happening, but we have no idea how! 
• Unstructured Data: Things like emails, documents, images, and videos are everywhere. Unstructured data makes up about 80-90% of all new data. It’s like trying to organise your sock drawer – where do you start with all the holey, single, Peppa Pig themed socks that have found themselves in your draw?
• Ethical Concerns: It’s critical to make sure AI systems are fair and unbiased. 
• Data Privacy and Security: Protecting sensitive data used in AI training and operations is essential.  

Do I need Data Governance before Artificial Intelligence?

Data governance is essential before implementing artificial intelligence (AI) if you want your AI initiatives to succeed. Here’s why:

1. Data Quality Drives AI Performance

AI systems depend on high-quality data to learn and make decisions. Without proper data governance, your AI models may be trained on inaccurate, incomplete, or biased data, leading to unreliable outputs and potentially harmful business decisions.

2. Ensuring Data Compliance and Security

AI often involves handling sensitive data. Data governance ensures compliance with regulations such as GDPR, CCPA, or industry-specific standards. This reduces the risk of legal penalties or reputational damage from improper data usage.

3. Establishing Data Accessibility and Consistency

Data governance creates a framework for consistent data definitions, ownership, and access control. AI thrives on integrated and well-structured datasets. Governance ensures that the right data is available to the right teams in a usable format, eliminating silos and duplication.

4. Mitigating Bias and Ethical Risks

Poorly governed data can introduce bias into AI models, leading to unfair or unethical outcomes. Data governance provides processes to identify, address, and monitor bias, ensuring AI systems make equitable decisions.

5. Cost and Efficiency Benefits

Investing in data governance upfront reduces the time and cost of preparing data for AI initiatives. It prevents expensive rework or failures by addressing data issues early, rather than discovering them during or after model development.

Read our blog on how you can demonstrate the ROI of Data Governance.

6. Scalability for Future AI Projects

Data governance provides a scalable foundation for expanding AI capabilities. It ensures that as data volumes grow, they remain manageable, trustworthy, and usable for future AI-driven projects.

Nothing’s perfect.. and that’s alright 

At Oakland, data governance consulting doesn’t mean you need perfect data to embark on or progress on your AI roadmap.    

AI gets your data talking.. you might not like what it says if your data is incomplete or of poor quality, but on the plus side, you can also use AI to flush out where your data issues are (every cloud has a silver lining) 

How do we manage AI regulatory requirements 

As with everything, a balanced approach is essential.  

Yes, it is a balancing act between being compliant and giving the business space to innovate. The good news is that, although the AI regulatory landscape is continuously evolving, a key theme is the ability to demonstrate that your AI is trustworthy—and a data governance framework allows you to demonstrate trustworthy AI data pipelines. 

“AI, Generative AI or Agentic AI holds great promise. However, many organisations currently feel held back by the necessity to validate their data thoroughly, unclear understanding of AI risks, and the potential for unforeseen issues, like bias and data security. Either making them hesitant to fully commit, i.e. death by POC, or unrealised sustainable value when deployed. 

These are unchartered waters for most. The good news is you can choose the pace you want to go. Navigating AI governance is like steering a ship through a storm; robust data governance is your compass, guiding you to safer, clearer waters”.

– Zareene Choudhury – Oakland Data Governance Lead

Want to find out more? 

If you want to know more about how to leverage Data Governance to get your AI Data trustworthy – then tune into our webinar, where Zareene Choudhury will be joined by Richard Adams of erwin by Quest. 

Get in touch

If you’d like to learn more about Oakland’s data governance consulting services, then get in touch by emailing hello@weareoakland.com

Want to read more about what a suitable Data Governance Framework looks like?

And if you want to find out how AI agents can make a difference to our business, read some of our case studies here, where we’re already delivering value to clients (well before AI agents became ‘the phrase of the year’! 

The post What is the Role of Data Governance in AI? Should You Even Care? appeared first on Oakland.

As a Microsoft Partner, Oakland’s Microsoft Purview consulting services give us access to the latest insights and best practices, providing our clients with unique support. We believe in the value of connecting data leaders with Microsoft experts to tackle pressing data challenges.

The workshop was a full house, largely due to Purview’s newest features and its updated billing model. One standout update is the ability to scan data sources without incurring additional costs. With Purview, billing now happens per unique governed data asset, which means that assets linked to multiple products are only charged once, and asset counts are taken daily—a change many have been eager to see.

Why Data Governance Tooling Matters

At Oakland, we know that data governance tooling can accelerate a data governance implementation. However, technology alone isn’t the answer. Our Microsoft Purview consulting services start with a strong foundation of people and processes. Before implementing any tool, it’s essential to establish a solid data ownership model and steering committees that ensure robust processes are in place, such as data quality management and improvement. With the right structure, tools like Purview can then help prioritise and streamline data governance.

Our data governance tooling guide also assists organisations in choosing the right solution by focusing on their most critical needs. No single tool is perfect for every data governance task, so understanding the organisation’s unique challenges allows for smarter decision-making when selecting a data governance solution.

Key Insights from Microsoft’s Team

Thanks to Microsoft’s Michael Robson and Mark Farrow-Smith, attendees received an in-depth keynote on Microsoft’s data governance journey. They emphasised that data governance is 92% people and process, and only 8% technology.

Michael presented three critical pillars for effective governance:

1. Data Discovery: Gaining a thorough understanding of the organisation’s data landscape, shared across functions, to avoid data silos.
2. Data Literacy: Building an organisation-wide understanding of data, creating a data-centric culture.
3. Data Quality: Ensuring that data entering systems is accurate, as this is essential for trustworthy business insights.

These pillars, combined with executive sponsorship, provide a solid foundation for a successful governance strategy. Michael also shared six lessons learned from Microsoft’s governance journey, from prioritising critical data domains to the importance of measuring governance metrics for continued executive buy-in.

Oakland’s Hands-On Demo of Microsoft Purview

After the keynote, Oakland’s data consultants, Alex Coulthard, Amy Farnfield, and Rehan Hussain, led an interactive demo showcasing Microsoft Purview’s capabilities. The demo began with Purview’s data scanning feature, highlighting how this capability accelerates data discovery by automating a process that might otherwise take weeks.

Our team also explored the Business Glossary, OKRs, and Critical Data Elements (CDE) features, discussing the advantages of Purview’s user-friendly glossary and explaining the enhancements we’ve built to streamline these processes. For example, Oakland has created a pipeline to bulk upload glossary terms, addressing Purview’s current lack of this capability.

The demo further showcased the roles and responsibilities supported by Purview, which allow for extensive customisation across different data ownership structures, making it adaptable to various organisational needs. Lastly, our consultants reviewed Purview’s handling of data products and assets, illustrating how documentation, such as data contracts, can be integrated within the tool.

Oakland’s team finished the session by discussing the role of data quality in data governance—a critical area for most organisations. Purview’s capabilities, combined with Oakland’s expertise in Microsoft Purview consulting services, offer a robust framework to help organisations accelerate their data governance journeys.

If you’d like to learn more about our Microsoft Purview consulting services, reach out to us, and let’s discuss how we can support your data governance strategy.

The post Microsoft Purview Latest Insights appeared first on Oakland.

But is it the right fit for your organisation? In this blog, we explore Purview’s features, deployment, and pricing, helping you determine if it can meet your data governance needs while ensuring compliance and scalability.

How Do You Govern Your Data?

Most organisations are exploding with data that has been collected, transformed, and reported on with the business requirement to improve decision making. However, this huge increase in the volume of data has come with a lack of accurate tracking which all too often hampers the actionable insights that the business stakeholders demand. As organisations become more data-driven, Oakland has seen a growth in 4 particular pains which have been increasing for the last few years: 

• How can we audit all this data to protect against data leaks and unexpected data loss which is especially crucial with the regulatory requirements of many organisations? 
• How can data users discover data and derive business value in an environment that changes constantly? 
• How can data consumers understand what the data collected means and turn this into business value? 
• How can we show the current data quality of key datasets? 

Plus, in a modern business environment, you may need on-premises and multi-cloud data governance solution which also easily integrates with your office 365 workloads.

Data Governance tooling can help mitigate these problems, and help with data management however, these tools are often complex to integrate to your entire data estate due to requiring: 

• The ability to scan a large variety of data sources. 
• A highly customised user interface. 
• A powerful search engine to find data assets by many different types of metadata attributes. 
• Technical experience to setup a Catalog, and Data Stewards with experience on maintaining a Catalog. 

These are just a few of the main requirements that create a software marketplace full of products that are often expensive and hard to implement and maintain which is why these are often not business friendly and/or very expensive.  

These products also need to ingest large amounts of sensitive data to meet user requirements, ironically creating a data governance concern in itself! 

Microsoft Purview aims to ease the pain of data governance by being feature-rich, easy to deploy, maintain and secure. But is it worth the cost, and can it compete with bespoke data governance companies that have a head start measured in years or even decades? 

What Are the Features of Microsoft Purview? 

Microsoft Purview has recently gone through a major new update which adds lots of features, so if you’ve dismissed Microsoft Purview before, we recommend looking again. 

• It’s connectors are very Microsoft-focused but cover most of its ecosystem: Azure, SQL Server, Power BI, and Office 365. If you’ve already bought heavily into Microsoft, you can scan most or all your data assets automatically. 
• It focuses less on connectors made by other companies but still covers many popular data products like SAP, Salesforce, Oracle, GCP Big Query, AWS S3, and Snowflake.  
• You can create and streamline business domains like sales, marketing, HR and supply chain, to help move data governance closer to the business  
• Data classification becomes more business friendly as you can classify data with 200+ pre-built classifications, as well as custom classifications. 
• The pre-built data governance compliance reports enable you to quickly check insights such as the percentage of data that has a data owner and the percentage of new data assets in the last month. 
• API and Python SDK enable you to create custom data sources where connectors don’t exist or mass updating existing scanned data assets. 
• Data quality is always a key issue Microsoft Purview has pre-defined and custom data quality tests. 
• Data Sharing allows users to give other users read-only data lake data access without having to copy data. 
• Data Governance can integrate with Master Data Management tooling like Profisee 
• AI powered with lots of integration with Microsoft Copilot to generate data documentation and data quality tests. 

How Do You Deploy Microsoft Purview? 

• Oakland has designed and built many data platforms, and we highly value any product that can be deployed quickly, has low maintenance, and will meet stringent client IT & security requirements. We believe Microsoft Purview is stronger than most data governance products when it comes to data protection. 
• It is as easy to deploy and maintain in Azure as any SaaS data governance product but also offers a choice – 20 plus regions to deploy into, including the UK. 
• Microsoft Purview can also keep all traffic in and out of its server on its private network using Private Endpoints, never touching the public internet, offering an extra layer of data security when creating a Data Catalog. 
• Scan Azure data via Managed Identity authentication which offers high-security data connections without worrying about managing passwords. 
• It can connect directly to scan on-premises and other public cloud data assets (for example, AWS and GCP), though it requires some technical knowledge to setup the networking. 

How Much Does Microsoft Purview Cost? 

Automated data governance tooling is expensive, with costs starting in the thousands of pounds for most products. Microsoft Purview arguably starts at a lower base: we’ve found it starts at about £250 per month. However, you will also be charged on top of the base cost for scanning data, which goes up the more data consumed. 

Due to the pricing being highly variable in Microsoft Purview we recommend building a proof of concept to road-test Microsoft Purview for a month or so to accurately measure costs. 

What Are The Alternatives to Microsoft Purview?

Note this isn’t a comprehensive list and is a quickly evolving space with new exciting start-ups, and apps entering all the time, but we hope it will help you make an informed decision. 

• Build your own: Building your own data governance tool offers complete customisation to your business’s unique needs, allowing you to tailor features and integrate seamlessly with existing systems. It gives you full control over security and privacy, ensuring sensitive data stays in-house, and avoids costly vendor licensing fees, providing better long-term ROI. You also avoid vendor lock-in, giving you the flexibility to evolve your tool as your governance requirements change.

While developing a custom tool requires a significant upfront investment, it fosters internal expertise and provides faster iteration when compliance needs or data regulations shift. By owning the solution, your organisation gains greater agility and control, avoiding reliance on third-party support or updates.

• Excel: low cost, low maintenance if data structures don’t update regularly, doesn’t require any specialist skills to build. While we suspect this is the most common type of data catalog used, we feel nervous about doing a data catalog in a data tool infamous for having poor data governance. It does not scale and requires lots of manual effort for any major changes to the organisation, creating data lineages and classification of sensitive data. 
• Automate your own solution by extracting schemas of databases and files. This is a nice quick way of generating a data catalog with low maintenance and little extra costs. You can also build a dashboard on top of the business intelligence (BI) platform of your choice. It requires minimum effort if the number of data assets is small. Although adding features like data lineage and classifying data sensitivity will require a reasonable amount of engineering effort, which makes buying off the shelf products more appealing.
  • You can combine this with a SharePoint Site to collect business information like Business Domains to make sure it is not just an IT exercise. 
• Databricks Unity Catalog – ideal for Databricks heavy data platforms, as it is a free extra. Though it will only scan what Databricks can scan. You can integrate with other data governance products, including Microsoft Purview, and update schema as they update in real time.  
• Mature products like Informatica and Talend. These tend to charge by the user and are more commonly found on-premise (though they can be configured and maintained in the cloud on Virtual Machines). They will likely cost the most; sometimes, this is significant, but these are feature rich, well-trusted and reliable.  
• New(er) products like Atlan and Immuta often focus on providing data governance to more recent cloud data tooling like Databricks and Snowflake but also often focus on making deployments into the cloud more accessible by offering deployments via Docker or Kubernetes.
  • Immuta also provides a single pane of glass for fine-grain data access across many popular data products that allows data access controls at a column and row level. 
• Open Source software like Datahub and Amundsen, both built by large tech companies (LinkedIn and Lfyt respectively). These are the go solutions if your organisation has the technical capacity to build and maintain complex workflows. They offer a lot of customisations and the possibility of no licence costs, so they can be much cheaper at scale and be more custom tailored to fit an organisation’s data governance needs. 

If you would like to see more tooling options and deep dive into how to select the right data governance tool to streamline your governance practices check out our data governance tooling guide.

Why Should You Choose Microsoft Purview?

In an increasingly busy data governance market, Azure Purview is a serious option to consider especially considering the new update which fills in some major gaps in its features. 

If you are looking for a data governance product that is easy to deploy, secure, catalogue, and classify data assets, and provides some customisation through APIs and user interface at a competitive cost, then we think Microsoft Purview is a good contender. 

However, and there is always a but, we do want to end on a cautionary note: we have found Microsoft Purview or indeed any other data catalog implementation fails more often because of either:  

a) Lack of data governance processes and people.  

b) Not knowing what business problems, you are exactly trying to solve. 

Rather than choosing the wrong tool. This is because data catalog requires constant maintenance to keep up with the evolving nature of any organisation, so needs to show a high level of return of investment and have the right processes and people to maintain it efficiently.  

Also remember to make sure that you select a tool based on the problems you have and how the tool can help you solve them and keep reviewing how it does can could add value.  If you would like to know more about how we help clients with data governance while achieving a quick return on investment, download our data governance guide.   

The post Is Microsoft Purview The Answer to Modern Data Governance? appeared first on Oakland.

Oakland, which has a proven track record in business improvement and transformation, has joined forces with Data Literacy Academy. This strategic partnership combines Oakland’s expertise in data strategy and technical implementation with the Data Literacy Academy’s specialised knowledge and resources in developing data literacy skills.

Why a Two-Way Street Approach to Data Literacy Matters

The partnership is built on the fundamental understanding that data literacy is a two-way street. It’s not just about data professionals understanding business needs; it’s equally important for business leaders and decision-makers to understand the potential and limitations of data. This two-way communication is essential for organisations to unlock the true value of their data and make informed decisions. Far too often, businesses work in siloed ways without communicating effectively, which can lead to missed opportunities.

Addressing the Evolving Landscape of Data and Business Leadership

The partnership acknowledges the evolving landscape of data roles and leadership. We are seeing a new wave of data leaders emerging, often with strong business acumen but limited technical expertise. This creates a crucial need to help these leaders bridge the gap between data capabilities and business strategy.

Everyone talks about the value gap of what people can achieve through data. Teams are particularly poor at defining or even trying to define ROI metrics. Back to the two-way street approach. Business people must share what problems or opportunities they are trying to create using data, and data people need to be able to translate the technology, what it delivers, and how it works.

Often, data literacy, or data training as most people call it, is seen along with data governance as a tick-box exercise. They don’t understand how to execute education in the right way to achieve the outcomes they are looking for. You don’t get endless chances to make people see the value in becoming data literate, so you have to get it right.

The Rise of AI and the Need for Support

The growth of AI is opening up the use of data to everyone. AI is probably powering half of the applications you use every day. It’s easy to think of AI as a chatbot or ChatGPT, but if you are tasked with how your business should use AI, that becomes tricky, and if you’re responsible for coming up with that solution, then it becomes even more challenging.

For AI to realise its full potential, it must move on from the chatbot. It must be process native. You need to design AI capabilities into your processes. It’s not just a technology solution. This involves a whole new level of data literacy!

There is a lot of groundwork to be done so businesses are truly ready for AI, and have the skillset needed to drive value from it. We believe this is an incredible opportunity; people shouldn’t be fearful. AI gives your data the opportunity to talk – just imagine what it can tell you. But you must be able to walk before you can run. You must be data literate before you can become AI literate.

Our Partnership’s Goals: Building Bridges and Fostering Growth

This collaboration aims to achieve two key objectives:

1. Embedding Data Literacy Early On: Unlike traditional IT projects, data initiatives often lack a focus on user training and adoption. The partnership aims to address this by highlighting the importance of data literacy and embedding it throughout the entire transformation journey, from initial planning to implementation and ongoing use.
2. Bridging the Imagination Gap and Empowering Growth: Through a clear understanding that people and technology have equal value. Our partnership aligns perfectly with that mission by equipping individuals with the data literacy skills needed to understand the “art of the possible” and make informed decisions that drive growth.

By bridging the gap between data and business understanding, this partnership empowers individuals and businesses to unlock the true potential of data. Ultimately, it’s about enabling freedom and growth through a deeper understanding of data and its potential applications.

What does our partnership look like?

Our customers will have the opportunity to get support on the technology side, as well as the people side. When training and data culture change is needed, Data Literacy Academy will set up an impactful change management programme to enable teams. Supported by Oakland through strategic planning and technical implementation.

All too many technology programmes end in disappointment.  With our combined expertise, our goal is that no business needs to experience failed technology projects again, because they had the professional resources to do it right the first time.

Mastering the 5 Stages of a Successful Data Project

We’re determined to turn the tide of lacklustre data projects, which is why we have created our new guide. A blueprint to mastering the 5 Stages of a Successful Data Project.

Download the guide, and you’ll learn:

• Insider tips to align your data and business strategies, avoiding the dreaded silo effect
• Actionable strategies to ensure your data projects are eagerly adopted across the board
• Real-life anecdotes to illustrate the dos and don’ts of data project management

The post Oakland Partners with Data Literacy Academy: Bridging the imagination gap appeared first on Oakland.

Now let’s extend this scenario beyond shopping and step into the shoes of an executive and a decision-maker in a business. Picture yourself, not just as a shopper but as a strategic buyer for the business. As you carefully assess items for their value, you also scrutinise potential investments, while ensuring that each item aligns with business goals and adds value.

Just as we carefully choose what adds value to our daily lives in a shop, businesses must critically assess and invest in assets, services, or strategies that bring tangible value to their operations and business objectives.  Therefore, being able to prove value to others in the business is vital. But how do we go about that for Data Governance?

Difficulties proving worth

There is little doubt that sometimes data governance can feel intangible. Isn’t Data Governance going to slow us down? It’s nice to have but not a must-have, isn’t it? What is the value of trust in data? What value does increased ability to collaborate bring? Giving the right people access and the ability to understand the data brings what value exactly?

Data and Analytics leaders often find it challenging to create a compelling business case for data and analytics governance improvement. Often, the business case is too technical and complex, which makes business stakeholders switch off and file it for another day.

So how do you go about showing why this is essential without sending your audience to sleep?

Tracking the impact of a change made by Data Governance to an outcome can be hard. For example, if we increase customer data quality, what impact does that have on sales or customer retention?

So, it is hard proving the worth of Data Governance. Just like the classic duck-rabbit image, people might see different things based on their understanding and the position they take towards it. How do we get the right people to understand Data Governance and see the value it brings?

Also be aware, the benefits of data governance can take time to materialise. How long before that is useful enough to be utilised for long-term planning and decision-making? When can you claim a saving in fines for adhering to regulations when they have never been directly sanctioned?

All these things make it difficult to quantify value in Data Governance, but it is by no means impossible.

What is the value of Data Governance?

Data Governance initiatives rarely start because someone just decides to do it. Businesses typically embark on these initiatives in response to growing concerns related to data risks, compliance challenges, and operational inefficiencies.

As defensive strategies, Data Governance initiatives act as preventive measures, safeguarding against data breaches, compliance violations and reputational damages.

They also pre-emptively address vulnerabilities and risks, which shows a strategic and forward-thinking approach. This proactive stance is necessary in the face of evolving data landscapes and the increasing complexity of regulatory requirements.

However, it is not just about defence. Offensive strategies that rely on high quality data, well-defined, accessible data often require Data Governance assistance. Businesses can leverage a well-established Data Governance framework to proactively drive business initiatives and gain a competitive edge. Governance can help to create business value through monetisation and exchange of information products in addition to helping you to allocate your scarce resources to high priority projects in line with the organisations strategic goals.

Now, when it comes to measuring the value of Data Governance, businesses often face the challenge of quantifying the value of this initiative. To effectively measure this value, we must understand the business needs. Why is your business in need of Data Governance? Is it because a project relies on high-quality data to be successful? Are customers getting a poor experience because advisors in one business function can’t see updated data entered by another function? Is the business struggling to meet regulatory demands?

Once we understand the business needs, it is easier to evaluate the value of Data Governance on key business outcomes, such as customer satisfaction and regulatory compliance.

Identifying the value of Data Governance is also a crucial step to measuring the value of data governance initiatives. In doing this, defining precise metrics and key performance indicators (KPIs) can be used to measure the value. If there has been prior identification, great! We’ve got our starting point for what value we are expecting to realise.

If not, then we can work out what value we are expecting to deliver by quantifying how we are going to contribute to the overall value. How much are we expecting to increase our colleague’s productivity by making data accessible? If a colleague cost £X per day, what does that productivity increase equate to?

What is the likelihood and severity of risks currently and how much will we reduce that? If the expected cost of the risk was Y, what does the reduced likelihood and severity reduce that to? How much will we increase customer satisfaction through making customer data more accurate and secure? If that is expected to increase total customer lifetime value or customer retention rates, what is the value of the data?

By regularly monitoring these metrics and analysing the results, we can effectively gauge the tangible benefits derived from Data Governance.

Let`s consider this scenario: A retail business is suffering from delayed customer shipment and invoicing due to inaccurate, incomplete and duplicate customer data on their system. They need to get better at managing this or they will lose business.

By adopting a Data Governance approach focused on Data Quality, the business was able to fix its customer data, significantly improving the Order-to-Delivery time and increase customer satisfaction, retention and loyalty.

The closer you can get to value that is measurable, the better. If you are struggling with that, here is a top tip: Make friends with a Management Accountant. Aside from the fact that they make great friends, they are also great at taking business decisions and working out what they mean in pounds and pence (or whatever denomination you work in).

Great, we have some value we can attribute to Data Governance effort, job done. Or maybe not. If other people are not aware of the value you are delivering then when those senior leadership discussions happen, how are they going to take it into account?

The final and most important bit – Communicate the value! For the value of data governance to be meaningful, it needs to be communicated effectively through compelling storytelling to engage the hearts and minds of your stakeholders. The best stories engage people at an emotional level by connecting personal and collective experiences that help you to get your message to resonate. Use storytelling to generate awareness of the value being delivered and make sure people understand how Data Governance efforts have contributed.

This can be substantiated with evidence, case studies, and testimonials from the people who have massively benefited from the initiatives. Bring your story to life with imagery, graphics, and videos.  Making these efforts visible and understandable ensures that the value is recognised and appreciated across the business. Highlight success stories such as increases in regulatory compliance or resolving of data quality issues. This shows the effectiveness of the initiatives and contributes to a positive narrative within the business.

In summary

Is proving the value Data Governance easy? Honestly, it depends on how you are set up to do so in your business as everyone will do it differently. However, if you are working on projects that are linked to delivering valuable outcomes to the business, it should be possible. Remember, it is an ongoing process and measuring its value takes continuous evaluation and feedback loops. Regular assessments ensure that Data Governance initiatives remain relevant and effective in the long run.

If you are looking for help with your Data Governance initiatives, please reach out to us for an assessment to kickstart your journey.

Authors:

Yejide is a Data Governance Consultant at Oakland. With a legal background spanning over a decade, Yejide combines her legal training with data governance to provide strategic advice and guidance to organizations. Beyond her professional achievements, Yejide is  passionate about sharing knowledge, insight and making a difference.

Rob Lancashire is a Data Consultant here at Oakland, focused on helping organisations with using data governance to release their data potential. Rob has worked in data for over 20 years, helping many companies improve their use of data to bring value to the whole business and has also produced a series of cartoon books to help others understand the data industry.

The post How can you demonstrate the ROI of Data Governance? appeared first on Oakland.

There is Something Human about Waste

Let’s start by setting the scene – an all too familiar one. Humans love waste. We throw away around 100 billion pieces of plastic every year in the UK, and around 1.9 billion tonnes of food that can be eaten is discarded on an annual basis, and even on a household level – between 9-16% of energy is wasted on standby.

This fascinating graphic from No Planet B by Mike Berners-Lee shows the world supply chain for food based on energy loss at each stage. As you can see, at every level of the picture, our food waste. The depressing part can be seen in the bottom three rungs; even if we’re lucky enough to have food we still waste around 50% of it.

The impact of this level of overconsumption is evidenced in the world around us: fires in Maui, droughts in Uruguay, and extreme flooding in Bangladesh – all directly attributed to climate change driven largely by Greenhouse gas emissions.

But what does this have to do with data?

It doesn’t come as much of a surprise that physical human behaviour also maps onto digital human behaviour. The attitude towards waste and overconsumption has transcended from the physical to the digital in the case of data – and that has led to some huge inefficiencies in the way that we currently manage data – directly resulting in some pretty stark impacts on the environment. We talk about inefficiency through a few lenses – the first being the underutilisation of resources which we can see represented by the fact that the utilisation rate of on-premises servers is only around 18%, and it doesn’t get much better when we talk about cloud storage and usage, where around 33% of paid-for services are considered waste.

Combine this with the fact that 55% of stored data is considered dark (currently unused, with no future use-case) – a frightening statistic! That means stored and processed data that will never realistically be used for operational or analytical purposes. It’s one of the darker legacies of the big data wave of 2017!

Finally let’s not forget the onslaught of Artificial Intelligence. The amount of data we are creating is out of control.

Data Centres – The Lords of Darkness

When you consider the artificially inflated needs of server racks, the ramifications of this wastage are very real – data centres are being constructed at a rate of knots and are having pretty dire consequences on the environment.

Think about all the utilities that you need to maintain a data centre, primarily you might just think about the energy costs of powering your servers. Ignoring the energy requirements for cooling and lighting, server energy usage on its own has the potential for improvement. Server configuration and distribution of workloads can often drive higher than necessary energy usage.

Due to the sheer amount of energy digital technologies power through, data centres have become the world’s second largest source of greenhouse gases (2.7%) – behind petrochemicals but ahead of the poster boy of climate change, the airline industry (2%).

With no signs of slowing down, the environmental costs of increasing energy consumption will be around 14% of the world’s share by 2030.

All of this has led to data and digital becoming a primary concern for those interested in sustainability initiatives. Most large businesses require decarbonisation initiatives to meet regulatory requirements (e.g. SECR or CSRD). As the issue becomes more and more severe, these regulations are expanding from not only being focused on direct emissions (e.g. owned vehicles, building emissions) but also including scope 3 or supply chain emissions. In the context of data, on-premises data centres can utilise as much as 25% of an organisation’s total energy expenditure, so when you add your cloud usage into the picture – you can see the scope for improvement.

So what?

So we’ve set the scene – but why should businesses care? The way Oakland views this, there are four main drivers of why you should care as a business – boiled down into direct and indirect impacts. If you are considering or are currently involved in any kind of digital transformation, these are the metrics you should consider.

Cost

Direct impacts are associated with cost and regulation. With wastage comes excess fees – you can see enormous waste in resource use across cloud estates, and there is an associated cost with this resource wastage. Waste is the key word here – it’s cost that isn’t being spent on business outcomes.

The main concern with many sustainability initiatives is that they cost a lot of money and negatively impact business processes. However, with data, many of the issues are down to businesses not operating as efficiently as possible, meaning both environmental and financial costs are spiraling for everyone! By tackling these problems, you could potentially have an environmental sustainability initiative that maintains business processes and operations and positively impacts your bottom line.

Regulatory

Regulations are coming in that expect businesses (corporate sustainability reporting directive) to report not only their emissions for scope 2 and 3 (direct emissions and supply chain emissions) but also carbon reduction initiatives. You become more compliant if you can show how you plan to reduce carbon emissions across scopes.

Reputation

Reputation is critical – businesses are now becoming more attractive to prospective employees and customers because of their sustainability credentials. Imagine a data engineer who wants to go and work for two competing organisations. Both roles are bog-standard data engineering, but one of the organisations is committed to its environmental footprint by making its data infrastructure as carbon neutral as possible. The same goes for customers; if you can offer a sustainable option against one that hasn’t even thought about it, there’s a clear winner.

Doing what is right for the planet

Arguably THE most important driver – the long-term ecological disasters that are likely to increase due to the ever-increasing number of data centres and uncontrollable data estates. We mentioned them earlier, but on a more local level, we are seeing huge impacts of data centre development in places like Ireland (18% of energy is used by data centres); and in the US, where new data centres are being constructed in sunny, arid areas to take advantage of PV, we’re seeing reliant water consumption leading to severe droughts.

So really we need to tackle this at the level that we do business and change how we do things. Head to our Water Utilities page for more information on how we work with the water sector.

How to Approach Change

We need to talk about a new way to look at data operations. Can you support the delivery of both cloud and on-premises data estates in the most efficient possible way and have a more muted impact on the environment whilst having a more positive impact on your finances?

Let’s talk about FinOps

There’s been an obsession with FinOps in the past five years – and what that tends to focus on is cloud efficiency, but we saw from the stats before that it’s not necessarily working.

Rates of adoption are pretty low, action is limited to just IT or data teams, and cost becomes once again “a data problem.” When things are just “a data problem,” we don’t see the action which is needed.

So, let’s widen the conversation – How is sustainability baked into your data strategy? – that’s one of the key drivers to create buy-in to data across the business. As we’ve seen the next levels of both regulation and CSR being rolled into key strategic goals for organisations, we can leverage our approach to data in a way that appeals to these initiatives. Cut the waste and your cost can drop whilst meeting your strategic sustainability goals.

The answer? Include Sustainability

So, we approach data with a fourth lens – a lens of sustainability.

This acknowledges there is an issue with how we do business: by handling processes more efficiently we can save energy and carbon emissions, and by driving down energy spend and resource usage, cost savings can be achieved!

By revolutionising how we look at and assess the efficiency of our data estate, as a vehicle to support organisational sustainability initiatives we can drive the value from our data and drive the business outcomes needed from the data itself! This revolution moves us from looking at FinOps, to GreenOps.

Alas this isn’t easy – although we can quite easily say “how much” we spend financially on data, it is difficult to get a full picture of what our energy actually costs.

Oakland and our partners at Interact have devised a four-step process that businesses and data teams can use to start to adopt a GreenOps approach.

There are four main stages: Display, Diagnose, Decide, Deliver.

Display

You can’t take action without first understanding where you currently are – think of this as a baselining stage, but there are some major challenges in this phase – starting with understanding the impact your cloud is having. You need to consider many things, such as replication factors and networking – things that aren’t necessarily covered by existing cloud reporting.

Diagnose

A much more straightforward process – once you have completed your assessment, the next stage is to identify the key areas that are having the biggest impact without reaping benefits. One of the critical things from a cloud perspective could be where you do the majority of your computing, on-premises you want to look at utilisation rates and potential for consolidation to provide short term wins.

An example of this would be when we talk about the statistics of average utilisation of servers; combined with the linear nature of power usage vs. utilisation, you can triple the existing “utilisation” of that server (making it 54% utilised), whilst only using around 50% more energy.

That could result in a 300% increase in productivity for just a 50% increase in energy usage. Think about the carbon impact of this change and the cost savings that can be attached to your energy bills! With the high cost of energy, this is well worth an investigation.

Decide

Take the insights and diagnosis which enable you to start making decisions, reviewing what processes are necessary and if it is possible to change the location of your servers and the times of your heaviest usage.

Consider the wider impact that this has on your business – are there applications or data that you store on-premises that would be better served by migrating to the cloud? Or alternatively, you may even find potential for saving energy and money by moving your “archival” data back to on-premises servers.

The decisions you make here can be broken down into two lenses – short-term and long-term. Short-term decisions are those largely around consolidation and immediate fixes, such as changing the time of an ETL process to when a grid is being run on more renewable energy. To make these kinds of impacts, you don’t always have to fundamentally change what you’re doing in the short term – in fact, with Oakland’s carbon efficiency tool, we have seen that by shifting your heavy compute to one day rather than another has the potential to drop associated carbon emissions by around 10% – which costs nothing!

Your longer-term view of things might be migrating on-premises data to the cloud, but to do this most efficiently you need to understand the potential environmental impact – but the most important thing is to create a plan.

You want to focus on the areas you see the most potential to support your business and data strategies. If you’re going for a big push on governance, perhaps you need to initially consider where all that pesky dark data is sat and create a plan to decommission it, if you’ve got some horribly inefficient applications that aren’t optimised for the cloud – then create a plan to refactor them.

Deliver

Migrate, consolidate, and even delete! You need to deliver to the plan. When it comes to the delivery of these larger scale migration projects, you need to consider business continuity – how are you going to maintain services whilst doing this migration? One way to approach this is through introducing new tooling – for example, our partners at Starburst provide a distributed analytics engine that enables you to continually query from both on-premises and cloud instances to maintain services during a migration process.

What challenges might you face?

A lack of transparency from cloud providers on the carbon impact their services are making (for example, when we talk about replication factors, you need to consider that AWS lambda instances need to be replicated six times across different servers – meaning your carbon impact is six-fold).

A lack of publicly available data to support on-premises data assessments. Without support and data, it will take a long time to understand the existing impact that your large on-prem data centres are having.

We face a number of challenges as an industry, and to take on a GreenOps approach we need to be equipped with the tools and capability to assess and benchmark where we are, from an on-premises perspective and a consolidated view across both cloud and on-premises.

What are the next steps?

To support businesses in taking their next steps towards GreenOps, Oakland and Interact have partnered to create a revolutionary service based on the four-step method we’ve talked about here. Interact are experts in on-premises data centre efficacy assessments and how you can save money and energy by consolidating and reconfiguring servers.

Oakland can support organisations in reporting on, understanding, and investigating their cloud estate using our carbon efficiency tool – an enhancement from the well-regarded Thoughtworks cloud carbon footprint tool to create an estimate of the carbon footprint of your data estate.

We work together to identify how you can streamline your data estate and optimise for both carbon and cost perspectives. It’s important to note that the platform itself is only a part of the battle; you need governance to ensure that retention policies are being fulfilled, which can have a huge impact on your dark data, which needs to be baked into your data strategy as a means of alignment, and from an analytics perspective, you have to start to report on data sustainability as a KPI to support carbon reduction initiatives.

As an industry, we need to do better for ourselves and the planet by working together to minimise the environmental impact of our data in a cost-efficient way.

If you’d like more information about Oakland’s revolutionary GreenOps service please get in touch by emailing Luke.sharma@theoaklandgroup.co.uk

The post What is the environmental impact of your data? appeared first on Oakland.

All these Data Quality issues add up, with bad Data Quality costing organisations on average $15mil a year.

Data Quality investment is also an investment in high-quality AI and ML, as you’ll likely get more accurate AI and ML results from improving Data Quality than changing your AI model and code.

Having Data Quality checks in place helps reduce “data downtime” for outages and fixes, which subsequently increases the overall reliability of the Data Platform. Highly reliable data leads to more trust in data and better-informed decision-making.

Better decision-making should increase profitability, productivity, and confidence of the whole organisation, which in turn usually leads to more investment in data and, as a result, going back to the start: Increasing the quality of data again.

All this creates a “Virtuous Cycle” of Data Quality, constantly improving your organisation:

If Data Quality decreases the opposite happens with a negative cycle.

Better Data Quality testing should also reduce the blast radius of the issue to a few Data Engineers rather than hundreds or thousands of users as more issues are being found earlier:

The fewer users impacted, the smaller the cost caused by the issue, which should again pay back any investment in Data Quality in large multiples.

Do I need a Data Quality Framework?

I know that setting out to build a framework for anything requires time, and you’ll have many competing concerns, so we understand if you feel reluctant to build one, especially if you are a small team with a limited budget.

Oakland Data Quality Framework. Source: https://www.theoaklandgroup.co.uk/services/data-governance/data-quality-assessment/

But there comes a point where fighting lots of local battles with Data Quality becomes more inefficient than building out a framework to reduce Data Quality issues over the long term.

We’re not going to do a deep dive on Data Quality frameworks here, as they are often tied to wider Data Governance frameworks (you can download our guide here) We will say that whatever framework you use, make sure it’s cyclical so that it’s always improving and you are acting on any emerging issues in a timely manner.

How Do I Test for Data Quality?

Classically, Data Quality tests are a set of rules that test between the actual and desired state of data. The desired state may not be perfect, but ‘good enough’. What counts as ‘good enough‘ varies from dataset to dataset, which makes Data Quality more challenging.

What do we normally test in a dataset, though? The DAMA International’s Guide to the Data Management Body of Knowledge says there are six dimensions to Data Quality:

• Accuracy: does the date look how we expect it to?
• Completeness: are there any unexpected missing values?
• Uniqueness: no duplicates!
• Consistency: does a person’s data match in two different datasets?
• Timeliness: is the data out of date?
• Validity: does the data conform to an expected format? Think postcodes, emails, etc.

Tracking all these dimensions for every dataset at every stage of your pipeline is a lot of work, probably too much work. Therefore, a trade-off is often required to focus on areas where Data Quality will have the most impact on the business.

You also have to beware of false positives or minor issues being blown out of proportion, overwhelming your engineers with too many issues. It can help if you categorise your Data Quality issues by severity just like other software issues.

You also have to take into account the mental wellbeing aspect too: few Data Engineers and Analysts want to spend a large percentage of their time fixing Data Quality issues over a long period of time.

There is help, though, with software frameworks to help you write Data Quality testing:

• Great Expectations
• dbt tests and dbt-expectations
• Datafold
• Soda
• Monte Carlo

Most of the above profile your data and setup recommended tests for you to use, saving you some time configuring them yourself.

But in reality, we see a lot of custom-made Data Quality testing, partly because Data Quality struggles for investment, so it is usually done in an organic, ad-hoc manner.

The above products work best in development and staging environments, so you can find issues before they enter production or use them as circuit breakers, to stop a Data Pipeline if the incoming or outgoing data is of poor quality.

It is also worth mentioning that you can use constraints in a Warehouse or Lakehouse schema, which have the benefits of not requiring another software library but are not as feature rich (you will likely have to setup your own notifications for alerting).

It is also important to inform your users of any Data Quality issues as soon as possible so they don’t waste time finding out for themselves or use data that is untrustworthy. This can be done through notifications and alerts, though we’ve also had a lot of success creating Data Quality dashboards that sit alongside existing reports and can be easily referred to by users.

Latest Concepts in Data Quality

There has been significant innovation in Data Quality in the last few years, so we present below the concepts to take your Data Quality process to the next level.

This will require more investment, but it will give you an edge over your competitors to make better informed decisions as you’ll have more trustworthy data. This investment should also pay back long term with less time wasted fixing Data Quality issues.

What is Data Reliability and do I Need it?

Data Reliability gives Data Quality more of a support focus, which makes sense as most Data Quality issues in production will be dealt with as a support issue to a Data Platform.

Data Reliability takes a lot of its thinking from Site Reliability Engineering (SRE), which treats support as more of a engineering problem, where you examine your past and current support tickets and look to decrease them with engineering or better processes.

With Data Reliability, you would look to get a baseline of Data Quality issues per week or month and then look at ways to reduce them and monitor to see if the changes have reduced the number of issues and/or reduced the amount of time spent on issues.

The changes to improve Data Reliability can be technology-based:

• New or updated tooling
• Better automation of when a pipeline fails or automated actions to respond to a data issue

Or the changes can be process-oriented:

• Writing better documentation to avoid common issues
• Incident playbooks so the whole team can more quickly respond to a issue in an consistent way.

You can rather cynically say Data Reliability is just Data Quality with a feedback loop and a time series graph, but it is there to make sure you avoid short term thinking about Data Quality and instead consider long term improvements that will make your data platform more efficient and trustworthy.

Data Reliability Cycle

You may also set targets such as “99.9% of data will refresh on time” or “A maximum of 33% of engineer time should be spent on support issues“ as well. As mentioned before, it can be impossible to achieve perfect Data Quality, so aiming for a reasonable target instead can avoid engineer burnout.

What is Data Observability and do I Need it?

Data Observability is about gaining a Data Platform or organisation-wide understanding of your Data Quality.

It arguably goes beyond Data Quality by adding metadata features normally found in a Data Catalog: cataloguing schemas of datasets and data lineage. These features allow you to more quickly find a Data Quality issue by tracing the lineage of the issue and also you gain the ability to see how much Data Quality is impacting your organisation.

Data Observability software can often also come with Machine Learning (ML) algorithms to detect anomalies in data, so you can be warned about issues you haven’t even thought of yet.

We’ve seen products either extend a Data Quality framework with Data Catalog features such as Monte Carlo and Big Eye. Or existing Data Catalogs add Data Quality functionality, such as Datahub, which imports Data Quality tests created by Great Expectations and dbt tests. Both Soda and Monte Carlo have integration with the Data Catalog Alation.

What are Data Contracts and do I Need Them?

Data Contracts make a contract between a data producer and a data consumer, so the consumer knows what data to expect from the producer.

While you can replicate some of a Data Contract’s benefits by tracking the schema of the data produced, a Data Contract is meant to go beyond that by giving you a full suite of metadata about the data:

• The data’s schema.
• How the data is calculated.
• Who owns the data?
• What is the data lineage?
• How to access the data.
• What is the data’s expected quality, availability, etc.
• Plus anything else that is relevant to the data.

You may think Data Contracts are redundant if you have a well-maintained Data Catalog, as they capture similar information, but Data Contracts are designed to be checked during every run of a Data Pipeline and have some action in the pipeline if the Data Contract is broken:

• Stop the pipeline with a circuit breaker.
• Alerting.
• Moving data that doesn’t meet the contract to a manual checking table.

For an example, Paypal has open-sourced their Data Contract template.

https://github.com/paypal/data-contract-template

This should create more positive collaboration between data producers and consumers because they have a collective agreement of what the data should look like. It is not uncommon to have a poor working relationship where a producer makes changes without telling consumers or consumers accessing data in way not recommended by the producer.

One issue with Data Contracts is that they are a new concept, so require more work to implement at present, though that will likely change in the near future as more companies adopt them.

Most of the examples of Data Contracts we’ve seen so far use Apache Flink and the Kafka Schema Registry, so assume you are using streaming, though there are some examples that use batch processing.

Data Governance and Data Quality

Good Data Governance can also improve quality of data. It is important to know where data is coming from, who owns it, for what purpose data is being transformed, and finally, what is the impact of poor availability and data quality: all helped by having Data Governance properly implemented.

Some of the above concepts (Data Contracts and Data Observability) can also improve Data Governance, so investing in Data Quality can also be an investment in good Governance too.

How Does This All Fit Together?

The diagram below is one example of how it all fits together:

• Any code changes are tested in development and/or test environments with Data Quality Tests to check that any changes won’t have a negative impact on Data Quality.
• Source Data at the start of the data pipeline is checked to see if the Data Contract is held; if not, a circuit breaker may kick in, stopping the data pipeline early to avoid processing unsuitable data.
• Data Quality tests are also run in production, which can feel like duplication from testing in development, but there may be changes caused by moving to a production environment (different data, etc.).
• Data is collected for observability checks by Data Observability software, looking for any anomalous data: a department budget that goes from £10k to £1mil or 10x increase in rows for a table, for example. This can replace a lot of tests, but not all of them.

You’ll also be collecting Data Quality metadata to improve your Data Reliability.

Making all this work together seamlessly isn’t cheap and will take time, but as mentioned, poor Data Quality will also cost an organisation a lot of money. So we recommend tackling this in an agile manner by improving Data Quality in small increments, one change at a time, starting where it will have the most impact.

Summary

Data Quality is a difficult subject to tackle, due to it being a slightly different problem in every organisation and never “perfect”. That said, there are lots of options to help improve the quality of your data, so you should be able to get to “good enough“ if you give Data Quality enough priority and forethought.

Jake Watson is a Principal Engineer at Oakland

The post Why Invest in Data Quality? appeared first on Oakland.