There is Something Human about Waste

Let’s start by setting the scene – an all too familiar one. Humans love waste. We throw away around 100 billion pieces of plastic every year in the UK, and around 1.9 billion tonnes of food that can be eaten is discarded on an annual basis, and even on a household level – between 9-16% of energy is wasted on standby.

This fascinating graphic from No Planet B by Mike Berners-Lee shows the world supply chain for food based on energy loss at each stage. As you can see, at every level of the picture, our food waste. The depressing part can be seen in the bottom three rungs; even if we’re lucky enough to have food we still waste around 50% of it.

The impact of this level of overconsumption is evidenced in the world around us: fires in Maui, droughts in Uruguay, and extreme flooding in Bangladesh – all directly attributed to climate change driven largely by Greenhouse gas emissions.

But what does this have to do with data?

It doesn’t come as much of a surprise that physical human behaviour also maps onto digital human behaviour. The attitude towards waste and overconsumption has transcended from the physical to the digital in the case of data – and that has led to some huge inefficiencies in the way that we currently manage data – directly resulting in some pretty stark impacts on the environment. We talk about inefficiency through a few lenses – the first being the underutilisation of resources which we can see represented by the fact that the utilisation rate of on-premises servers is only around 18%, and it doesn’t get much better when we talk about cloud storage and usage, where around 33% of paid-for services are considered waste.

Combine this with the fact that 55% of stored data is considered dark (currently unused, with no future use-case) – a frightening statistic! That means stored and processed data that will never realistically be used for operational or analytical purposes. It’s one of the darker legacies of the big data wave of 2017!

Finally let’s not forget the onslaught of Artificial Intelligence. The amount of data we are creating is out of control.

Data Centres – The Lords of Darkness

When you consider the artificially inflated needs of server racks, the ramifications of this wastage are very real – data centres are being constructed at a rate of knots and are having pretty dire consequences on the environment.

Think about all the utilities that you need to maintain a data centre, primarily you might just think about the energy costs of powering your servers. Ignoring the energy requirements for cooling and lighting, server energy usage on its own has the potential for improvement. Server configuration and distribution of workloads can often drive higher than necessary energy usage.

Due to the sheer amount of energy digital technologies power through, data centres have become the world’s second largest source of greenhouse gases (2.7%) – behind petrochemicals but ahead of the poster boy of climate change, the airline industry (2%).

With no signs of slowing down, the environmental costs of increasing energy consumption will be around 14% of the world’s share by 2030.

All of this has led to data and digital becoming a primary concern for those interested in sustainability initiatives. Most large businesses require decarbonisation initiatives to meet regulatory requirements (e.g. SECR or CSRD). As the issue becomes more and more severe, these regulations are expanding from not only being focused on direct emissions (e.g. owned vehicles, building emissions) but also including scope 3 or supply chain emissions. In the context of data, on-premises data centres can utilise as much as 25% of an organisation’s total energy expenditure, so when you add your cloud usage into the picture – you can see the scope for improvement.

So what?

So we’ve set the scene – but why should businesses care? The way Oakland views this, there are four main drivers of why you should care as a business – boiled down into direct and indirect impacts. If you are considering or are currently involved in any kind of digital transformation, these are the metrics you should consider.

Cost

Direct impacts are associated with cost and regulation. With wastage comes excess fees – you can see enormous waste in resource use across cloud estates, and there is an associated cost with this resource wastage. Waste is the key word here – it’s cost that isn’t being spent on business outcomes.

The main concern with many sustainability initiatives is that they cost a lot of money and negatively impact business processes. However, with data, many of the issues are down to businesses not operating as efficiently as possible, meaning both environmental and financial costs are spiraling for everyone! By tackling these problems, you could potentially have an environmental sustainability initiative that maintains business processes and operations and positively impacts your bottom line.

Regulatory

Regulations are coming in that expect businesses (corporate sustainability reporting directive) to report not only their emissions for scope 2 and 3 (direct emissions and supply chain emissions) but also carbon reduction initiatives. You become more compliant if you can show how you plan to reduce carbon emissions across scopes.

Reputation

Reputation is critical – businesses are now becoming more attractive to prospective employees and customers because of their sustainability credentials. Imagine a data engineer who wants to go and work for two competing organisations. Both roles are bog-standard data engineering, but one of the organisations is committed to its environmental footprint by making its data infrastructure as carbon neutral as possible. The same goes for customers; if you can offer a sustainable option against one that hasn’t even thought about it, there’s a clear winner.

Doing what is right for the planet

Arguably THE most important driver – the long-term ecological disasters that are likely to increase due to the ever-increasing number of data centres and uncontrollable data estates. We mentioned them earlier, but on a more local level, we are seeing huge impacts of data centre development in places like Ireland (18% of energy is used by data centres); and in the US, where new data centres are being constructed in sunny, arid areas to take advantage of PV, we’re seeing reliant water consumption leading to severe droughts.

So really we need to tackle this at the level that we do business and change how we do things. Head to our Water Utilities page for more information on how we work with the water sector.

How to Approach Change

We need to talk about a new way to look at data operations. Can you support the delivery of both cloud and on-premises data estates in the most efficient possible way and have a more muted impact on the environment whilst having a more positive impact on your finances?

Let’s talk about FinOps

There’s been an obsession with FinOps in the past five years – and what that tends to focus on is cloud efficiency, but we saw from the stats before that it’s not necessarily working.

Rates of adoption are pretty low, action is limited to just IT or data teams, and cost becomes once again “a data problem.” When things are just “a data problem,” we don’t see the action which is needed.

So, let’s widen the conversation – How is sustainability baked into your data strategy? – that’s one of the key drivers to create buy-in to data across the business. As we’ve seen the next levels of both regulation and CSR being rolled into key strategic goals for organisations, we can leverage our approach to data in a way that appeals to these initiatives. Cut the waste and your cost can drop whilst meeting your strategic sustainability goals.

The answer? Include Sustainability

So, we approach data with a fourth lens – a lens of sustainability.

This acknowledges there is an issue with how we do business: by handling processes more efficiently we can save energy and carbon emissions, and by driving down energy spend and resource usage, cost savings can be achieved!

By revolutionising how we look at and assess the efficiency of our data estate, as a vehicle to support organisational sustainability initiatives we can drive the value from our data and drive the business outcomes needed from the data itself! This revolution moves us from looking at FinOps, to GreenOps.

Alas this isn’t easy – although we can quite easily say “how much” we spend financially on data, it is difficult to get a full picture of what our energy actually costs.

Oakland and our partners at Interact have devised a four-step process that businesses and data teams can use to start to adopt a GreenOps approach.

There are four main stages: Display, Diagnose, Decide, Deliver.

Display

You can’t take action without first understanding where you currently are – think of this as a baselining stage, but there are some major challenges in this phase – starting with understanding the impact your cloud is having. You need to consider many things, such as replication factors and networking – things that aren’t necessarily covered by existing cloud reporting.

Diagnose

A much more straightforward process – once you have completed your assessment, the next stage is to identify the key areas that are having the biggest impact without reaping benefits. One of the critical things from a cloud perspective could be where you do the majority of your computing, on-premises you want to look at utilisation rates and potential for consolidation to provide short term wins.

An example of this would be when we talk about the statistics of average utilisation of servers; combined with the linear nature of power usage vs. utilisation, you can triple the existing “utilisation” of that server (making it 54% utilised), whilst only using around 50% more energy.

That could result in a 300% increase in productivity for just a 50% increase in energy usage. Think about the carbon impact of this change and the cost savings that can be attached to your energy bills! With the high cost of energy, this is well worth an investigation.

Decide

Take the insights and diagnosis which enable you to start making decisions, reviewing what processes are necessary and if it is possible to change the location of your servers and the times of your heaviest usage.

Consider the wider impact that this has on your business – are there applications or data that you store on-premises that would be better served by migrating to the cloud? Or alternatively, you may even find potential for saving energy and money by moving your “archival” data back to on-premises servers.

The decisions you make here can be broken down into two lenses – short-term and long-term. Short-term decisions are those largely around consolidation and immediate fixes, such as changing the time of an ETL process to when a grid is being run on more renewable energy. To make these kinds of impacts, you don’t always have to fundamentally change what you’re doing in the short term – in fact, with Oakland’s carbon efficiency tool, we have seen that by shifting your heavy compute to one day rather than another has the potential to drop associated carbon emissions by around 10% – which costs nothing!

Your longer-term view of things might be migrating on-premises data to the cloud, but to do this most efficiently you need to understand the potential environmental impact – but the most important thing is to create a plan.

You want to focus on the areas you see the most potential to support your business and data strategies. If you’re going for a big push on governance, perhaps you need to initially consider where all that pesky dark data is sat and create a plan to decommission it, if you’ve got some horribly inefficient applications that aren’t optimised for the cloud – then create a plan to refactor them.

Deliver

Migrate, consolidate, and even delete! You need to deliver to the plan. When it comes to the delivery of these larger scale migration projects, you need to consider business continuity – how are you going to maintain services whilst doing this migration? One way to approach this is through introducing new tooling – for example, our partners at Starburst provide a distributed analytics engine that enables you to continually query from both on-premises and cloud instances to maintain services during a migration process.

What challenges might you face?

A lack of transparency from cloud providers on the carbon impact their services are making (for example, when we talk about replication factors, you need to consider that AWS lambda instances need to be replicated six times across different servers – meaning your carbon impact is six-fold).

A lack of publicly available data to support on-premises data assessments. Without support and data, it will take a long time to understand the existing impact that your large on-prem data centres are having.

We face a number of challenges as an industry, and to take on a GreenOps approach we need to be equipped with the tools and capability to assess and benchmark where we are, from an on-premises perspective and a consolidated view across both cloud and on-premises.

What are the next steps?

To support businesses in taking their next steps towards GreenOps, Oakland and Interact have partnered to create a revolutionary service based on the four-step method we’ve talked about here. Interact are experts in on-premises data centre efficacy assessments and how you can save money and energy by consolidating and reconfiguring servers.

Oakland can support organisations in reporting on, understanding, and investigating their cloud estate using our carbon efficiency tool – an enhancement from the well-regarded Thoughtworks cloud carbon footprint tool to create an estimate of the carbon footprint of your data estate.

We work together to identify how you can streamline your data estate and optimise for both carbon and cost perspectives. It’s important to note that the platform itself is only a part of the battle; you need governance to ensure that retention policies are being fulfilled, which can have a huge impact on your dark data, which needs to be baked into your data strategy as a means of alignment, and from an analytics perspective, you have to start to report on data sustainability as a KPI to support carbon reduction initiatives.

As an industry, we need to do better for ourselves and the planet by working together to minimise the environmental impact of our data in a cost-efficient way.

If you’d like more information about Oakland’s revolutionary GreenOps service please get in touch by emailing Luke.sharma@theoaklandgroup.co.uk

The post What is the environmental impact of your data? appeared first on Oakland.

In this blog, we’ll guide you through creating a data product-focused data strategy. We’ll cover key concepts like defining data products, setting up domain-specific teams, and implementing a federated ownership model to enhance data management and efficiency within your organisation.

Are you looking to write a data strategy but unsure how? Read our guide on how to write your data strategy for further information.

What is a Data Product?

Data products are reusable assets that deliver trusted datasets for specific purposes, ensuring the right data reaches the right people in the right format. The unique aspect of data products is the federated ownership model, where domain-specific teams manage their data from production to consumption, ensuring better efficiency and responsibility.

Domain driven ownership

Figure 1 A Domain-Oriented Ownership model

Source: https://www.starburst.io/blog/data-mesh-and-starburst-domain-oriented-ownership-architecture/ 

What is a Data Product-Focused Data Strategy?

A data product-focused data strategy involves creating reusable data assets, known as data products, to solve specific business problems. This strategy enhances data quality, reduces silos, and improves security through an ownership model. 

It shifts the approach from treating data as an output of projects to managing data as products, ensuring ongoing maintenance and updates. This strategy includes defining domains, building data products, and continuously engaging with consumers to improve and deliver valuable data solutions.

Why Do You Need a Data Product-Focused Data Strategy?

In today’s data-centric world, having a solid data product strategy is a necessity for your organisation. 

1. Improves Data Trustworthiness: Ensures accountability for data quality through domain ownership.
2. Reduces Data Silos: Facilitates autonomous access to managed data, enhancing self-service capabilities.
3. Enhances Security: Allows domain teams to manage access and compliance, optimising risk management.
4. Supports Continuous Improvement: Encourages ongoing maintenance and updates of data assets.
5. Drives Business Value: Focuses on solving specific business problems, leading to more impactful data solutions.

Read our blog to find out more information about the purpose of a company’s data strategy.

Is the Data Product Approach Suitability for Mature Data Estates?

The data product approach may not suit less mature data estates due to its reliance on advanced processes, skills, and infrastructure. However, adopting it as a long-term goal can improve existing practices and lay the groundwork for future data product delivery.

How to Integrate a Product Data Strategy into your Organisation

Incorporating a data product approach into your data strategy has people, process and technology implications – so the following tips are designed to provide you with some ideas of steps you might want to take if you are considering taking the plunge. 

We mentioned earlier the concept of domain driven ownership. Given the nature of data products being focused on how each domain can solve issues for the business, your first port of call is deciding how to divide the organisation up into domains. 

A good starting point for this is to investigate the existing subject areas that sit within your data estate – most organisations will have these. Once you have a picture of what your domains are, you can start to look at the process of building a data product. 

If your data strategy is led by IT or Data, this is where you start to find friends in the domain areas. Once you have identified a domain that is open to the data product approach, you can work with them to define the data products that would support the business needs from that domain. 

When pitching a product, it must solve a specific problem. Data products are no different. Start by identifying and addressing business challenges.

Key Steps to Build Data Products

You don’t see people on Dragon’s Den pitching a product that doesn’t solve a specific problem. In fact, the most successful companies will think about the problem first and work to build a solution to that problem. Data Products are no different, and building a strategy for these data products should align with that view. 

You will want to build products that solve specific problems and challenges within the business, which can be marketed and maintained. 

Understand Consumer Needs

Get to know your consumers, do your research, understand the pains they feel when engaging with your data. 

Establish Feedback Processes

Establish processes that enable consumers to contact and feedback to data product teams so that every data product is built either toward solving a specific problem or responding to market demand quickly by creating new business opportunities. 

Prioritise

You can then set out a prioritisation exercise to understand where to focus efforts to deliver your first data products! 

Why Should You Take the Lighthouse Approach to Data Strategy?

As with any new process within an organisation, at Oakland – we find that delivering value rapidly is critical. We can do this by taking a lighthouse approach, where you take a narrow but deep slice of your data estate and design a small but valuable data product.

Luke Sharma, Senior Data Consultant at Oakland, states

‘’We achieve this by taking a lighthouse approach, where we focus on a narrow but deep slice of our data estate to design a small yet valuable data product. This method allows us to set up the rituals and roles of product management to fit within our existing operating model. Once the lighthouse project is delivered, we assess the supporting processes and roles, as well as the value gained from the product.’’

He goes on to mention that, ‘’These learnings are then embedded into our Target Operating Model, ensuring the product management approach is integrated into the organisation’s ways of working. The key here is that the approach is tested and drives value for your organisation with a more moderate investment. This lighthouse data product and way of working can then be demonstrated to other domains, increasing appetite and excitement.” 

Breaking the Project Mindset

This is where the strategy element of “data product strategy” becomes vital because it involves cultural and process change. Once you have delivered a data product, you can rapidly see it fall back into the traditional cycle of standard data development because the organisation’s mentality hasn’t moved forward with the technology. 

Data is unfortunately often seen as an output of a project, be it a report, dashboard, or ML algorithm. The project is delivered, signed off by stakeholders and then handed over to the business unit. Inevitably the deliverables from these projects are outdated pretty quickly – and the response? Set up another project, build another report – rinse and repeat. 

You need to stop thinking of data as an output and start to think about data as a product. 

“Products need to be maintained, updated, and associated with SLAs/data contracts that guarantee a certain level of uptime and data quality as part of a standard product lifecycle, managed by the product management team. By taking the approach learned from the lighthouse project and embedding it into a Target Operating Model (TOM), we ensure that accountability for data product maintenance is prioritised.”

Luke Sharma, Senior Data Consultant

The whole aim of data products is they are built and maintained in order to be reusable, shareable assets!

Focusing on Value

Data products must deliver specific outcomes. Engage with consumers to understand their needs and guide product development.

Using data products in your strategy will drive adoption and accountability within your organisation. Engage regularly with consumers to refine and prioritise data products, overcoming technological and process complexities.

For more information, contact us at hello@theoaklandgroup.co.uk. For the original blog post, visit Oakland Blog.

The post How To Create a Data Product-Focused Data Strategy appeared first on Oakland.