Zareene Choudhury, Author at Oakland Thu, 03 Apr 2025 12:50:06 +0000 en-GB hourly 1 https://wordpress.org/?v=6.9.4 https://weareoakland.com/wp-content/uploads/2024/01/cropped-oakland-favicon-150x150.jpg Zareene Choudhury, Author at Oakland 32 32 Data Governance in the Age of AI https://weareoakland.com/blog/data-governance-in-the-age-of-ai/ Thu, 27 Feb 2025 10:46:12 +0000 https://weareoakland.com/?p=9350 In an era where artificial intelligence (AI) is revolutionising industries, many organisations embark on ambitious AI projects with the hope of driving efficiency, automation, and competitiveness. However, a significant portion of these projects falter due to foundational data challenges. According to Gartner, by 2025, 30% of generative AI projects will be abandoned due to poor...

The post Data Governance in the Age of AI appeared first on Oakland.

]]>
In an era where artificial intelligence (AI) is revolutionising industries, many organisations embark on ambitious AI projects with the hope of driving efficiency, automation, and competitiveness. However, a significant portion of these projects falter due to foundational data challenges. According to Gartner, by 2025, 30% of generative AI projects will be abandoned due to poor data quality, inadequate risk controls, and unclear business use cases. This statistic underscores a fundamental truth: AI is only as effective as the data that fuels it. 

For organisations to harness AI’s potential, robust data governance must be in place. Here at Oakland, like many data management consultancies we’ve been exploring AI-driven transformations, the differences between data governance and AI governance, and how businesses can overcome common challenges to ensure AI success. 

The Critical Role of Data in AI Success 

AI, at its core, is a data product. Whether in machine learning models, large language models (LLMs), or generative AI applications, the principle of “garbage in, garbage out” remains true. If AI systems are trained on poor-quality data, the outcomes will be flawed, potentially leading to misinformed decision-making, regulatory compliance risks, and reputational damage. 

Key data challenges impacting AI initiatives are: 

  1. Data Quality Issues – AI models depend on accurate, complete, and consistent data. Missing values, duplication, and outdated records compromise AI outputs. 
  2. Data Ownership Ambiguities – A lack of clear data ownership makes accountability difficult and slows decision-making. 
  3. Understanding Data Landscapes – Without a well-documented data architecture, it is challenging to trace data lineage, assess data integrity, and ensure compliance. 
  4. Bridging the Gap Between Vision and Execution – Business leaders often have ambitious AI visions, but data professionals struggle to translate these into feasible implementations due to underlying data constraints. 
  5. Unclear Business Value – Lack of alignment to wider business goals and challenges to determine where AI efforts will deliver most value. 

Additionally, there is often a disconnect between executives pushing for AI adoption and the data teams responsible for managing the infrastructure. Many assume that AI will work out of the box, failing to recognise the need for strong governance principles that allow AI initiatives to scale successfully. Without this, projects often result in frustration, misalignment, and ultimately abandonment. 

Data Governance as the Foundation of AI Success 

This is where working with a data management consultancy like Oakland can play a crucial role in addressing these challenges. We provide organisations with structured methodologies to manage data effectively, ensuring data is accurate, reliable, and accessible for AI applications. 

Key Elements of Data Governance for AI 

  1. Data Lineage and Provenance – Organisations must track where data originates, how it has been transformed, and who is responsible for it. 
  2. Data Quality Management – Implementing data validation, cleansing, and monitoring processes ensures high-quality input for AI models. 
  3. Ownership and Accountability – Establishing clear roles for data stewardship fosters responsibility and trust. 
  4. Privacy and Security Controls – Protecting sensitive data and ensuring AI models comply with regulations like GDPR and the EU AI Act is critical. 
  5. Feedback Mechanisms – Continuous monitoring and improvement of data assets and AI outcomes create an adaptive governance framework. 

Organisations that embrace these principles avoid the common pitfalls of AI projects, such as reliance on poor-quality datasets, uncertainty around data sources, and misalignment between business strategy and AI deployment. 

Data Governance vs. AI Governance: What’s the Difference? 

While data governance focuses on managing data assets within an organisation, AI governance extends beyond data to include: 

  • Model Lifecycle Management – Ensuring AI models are built, tested, deployed, and monitored according to best practices. 
  • Regulatory Compliance – Adhering to AI-specific regulations such as the EU AI Act and industry guidelines. 
  • Ethical Considerations – Addressing fairness, bias, and transparency in AI decision-making. 
  • Risk Management – Identifying and mitigating risks associated with AI models, including unintended consequences and security threats. 

Ultimately, AI governance builds on data governance. Organisations with a strong data governance framework can more easily scale AI governance practices, ensuring that AI models operate with integrity and compliance. 

As organisations navigate this landscape, they must also consider how AI itself can assist in governance efforts. AI-driven tools can help automate data lineage mapping, anomaly detection, and compliance monitoring, making governance more scalable and efficient. 

The Business Case for Investing in AI and Data Governance 

To secure executive buy-in for AI and data governance initiatives, organisations should: 

  • Align with Business Objectives – AI initiatives should support core strategic goals, such as improving customer experience, increasing operational efficiency, or enabling new revenue streams. 
  • Prioritise Use Cases – Focusing on specific, high-impact AI applications rather than boiling the ocean ensures measurable results and quick wins. 
  • Leverage Existing Capabilities – Organisations with mature data governance practices can extend them to AI governance, optimising resource utilisation. 
  • Demonstrate ROI – Quantifying cost savings, risk reduction, and performance improvements strengthens the case for investment. 

How Data Governance and AI Governance Enhance Business Performance 

Improved Decision-Making 

With well-governed data, organisations can confidently rely on AI-driven insights to make strategic business decisions, reducing uncertainty and improving outcomes. 

Increased Efficiency and Scalability 

AI-powered automation can eliminate repetitive tasks, but only if the underlying data is trustworthy. Data governance ensures that automation is built on a solid foundation, allowing businesses to scale AI initiatives seamlessly. 

Regulatory Compliance and Risk Mitigation 

AI regulations are rapidly evolving, and organisations that fail to comply face fines and reputational damage. Data governance ensures that AI models adhere to legal and ethical standards, reducing regulatory risks. 

Competitive Advantage 

Organisations that integrate AI governance with strong data management practices differentiate themselves by delivering AI solutions that are accurate, reliable, and compliant, gaining an edge over competitors. 

Implementing a Structured Data Governance and AI Governance Framework 

A structured, standardised approach to governance is key to AI success. Best practices include: 

  1. Building a Cross-Functional AI Governance Team – Engage data stewards, compliance officers, IT leaders, and business stakeholders. 
  2. Developing a Clear Data Strategy – Define data ownership, quality standards, and governance policies. 
  3. Leveraging AI for Data Governance – Use AI to automate data lineage mapping, quality assessments, and compliance monitoring. 
  4. Establishing Trust and Transparency – Ensure AI decisions are explainable, auditable, and aligned with ethical standards. 
  5. Continuously Refining Governance Practices – AI and data governance should evolve with changing business needs and regulatory landscapes. 

Organisations that view AI as a data-driven capability rather than just a technological innovation position themselves for long-term success. Data governance provides the critical foundation needed for AI initiatives to thrive, ensuring high-quality data, clear accountability, and compliance with regulations. By aligning data governance and AI governance, businesses can unlock AI’s full potential while mitigating risks, ultimately driving better decision-making, efficiency, and competitive advantage. 

For enterprises looking to embark on or refine their AI journey, investing in data governance is not optional it’s a necessity. 

“AI, Generative AI or Agentic AI holds great promise. However, many organisations currently feel held back by the necessity to validate their data thoroughly, unclear understanding of AI risks, and the potential for unforeseen issues, like bias and data security. Either making them hesitant to fully commit, i.e. death by POC, or unrealised sustainable value when deployed. 

These are unchartered waters for most. The good news is you can choose the pace you want to go. Navigating AI governance is like steering a ship through a storm; robust data governance is your compass, guiding you to safer, clearer waters”.

Zareene Choudhury – Oakland Data Governance Lead

Want to find out more?

If you’d like to learn more about Oakland’s data governance consulting services, then get in touch by emailing hello@weareoakland.com If you’d like to learn more about what a suitable data governance framework looks like, click on the link.

The post Data Governance in the Age of AI appeared first on Oakland.

]]>
What is the Role of Data Governance in AI? Should You Even Care? https://weareoakland.com/blog/role-of-data-governance-in-ai/ Tue, 04 Feb 2025 08:24:15 +0000 https://weareoakland.com/?p=9311  Data Governance is having a moment or, at the very least, is about to. Governance was one of the key topics discussed in the opening address at the recent Gartner IT symposium. Obviously, it goes without saying that the main topic was Artificial Intelligence and Agentic AI. But before we get carried away, here’s some context...

The post What is the Role of Data Governance in AI? Should You Even Care? appeared first on Oakland.

]]>
 Data Governance is having a moment or, at the very least, is about to. Governance was one of the key topics discussed in the opening address at the recent Gartner IT symposium. Obviously, it goes without saying that the main topic was Artificial Intelligence and Agentic AI. But before we get carried away, here’s some context on the impact Artificial Intelligence is having in the business landscape: 

  • The UK AI Market is worth more than £16.6B and is projected to grow to over £700B by 2035 
  • 28% of businesses have built their own AI solutions or are using existing AI tools like CoPilot or ChatGPT 
  • In the Autumn Budget in November 2024, the UK government laid out plans for an AI Opportunities Action Plan, which should provide a ‘roadmap to capture the opportunities of AI to enhance growth and productivity. 

Are you in the AI game? 

There’s no question that ‘AI-driven’ business is here to stay… much like ‘data-driven’ (insert your own marketing line here). But Data Governance Consulting at Oakland is very much rooted in the operational and practical side of real-world business. 

We prefer to consider it as “AI-enabled: A business that gains real value from their AI solutions’’. 

Back to our earlier point, 2025 AI is all about Agentic AIs: But is Agentic AI the future or just another AI technology? (although we have been developing intelligent agent solutions way before they were renamed Agentic AI, and all the major tech providers got on board)  

The beauty of AI Agents is that they are so much more than chatbots. They enable organisations to deliver focused value and target specific problems and opportunities in the data ecosystem. This is why honing down on use cases and relating them to the pains of business users is crucial in creating lasting AI solutions. 

Read the small print! 

So yes, AI promises to improve decision-making and deliver real value in a game-changing way. But… make sure to pause and read the small print on this promise, in particular: 

‘Your data quality and your AI governance capabilities will directly impact the value you extract from your investments in AI’ 

A study from RAND concludes that 80% of AI projects will likely fail. This prognosis is not dissimilar to Gartner’s, which predicts that 30% of GenAI projects will be abandoned after POC.  A key reason mentioned in both studies is the lack of high-quality data. 

What is high-quality data? (A holy grail..? Of course not!) 

  • First and foremost, it’s important to know what data you have, where it exists, and who in the business has accountability for it. You don’t need to boil the ocean. Start with critical data domains related to your AI use cases.
  • It knows what benchmarks and rules need to be applied to your data to classify it as good or bad: it is unlikely that you need 100% completeness for some of your data sets – understand what business purpose the data caters for, and this helps you define your benchmarks. 
  • Having a view of your data eco-system is not just about having an architecture map that details the technical integrations. Have a view of the business processes and information requirements that underpin this architecture. Highlight where the risks are, e.g., because of duplication, manual interventions, or siloed processes. 
  • Having mechanisms to monitor, log, and address data issues as they get flagged. This allows the business to mitigate risk and avoid confusion when resolving issues quickly but also allows it to be more strategic in improving business capabilities. 
  • Documenting your critical data dependencies and how your data flows cross-functionally, i.e., your data lineage. This not only helps to make informed decisions on what protocols are needed to keep your data secure and address issues but also allows you to nail down requirements when scaling your data capabilities quickly. 
  • If you are in a heavily regulated industry, then data security and regulatory compliance will require more robust data governance unless you want to risk £££ in fines and loss of your reputation. 
  • People, People, People: Do not forget that ultimately, it is your business users who will make or break any data initiative. Training, engagement, data mindset, and change management are all key contributors to creating and sustaining good data. 

AI Governance vs Data Governance

Governing AI solutions is indeed wider than just governing your AI data. Trust is perhaps the greatest challenge to business adoption of AI, which is why it is just as important to govern the technical side of things, for example, how the models and algorithms are developed, trained, and deployed, as well as the architecture they rely on. 

AI governance faces a few challenges, including: 

  • Explainability: A lot of AI models can feel like mysterious black boxes – we know something is happening, but we have no idea how! 
  • Unstructured Data: Things like emails, documents, images, and videos are everywhere. Unstructured data makes up about 80-90% of all new data. It’s like trying to organise your sock drawer – where do you start with all the holey, single, Peppa Pig themed socks that have found themselves in your draw?
  • Ethical Concerns: It’s critical to make sure AI systems are fair and unbiased. 
  • Data Privacy and Security: Protecting sensitive data used in AI training and operations is essential.  

Do I need Data Governance before Artificial Intelligence?

Data governance is essential before implementing artificial intelligence (AI) if you want your AI initiatives to succeed. Here’s why:

1. Data Quality Drives AI Performance

AI systems depend on high-quality data to learn and make decisions. Without proper data governance, your AI models may be trained on inaccurate, incomplete, or biased data, leading to unreliable outputs and potentially harmful business decisions.

2. Ensuring Data Compliance and Security

AI often involves handling sensitive data. Data governance ensures compliance with regulations such as GDPR, CCPA, or industry-specific standards. This reduces the risk of legal penalties or reputational damage from improper data usage.

3. Establishing Data Accessibility and Consistency

Data governance creates a framework for consistent data definitions, ownership, and access control. AI thrives on integrated and well-structured datasets. Governance ensures that the right data is available to the right teams in a usable format, eliminating silos and duplication.

4. Mitigating Bias and Ethical Risks

Poorly governed data can introduce bias into AI models, leading to unfair or unethical outcomes. Data governance provides processes to identify, address, and monitor bias, ensuring AI systems make equitable decisions.

5. Cost and Efficiency Benefits

Investing in data governance upfront reduces the time and cost of preparing data for AI initiatives. It prevents expensive rework or failures by addressing data issues early, rather than discovering them during or after model development.

Read our blog on how you can demonstrate the ROI of Data Governance.

6. Scalability for Future AI Projects

Data governance provides a scalable foundation for expanding AI capabilities. It ensures that as data volumes grow, they remain manageable, trustworthy, and usable for future AI-driven projects.

Nothing’s perfect.. and that’s alright 

At Oakland, data governance consulting doesn’t mean you need perfect data to embark on or progress on your AI roadmap.    

AI gets your data talking.. you might not like what it says if your data is incomplete or of poor quality, but on the plus side, you can also use AI to flush out where your data issues are (every cloud has a silver lining) 

How do we manage AI regulatory requirements 

As with everything, a balanced approach is essential.  

Yes, it is a balancing act between being compliant and giving the business space to innovate. The good news is that, although the AI regulatory landscape is continuously evolving, a key theme is the ability to demonstrate that your AI is trustworthy—and a data governance framework allows you to demonstrate trustworthy AI data pipelines. 

“AI, Generative AI or Agentic AI holds great promise. However, many organisations currently feel held back by the necessity to validate their data thoroughly, unclear understanding of AI risks, and the potential for unforeseen issues, like bias and data security. Either making them hesitant to fully commit, i.e. death by POC, or unrealised sustainable value when deployed. 

These are unchartered waters for most. The good news is you can choose the pace you want to go. Navigating AI governance is like steering a ship through a storm; robust data governance is your compass, guiding you to safer, clearer waters”.

Zareene Choudhury – Oakland Data Governance Lead

Want to find out more? 

If you want to know more about how to leverage Data Governance to get your AI Data trustworthy – then tune into our webinar, where Zareene Choudhury will be joined by Richard Adams of erwin by Quest. 

Get in touch

If you’d like to learn more about Oakland’s data governance consulting services, then get in touch by emailing hello@weareoakland.com

Want to read more about what a suitable Data Governance Framework looks like?

And if you want to find out how AI agents can make a difference to our business, read some of our case studies here, where we’re already delivering value to clients (well before AI agents became ‘the phrase of the year’! 

The post What is the Role of Data Governance in AI? Should You Even Care? appeared first on Oakland.

]]>
How to Harness Generative AI for Data-Driven Decision-Making https://weareoakland.com/blog/harnessing-generative-ai-for-data-driven-decision-making/ https://weareoakland.com/blog/harnessing-generative-ai-for-data-driven-decision-making/#respond Tue, 10 Oct 2023 14:17:15 +0000 https://www.theoaklandgroup.co.uk/?p=7701 “In the boardroom, one fact remains constant: data is the lynchpin of contemporary business. How many organisations can genuinely claim to have fully operationalised their data assets, though? If your data strategy still resides on the periphery of business operations, rather than being its driving force, you’re in good company.”  That illuminating passage was written...

The post How to Harness Generative AI for Data-Driven Decision-Making appeared first on Oakland.

]]>
“In the boardroom, one fact remains constant: data is the lynchpin of contemporary business. How many organisations can genuinely claim to have fully operationalised their data assets, though? If your data strategy still resides on the periphery of business operations, rather than being its driving force, you’re in good company.” 

That illuminating passage was written by ChatGPT (should we be surprised that even artificial intelligence (AI) calls for cleaner data?). Yet, while amusing, and with many of us having had a good play around with the technology, it really is time to consider generative AI as a key strategic asset for your business. 

Generative AI isn’t just another buzzword to add to your corporate vocabulary—it’s at the forefront of intelligent decision-making. You may already be familiar with analytical AI technologies like machine learning, but generative AI offers something more: the ability to create new, actionable insights by synthesising vast realms of data through custom AI systems.

What Are Some Key Gen AI Enterprise Use-Cases?

There are thousands of generative AI use cases out there – every organisation and its challenges are different after all. Yet, these are some of the most common that our AI consultants have worked with  over the past few years:

  • Automated Customer Interactions: Generative AI can power customer service solutions that offer unprecedented personalisation while streamlining operations. Recent surveys indicate that customers in some sectors prefer AI’s efficient and targeted responses over human responses.
  • Gen AI in marketing: GenAI suddenly makes life in the Marketing world much less resource and time-intensive. If you can think it, you can produce it! If you want a celebrity in your marketing campaign, no problem. For Nike’s 50th anniversary, we could see Serena Williams and a younger version of herself battling it out on the tennis court. Avatars can now also provide personalised and interactive messages. Imagine running campaigns that are not only data-driven but also continuously optimise themselves. Generative AI can produce innovative, personalised marketing content at scale.
  • AI in healthcare: Generative AI has dramatically reduced drug development and medical research cycles in the pharmaceutical and healthcare sectors. Accelerated healthcare research isn’t just about speed; it’s about enabling new avenues of research that were previously unthinkable.
  • AI for data governance: In a recent use case, a financial institution substantially leveraged Generative AI to improve its data quality. By creating simulated but realistic data sets, the institution could test the robustness of its fraud detection algorithms under various conditions before actual deployment. This enabled them to pinpoint weaknesses in their governance framework, tighten control mechanisms, and gain more reliable risk assessment insights.

What are the Challenges of Generative AI? 

In order for it to provide return on investment, generative AI brings with it several distinct challenges that businesses must be vigilant about: 

  • Lack of data integrity that produces untrue or biased outcomes
  • Unethical use that may lead to societal bias or human rights infringements
  • Copyright and intellectual property infringements and opportunities
  • Security and privacy rights
  • ESG Impact

Although there is currently no explicit law or legal framework to regulate AI use, the EU and UK are intent on releasing these soon. What’s more, there’s no getting around the fact that underlying data assets need to be fit for purpose to set foundations for compliant tools.  

How can AI be used in Data Governance?

In 2023, Gartner polled 2,500 executives, asking what the primary focus for GenAI initiatives in their businesses had been. Customer experience and retention (38% of all initiatives), and revenue growth (26%) came out on top, followed by cost optimisation (17%) and business continuity (7%).   

However, all this begs the follow-up question: how well did these initiatives do? 

A decade ago, dashboards were considered the pinnacle of data-driven decision-making. The limitations, however, became apparent when executives realised that the quality of underlying data was often inadequate. For AI implementations, the governance requirements are similar but exponentially more complex. Businesses must ensure that data governance policies are robust enough to manage the capabilities and risks of AI-driven decision-making processes.  

Bluntly put,  “Garbage in – Garbage out”. 

AI tools need to be managed as data tools. To make them fit for purpose, companies must have rigid oversight of the processes, policies, and governance of the data involved (both at ingress and egress) and the tool’s performance itself.  

An AI data governance framework should include: 

  • Data Quality and Integrity: Ensuring the data is unbiased and accurate.
  • Accountabilities and owners: For the data itself but also the development specifications and the outcomes.
  • Transparency: Where is the data being sourced from, how are the outcomes being used, and who can access these?
  • Ethical use: Does it have a negative impact, or is it a threat to people’s security and rights?
  • Human Oversight: AI doesn’t and shouldn’t fix its data problems to ringfence what the tool can and cannot produce, as hallucinations of the AI can become the norm.

Is your organisation ready for generative AI? 

There are four key questions every organisation should ask itself before embarking on a generative AI project:

  • Do you understand your business capability landscape adequately to scope where AI can improve your business performance?
  • Do you have a clear map of your information flow and data dependencies?
  • Does your business have a working Data Strategy or is it currently just a document sitting on the intranet?
  • Are your data platforms up to the task?

If you can answer yes to these questions, piloting Generative AI initiatives could be the next logical step. In today’s data-driven world, harnessing the power of AI is no longer an option; it’s a necessity. 

Generative AI is not just a buzzword; it’s a game-changer. It can transform how you handle data, making your operations smarter, faster, and more efficient. Whether you’re looking to automate tasks, enhance decision-making, or innovate your products and services, Generative AI is the key to unlocking these opportunities.

The next important question, though, is: where does Generative AI fit into your unique data landscape?

How Oakland can help you drive value from generative AI

Our experts understand how Generative AI can seamlessly integrate into your organisation’s data strategy and governance framework.

We understand that every organisation is unique. That’s why we don’t offer one-size-fits-all solutions. Instead, we take the time to understand your specific goals, challenges, and data ecosystem. Then, we craft a customised strategy that aligns with your business objectives, ensuring maximum ROI.

Data governance is the cornerstone of effective AI implementation. Our team specialises in developing robust data governance programs that ensure your data is secure, compliant, and ready for AI-driven insights. With our guidance, you can confidently navigate the complex world of data regulations. 

If you’re struggling to envision where Generative AI could fit into your data strategy or are ready to implement a data governance program that sets you up for AI success, have a chat with one of our team.

Zareene Choudhury and Lea Gorgulu Webb are senior consultants here at Oakland

Frequently asked questions

Want to know more about generative AI? Here are answers to some common questions. Don’t see the answer you’re looking for? Get in touch with one of our team.

What is Generative AI?

Generative AI is a type of AI focused on content creation. Gen AI systems are trained on existing data and use it to create original content such as text, images, videos, code, or audio, based on the patterns inherent to the training data. This is different to traditional AI, which only uses the data to make predictions or decisions.

The technology has grown significantly in popularity since November 2022, when ChatGPT was released to the public. While the initial novelty and hype have somewhat subsided, the potential for the technology to drive productivity gains has meant many organisations are adopting generative AI in their operations. A global INSEAD survey of business alumni in 2024 found that just over half of respondents’ organisations were using generative AI and only 21% had no plans to use it in the future.

How Does Generative AI Work?

The workings of generative AI can differ significantly from one use case to another, but in general, they are created using a four-step process that utilises neural networks, and data architectures like transformers:

  1. Data Collection: Significant volumes of data relevant to the desired output are collected. For example, if the goal is to generate customer service responses, transcripts and training materials might be used in the dataset.
  2. Training: The AI model is trained on this dataset using machine learning algorithms. This allows it to notice and replicate patterns in the data.
  3. Generation: Once trained, the model can generate new data that draws on and mimics patterns in the training data.
  4. Fine-Tuning: The model can be fine-tuned for specific applications or industries, ensuring the generated content meets particular standards or requirements.

Is generative AI machine learning?

Generative AI is a system enabled by the broader concept of machine learning. Machine learning algorithms allow generative AI systems to learn from data and then apply these learnings to complete tasks. Generative AI specifically uses those learnings to create original content. 

There are three main ways machine learning algorithms are trained: 

  • Reinforcement learning – Where the machine learning model is trained using rewards and penalties based on its actions. This reward system hones its understanding.
  • Supervised learning – Where data sets are labelled, giving the algorithm insight into the meaning and relationships between different data.
  • Unsupervised learning – Here, the algorithm is given unstructured data without any human input, and allowed to discover relationships and patterns on its own.

Generative AI systems are often developed with the final unsupervised method. This allows them to be more ‘creative’ with their outputs than other machine learning algorithms.

Is generative AI deep learning?

Generative AI is provided using deep learning techniques which involve layered architectures (so-called neural networks) to identify and remember complex patterns in data. Deep learning techniques include things like: 

  • Transformers: Transformers are a neural network architecture that transforms an input sequence (a prompt) into an output sequence (content) based on the learned relationships between the components in the two sequences. They are a key part of large language models like ChatGPT.
  • Generative Adversarial Networks (GANs): This system is formed from a generator, which creates fake outputs that mirror real data, and a discriminator, which tries to guess which are fake or real. The discriminator receives rewards or penalties (via reinforcement learning) if it’s correct or incorrect. GANs can create very lifelike outputs but can be unstable.
  • Variational Autoencoders (VAEs): These unsupervised models understand the structure of data using an encoder, decoder and loss function. The encoder compresses data and gives it a summary (of the image, text, audio, etc.). The system creates a bank of these summaries logged to each piece of compressed data, ready to draw on – red flowers, happy faces, etc.

    The user then provides a prompt, which is given to the decoder. It then tries to reconstruct the data as accurately as possible, based on the summary characteristics. The loss function then measures how much data was lost during reconstruction, distributing data smoothly. In practice, this all means that VAEs can create highly imaginative content (happy red faces, surrounded by petals), but that might be blurry or garbled, due to the loss function. 

The post How to Harness Generative AI for Data-Driven Decision-Making appeared first on Oakland.

]]>
https://weareoakland.com/blog/harnessing-generative-ai-for-data-driven-decision-making/feed/ 0