Let's talk

LinkedIn

Twitter

Medium

Youtube

<h2>Get a Detailed View with Datatonic + Looker Dashboards</h2>
Keeping track of Google Cloud consumption is key to monitoring your business&#8217; expenses. However, getting a detailed view of where the costs occur can be a challenge. By implementing a LookML (Looker&#8217;s native modelling language) wrapper, there is a simpler solution.
Datatonic, Looker&#8217;s go-to partner for all things Google Cloud, can help. See how you can be up and running within half an hour using our dashboards. Get clear visibility and optimise your Google Cloud consumption across projects, products and resources.  
<h2>Get a sneak peek</h2>
Datatonic can help with implementing two types of blocks for your business &#8211; the Basic Features version and the Customised Features version. 
Our Basic Features block covers: 
<ul>
<li style="font-weight: 400;">Billing costs allocated overall projects and products</li>
<li style="font-weight: 400;">Historical and period-over-period billing analysis</li>
<li style="font-weight: 400;">Queries issued and gigabytes scanned in a selected timeframe</li>
<li style="font-weight: 400;">Average query time across projects and users</li>
<li style="font-weight: 400;">Query inspector to dissect the mechanics and cost behind the most troublesome of expensive queries</li>
</ul>
<div id="attachment_3089" style="width: 2692px" class="wp-caption aligncenter"><img aria-describedby="caption-attachment-3089" decoding="async" loading="lazy" class="wp-image-3089 size-full" src="https://cms.datatonic.com/app/uploads/2023/03/looker_blog_basic.png" alt="Optimise your Google Cloud costs with Looker" width="2682" height="1305" />Figure 1. Looker Google Cloud Costs Dashboard &#8211; Basic Features</div>
Our Customized block includes:
<ul>
<li style="font-weight: 400;">Individual user-level cost analysis</li>
<li style="font-weight: 400;">Costs associated with each user and query/service</li>
<li style="font-weight: 400;">Detailed service logs analytics</li>
<li style="font-weight: 400;">Custom billing logic, such as applying discounts across SKUs</li>
<li style="font-weight: 400;">Organisation wide consumption with drill-downs through project/service/SKUs</li>
</ul>
<div id="attachment_3090" style="width: 3137px" class="wp-caption aligncenter"><img aria-describedby="caption-attachment-3090" decoding="async" loading="lazy" class="wp-image-3090 size-full" src="https://cms.datatonic.com/app/uploads/2023/03/Looker_blog_pro.png" alt="Figure 2. Looker GCP Cost Dashboard - Customized Features" width="3127" height="1209" />Figure 2. Looker GCP Cost Dashboard &#8211; Customized Features</div>
Follow us on <a href="https://www.linkedin.com/company/datatonic/">LinkedIn</a> to keep up to date with our latest blogs. 
Are you interested in managing your Google Cloud costs with Looker? Get in touch <a href="https://datatonic.com/contact/">here</a>.

<h1>Download 
the guide</h1>
A guide by data leaders, for data leaders, looking to turn Data Mesh principles into practice on Google Cloud.

Optimise Your Google Cloud Costs Using Looker

Data is one of the most important assets in a business. Data fluency is about being able to describe and interact with your data to bring value to the business, accurately and efficiently. Currently, only <a href="https://www.gov.uk/government/publications/quantifying-the-uk-data-skills-gap/quantifying-the-uk-data-skills-gap-full-report" target="_blank" rel="noopener">58% of businesses</a> think that their organisation has sufficient data skills to meet their current and future needs. Without fluency, data remains an untapped resource, and decision-making across the business is limited. To find out more, we spoke to Data Science + ML Trainer, Tom Udale.
<h4>Part 1: What is data fluency?</h4>
What do we mean when we talk about data fluency and why is it important?
“The first main part is describing data; employees should be able to identify the data within their area of the organisation and also be able to understand fundamental statistics about their data. And the second main area is interacting with that data. Employees should be able to identify issues with that data and know how to resolve them (or who to ask!)”
“This also means employees should be able to manipulate data to answer business questions and suggest ways to use their data to technical teams.”
While technical teams need to understand and be able to explore their data, for truly data-driven businesses, this also applies to business users.
&nbsp;
Does data fluency need to be business-wide?
“Data science teams don’t have all the business knowledge of users. Each employee will have specific knowledge about their area of expertise, which allows them to identify opportunities within data.
The amount of data in an organisation is massive! A small team of analysts and data scientists won’t be able to keep track of all the data generated in a company, so having active data citizens is vital.”
This means that business users will be able to identify innovative and realistic opportunities. If business users aren’t aware of what can be done with data, they won’t raise it with the technical teams. Equally, users need to be aware of what makes a good data opportunity, either for analytics, advanced machine learning or AI.
<h4>Part 2: How to build data fluency</h4>
What are the challenges to building data fluency?
&nbsp;
Building data fluency is a challenge that needs to be approached strategically and with best practices in mind. Here are four key attributes of successful data fluency development: 
1. Large-scale learning
Data fluency is a programme which needs to upskill whole organisations, which often have different schedules and restrictions on time. With proper planning, entire organisations can be prepared to use data effectively.
2. Specialized
The programme should be specific to different departments (Business, HR, Finance), with tailored use cases and exercises to show real-life examples and prepare employees with the most required skills for their role. 
3. At the right level
The programme should not focus too much, or spend too much time on technical implementation. Rather, it should focus on identifying business value, and understanding how data fluency can be used to make a difference.
4. Continuous training + upskilling
Instead of a one-off session, the programme should enable new employees on an ongoing basis, with a rolling programme to facilitate continuous improvement and skill retention.
&nbsp;
How can training solve your fluency shortfall?
“Datatonic Academy can align training to business objectives around data specific to your business. Training can be further tailored to each business unit, with relevant examples and exercises, and can be delivered in both an on-demand or instructor-lead format allowing for flexibility around users&#8217; other commitments”.
Furthermore, we can provide materials for ongoing in-house training, as well as create “super-users” within the organisation who can upskill colleagues, including new joiners.
<h4>Part 3: Leveraging data fluency to create AI + ML innovation in your business</h4>
It’s important to consider, once your team has been trained and upskilled: how can businesses measure a successful data fluency programme? And what is the impact we’ll see in the long run? 
There are three main areas that businesses who immerse themselves in a successful data fluency programme can expect to see:
&nbsp;
1. Data Opportunities
With a strong data fluency programme in place, you would see more opportunities for analytics and ML to come to the technical team. This will lead to higher productivity and efficiency
2. Higher quality data
As business users who interact with datasets across the organisation grow in fluency, they will be able to identify issues with data and help technical teams resolve problems.
3. Data-first culture
If the entire organisation is aware of the value of data, and how that can be realised, then business challenges of all types will be addressed with the relevant data at hand. This means faster and more accurate decision-making throughout your company.
&nbsp;
How can training accelerate your ML workflow and generate value for your business?
“Training doesn’t just have an impact at the foundational and cultural level within an organisation, realising the potential of AI, machine learning and generative AI all require a base level of knowledge, along with continuous upskilling and training.”
Strong data fluency and data foundations make it much easier for people to first identify new ways of using their data, and how this can be done with AI + ML, and then much easier to get started as the data is ready to use.
&nbsp;
Conclusion
Datatonic Academy can provide training at all levels, from fluency to state-of-the-art ML techniques, from experimentation to <a href="https://datatonic.com/events/mlops-101-ml-at-scale/">MLOps</a> and from on-premises to the cloud. As tools and technologies improve, help your workforce improve and stay motivated in AI and ML. See our upcoming public training events <a href="https://datatonic.com/events/">here</a>. 
&nbsp;
Find out more about <a href="https://datatonic.academy/">Datatonic Academy</a> and <a href="https://datatonic.academy/courses/#contact-section">get in touch</a> to book a free scoping session to kick-start your journey to building data fluency and secure 20% off any training courses booked before the end of January 2024.

Building Data Fluency Throughout Your Business

MLOps encompasses the set of practices, technologies, and culture that enable Machine Learning to deliver real business value in an organisation. Borrowing concepts from DevOps and applying them to Machine Learning, MLOps enables teams to move rapidly from offline experimentation to a deployed production system and beyond, unlocking business potential from Data Science.
In this webinar, Datatonic’s Machine Learning Engineering Lead, Felix Schaumann and Google Cloud&#8217;s Customer Engineer, Ivan Nardini, show you how to use <a href="https://datatonic.com/insights/datatonic-updated-turbo-templates-mlops-adoption-2024/" target="_blank" rel="noopener">Datatonic&#8217;s new Turbo Templates</a> to kickstart your Machine Learning development. 
&nbsp;
<iframe loading="lazy" title="MLOps in action: Operationalize your ML workflow using pipeline templates" width="500" height="281" src="https://www.youtube.com/embed/SdR5P2BDGow?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
&nbsp;
Looking to get started with MLOps but need help finding where to start? Datatonic is Google Cloud&#8217;s 5x Partner of the Year and 2x Machine Learning Partner of the Year. <a href="https://datatonic.com/contact/" target="_blank" rel="noopener">Get in touch</a> to start planning your next data + AI project.

MLOps in Action: Operationalize Your ML Workflow Using Pipeline Templates

London, 21st November 2023 &#8211; Datatonic announced today that it has released a new version of its open-source MLOps Turbo Templates, developed in partnership with Google Cloud’s Vertex Pipelines product team, to help data teams kickstart their Vertex AI MLOps initiatives. Customers like <a href="https://datatonic.com/case-studies/reliable-predictive-models-mlops-delivery-hero/" target="_blank" rel="noopener">Delivery Hero</a>, <a href="https://datatonic.com/case-studies/ai-booster-vodafone-supercharging-ai-ml-at-scale/" target="_blank" rel="noopener">Vodafone</a>, <a href="https://datatonic.com/case-studies/bt-ai-accelerator-artificial-intelligence/" target="_blank" rel="noopener">BT</a>, and many more have already adopted MLOps and so can you. 
Join today’s <a href="https://goo.gle/mlops-event" target="_blank" rel="noopener">Google x Datatonic webinar</a> to experience the MLOps Turbo Templates in action and continue reading to get started with MLOps.
<h4>What is MLOps?</h4>
MLOps encompasses the set of practices, technologies, and culture that enable Machine Learning to deliver real business value in an organisation. Borrowing concepts from DevOps and applying them to Machine Learning, MLOps enables teams to move rapidly from offline experimentation to a deployed production system and beyond, unlocking business potential from Data Science. 
Even in the era of Generative AI, MLOps is a valuable investment for companies. Core MLOps capabilities for traditional Machine Learning can be used for <a href="https://datatonic.com/services/generative-ai/" target="_blank" rel="noopener">Generative AI</a> use cases. Watch the Vertex AI product team talking at Google Cloud Next 2023 about <a href="https://www.youtube.com/watch?v=UfUoG_Ijgq4" target="_blank" rel="noopener">Building your MLOps strategy for Generative AI</a> to understand common MLOps components and unique requirements for Generative AI.
<h4>What is Vertex AI?</h4>
<a href="https://cloud.google.com/vertex-ai" target="_blank" rel="noopener">Vertex AI</a> is Google Cloud’s unified platform for AI + ML. It helps teams build, deploy, and scale ML systems faster, with tooling to support the entire ML lifecycle, from experimentation to production. Since its release in 2021, Vertex AI has continued to evolve, introducing powerful new features such as model monitoring, model versioning, and experiment tracking. Most recently, a new set of tooling has been released to support generative AI including <a href="https://cloud.google.com/generative-ai-studio" target="_blank" rel="noopener">Generative AI Studio</a>, <a href="https://cloud.google.com/model-garden" target="_blank" rel="noopener">Model Garden</a>, and <a href="https://cloud.google.com/blog/products/ai-machine-learning/create-generative-apps-in-minutes-with-gen-app-builder" target="_blank" rel="noopener">Gen App Builder</a>.
<h4>What are the “MLOps Turbo Templates” and why were they created? </h4>
The MLOps Turbo Templates are an open-source codebase that provides a reference implementation of the end-to-end ML lifecycle. Co-developed by Datatonic and Google Cloud’s Vertex Pipelines product team to help businesses kickstart their MLOps initiatives, they enable both Data Scientists and ML Engineers to use Google Cloud better and faster. Today, Datatonic is announcing a new version, bringing new features and capabilities to help organisations adopt MLOps practices on Google Cloud.
With more clients deploying ML models to production than ever before, our teams realised the power of MLOps solutions for maximising the value of clients’ ML models early on. Our MLOps solution for <a href="https://datatonic.com/case-studies/ai-booster-vodafone-supercharging-ai-ml-at-scale/" target="_blank" rel="noopener">Vodafone</a>, for example, has helped them significantly reduce time to production, and <a href="https://datatonic.com/case-studies/reliable-predictive-models-mlops-delivery-hero/" target="_blank" rel="noopener">Delivery Hero</a>’s MLOps Platform allows their teams to seamlessly leverage Machine Learning best practices. 
The MLOps Turbo Templates combine Datatonic’s experience of implementing MLOps solutions with Google Cloud’s technical excellence to help reduce technical debt and adoption challenges many businesses face.
The templates can be used to create a production-ready MLOps solution on Google Cloud, covering the key components of a production ML system, including:
<ul>
<li style="font-weight: 400;" aria-level="1">Infrastructure-as-Code</li>
<li style="font-weight: 400;" aria-level="1">ML code versioning and deployment with CI/CD</li>
<li style="font-weight: 400;" aria-level="1">Data preprocessing</li>
<li style="font-weight: 400;" aria-level="1">Training a model</li>
<li style="font-weight: 400;" aria-level="1">Scheduled ML batch predictions</li>
<li style="font-weight: 400;" aria-level="1">Skew/drift detection</li>
</ul>
&nbsp;
Although the Turbo Templates can be used in isolation for individual ML use cases, they can also form one of the key ingredients needed for adopting MLOps at scale across an organisation. The diagram below shows how a strong MLOps blueprint can fit into the wider picture of enterprise MLOps adoption.
&nbsp;
<img decoding="async" loading="lazy" class="alignnone wp-image-23282" src="https://cms.datatonic.com/app/uploads/2023/11/MLOps-Turob-Templates-Diagram-300x140.png" alt="MLOps-Turob-Templates-Diagram" width="750" height="350" srcset="https://cms.datatonic.com/app/uploads/2023/11/MLOps-Turob-Templates-Diagram-300x140.png 300w, https://cms.datatonic.com/app/uploads/2023/11/MLOps-Turob-Templates-Diagram-1024x479.png 1024w, https://cms.datatonic.com/app/uploads/2023/11/MLOps-Turob-Templates-Diagram-768x359.png 768w, https://cms.datatonic.com/app/uploads/2023/11/MLOps-Turob-Templates-Diagram-1536x718.png 1536w, https://cms.datatonic.com/app/uploads/2023/11/MLOps-Turob-Templates-Diagram.png 1600w" sizes="(max-width: 750px) 100vw, 750px" />
<h4>What’s new in the MLOps Turbo Templates?</h4>
Since the initial release last year, the MLOps Turbo Templates have continued to evolve, building on Datatonic’s experience implementing MLOps with leading enterprises, as well as Google’s continued investment and innovation in state-of-the-art ML tooling. The new enhancements to the template cover two key themes:
<ol>
<li style="font-weight: 400;" aria-level="1">Adopting the latest and greatest in Google Cloud</li>
<li style="font-weight: 400;" aria-level="1">Improving the developer experience, making the template more lightweight and easy to use.</li>
</ol>
In terms of new functionality, the MLOps Turbo Template now uses Vertex AI’s model versioning to make managing your trained models much easier and neater, with model evaluations now viewable natively in the Vertex Model Registry. 
Vertex AI Model Monitoring has replaced Tensorflow Data Validation to provide a built-in solution for monitoring data skew/drift, with added support for alerting. The Kubeflow Pipelines (KFP) SDK has been updated to the new KFP V2, which adds support for custom container components, pipelines-as-components, and the ability to store and version pipelines in Artifact Registry.
A lot of work has been done to improve the developer experience. By simplifying the pipelines and using more native Vertex AI functionality, the run time of the pipelines has been reduced by over 50%, allowing for faster iteration and less time spent waiting for pipelines to finish. The codebase has been significantly refactored, reducing the number of custom pipeline components and simplifying the approach to testing, both of which contribute to a lower barrier to entry and a reduced maintenance burden for teams using the template. 
Lastly, the Infrastructure-as-Code has been rewritten from the ground up to provide full cloud infrastructure for dev, test and production environments using Terraform, making it even easier for Data Science teams to get started on Vertex AI.
<h4>Try it out for yourself!</h4>
The new release of the MLOps Turbo Template is now available in Datatonic’s open-source repository <a href="https://github.com/teamdatatonic/vertex-pipelines-end-to-end-samples" target="_blank" rel="noopener">here</a> (and <a href="https://github.com/GoogleCloudPlatform/vertex-pipelines-end-to-end-samples" target="_blank" rel="noopener">Google Cloud’s repository</a> in future), including all the documentation needed to help teams start with their use cases. 
Get started today with the new <a href="https://github.com/teamdatatonic/vertex-pipelines-end-to-end-samples/blob/develop/docs/notebooks/01_infrastructure_setup.ipynb" target="_blank" rel="noopener">three-part notebook series</a> which will guide you through deploying your own MLOps project on Google Cloud and the productionisation of an ML model on Vertex AI in minutes.
Datatonic enables leading companies, like Vodafone, Sky, and Inmarsat, to make better business decisions with a powerful Modern Data Stack and accelerate impactful solutions with MLOps. For enquiries about new projects, <a href="http://datatonic.com/contact" target="_blank" rel="noopener">get in touch</a>.

Datatonic Launches Updated “Turbo Templates” to Supercharge MLOps Adoption in 2024

<img decoding="async" loading="lazy" class="wp-image-23160 alignleft" src="https://cms.datatonic.com/app/uploads/2023/11/Marketing-Analytics-Specialization-300x300.png" alt="" width="178" height="178" srcset="https://cms.datatonic.com/app/uploads/2023/11/Marketing-Analytics-Specialization-300x300.png 300w, https://cms.datatonic.com/app/uploads/2023/11/Marketing-Analytics-Specialization-150x150.png 150w, https://cms.datatonic.com/app/uploads/2023/11/Marketing-Analytics-Specialization.png 726w" sizes="(max-width: 178px) 100vw, 178px" />
London, 15th November 2023 — Datatonic today announced that it has achieved the Marketing Analytics Partner Specialization in Google Cloud Partner Advantage. By earning the Partner Specialization, Datatonic has proven its expertise and success in building powerful Marketing Analytics solutions for customers using Google Cloud technology.
Datatonic has received this Specialization following a range of advanced marketing analytics projects for industry-leading businesses, including: 
<ul>
<li style="font-weight: 400;" aria-level="1">Developing a Customer Marketing Dataset to process over one billion events daily for a <a href="https://datatonic.com/industries/retail/?utm_source=linkedin&amp;utm_medium=organic&amp;utm_campaign=Announcement_MASpecialization_202311&amp;utm_term=" target="_blank" rel="noopener">leading UK retailer</a></li>
<li style="font-weight: 400;" aria-level="1">Enabling a leading global <a href="https://datatonic.com/industries/telecommunications/?utm_source=linkedin&amp;utm_medium=organic&amp;utm_campaign=Announcement_MASpecialization_202311&amp;utm_term=" target="_blank" rel="noopener">telecommunications</a> company to monitor + reduce customer churn on its broadband products</li>
<li style="font-weight: 400;" aria-level="1">Delivering real-time analytics and innovative new <a href="https://datatonic.com/partners/looker/?utm_source=linkedin&amp;utm_medium=organic&amp;utm_campaign=Announcement_MASpecialization_202311&amp;utm_term=" target="_blank" rel="noopener">Looker</a> dashboards and Explores to improve marketing metrics for a <a href="https://datatonic.com/industries/retail/?utm_source=linkedin&amp;utm_medium=organic&amp;utm_campaign=Announcement_MASpecialization_202311&amp;utm_term=" target="_blank" rel="noopener">global luxury retailer</a></li>
</ul>
Datatonic’s solutions have enabled clients to make data-driven decisions to generate value at scale. We’ve given our customers the ability to transition from disparate datasets to data-driven marketing.
“At Datatonic, we&#8217;re passionate about impact and feel privileged that our customers choose us to accelerate and support their data and AI journeys &#8211; allowing us to realise new value for their businesses, delivering real-time analytics and powerful marketing dashboards using Looker. We are proud of our deep collaboration with Google and are delighted to have been awarded the Marketing Analytics Specialization.” &#8211; Andy Harding, CTO at Datatonic
We look forward to our continued collaboration with Google Cloud, helping clients drive value from their data from cloud migrations to advanced analytics, <a href="https://datatonic.com/services/generative-ai/?utm_source=linkedin&amp;utm_medium=organic&amp;utm_campaign=Announcement_MASpecialization_202311&amp;utm_term=" target="_blank" rel="noopener">Generative AI</a> and beyond.
Discover <a href="https://datatonic.com/services/?utm_source=linkedin&amp;utm_medium=organic&amp;utm_campaign=Announcement_MASpecialization_202311&amp;utm_term=" target="_blank" rel="noopener">our services</a> to start your next data + AI project.

Datatonic Achieves the Marketing Analytics Partner Specialization in Google Cloud Partner Advantage 

Didn&#8217;t get a chance to attend <a href="https://coalesce.getdbt.com/" target="_blank" rel="noopener">dbt Coalesce</a>?
Hear from Datatonic’s Lead Data Architect, Ash Sultan, DevOps Lead, Alex Peissel, and Senior Cloud Architect, Stan Hill, as they explore using modern data analytics at scale with dbt!
Learn how you can reduce your time to insights, increase the productivity of your data teams, and optimise your costs.
<iframe loading="lazy" title="Enterprise MDS deployment at scale: dbt &amp; DevOps - Coalesce 2023" width="500" height="281" src="https://www.youtube.com/embed/6NbQlbp_JDM?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
<div class="content single-post__content">
&nbsp;
Datatonic, 5x Google Cloud Partner of the year, is the leading cloud data + AI consultancy for the world’s most ambitious brands, challenging traditions to deliver tangible innovation at the leading edge of Google Cloud. Learn about our experience with <a href="https://datatonic.com/partners/dbt/" target="_blank" rel="noopener">dbt</a>, and <a href="https://datatonic.com/contact/" target="_blank" rel="noreferrer noopener">contact us</a> to find out more.
</div>

Enterprise MDS Deployment at Scale: dbt + DevOps

Generative AI is enabling businesses to provide innovative new customer experiences, eliminate repetitive manual tasks, and accelerate time-to-insights for data-driven businesses.
In this online lecture, Datatonic’s Machine Learning Engineering Lead, Felix Schaumann shows you how to find the right Generative AI use case for your business, and walks you through how you can easily get started today with Snowflake’s <a href="https://www.snowflake.com/en/data-cloud/snowpark/" target="_blank" rel="noopener">Snowpark</a>. 
<iframe loading="lazy" title="Unlocking Generative AI Use Cases with Snowpark" width="500" height="281" src="https://www.youtube.com/embed/Cf2FSMGuf7o?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
&nbsp;
Looking to get started with <a href="https://datatonic.com/services/generative-ai/" target="_blank" rel="noopener">Generative AI</a> but need help finding where to start? Request a free two-hour <a href="https://datatonic.com/solutions/generative-ai/art-of-the-possible/" target="_blank" rel="noopener">Art of the Possible workshop</a> to explore how GenAI can transform customer experiences, unlock new revenue streams, and increase efficiency.

Unlocking Generative AI Use Case with Snowpark

AI has been accelerating rapidly over the past decade. Most recently, the rise of <a href="https://datatonic.com/solutions/generative-ai/" target="_blank" rel="noopener">Generative AI</a> has captured the fascination and imagination of us all &#8211; with Google at the forefront of creating Generative models that empower users to revolutionise how we complete many tasks, such as Code Assistants and Image Creation models. 

This trend is predicted to continue throughout this decade, with a <a href="https://www.nextmsc.com/report/artificial-intelligence-market" target="_blank" rel="noopener">recent study</a> showing that the AI market will grow 20-fold in the next seven years. 
<img decoding="async" loading="lazy" class="alignnone wp-image-22825" src="https://cms.datatonic.com/app/uploads/2023/10/Carbon-Impact-AI-Graph-300x181.png" alt="Carbon Impact AI Graph" width="701" height="423" srcset="https://cms.datatonic.com/app/uploads/2023/10/Carbon-Impact-AI-Graph-300x181.png 300w, https://cms.datatonic.com/app/uploads/2023/10/Carbon-Impact-AI-Graph-1024x617.png 1024w, https://cms.datatonic.com/app/uploads/2023/10/Carbon-Impact-AI-Graph-768x462.png 768w, https://cms.datatonic.com/app/uploads/2023/10/Carbon-Impact-AI-Graph.png 1096w" sizes="(max-width: 701px) 100vw, 701px" />
Fig 1: Predicted growth of the AI market from 2023-2030
&nbsp;
We are creating more and more models that cover every aspect of our lives. This is fantastic from an innovation perspective, but a subject that is often overlooked, or forgotten about completely, is the energy that goes into both training AI + ML models and the energy required for model inference.
Before we dig into how we can reduce the carbon footprint of our AI estate by using Intel’s state-of-the-art infrastructure, let’s go back to basics &#8211; how does Cloud infrastructure create emissions?
<h3>Carbon Footprint of the Cloud</h3>
Over the past decade, an increasing percentage of IT infrastructure has been migrated to the Cloud. All software running in the Cloud, from applications to data pipelines to AI, runs in Data Centres (DCs) and consumes electricity.
In short, electricity can either come from dirty sources (i.e., fossil fuels) or clean sources (i.e., renewables, such as solar and wind). This means that calculating the carbon impact of a particular piece of software is dependent on several factors, including the electricity make-up of the grid at runtime. To encapsulate all of the various factors that go into calculating the carbon impact of software, we use the SCI equation.
The SCI is a method of scoring any software application, regardless of where it runs, comprising of the following:
SCI = ((E * I ) + M) per R
&nbsp;
Where:
<ul>
<li style="font-weight: 400;" aria-level="1">E = Energy consumed by software in kWh</li>
<li style="font-weight: 400;" aria-level="1">I = Carbon emitted per kWh of energy (gCO²/kWh)</li>
<li style="font-weight: 400;" aria-level="1">M = Carbon emitted through the hardware that the software is running on</li>
<li style="font-weight: 400;" aria-level="1">R = Functional Unit e.g., per user or device</li>
</ul>
&nbsp;
From this equation, we can see that the part (E * I) is heavily influential when calculating the carbon footprint of a particular piece of software.
In the simplest form, E is dependent on the execution of the software itself. Logically, if software can run more efficiently, then it will consume less energy (this is important for later!).
The I part of the equation is all about the make-up of the grid that was mentioned earlier &#8211; i.e., the “carbon intensity” of the energy being used by the Cloud provider’s DC at that time. In the case of Google Cloud, these are published for all to see <a href="https://cloud.google.com/sustainability/region-carbon" target="_blank" rel="noopener">here</a>.
So now that we’ve covered the basics of Cloud carbon impact, let’s show how we can reduce the carbon impact of our AI inferencing!
<h3>Reducing the Carbon Impact of AI Inferencing on Google Cloud</h3>
The scene is set: AI is growing exponentially, it uses a lot of energy, and energy use is a key part of calculating the carbon footprint on a particular software (in our case, AI Inference).
As mentioned previously, the more efficiently a piece of software runs, the less energy it will consume. This is where Intel’s new state-of-the-art infrastructure on Google Cloud comes in for AI inference!
<h3>Introducing the new C3 machine series on Google Cloud</h3>
A new addition to Google’s ecosystem of workload-optimised infrastructure is the C3 VM instance type &#8211; powered by the 4th Gen Intel Xeon Scalable processor and Google’s custom Intel Infrastructure Processing Unit (IPU).
C3 instances were the first VMs in the public cloud with this next-gen processor which also features Intel(R) AMX (Advanced Matrix Technology) &#8211; a built-in accelerator for deep-learning training and inference workloads. Combining “Tiles” (hardware registers for storing larger chunks of data) and “TMUL” (matrix multiplication instructions), and with support for INT8 and BF16 data types, Intel AMX enables 4th Gen Xeons to deliver significantly higher training and inference performance than prior processor generations.
But how does this translate into impact in the real world? In short, it makes the C3 instance particularly good for AI Inferencing &#8211; the process of a model and receiving results.
It’s worth pointing out that the C3 VM provides us with CPU-based compute. You might be thinking “Why not use a GPU to improve efficiency?” &#8211; and GPUs are indeed very effective for AI workloads. However, GPU-based workloads tend to be considerably more expensive. This positions the C3 Instance in a unique position, where you can take advantage of the price advantage of using CPU compute for AI inference, with incredible performance.
To demonstrate the performance improvements for AI inference with the C3 instance type, Datatonic ran a benchmarking exercise against its predecessors!
<h3>Putting C3 to the test</h3>
To highlight the benefit of running AI inference on the new C3 instance, we compared it to its older cousin, the N2 instance (all running on Google Cloud). In our experiments, we ran two tests: one RESNET model and one BERT model (both with INT8 quantization and utilising the Intel Extension for PyTorch to enable AMX support) &#8211; measuring for CO² produced and time taken.
Note: to harness metrics about CO², we used the open-source Python library <a href="https://github.com/mlco2/codecarbon" target="_blank" rel="noopener">CodeCarbon</a>.
&nbsp;
The results are quite clear to see:
<table>
<tbody>
<tr>
<td>Instance Type</td>
<td>RESNET &#8211; CO² (kgCO2e)</td>
<td>RESNET &#8211; Time (s)</td>
<td>BERT  &#8211; CO² (kgCO2e)</td>
<td>BERT  &#8211; Time (s)</td>
</tr>
<tr>
<td>c3-standard-8</td>
<td>4.64E-06</td>
<td>3.957773924</td>
<td>5.26E-05</td>
<td>30.76444554</td>
</tr>
<tr>
<td>n2-standard-8</td>
<td>9.87E-06</td>
<td>6.932961464</td>
<td>1.24E-04</td>
<td>70.95138311</td>
</tr>
<tr>
<td>Reduction (%)</td>
<td>53.0%</td>
<td>42.9%</td>
<td>57.7%</td>
<td>56.6</td>
</tr>
</tbody>
</table>
 
When compared to the N2 Instance type, the C3 reduces the carbon impact of AI inference by more than 50%! In addition, we received results roughly twice as quickly too &#8211; meaning that we cannot only serve predictions quicker, but we can more than halve our carbon impact in the process by simply switching the underlying infrastructure!
<h3>Final Thoughts</h3>
AI will be one of the key influences that shape the future of humankind, however, the environmental impact of AI is a topic that is often pushed to one side. If we are to live in a future where AI models are responsible for assisting or even fully owning everyday tasks, we must ensure that we build AI in a responsible manner &#8211; especially from an environmental perspective.
There are many ways to <a href="https://datatonic.com/insights/predicting-reducing-energy-consumption-machine-learning-models/" target="_blank" rel="noopener">reduce the carbon footprint of your AI</a>, such as selecting your Cloud regions to deploy AI into the greenest regions possible or utilising more efficient model architectures, but one super simple way to reduce the carbon impact of AI inference, while looking after cost, is to simply switch the underlying CPU infrastructure.
We have demonstrated that switching from an N2 instance to the new C3 instance, powered by Intel’s 4th Gen Intel Xeon Scalable processor, more than halves the carbon emissions of AI inference! 
&nbsp;
To find out how Data, AI, and Machine Learning can benefit your business and help you to become more sustainable, <a href="http://datatonic.com/contact" target="_blank" rel="noopener">get in touch here</a>.

Reducing the Carbon Impact of AI Inference Using State-of-the-Art Infrastructure

In an ever-evolving landscape, the gaming industry has been at the forefront of technological innovation and transformation, with rapidly emerging and changing trends. By 2027, an astonishing <a href="https://www.statista.com/statistics/667694/number-mobile-gamers-worldwide/">2.3 billion people</a> are projected to engage in mobile gaming, surpassing the combined numbers of console and PC gamers. 
The games industry is a driving force for technological innovation, unleashing human imagination through cutting-edge advancements. Game developers create experiences that captivate and inspire over 3 billion players worldwide (<a href="https://newzoo.com/resources/blog/the-games-market-in-2022-the-year-in-numbers">Newzoo</a>). From the latest AAA titles to mobile games, the games industry is constantly pushing the boundaries of what is possible.
In this blog, we delve into the exciting world where artificial intelligence and gaming converge, exploring how Generative AI is revolutionising game development and player experiences.
<h3>Generative AI in Gaming</h3>
When exploring how Generative AI can be used in the gaming industry, use cases can be split into two main areas. The first is Generative AI which will be used by developers in the gaming industry to accelerate and innovate the process of game creation. The second is for the players to improve, and potentially revolutionise, their gaming experience to make gaming more fun and accessible.
<h4>GenAI will help those who work in the Industry</h4>
With the ability to augment and scale game development teams, Generative AI is empowering developers to craft richer gaming experiences at a pace that was previously unattainable. Serving as powerful coding assistants, GenAI models can help streamline the technical aspects of game creation, enabling teams to focus more on creative and impactful aspects, such as developing and improving new missions or working on new features to be added to games. 
It&#8217;s important to note that while GenAI can aid creative teams, it&#8217;s not a replacement for skilled designers and creators. The true winners in the evolving landscape of game development will be those who seamlessly integrate Generative AI into their creative processes, leveraging its power to amplify their artistic visions and deliver unparalleled gaming experiences for their players.
<h4>GenAI will provide new innovative adventures for players</h4>
Generative AI is also enabling a completely new era of gameplay within the gaming industry. It&#8217;s not just a tool for developers; it&#8217;s becoming an integral part of the player experience. 
GenAI is unlocking gaming experiences that were previously unimaginable. One example of this is the metaverse gaming platform, <a href="https://datatonic.com/case-studies/hiber-gaming-generative-ai-world-creation/">HiberWorld</a>, where users will soon be able to create entirely new worlds by simply typing a description of what they are imagining using natural language. (Find out more in our <a href="https://datatonic.com/case-studies/hiber-gaming-generative-ai-world-creation/">latest customer success story</a>).
&nbsp;
“As development teams integrate GenAI into their production process, some have begun integrating it into the game itself. This has the potential of revolutionizing player experiences, with GenAI running in real-time as the game is played.” Jack Buser, Director of Game Industry Solutions, Google Cloud
&nbsp;
<iframe loading="lazy" title="Hiber shows off generative AI for 3D world creation" width="500" height="281" src="https://www.youtube.com/embed/y_AS0SKhXnE?start=1&#038;feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
&nbsp;
In the future, GenAI in gaming could be taken a step further, where players can engage with in-game scenarios that adapt to their choices and interact with non-playable characters (NPCs) in a way that feels like they are holding a real conversation. This could be used during tutorials to help users when they first start playing a new game, later on, to assist with difficult missions or levels, or simply to enhance player experience with more interactive NPCs. 
&nbsp;
<h4>What about Intellectual Property?</h4>
The Challenge
Incorporating Generative AI into the gaming industry comes with its share of challenges, notably the concern that developers have regarding their valuable data and intellectual property. Game development studios invest significant resources in creating and curating data, and some fear that this data could be misused or exploited if it falls into the wrong hands. 
Additionally, there&#8217;s worry about the unauthorised usage of copyrighted assets, textures, and artwork. Game companies and artists put in effort to develop unique content, and they are rightfully cautious about any potential accidental infringement by AI on their intellectual property.
&nbsp;
The Solution
Despite these concerns, the required infrastructure and tools have emerged to address these issues. One such solution is exemplified by Hiber, a gaming platform that uses Generative AI to enable its users to create new 3D assets and worlds. Hiber takes a proactive approach by creating its own graphics, objects, and textures in-house and fine-tuning its Large Language Model to its asset library, ensuring that external copyrighted or proprietary materials are not at risk of being generated by its users. The model only uses assets that have already been created and approved.
Moreover, Hiber&#8217;s integration with Google&#8217;s Generative AI tooling allows for private, secure content generation. This means that the generated content that makes use of the asset library will not reappear elsewhere without explicit consent, removing any fears of businesses worried about their assets being used without permission, such as with open-source LLMs. Hiber&#8217;s approach not only embraces the potential of Generative AI but also prioritises data security and intellectual property protection, paving the way for a more secure and innovative future in the gaming industry.
<h4>Conclusion</h4>
As the gaming industry explores the potential of Generative AI and the infrastructure and governance improve, it seems that developers who use Generative AI to accelerate game production and provide innovative new experiences will be the ones to succeed in this competitive industry. 
Need help finding where to start? Request a free two-hour <a href="https://datatonic.com/solutions/generative-ai/art-of-the-possible/">Art of the Possible workshop</a> to explore how GenAI can transform customer experiences, unlock new revenue streams, and increase efficiency.

Enhancing Gaming with Generative AI

<a href="https://bigdataldn.com/" target="_blank" rel="noopener">Big Data LDN</a> is the UK’s leading free data, analytics + AI conference and exhibition, hosting leading data, analytics + AI experts, discussing how they deliver their most effective data-driven strategies. 300+ speakers and 10,000+ attendees from businesses around the world discussed their most pressing data + AI challenges and the brand-new solutions they have developed to solve them. 
Our team was invited by Snowflake and Fivetran to showcase how their technologies enable GenAI use cases, data monetisation, and deeper insights, and learned about the biggest trends right now. Have a read of the highlights of the two days of sessions below covering various sessions over the two days, from how to build strong data fundamentals before using AI, how to get started with the latest Generative AI technologies, to how to get your teams on board with a powerful data culture. 
<h4>1. Building strong data fundamentals</h4>
Setting the Foundations of Your Data Strategy to Maximise the Power of AI + ML
Successful adoption of AI + ML is increasingly essential for competitive success. Snowflake’s Field CTO, Hilda Davies, highlighted how the right Data Cloud strategy can accelerate and scale your AI + ML adoption. This is particularly important with people rushing into innovative new Generative AI use cases, without considering their underlying data platform and best practices. 
Snowflake allows you to store large volumes of data, and run ML models directly on what you already have, without needing to make copies of the data, maintaining a single source of truth. The main message from the talk was an important one: as you rush into Generative AI, do not compromise on data governance principles, and ensure that you use a platform that gives security as well as efficiency.
<img decoding="async" loading="lazy" class="wp-image-22646 alignleft" src="https://cms.datatonic.com/app/uploads/2023/09/BDL3-1024x768.jpeg" alt="Big Data LDN Snowflake" width="307" height="230" srcset="https://cms.datatonic.com/app/uploads/2023/09/BDL3-1024x768.jpeg 1024w, https://cms.datatonic.com/app/uploads/2023/09/BDL3-300x225.jpeg 300w, https://cms.datatonic.com/app/uploads/2023/09/BDL3-768x576.jpeg 768w, https://cms.datatonic.com/app/uploads/2023/09/BDL3-1536x1152.jpeg 1536w, https://cms.datatonic.com/app/uploads/2023/09/BDL3-2048x1536.jpeg 2048w" sizes="(max-width: 307px) 100vw, 307px" />
&nbsp;
Datatonic’s Lead Data Architect, Ash Sultan, and Senior Analytics Engineer, Scott Hobbs, discussed how users can get started with data monetisation on Snowflake, enabling them to diversify revenue streams and increase the profitability of their data team. 
&nbsp;
&nbsp;
&nbsp;
<img decoding="async" loading="lazy" class="alignnone wp-image-22647 size-large" src="https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-1024x574.png" alt="" width="1024" height="574" srcset="https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-1024x574.png 1024w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-300x168.png 300w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-768x431.png 768w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-1536x861.png 1536w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-2048x1148.png 2048w" sizes="(max-width: 1024px) 100vw, 1024px" />
Data monetisation is a great way for any company to increase its revenues quickly, and Snowflake provides the right infrastructure to do this securely in many different ways. Find out more about our <a href="https://datatonic.com/solutions/snowflake-data-sharing-package/" target="_blank" rel="noopener">Snowflake Data Sharing package</a>. 
<img decoding="async" loading="lazy" class="wp-image-22648 alignleft" style="font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif;" src="https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-1024x768.jpeg" alt="" width="307" height="230" srcset="https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-1024x768.jpeg 1024w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-300x225.jpeg 300w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-768x576.jpeg 768w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-1536x1152.jpeg 1536w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Snowflake-2048x1536.jpeg 2048w" sizes="(max-width: 307px) 100vw, 307px" />
Head of Machine Learning Engineering, Felix Schaumann, explored how to get started with Generative AI use cases on Snowpark. Businesses from all industries are unlocking the power of Generative AI with innovative new use cases. Snowflake is enabling companies to get started securely, without needing to worry about what happens to their data &#8211; see our <a href="https://datatonic.com/solutions/generative-ai/art-of-the-possible/" target="_blank" rel="noopener">Art of the Possible workshop</a> for more information. Both talks will be available on demand soon!<img decoding="async" loading="lazy" class="alignnone wp-image-22649 size-large" src="https://cms.datatonic.com/app/uploads/2023/09/Screenshot-2023-09-27-at-11.43.36-1024x601.png" alt="" width="1024" height="601" srcset="https://cms.datatonic.com/app/uploads/2023/09/Screenshot-2023-09-27-at-11.43.36-1024x601.png 1024w, https://cms.datatonic.com/app/uploads/2023/09/Screenshot-2023-09-27-at-11.43.36-300x176.png 300w, https://cms.datatonic.com/app/uploads/2023/09/Screenshot-2023-09-27-at-11.43.36-768x451.png 768w, https://cms.datatonic.com/app/uploads/2023/09/Screenshot-2023-09-27-at-11.43.36-1536x902.png 1536w, https://cms.datatonic.com/app/uploads/2023/09/Screenshot-2023-09-27-at-11.43.36-2048x1203.png 2048w" sizes="(max-width: 1024px) 100vw, 1024px" />
Understanding people, processes and platforms
With Generative AI, the technology and best practices are constantly evolving, as noted in a panel discussion with Di Mayze, Global Head of Data and AI at WPP, Natalie Cramp, CEO of Profusion, and Richard Davis, Chief Data Officer of Ofcom. The discussion explored the evolution of data culture caused by the GenAI movement and the implications of generative AI on various aspects of organisations, such as workforce dynamics, decision-making processes, and the role of human creativity. 
Richard Davis discussed how internal analytics assistants and automation can rapidly improve operational processes, to allow workers to conduct more interesting tasks, rather than repetitive manual work. 
Di Mayze spoke about WPP’s Generative AI roadmap, with a use case roadmap stretching as far as 2028. Addressing concerns about AI replacing jobs, Di said “You’re not going to lose your job to AI; you’re going to lose your job to someone who uses it”. Di also stressed that CEOs need fundamentals before GenAI. 
With Generative AI, there are still currently lots of uninformed users, and it is important to <a href="https://datatonic.academy/" target="_blank" rel="noopener">educate your organisation</a>, ensuring that technology is a tool to achieve your business goals, not the endgame product. 
“The AI had given their humanity back because it’s allowing them to do the things that people are good at” &#8211; Richard Davis, CDO at Ofcom
The main takeaway from all speakers on the panel was that before getting started with Generative AI, your business needs good data fundamentals, and your team needs to understand data and AI more widely. Education is also required to share best practices, as well as the risks and how to minimise them. 
<h4>2. Using data science + Generative AI</h4>
How Harrods Hits Profitability Targets with Data Science + MLOps
Carys Lees, Head of Data Strategy at Harrods, shared Harrods’ journey of building a data science capability in one of Britain&#8217;s leading luxury retailers. Tasked with high targets in terms of business value, Carys shared how they went on a journey to delivering tangible benefits to the business. 

AI and ML unlock many opportunities for <a href="https://datatonic.com/insights/generative-ai-personalisation-for-retail/" target="_blank" rel="noopener">retail</a>, ranging from traditional AI + ML such as <a href="https://datatonic.com/case-studies/product-recommendations-mulberry/" target="_blank" rel="noopener">propensity scoring</a> to more recent <a href="https://datatonic.com/solutions/generative-ai/" target="_blank" rel="noopener">Generative AI</a> solutions. Harrods discussed the need in retail to get the right data to the right people at the right time at the right cost. 
For them, this involves using data to predict the stock required for each size of luxury clothing items. This significantly impacts profitability; they do not want to understock a size and miss out on revenue by not meeting demand, but they do not want to over-order and have to significantly discount or throw away leftover stock at the end of the season. With AI + ML, they can <a href="https://datatonic.com/case-studies/retail-store-segmentation-intersport/" target="_blank" rel="noopener">optimise stock levels</a> and locations. 
Different businesses with have different high-impact use cases depending on their current challenges. For other retail businesses, this could be using ML to enhance operations, such as <a href="https://datatonic.com/case-studies/demand-forecasting-optimise-stock/" target="_blank" rel="noopener">demand forecasting</a>, <a href="https://datatonic.com/events/ad-spend-optimisation-machine-learning/" target="_blank" rel="noopener">ad spend optimisation</a>, and <a href="https://datatonic.com/insights/helping-mandm-direct-select-the-right-audience-for-google-ad-campaigns/" target="_blank" rel="noopener">improving advertising audiences</a>. Generative AI in retail also has a <a href="https://datatonic.com/insights/generative-ai-personalisation-for-retail/" target="_blank" rel="noopener">number of applications</a>.
&nbsp;
Every company needs its own GPT
As GenerativeAI takes the tech world by storm, companies must take control of their data, algorithms, and models. H2o.ai described how organisations now have the power to create Generative AI capabilities while maintaining their data integrity with fine-tuned state-of-the-art LLMs. Any business can develop a knowledge worker which maintains their data privacy and security and reduces the strain on internal teams, such as Analytics and HR. (Find out more about <a href="https://datatonic.com/solutions/generative-ai/" target="_blank" rel="noopener">Generative AI use cases</a>).
&nbsp;
How Anheuser-Busch InBev Unlocked Insights on Tap with a Gen AI Assistant
Global beverage leader, Anheuser-Busch InBev, also spoke about GenAI, showing how it’s transforming its workflows with AI assistants for automated and ad hoc analysis and insights. Instead of waiting hours or days for an analyst to run queries and develop reports, AB InBev’s internal analytics assistant rapidly generates ad hoc questions, freeing up time for its data teams, and accelerating the time to insights for everyone else. 
An often overlooked point is that these assistants do not work a traditional 9-5; they can be used 24/7, and are never busy like a person would be. These analytics assistants can be used to remove the guesswork from the insight process and accelerate time-to-value.
<img decoding="async" loading="lazy" class="wp-image-22650 alignleft" src="https://cms.datatonic.com/app/uploads/2023/09/BDL4-225x300.jpg" alt="Big Data LDN Fivetran" width="228" height="304" srcset="https://cms.datatonic.com/app/uploads/2023/09/BDL4-225x300.jpg 225w, https://cms.datatonic.com/app/uploads/2023/09/BDL4-768x1024.jpg 768w, https://cms.datatonic.com/app/uploads/2023/09/BDL4-1152x1536.jpg 1152w, https://cms.datatonic.com/app/uploads/2023/09/BDL4-1536x2048.jpg 1536w, https://cms.datatonic.com/app/uploads/2023/09/BDL4-scaled.jpg 1920w" sizes="(max-width: 228px) 100vw, 228px" />
&nbsp;
&nbsp;
Christelle Xu, Datatonic’s Senior Global Product Manager, presented how simple it is to set up an analytics assistant to rapidly reduce the strain on your internal analytics teams and provide quick insights across the business. <img decoding="async" loading="lazy" class="alignnone wp-image-22651 size-large" src="https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Fivetran-1024x576.png" alt="Big Data LDN Fivetran" width="1024" height="576" srcset="https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Fivetran-1024x576.png 1024w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Fivetran-300x169.png 300w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Fivetran-768x432.png 768w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Fivetran-1536x864.png 1536w, https://cms.datatonic.com/app/uploads/2023/09/Big-Data-LDN-Fivetran-2048x1152.png 2048w" sizes="(max-width: 1024px) 100vw, 1024px" />
This will enable your analytics team to focus on more human-centric tasks such as planning new ways to improve customer experience, or collaborating with other teams on their analytics-focused tasks, rather than spending time manually compiling required data and developing reports. Stay tuned for the full talk.
<h4>3. Enhancing culture and adoption</h4>
The building blocks of data culture during transformation
Aside from Generative AI, data + AI leaders spoke more widely about their current and upcoming data initiatives. Admiral’s Head of Data Academy, Hannah Davies, discussed what people mean when they say data culture and transformation, and how businesses can adopt tools, and initiatives successfully.
Companies often roll out new tech without changing the behaviours surrounding them. Change programmes must consider all aspects of an organisation, especially the people. People are at the heart of everything, and we use data to improve business outcomes and form insights and actions.
This is a collaborative and holistic process across various departments. Quoting Walt Disney, Hannah presented that “The way to get started is to quit talking and begin doing”. New initiatives require support from across the business, and this doesn’t need to be a data team; it can also be from marketing or recruiters who have ideas about how data could help them. Find out more about analytics adoption in our <a href="https://datatonic.com/events/driving-analytics-adoption-with-admiral-insurance/" target="_blank" rel="noopener">recent webinar with Hannah</a>. 
&nbsp;
Prioritising Responsible AI
With the rapid advancement of new technologies, the need to develop them safely and responsibly must not be forgotten. 
Many companies discussed the goals of <a href="https://datatonic.com/insights/responsible-ai-101-what-you-need-to-know/" target="_blank" rel="noopener">Responsible AI</a>, roles in Responsible AI, and the first steps for businesses to achieve more Responsible AI, especially when it comes to using <a href="https://datatonic.com/insights/generative-ai-large-language-models-responsibly/" target="_blank" rel="noopener">Generative AI</a>. This includes concepts such as keeping a human in the loop and ensuring that data maintains its privacy without becoming public due to training a model. 
To implement Responsible AI successfully, these models must also be fair and unbiased to prevent any issues down the line, and governance principles must be embedded throughout, reiterating the importance of accountability, privacy and compliance. In the future, as AI systems develop and become more widely used, businesses must be even more careful about how they store their data and customer data, as well as being selective over which data they use to train their ML models. 
<h4>Conclusion</h4>
Big Data LDN brings together thousands of the brightest minds to discuss the latest challenges and solutions within data + AI. While the focus this year was undoubtedly the impact of Generative AI, the need for strong data foundations and best practices is crucial to ensuring successful adoption. Furthermore, ensuring your business is enabled with the right skills and knowledge will accelerate your data initiatives, and help you to achieve business value faster.
Datatonic, 5 x Google Cloud Partner of the Year, is the leading cloud data + AI consultancy for the world’s most ambitious businesses, challenging traditions to deliver tangible innovation at the leading edge of Google Cloud. Request a <a href="https://datatonic.com/solutions/generative-ai/art-of-the-possible/" target="_blank" rel="noopener">free Generative AI workshop</a>, or to discuss your current challenges or a project you have in mind, <a href="https://datatonic.com/contact/" target="_blank" rel="noopener">contact us</a>.

Big Data LDN: Our Top Insights

Configuring Looker for Performance and Scalability

Blue-Green deployment is a major upgrade for organisations looking for more stability in their dbt continuous integration/continuous deployment (CI/CD) pipeline. It allows you to test before publishing your data, and it always gives you a readily available backup for when things go awry. This can save you from damaging downtimes and after-the-fact test failures, which will help your stakeholders be more confident in your data pipeline and reports.
Beyond just dbt, leveraging Snowflake’s unique features like <a class="af os" href="https://docs.snowflake.com/en/sql-reference/sql/alter-table.html#parameters" target="_blank" rel="noopener ugc nofollow">swap</a> and<a class="af os" href="https://docs.snowflake.com/en/sql-reference/sql/alter-table.html#parameters" target="_blank" rel="noopener ugc nofollow"> zero-copy cloning</a> makes implementing Blue-Green deployment more effortless and efficient. This article will cover what Blue-Green deployment is and guidelines on how to implement it using dbt and Snowflake.
<h3 id="8250" class="mv mw fo be mx my mz go na nb nc gr nd ne nf ng nh ni nj nk nl nm nn no np nq bj">What is Blue-Green deployment?</h3>
Blue-Green deployment is a methodology that aims to create a stable and reliable production environment by running and rotating two identical databases. One database (prod) has the most up-to-date data and is the target of all downstream applications like Looker, Sigma, or any other production use case. The other database (stage) has the previous, slightly stale state of data, which is not the target of any querying and will stand idle until the next swap.
There are many benefits to implementing this methodology. To name a few, Blue-Green deployment allows you to push new deployments without risking any downtime caused by broken data, is an easy way to roll back if the deployment did not go as expected, and it allows you to complete tests and validations before publishing changes to production.
The worst thing that can happen with Blue-Green is that your data warehouse becomes slightly stale, which is better than leaving off for the weekend with a non-functioning pipeline, and is easy to fix.
Blue-Green deployment is implemented by executing dbt commands in the following order:
<ol class="">
<li id="1a03" class="nr ns fo nt b gm on nv nw gp oo ny nz oa ou oc od oe ov og oh oi ow ok ol om ox oy oz bj" data-selectable-paragraph=""><code class="cw pa pb pc pd b">dbt run</code>: When a dbt job is executed, dbt will target the staging database to perform the run without affecting the production database.</li>
<li id="e7ff" class="nr ns fo nt b gm pe nv nw gp pf ny nz oa pg oc od oe ph og oh oi pi ok ol om ox oy oz bj" data-selectable-paragraph=""><code class="cw pa pb pc pd b">dbt test</code>: After the dbt run is completed in the staging database, dbt will perform all tests on models to make sure the project passes all requirements. If the run fails, the job will exit, and it will not proceed to the next step.</li>
<li id="8051" class="nr ns fo nt b gm pe nv nw gp pf ny nz oa pg oc od oe ph og oh oi pi ok ol om ox oy oz bj" data-selectable-paragraph=""><code class="cw pa pb pc pd b">dbt run-operation swap_database</code>: At this final step, the staging database gets promoted to host production data, and the previous hosting database gets demoted to staging and becomes idle.</li>
</ol>
<figure class="mj mk ml mm mn mo mg mh paragraph-image"></figure>
&nbsp;
<h3 id="39db" class="mv mw fo be mx my mz go na nb nc gr nd ne nf ng nh ni nj nk nl nm nn no np nq bj">Tutorial</h3>
<h4 id="310f" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">1. Create and configure the staging database</h4>
The first step of implementing Blue-Green Deployment is to create a staging database with the same configuration as production. The role used by the tool that runs the dbt project must be granted permission to write and read on both production and staging databases. Cloning all schemas and tables from production to staging is cost-efficient. However, dbt should rebuild all models from scratch.
Below is an example of Snowflake SQL commands to create a new staging database named <code class="cw pa pb pc pd b">stage_db</code>and grant ownership to <code class="cw pa pb pc pd b">dbtcloud_role</code>, the exact role we set up dbt Cloud to run transformations.
<code>-- Create a new database and grant ownership 
create or replace database stage_db; 
grant ownership on database stage_db to role dbtcloud_role;</code>
<code> 
-- Optional step: clone production schemas to the staging database 
create or replace schema 
Stage_db.dbt_analytics 
Clone 
analytics_db.dbt_analytics;</code>
<h4 id="0ca6" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">2. Configure dbt environment to target the staging database</h4>
Next, configure the environment that dbt uses to connect with the data warehouse. This is to target the staging database when dbt is executed. This step can be done in two ways depending on the orchestration tool used to run dbt transformation.
dbt Cloud
dbt Cloud has a native interface to configure dbt environments. Create a new environment and set the target database to use the staging database. This will be the environment to push new deployments to production and perform database swaps.
&nbsp;
<img decoding="async" loading="lazy" class="alignnone wp-image-22502" src="https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog-1024x993.png" alt="" width="700" height="679" srcset="https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog-1024x993.png 1024w, https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog-300x291.png 300w, https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog-768x744.png 768w, https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog.png 1044w" sizes="(max-width: 700px) 100vw, 700px" />
&nbsp;
Other tools
The configuration for other tools will most likely be in the <code class="cw pa pb pc pd b">profiles.yml</code> file. Input the newly created staging database as the default target.
<code class="cw pa pb pc pd b">database: stage_db</code>
<h4 id="f1d4" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">3. Create swap_database operation</h4>
Create a dbt<a class="af os" href="https://docs.getdbt.com/docs/building-a-dbt-project/hooks-operations#operations" target="_blank" rel="noopener ugc nofollow"> operation</a> named <code class="cw pa pb pc pd b">swap_database</code> in the macro folder. After executing all runs and tests, this operation will promote the staging database to production. The <code class="cw pa pb pc pd b">swap with</code> function is a command unique to Snowflake that performs a rename of both databases as a single operation.
<code>{% macro swap_database() %} 
{% set sql='alter database analytics_db swap with stage_db' %} 
{% do run_query(sql) %} 
{{ log("database swapped", info=True) }} 
{% endmacro %}</code>
<h4><code></code><code></code>4. Dealing with Snapshots</h4>
The best practice for Snapshot models is to have a dedicated database for them. The swapping of databases will break the continuity of any single snapshot table leading to incomplete histories. By having snapshot models written to a third database, it will not be affected by the swap.
Below are four steps to configure snapshot models for Blue-Green deployment and migrate existing snapshots to the new database. If your current dbt project does not have any snapshot models, only follow steps A and B.
A. In Snowflake, create a new database and a schema to store the snapshot models.
<code>-- Create a new database and grant ownership 
create or replace database snapshots_db; 
grant ownership on database snapshots_db to role dbtcloud_role;</code>
<code>-- Create a new schema 
create or replace schema snapshots_db.dbt_snapshots;</code>
&nbsp;
B. In your dbt project, configure all snapshots to write to the new target database and schema. This could be done in the <code class="cw pa pb pc pd b">dbt_project.yml</code> file by defining the target schema and database destination under snapshots.
<code>snapshots:</code> 
<code>  datatonic: 
+target_schema: snapshots_db 
+target_database: dbt_snapshots</code>
&nbsp;
C. In Snowflake, migrate all existing snapshot tables from the prod and database to the new snapshot database and schema, if there are any. Snapshot models are stored as transient tables in Snowflake; therefore, the <code class="cw pa pb pc pd b">clone</code> function does not work. Instead, use the classic insert method like the example below to copy records to the new target destination.
<code>-- Create a new snapshot table and insert records from original target 
create or replace transient table snapshots_db.dbt_snapshots.cdc_datatonic__subscribers as 
select * from analytics_db.dbt_analytics.cdc_datatonic__subscribers;</code>
&nbsp;
D. This step is optional, but it is highly recommended to delete the old snapshot tables from the prod and staging database.
 
<code>-- Drop table from the old target</code> 
<code>drop table if exists analytics_db.dbt_analytics.cdc_datatonic__subscribers;</code> 
<code>drop table if exists stage_db.dbt_analytics.cdc_datatonic__subscribers;</code>
<h4 id="976f" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">5. Overwrite your ref macro</h4>
When using Blue-Green deploy, databases are renamed on every successful run. The renaming of databases will cause issues whenever a view is executed in production. That is because views are saved as queries in Snowflake along with many other database services. In dbt, the <code class="cw pa pb pc pd b">ref</code> function includes the complete object address (database.schema.object) by default after compilation. When a view is executed, the query will reference the staging database instead of the production database because it is set as the default target in step 2. To have the <code class="cw pa pb pc pd b">ref</code> function only reference the schema.object, overwrite the <code class="cw pa pb pc pd b">ref</code> macro to exclude the database name. Create a new macro in the <code class="cw pa pb pc pd b">macros</code> folder named <code class="cw pa pb pc pd b">ref</code> and include the following code snippet:
<code>{% macro ref(model_name) %}</code>
<code> 
  {% set rel = builtins.ref(model_name) %}</code>
<code> 
  {% if rel.database == 'snapshots_db' %} 
   {% do return(builtins.ref(model_name)) %} 
  {% else %} 
   {% do return(builtins.ref(model_name).include(database=false)) %} 
  {% endif %}</code>
<code> 
{% endmacro %}</code>
With the ref macro being overwritten, the macro will compile the Snowflake address without the database name like the example below. With this setup, views will no longer cause issues relating to database swapping. The database name does not need to be referenced as long as the schema and table exist in prod and stage databases.
 
<code>-- Before dbt Compile</code> 
<code>select * from {{ ref('dim_subscribers') }}</code>
<code> 
-- After dbt Compile 
select * from dbt_analytics.dim_subscribers</code>
Note: We added a condition to the ref macro where if the database name of the referenced table is <code class="cw pa pb pc pd b">snapshot_db</code>, then include the database name. This condition is needed to tell Snowflake where to look for referenced tables outside the prod and stage database, like snapshot models, because they are not part of the swapping databases.
<code>-- Before dbt Compile</code> 
<code>select * from {{ ref('cdc_datatonic__subscribers') }}</code>
&nbsp;
<code>-- After dbt Compile 
select * from snapshots_db.dbt_snapshots.cdc_datatonic__subscribers</code>
<h4 id="dd8c" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">6. Configure dbt job commands</h4>
The final step of implementing Blue-Green is adding the <code class="cw pa pb pc pd b">swap_database</code> operation as the last command in the dbt Cloud deployment process. Create a dbt job using the same order of the dbt commands below, and don’t forget to set the job using the same environment created in step 2.
Jobs that do partial dbt runs should use an environment that writes directly to the production database. Only apply the <code class="cw pa pb pc pd b">swap_database</code> operation on jobs that perform full runs. Otherwise, end-users will receive incomplete data.
<img decoding="async" loading="lazy" class="alignnone wp-image-22517" src="https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-1024x1017.png" alt="Blue-Green Snowflake" width="700" height="695" srcset="https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-1024x1017.png 1024w, https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-300x298.png 300w, https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-150x150.png 150w, https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-768x762.png 768w, https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1.png 1376w" sizes="(max-width: 700px) 100vw, 700px" />
<h3 id="b009" class="mv mw fo be mx my mz go na nb nc gr nd ne nf ng nh ni nj nk nl nm nn no np nq bj">Conclusion</h3>
By creating two new databases, two new macros, and some tweaks in dbt Cloud settings, there you have Blue-Green deployment, a simple and robust technique to add to your dbt and Snowflake project. It allows stakeholders to have more confidence in your data stack by providing a more stable production environment without risking damaging downtimes.
Datatonic has a wealth of experience in data + AI, including dbt and Snowflake migrations and optimisation. To discuss your current challenges or work on a project together, contact us <a href="https://datatonic.com/contact/">here</a>.

Blue-Green Deployment with dbt and Snowflake

In recent years, the retail industry has faced an ever-evolving landscape marked by several challenges. The rapid growth of e-commerce, shifting consumer preferences, and fast production mean that consumers are demanding increased personalisation and more satisfying customer journeys. In fact, <a href="https://www.growthengineering.co.uk/retail-challenges/" target="_blank" rel="noopener">86%</a> of buyers are willing to pay more for products if they have a great customer experience.
In this era of constant change, understanding and effectively addressing these challenges with advanced technologies has become essential for success, leading to the rise of <a href="https://datatonic.com/case-studies/nordward-machine-learning-recommender-system-recommendations/" target="_blank" rel="noopener">Recommender Systems</a> and <a href="https://datatonic.com/case-studies/product-recommendations-mulberry/" target="_blank" rel="noopener">personalised outreach</a>. While AI + ML has given retailers the opportunity to build stronger connections with customers, Generative AI has the power to go beyond this and solve many common challenges, ranging from providing innovative personalised customer experiences to helping business users get rapid value out of their consumer data. 
<h3>AI + ML in Retail</h3>
Different types of AI + ML have been used within the retail industry for many years, from stock optimisation to reduce supply chain costs, to recommender systems to provide personalised product recommendations for customers shopping online, and more. 
For example, LUSH uses Computer Vision for a feature on its mobile app, <a href="https://www.youtube.com/watch?v=Lpe410kderk" target="_blank" rel="noopener">LUSH Lens</a>, which allows users to scan cosmetic products to learn more, such as how to use them, or a full list of ingredients. While this provides the customer with an innovative experience, it also enables LUSH to remove its in-store packaging, reducing costs and cutting out thousands of unnecessary plastic wrappers and labels. Find out more about how this is possible <a href="https://datatonic.com/case-studies/image-recognition-lush/" target="_blank" rel="noopener">here</a>. 
<img decoding="async" loading="lazy" class="alignnone wp-image-22495" src="https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens-1024x504.jpeg" alt="LUSH Lens Generative AI" width="699" height="344" srcset="https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens-1024x504.jpeg 1024w, https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens-300x148.jpeg 300w, https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens-768x378.jpeg 768w, https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens.jpeg 1270w" sizes="(max-width: 699px) 100vw, 699px" />
&nbsp;
Machine Learning can also be used for supply chain management, and to improve stocking decisions for brick-and-mortar stores. Global sports retail chain, <a href="https://datatonic.com/case-studies/retail-store-segmentation-intersport/">INTERSPORT</a>, uses Machine Learning to segment its stores across all geographies, creating groups of stores with a similar mix of best-selling product categories. This provides INTERSPORT with the ability to develop bespoke sales strategies on the segment level rather than strategies based on geographies or sales volume, increasing sales and simplifying supply chains.
<h4>Personalisation with AI + ML</h4>
Yet, the AI + ML use cases that are most commonly used within retail are those targeted at increasing the level of personalisation for its customers, such as recommender systems and propensity scoring models. Recommender systems can be used to display products based on what a user may be interested in buying, while propensity scoring helps to predict and target users who will buy certain products, informing ads and direct outreach.
Luxury fashion retailer, <a href="https://datatonic.com/case-studies/product-recommendations-mulberry/" target="_blank" rel="noopener">Mulberry</a>, uses propensity scoring to run personalised email campaigns, achieving a 37% increase in click-through rates, and increased ROAS by 110% for its Facebook ads. Mulberry collaborated with Datatonic to leverage its data and develop a model for predicting who will purchase different bag families and types. It is now able to generate propensity scores for customers against several product attributes to improve its advertising efficiency. (Find out more <a href="https://datatonic.com/case-studies/product-recommendations-mulberry/" target="_blank" rel="noopener">here</a>).
With Generative AI providing even more innovative and personalised customer experiences, retail businesses are eager to explore how this can work for them.
<h4>Personalisation with Generative AI</h4>
Generative AI is helping companies from all industries provide their customers with enhanced, more personalised experiences. This is particularly relevant for the retail industry, where <a href="https://ww.fashionnetwork.com/news/Experiential-retail-economy-to-reach-tipping-point-in-2025,1175190.html#michael-kors" target="_blank" rel="noopener">nearly 60%</a> of consumers expect more than half of retail space to be devoted to experience rather than product by 2025. Increasing customer engagement, retention and lifetime value has never been more crucial. 
&nbsp;
<ol>
<li> Tailored advertising</li>
</ol>
In traditional ML, <a href="https://datatonic.com/insights/propensity-modelling-tensorflow-cloud-ai/" target="_blank" rel="noopener">propensity modelling</a> is often used to better target customers in email ads. The propensity model helps the business understand who has the highest likelihood to buy different products. And now with GenAI, you can take it to that next level by rapidly creating tailored content.
With Generative AI, online brands can modify advertising imagery to better fit their website or magazine, adapting to different target audiences and mediums. This includes imagery or rewriting copy to appeal to the chosen audience, increasing revenue and enhancing customer experience through better personalisation. 
This also greatly reduces the strain on designers and copywriters, enabling them to focus on improving their overall branding, thought leadership content pieces, or more research about what resonates with customers. 
&nbsp;
<ol start="2">
<li> Help-me-buy chatbots</li>
</ol>
With Generative AI, customers can interact with website chatbots that consistently use brand-aligned language and experience reliable expertise from a finely-tuned chatbot, tuned to a company’s products and services, to guide them to what they are looking for.
This creates a more enjoyable customer experience and more sales leading to increased revenue. If customers can easily find their desired products, they are <a href="https://www.sciencedirect.com/science/article/abs/pii/S0969698919307039" target="_blank" rel="noopener">more likely to</a> recommend the site to a friend or return themselves, ultimately boosting customer acquisition and lifetime value.
&nbsp;
<ol start="3">
<li> Analytics summarisers</li>
</ol>
Analytics summarisers help business users to summarise sales outcomes to quickly gather information to inform decision-making on what to stock, reducing the time required to compile and analyse relevant data. Large Language Models can be trained on company data to unlock this with Generative AI. This allows you to have conversations with your data to extract rapid insights on sales, customers, and revenue. 
Furthermore, these summarisers can then be used to increase personalisation. For example, asking your data “Which of our products is selling the best among 18-25-year-olds in the UK?” can help marketing teams develop personalised advertising campaigns and improve outreach. It also reduces the strain on internal analytics and finance teams and accelerates decision-making within marketing and across the business. 
&nbsp;
<h4>Conclusion</h4>
Generative AI provides businesses in the retail industry with new opportunities to deliver innovative personalised experiences to their online customers. Getting started simply requires finding the right use case for your business. From there, <a href="https://datatonic.com/insights/google-cloud-next-top-10-announcement-highlights/" target="_blank" rel="noopener">Google’s latest updates</a> make it easy to generate business impact in your business with Generative AI.
Need help finding where to start? Request a free two-hour <a href="https://datatonic.com/solutions/generative-ai/art-of-the-possible/" target="_blank" rel="noopener">Art of the Possible workshop</a> to explore how GenAI can transform customer experiences, unlock new revenue streams, and increase efficiency.

Retail

Telecommunications

Gaming

Media

Financial Services

Events

Whitepapers

Insights

Optimise Your Google Cloud Costs Using Looker

Get a Detailed View with Datatonic + Looker Dashboards

Get a sneak peek

Related

Building Data Fluency Throughout Your Business

MLOps in Action: Operationalize Your ML Workflow Using Pipeline Templates

Enterprise MDS Deployment at Scale: dbt + DevOps