Let's talk

LinkedIn

Twitter

Medium

Youtube

<a href="https://bigdataldn.com/" target="_blank" rel="noopener">Big Data LDN</a> is the UK’s leading free-to-attend data, analytics + AI conference and exhibition, hosting leading data, analytics &amp; AI experts, ready to arm you with the tools to deliver your most effective data-driven strategy.
Join us on September 20th and 21st, where we’ll be presenting three talks to help you unlock new value from your data:
&nbsp;
20th Sept 11:30am: <a href="https://calendar.google.com/calendar/u/0/r/eventedit?dates=20230920T103000Z/20230920T110000Z&amp;details=Speaker:+Felix+Schaumann,+Machine+Learning+Engineering+Lead+at+Datatonic%0A%0AIn+this+session,+you%E2%80%99ll+learn+how+to+identify+GenAI+opportunities+within+your+business+and+gain+a+comprehensive+understanding+of+the+core+technical+principles+underpinning+Generative+AI.+Learn+how+Snowflake%E2%80%99s+capabilities+facilitate+the+successful+implementation+of+Generative+AI,+and+acquire+pragmatic+insights+into+constructing+robust,+future-ready+Generative+AI+solutions+on+Snowflake.&amp;location=Stand+514,+Olympia+London,+Hammersmith+Rd,+London+W14+8UX,+UK&amp;text=Unlocking+Generative+AI+Use+Cases+with+Snowpark" target="_blank" rel="noopener">Unlocking Generative AI Use Cases with Snowpark</a> (Stand 514)
Speaker: Felix Schaumann, Machine Learning Engineering Lead at Datatonic
In this session, you’ll learn how to identify GenAI opportunities within your business and gain a comprehensive understanding of the core technical principles underpinning Generative AI. Learn how Snowflake’s capabilities facilitate the successful implementation of Generative AI, and acquire pragmatic insights into constructing robust, future-ready Generative AI solutions on Snowflake.
<a href="https://calendar.google.com/calendar/u/0/r/eventedit?dates=20230920T103000Z/20230920T110000Z&amp;details=Speaker:+Felix+Schaumann,+Machine+Learning+Engineering+Lead+at+Datatonic%0A%0AIn+this+session,+you%E2%80%99ll+learn+how+to+identify+GenAI+opportunities+within+your+business+and+gain+a+comprehensive+understanding+of+the+core+technical+principles+underpinning+Generative+AI.+Learn+how+Snowflake%E2%80%99s+capabilities+facilitate+the+successful+implementation+of+Generative+AI,+and+acquire+pragmatic+insights+into+constructing+robust,+future-ready+Generative+AI+solutions+on+Snowflake.&amp;location=Stand+514,+Olympia+London,+Hammersmith+Rd,+London+W14+8UX,+UK&amp;text=Unlocking+Generative+AI+Use+Cases+with+Snowpark" target="_blank" rel="noopener">Add to your calendar.</a>
&nbsp;
20th Sept 2:00pm: <a href="https://calendar.google.com/calendar/u/0/r/eventedit?dates=20230920T130000Z/20230920T133000Z&amp;details=Speaker:+Christelle+Xu,+Senior+Global+Product+Manager%0A%0AHow+can+businesses+use+Generative+AI+to+make+decisions+better+and+faster+than+ever+before?+In+this+session,+Datatonic%E2%80%99s+GenAI+experts+will+give+you+the+roadmap+to+successful+GenAI+adoption+-+from+amplifying+analytics+to+enhancing+customer+experiences.+You%E2%80%99ll+also+see+a+demo+for+a+fully+secure,+enterprise-grade+Large+Language+Model+that+helps+organisations+accelerate+Looker+adoption,+better+enable+their+users+and+deliver+a+much+faster+ROI+on+their+investment.&amp;location=Stand+406,+Olympia+London,+Hammersmith+Rd,+London+W14+8UX,+UK&amp;text=Amplifying+Insights:+Your+Roadmap+to+GenAI" target="_blank" rel="noopener">Amplifying Insights: Your roadmap to Generative AI</a> (Stand 406)
Speaker: Christelle Xu, Senior Global Product Manager
How can businesses use Generative AI to make decisions better and faster than ever before? In this session, Datatonic’s GenAI experts will give you the roadmap to successful GenAI adoption &#8211; from amplifying analytics to enhancing customer experiences. You’ll also see a demo for a fully secure, enterprise-grade Large Language Model that helps organisations accelerate Looker adoption, better enable their users and deliver a much faster ROI on their investment.
<a href="https://calendar.google.com/calendar/u/0/r/eventedit?dates=20230920T130000Z/20230920T133000Z&amp;details=Speaker:+Christelle+Xu,+Senior+Global+Product+Manager%0A%0AHow+can+businesses+use+Generative+AI+to+make+decisions+better+and+faster+than+ever+before?+In+this+session,+Datatonic%E2%80%99s+GenAI+experts+will+give+you+the+roadmap+to+successful+GenAI+adoption+-+from+amplifying+analytics+to+enhancing+customer+experiences.+You%E2%80%99ll+also+see+a+demo+for+a+fully+secure,+enterprise-grade+Large+Language+Model+that+helps+organisations+accelerate+Looker+adoption,+better+enable+their+users+and+deliver+a+much+faster+ROI+on+their+investment.&amp;location=Stand+406,+Olympia+London,+Hammersmith+Rd,+London+W14+8UX,+UK&amp;text=Amplifying+Insights:+Your+Roadmap+to+GenAI" target="_blank" rel="noopener">Add to your calendar.</a>
&nbsp;
21st Sept 10:30am: <a href="https://calendar.google.com/calendar/u/0/r/eventedit?dates=20230921T093000Z/20230921T100000Z&amp;details=Speakers:+Ash+Sultan,+Lead+Data+Architect+at+Datatonic+and+Scott+Hobbs,+Senior+Analytics+Engineer+at+Datatonic%0A%0ASnowflake%E2%80%99s+Data+Marketplace+offers+a+powerful+platform+for+data+monetisation,+enabling+organisations+to+derive+significant+benefits+from+sharing+and+selling+their+data+assets.+Join+us+in+this+discussion+featuring+a+demo+on+Snowflake,+to+see+how+you+can+harness+your+data+assets+for+enhanced+business+growth+and+success+by+opening+lucrative+revenue+streams,+expanding+market+reach,+and+fostering+collaboration+and+innovation.&amp;location=Stand+514,+Olympia+London,+Hammersmith+Rd,+London+W14+8UX,+UK&amp;text=Monetise+Your+Data+Securely+with+Snowflake" target="_blank" rel="noopener">Monetise Your Data Securely with Snowflake</a> (Stand 514)
Speakers: Ash Sultan, Lead Data Architect at Datatonic and Scott Hobbs, Senior Analytics Engineer at Datatonic
Snowflake’s Data Marketplace offers a powerful platform for data monetisation, enabling organisations to derive significant benefits from sharing and selling their data assets. Join us in this discussion featuring a demo on Snowflake, to see how you can harness your data assets for enhanced business growth and success by opening lucrative revenue streams, expanding market reach, and fostering collaboration and innovation.
<a href="https://calendar.google.com/calendar/u/0/r/eventedit?dates=20230921T093000Z/20230921T100000Z&amp;details=Speakers:+Ash+Sultan,+Lead+Data+Architect+at+Datatonic+and+Scott+Hobbs,+Senior+Analytics+Engineer+at+Datatonic%0A%0ASnowflake%E2%80%99s+Data+Marketplace+offers+a+powerful+platform+for+data+monetisation,+enabling+organisations+to+derive+significant+benefits+from+sharing+and+selling+their+data+assets.+Join+us+in+this+discussion+featuring+a+demo+on+Snowflake,+to+see+how+you+can+harness+your+data+assets+for+enhanced+business+growth+and+success+by+opening+lucrative+revenue+streams,+expanding+market+reach,+and+fostering+collaboration+and+innovation.&amp;location=Stand+514,+Olympia+London,+Hammersmith+Rd,+London+W14+8UX,+UK&amp;text=Monetise+Your+Data+Securely+with+Snowflake" target="_blank" rel="noopener">Add to your calendar.</a>
&nbsp;
We hope to see you soon!

<h1>Download 
the guide</h1>
A guide by data leaders, for data leaders, looking to turn Data Mesh principles into practice on Google Cloud.

Datatonic at Big Data LDN

Blue-Green deployment is a major upgrade for organisations looking for more stability in their dbt continuous integration/continuous deployment (CI/CD) pipeline. It allows you to test before publishing your data, and it always gives you a readily available backup for when things go awry. This can save you from damaging downtimes and after-the-fact test failures, which will help your stakeholders be more confident in your data pipeline and reports.
Beyond just dbt, leveraging Snowflake’s unique features like <a class="af os" href="https://docs.snowflake.com/en/sql-reference/sql/alter-table.html#parameters" target="_blank" rel="noopener ugc nofollow">swap</a> and<a class="af os" href="https://docs.snowflake.com/en/sql-reference/sql/alter-table.html#parameters" target="_blank" rel="noopener ugc nofollow"> zero-copy cloning</a> makes implementing Blue-Green deployment more effortless and efficient. This article will cover what Blue-Green deployment is and guidelines on how to implement it using dbt and Snowflake.
<h3 id="8250" class="mv mw fo be mx my mz go na nb nc gr nd ne nf ng nh ni nj nk nl nm nn no np nq bj">What is Blue-Green deployment?</h3>
Blue-Green deployment is a methodology that aims to create a stable and reliable production environment by running and rotating two identical databases. One database (prod) has the most up-to-date data and is the target of all downstream applications like Looker, Sigma, or any other production use case. The other database (stage) has the previous, slightly stale state of data, which is not the target of any querying and will stand idle until the next swap.
There are many benefits to implementing this methodology. To name a few, Blue-Green deployment allows you to push new deployments without risking any downtime caused by broken data, is an easy way to roll back if the deployment did not go as expected, and it allows you to complete tests and validations before publishing changes to production.
The worst thing that can happen with Blue-Green is that your data warehouse becomes slightly stale, which is better than leaving off for the weekend with a non-functioning pipeline, and is easy to fix.
Blue-Green deployment is implemented by executing dbt commands in the following order:
<ol class="">
<li id="1a03" class="nr ns fo nt b gm on nv nw gp oo ny nz oa ou oc od oe ov og oh oi ow ok ol om ox oy oz bj" data-selectable-paragraph=""><code class="cw pa pb pc pd b">dbt run</code>: When a dbt job is executed, dbt will target the staging database to perform the run without affecting the production database.</li>
<li id="e7ff" class="nr ns fo nt b gm pe nv nw gp pf ny nz oa pg oc od oe ph og oh oi pi ok ol om ox oy oz bj" data-selectable-paragraph=""><code class="cw pa pb pc pd b">dbt test</code>: After the dbt run is completed in the staging database, dbt will perform all tests on models to make sure the project passes all requirements. If the run fails, the job will exit, and it will not proceed to the next step.</li>
<li id="8051" class="nr ns fo nt b gm pe nv nw gp pf ny nz oa pg oc od oe ph og oh oi pi ok ol om ox oy oz bj" data-selectable-paragraph=""><code class="cw pa pb pc pd b">dbt run-operation swap_database</code>: At this final step, the staging database gets promoted to host production data, and the previous hosting database gets demoted to staging and becomes idle.</li>
</ol>
<figure class="mj mk ml mm mn mo mg mh paragraph-image"></figure>
&nbsp;
<h3 id="39db" class="mv mw fo be mx my mz go na nb nc gr nd ne nf ng nh ni nj nk nl nm nn no np nq bj">Tutorial</h3>
<h4 id="310f" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">1. Create and configure the staging database</h4>
The first step of implementing Blue-Green Deployment is to create a staging database with the same configuration as production. The role used by the tool that runs the dbt project must be granted permission to write and read on both production and staging databases. Cloning all schemas and tables from production to staging is cost-efficient. However, dbt should rebuild all models from scratch.
Below is an example of Snowflake SQL commands to create a new staging database named <code class="cw pa pb pc pd b">stage_db</code>and grant ownership to <code class="cw pa pb pc pd b">dbtcloud_role</code>, the exact role we set up dbt Cloud to run transformations.
<code>-- Create a new database and grant ownership 
create or replace database stage_db; 
grant ownership on database stage_db to role dbtcloud_role;</code>
<code> 
-- Optional step: clone production schemas to the staging database 
create or replace schema 
Stage_db.dbt_analytics 
Clone 
analytics_db.dbt_analytics;</code>
<h4 id="0ca6" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">2. Configure dbt environment to target the staging database</h4>
Next, configure the environment that dbt uses to connect with the data warehouse. This is to target the staging database when dbt is executed. This step can be done in two ways depending on the orchestration tool used to run dbt transformation.
dbt Cloud
dbt Cloud has a native interface to configure dbt environments. Create a new environment and set the target database to use the staging database. This will be the environment to push new deployments to production and perform database swaps.
&nbsp;
<img decoding="async" loading="lazy" class="alignnone wp-image-22502" src="https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog-1024x993.png" alt="" width="700" height="679" srcset="https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog-1024x993.png 1024w, https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog-300x291.png 300w, https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog-768x744.png 768w, https://cms.datatonic.com/app/uploads/2023/09/dbt-Snowflake-Blog.png 1044w" sizes="(max-width: 700px) 100vw, 700px" />
&nbsp;
Other tools
The configuration for other tools will most likely be in the <code class="cw pa pb pc pd b">profiles.yml</code> file. Input the newly created staging database as the default target.
<code class="cw pa pb pc pd b">database: stage_db</code>
<h4 id="f1d4" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">3. Create swap_database operation</h4>
Create a dbt<a class="af os" href="https://docs.getdbt.com/docs/building-a-dbt-project/hooks-operations#operations" target="_blank" rel="noopener ugc nofollow"> operation</a> named <code class="cw pa pb pc pd b">swap_database</code> in the macro folder. After executing all runs and tests, this operation will promote the staging database to production. The <code class="cw pa pb pc pd b">swap with</code> function is a command unique to Snowflake that performs a rename of both databases as a single operation.
<code>{% macro swap_database() %} 
{% set sql='alter database analytics_db swap with stage_db' %} 
{% do run_query(sql) %} 
{{ log("database swapped", info=True) }} 
{% endmacro %}</code>
<h4><code></code><code></code>4. Dealing with Snapshots</h4>
The best practice for Snapshot models is to have a dedicated database for them. The swapping of databases will break the continuity of any single snapshot table leading to incomplete histories. By having snapshot models written to a third database, it will not be affected by the swap.
Below are four steps to configure snapshot models for Blue-Green deployment and migrate existing snapshots to the new database. If your current dbt project does not have any snapshot models, only follow steps A and B.
A. In Snowflake, create a new database and a schema to store the snapshot models.
<code>-- Create a new database and grant ownership 
create or replace database snapshots_db; 
grant ownership on database snapshots_db to role dbtcloud_role;</code>
<code>-- Create a new schema 
create or replace schema snapshots_db.dbt_snapshots;</code>
&nbsp;
B. In your dbt project, configure all snapshots to write to the new target database and schema. This could be done in the <code class="cw pa pb pc pd b">dbt_project.yml</code> file by defining the target schema and database destination under snapshots.
<code>snapshots:</code> 
<code>  datatonic: 
+target_schema: snapshots_db 
+target_database: dbt_snapshots</code>
&nbsp;
C. In Snowflake, migrate all existing snapshot tables from the prod and database to the new snapshot database and schema, if there are any. Snapshot models are stored as transient tables in Snowflake; therefore, the <code class="cw pa pb pc pd b">clone</code> function does not work. Instead, use the classic insert method like the example below to copy records to the new target destination.
<code>-- Create a new snapshot table and insert records from original target 
create or replace transient table snapshots_db.dbt_snapshots.cdc_datatonic__subscribers as 
select * from analytics_db.dbt_analytics.cdc_datatonic__subscribers;</code>
&nbsp;
D. This step is optional, but it is highly recommended to delete the old snapshot tables from the prod and staging database.
 
<code>-- Drop table from the old target</code> 
<code>drop table if exists analytics_db.dbt_analytics.cdc_datatonic__subscribers;</code> 
<code>drop table if exists stage_db.dbt_analytics.cdc_datatonic__subscribers;</code>
<h4 id="976f" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">5. Overwrite your ref macro</h4>
When using Blue-Green deploy, databases are renamed on every successful run. The renaming of databases will cause issues whenever a view is executed in production. That is because views are saved as queries in Snowflake along with many other database services. In dbt, the <code class="cw pa pb pc pd b">ref</code> function includes the complete object address (database.schema.object) by default after compilation. When a view is executed, the query will reference the staging database instead of the production database because it is set as the default target in step 2. To have the <code class="cw pa pb pc pd b">ref</code> function only reference the schema.object, overwrite the <code class="cw pa pb pc pd b">ref</code> macro to exclude the database name. Create a new macro in the <code class="cw pa pb pc pd b">macros</code> folder named <code class="cw pa pb pc pd b">ref</code> and include the following code snippet:
<code>{% macro ref(model_name) %}</code>
<code> 
  {% set rel = builtins.ref(model_name) %}</code>
<code> 
  {% if rel.database == 'snapshots_db' %} 
   {% do return(builtins.ref(model_name)) %} 
  {% else %} 
   {% do return(builtins.ref(model_name).include(database=false)) %} 
  {% endif %}</code>
<code> 
{% endmacro %}</code>
With the ref macro being overwritten, the macro will compile the Snowflake address without the database name like the example below. With this setup, views will no longer cause issues relating to database swapping. The database name does not need to be referenced as long as the schema and table exist in prod and stage databases.
 
<code>-- Before dbt Compile</code> 
<code>select * from {{ ref('dim_subscribers') }}</code>
<code> 
-- After dbt Compile 
select * from dbt_analytics.dim_subscribers</code>
Note: We added a condition to the ref macro where if the database name of the referenced table is <code class="cw pa pb pc pd b">snapshot_db</code>, then include the database name. This condition is needed to tell Snowflake where to look for referenced tables outside the prod and stage database, like snapshot models, because they are not part of the swapping databases.
<code>-- Before dbt Compile</code> 
<code>select * from {{ ref('cdc_datatonic__subscribers') }}</code>
&nbsp;
<code>-- After dbt Compile 
select * from snapshots_db.dbt_snapshots.cdc_datatonic__subscribers</code>
<h4 id="dd8c" class="pp mw fo be mx pq pr ps na pt pu pv nd oa pw px py oe pz qa qb oi qc qd qe qf bj">6. Configure dbt job commands</h4>
The final step of implementing Blue-Green is adding the <code class="cw pa pb pc pd b">swap_database</code> operation as the last command in the dbt Cloud deployment process. Create a dbt job using the same order of the dbt commands below, and don’t forget to set the job using the same environment created in step 2.
Jobs that do partial dbt runs should use an environment that writes directly to the production database. Only apply the <code class="cw pa pb pc pd b">swap_database</code> operation on jobs that perform full runs. Otherwise, end-users will receive incomplete data.
<img decoding="async" loading="lazy" class="alignnone wp-image-22517" src="https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-1024x1017.png" alt="Blue-Green Snowflake" width="700" height="695" srcset="https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-1024x1017.png 1024w, https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-300x298.png 300w, https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-150x150.png 150w, https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1-768x762.png 768w, https://cms.datatonic.com/app/uploads/2023/09/Blue-Green-Snowflake-1.png 1376w" sizes="(max-width: 700px) 100vw, 700px" />
<h3 id="b009" class="mv mw fo be mx my mz go na nb nc gr nd ne nf ng nh ni nj nk nl nm nn no np nq bj">Conclusion</h3>
By creating two new databases, two new macros, and some tweaks in dbt Cloud settings, there you have Blue-Green deployment, a simple and robust technique to add to your dbt and Snowflake project. It allows stakeholders to have more confidence in your data stack by providing a more stable production environment without risking damaging downtimes.
Datatonic has a wealth of experience in data + AI, including dbt and Snowflake migrations and optimisation. To discuss your current challenges or work on a project together, contact us <a href="https://datatonic.com/contact/">here</a>.

Blue-Green Deployment with dbt and Snowflake

In recent years, the retail industry has faced an ever-evolving landscape marked by several challenges. The rapid growth of e-commerce, shifting consumer preferences, and fast production mean that consumers are demanding increased personalisation and more satisfying customer journeys. In fact, <a href="https://www.growthengineering.co.uk/retail-challenges/" target="_blank" rel="noopener">86%</a> of buyers are willing to pay more for products if they have a great customer experience.
In this era of constant change, understanding and effectively addressing these challenges with advanced technologies has become essential for success, leading to the rise of <a href="https://datatonic.com/case-studies/nordward-machine-learning-recommender-system-recommendations/" target="_blank" rel="noopener">Recommender Systems</a> and <a href="https://datatonic.com/case-studies/product-recommendations-mulberry/" target="_blank" rel="noopener">personalised outreach</a>. While AI + ML has given retailers the opportunity to build stronger connections with customers, Generative AI has the power to go beyond this and solve many common challenges, ranging from providing innovative personalised customer experiences to helping business users get rapid value out of their consumer data. 
<h3>AI + ML in Retail</h3>
Different types of AI + ML have been used within the retail industry for many years, from stock optimisation to reduce supply chain costs, to recommender systems to provide personalised product recommendations for customers shopping online, and more. 
For example, LUSH uses Computer Vision for a feature on its mobile app, <a href="https://www.youtube.com/watch?v=Lpe410kderk" target="_blank" rel="noopener">LUSH Lens</a>, which allows users to scan cosmetic products to learn more, such as how to use them, or a full list of ingredients. While this provides the customer with an innovative experience, it also enables LUSH to remove its in-store packaging, reducing costs and cutting out thousands of unnecessary plastic wrappers and labels. Find out more about how this is possible <a href="https://datatonic.com/case-studies/image-recognition-lush/" target="_blank" rel="noopener">here</a>. 
<img decoding="async" loading="lazy" class="alignnone wp-image-22495" src="https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens-1024x504.jpeg" alt="LUSH Lens Generative AI" width="699" height="344" srcset="https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens-1024x504.jpeg 1024w, https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens-300x148.jpeg 300w, https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens-768x378.jpeg 768w, https://cms.datatonic.com/app/uploads/2023/09/Lush-Lens.jpeg 1270w" sizes="(max-width: 699px) 100vw, 699px" />
&nbsp;
Machine Learning can also be used for supply chain management, and to improve stocking decisions for brick-and-mortar stores. Global sports retail chain, <a href="https://datatonic.com/case-studies/retail-store-segmentation-intersport/">INTERSPORT</a>, uses Machine Learning to segment its stores across all geographies, creating groups of stores with a similar mix of best-selling product categories. This provides INTERSPORT with the ability to develop bespoke sales strategies on the segment level rather than strategies based on geographies or sales volume, increasing sales and simplifying supply chains.
<h4>Personalisation with AI + ML</h4>
Yet, the AI + ML use cases that are most commonly used within retail are those targeted at increasing the level of personalisation for its customers, such as recommender systems and propensity scoring models. Recommender systems can be used to display products based on what a user may be interested in buying, while propensity scoring helps to predict and target users who will buy certain products, informing ads and direct outreach.
Luxury fashion retailer, <a href="https://datatonic.com/case-studies/product-recommendations-mulberry/" target="_blank" rel="noopener">Mulberry</a>, uses propensity scoring to run personalised email campaigns, achieving a 37% increase in click-through rates, and increased ROAS by 110% for its Facebook ads. Mulberry collaborated with Datatonic to leverage its data and develop a model for predicting who will purchase different bag families and types. It is now able to generate propensity scores for customers against several product attributes to improve its advertising efficiency. (Find out more <a href="https://datatonic.com/case-studies/product-recommendations-mulberry/" target="_blank" rel="noopener">here</a>).
With Generative AI providing even more innovative and personalised customer experiences, retail businesses are eager to explore how this can work for them.
<h4>Personalisation with Generative AI</h4>
Generative AI is helping companies from all industries provide their customers with enhanced, more personalised experiences. This is particularly relevant for the retail industry, where <a href="https://ww.fashionnetwork.com/news/Experiential-retail-economy-to-reach-tipping-point-in-2025,1175190.html#michael-kors" target="_blank" rel="noopener">nearly 60%</a> of consumers expect more than half of retail space to be devoted to experience rather than product by 2025. Increasing customer engagement, retention and lifetime value has never been more crucial. 
&nbsp;
<ol>
<li> Tailored advertising</li>
</ol>
In traditional ML, <a href="https://datatonic.com/insights/propensity-modelling-tensorflow-cloud-ai/" target="_blank" rel="noopener">propensity modelling</a> is often used to better target customers in email ads. The propensity model helps the business understand who has the highest likelihood to buy different products. And now with GenAI, you can take it to that next level by rapidly creating tailored content.
With Generative AI, online brands can modify advertising imagery to better fit their website or magazine, adapting to different target audiences and mediums. This includes imagery or rewriting copy to appeal to the chosen audience, increasing revenue and enhancing customer experience through better personalisation. 
This also greatly reduces the strain on designers and copywriters, enabling them to focus on improving their overall branding, thought leadership content pieces, or more research about what resonates with customers. 
&nbsp;
<ol start="2">
<li> Help-me-buy chatbots</li>
</ol>
With Generative AI, customers can interact with website chatbots that consistently use brand-aligned language and experience reliable expertise from a finely-tuned chatbot, tuned to a company’s products and services, to guide them to what they are looking for.
This creates a more enjoyable customer experience and more sales leading to increased revenue. If customers can easily find their desired products, they are <a href="https://www.sciencedirect.com/science/article/abs/pii/S0969698919307039" target="_blank" rel="noopener">more likely to</a> recommend the site to a friend or return themselves, ultimately boosting customer acquisition and lifetime value.
&nbsp;
<ol start="3">
<li> Analytics summarisers</li>
</ol>
Analytics summarisers help business users to summarise sales outcomes to quickly gather information to inform decision-making on what to stock, reducing the time required to compile and analyse relevant data. Large Language Models can be trained on company data to unlock this with Generative AI. This allows you to have conversations with your data to extract rapid insights on sales, customers, and revenue. 
Furthermore, these summarisers can then be used to increase personalisation. For example, asking your data “Which of our products is selling the best among 18-25-year-olds in the UK?” can help marketing teams develop personalised advertising campaigns and improve outreach. It also reduces the strain on internal analytics and finance teams and accelerates decision-making within marketing and across the business. 
&nbsp;
<h4>Conclusion</h4>
Generative AI provides businesses in the retail industry with new opportunities to deliver innovative personalised experiences to their online customers. Getting started simply requires finding the right use case for your business. From there, <a href="https://datatonic.com/insights/google-cloud-next-top-10-announcement-highlights/" target="_blank" rel="noopener">Google’s latest updates</a> make it easy to generate business impact in your business with Generative AI.
Need help finding where to start? Request a free two-hour <a href="https://datatonic.com/solutions/generative-ai/art-of-the-possible/" target="_blank" rel="noopener">Art of the Possible workshop</a> to explore how GenAI can transform customer experiences, unlock new revenue streams, and increase efficiency.

Generative AI Personalisation for Retail

Google Cloud Next: Our Top 10 Announcement Highlights

London, 29th August 2023 — Datatonic, a global leader in Cloud Data + AI consulting announced today that it has been named 2023 Google Cloud Industry Solution Services Partner of the Year for Telecommunications.
Together with Google Cloud, Datatonic has helped Telcos solve their most pressing challenges with powerful data analytics and AI solutions &#8211; from strengthening their customer experience and uncovering new sources of revenue, to improving network efficiency and connectivity.
“Google Cloud’s partner awards recognize the significant impact and customer success that our partners have driven over the past year,&#8221; said Kevin Ichhpurani, Corporate Vice President, Global Ecosystem and Channels at Google Cloud. &#8220;We&#8217;re delighted to recognize Datatonic  as a 2023 Google Cloud Partner Award winner, and look forward to a continued strong partnership in support of our mutual customers.&#8221;
Throughout the year, Datatonic has worked with industry-leading clients, such as <a href="https://datatonic.com/case-studies/ai-booster-vodafone-supercharging-ai-ml-at-scale/" target="_blank" rel="noopener">Vodafone</a>, <a href="https://datatonic.com/case-studies/bt-ai-accelerator-artificial-intelligence/" target="_blank" rel="noopener">BT</a>, and <a href="https://datatonic.com/case-studies/inmarsat-mlops-satellite-anomalies/" target="_blank" rel="noopener">Inmarsat</a>, to remain competitive in the rapidly evolving landscape of telecommunications, scale their operations, and bring better services to customers through data-driven decisions and continuous innovation.
Some examples of industry-leading impact include: Datatonic’s collaboration with BT Digital on the telco giant’s AI Accelerator Platform, which enabled the business to improve production time of AI models from <a href="https://inform.tmforum.org/features-and-opinion/bt-groups-digital-unit-accelerates-ai-use-cases-and-navigates-genai/" target="_blank" rel="noopener">6 months to 14 days</a>, and Datatonic’s work with leading satellite service provider, Inmarsat, which allowed the team to detect real-time satellite anomalies, and reduce malfunction detection time.
“We’re honoured to be recognised for our global impact in the telecommunications sector, which will be our fifth accolade from Google Cloud. This award celebrates Datatonic’s work with industry leaders, such as BT, Vodafone, Sky and Inmarsat, where we’ve driven substantial and sustained business impact over the years, through innovative Data + AI solutions.” &#8211; Louis Decuypere, CEO, Datatonic
As well as delivering projects with tangible business impact to help scale clients’ businesses, Datatonic has provided strategy and advisory to business leaders on the best use of data and AI to maximise business value as well as upskilling client teams in collaboration with <a href="https://datatonic.academy/" target="_blank" rel="noopener">Datatonic Academy</a>.
About Datatonic
Datatonic is the leading cloud data + AI consultancy for the world’s most ambitious businesses, challenging traditions to deliver tangible innovation at the leading edge of Google Cloud. Datatonic works with industry-leading organisations to unlock the value of data and solve their most complex business challenges fast; delivering more accurate predictions, automating decisions, optimising processes and improving customer experiences, and helping clients do the same by upskilling tech teams to stay ahead of what’s next.
For more information, visit <a href="https://datatonic.com/" target="_blank" rel="noopener">https://datatonic.com/</a>.
Register for our <a href="https://datatonic.com/events/genai-telco-media/?utm_source=website&amp;utm_medium=organic&amp;utm_campaign=Event_GenAI_Telco_Webinar_20230928&amp;utm_term=Announcement" target="_blank" rel="noopener">upcoming webinar</a> to learn how Telcos can benefit from Generative AI.
For press enquiries, contact <a href="mailto:krisztina.gyure@datatonic.com" target="_blank" rel="noopener">krisztina.gyure@datatonic.com</a>.

Datatonic is recognised for the fifth time as a Google Cloud Partner of the Year; being awarded this year for the company&#8217;s global impact in the Telecommunications industry.

Datatonic Wins Google Cloud Industry Solution Services Partner of the Year Award for Telecommunications 

Generative AI, like in many industries, has the potential to help telecommunications providers stay ahead of the curve in an increasingly competitive market. By using Generative AI, telcos can improve network efficiency, optimise customer experiences, and create new revenue streams. With <a href="https://www.ey.com/en_gl/telecommunications/top-ten-risks-for-telecommunications-in-2023">60% of households</a> concerned about monthly price increases and <a href="https://www.ey.com/en_gl/telecommunications/top-ten-risks-for-telecommunications-in-2023">44%</a> believing their broadband provider could do more to direct them to the best deal, there has never been more pressure to provide exceptional customer service at low prices. 
In this blog post, we&#8217;ll explore how providers can tackle pressing challenges using Generative AI, and look at the approach needed to successfully adopt and drive value from this rapidly growing technology.
<h4>Current Challenges in the Telecommunications Industry</h4>
The telecommunications industry faces various challenges, ranging from common data + AI deployment issues to others that are more industry-specific. These include:
<ul>
<li style="font-weight: 400;" aria-level="1">Low data impact &#8211; dealing with high data volumes that are unfriendly and unusable to most, or require significant time and effort to process.</li>
<li style="font-weight: 400;" aria-level="1">Network efficiency and automation &#8211; growing consumption requires new capabilities to <a href="https://datatonic.com/insights/reducing-emissions-operating-costs-5g/">meet demand at a low cost</a>.</li>
<li style="font-weight: 400;" aria-level="1">Customer-centricity &#8211; the need to provide value to customers, with a seamless experience at every stage of their journey.</li>
<li style="font-weight: 400;" aria-level="1">Monetisation and revenue challenges &#8211; pressure to create innovative revenue streams after reduced Average Revenue Per User (ARPU) + monetise data through partnerships.</li>
</ul>
&nbsp;
<h4>AI + ML is helping Telcos optimise operations and satisfy customers…</h4>
Traditional AI + ML has already been applied in the telco industry in various ways. Businesses, such as Inmarsat, are using <a href="https://datatonic.com/case-studies/inmarsat-mlops-satellite-anomalies/">ML for anomaly detection</a> to address any anomalies before they cause an issue and impact customer service. 
Our work with <a href="https://datatonic.com/case-studies/ai-booster-vodafone-supercharging-ai-ml-at-scale/">Vodafone</a> and <a href="https://datatonic.com/case-studies/bt-ai-accelerator-artificial-intelligence/">BT</a> has involved developing large-scale MLOps platforms to reduce the time it takes to deploy AI models and enable faster and more innovative data science projects at scale across the business. 
Yet, Generative AI is beginning to uncover a wide range of use cases that have not been accessible with other types of AI + ML. In the next section, we’ll look at how Telcos can get started with GenAI from the perspective of people, processes, and technology. 
<h4>Enabling your teams with the right skills for GenAI</h4>
Before diving in and developing key use cases, it’s important to ensure that teams are enabled with the right skills and familiarity with the tools that they will be required to use. This is even more relevant for Generative AI, as the tools and services, along with the skills required to use them, are constantly evolving. 
The requirement for upskilling extends beyond just technical teams; in order to effectively implement Generative AI use cases and ensure the business is ready to adopt them, it is essential to build a culture of AI fluency, by:
<ul>
<li style="font-weight: 400;" aria-level="1">Giving your non-technical teams the knowledge to identify opportunities for Generative AI within their roles.</li>
<li style="font-weight: 400;" aria-level="1">Enabling technical teams with the skills and expertise to implement use cases effectively.</li>
</ul>
&nbsp;
Building technical capability in-house is challenging, and our in-house Academy was built to solve this problem. With courses customised to your business needs, we provide instructor-led upskilling to turbocharge your teams. 
Datatonic Academy is an Authorised Training Partner for Google Cloud and AWS, delivering world-class training to enable businesses to upskill their existing teams and new hires. To find out more, visit <a href="https://datatonic.academy/">Datatonic Academy</a> to hear how you can prepare your team for Generative AI adoption. 
<h4>Generative AI must be approached with best practices in mind… </h4>
Processes are an important, yet often overlooked, part of technology development. Especially with innovative and less-tested technologies, users must approach them with best practices in mind; having clear processes can reduce the risk of errors, and improve security &amp; transparency. 
With Generative AI in particular, this includes considering various aspects of <a href="https://datatonic.com/insights/generative-ai-large-language-models-responsibly/">Responsible AI</a>, such as thorough testing and implementing guardrails to ensure data is used responsibly and ethically, and to prevent hallucinations (incorrect information that may appear to be true). 
Some easy first steps to ensure the successful adoption of Generative AI are: 
<ul>
<li style="font-weight: 400;" aria-level="1">Choose your use cases carefully &#8211; prioritise use cases carefully while considering the risk of each option combined with the need or value of the use case to your business.</li>
<li style="font-weight: 400;" aria-level="1">Set expectations correctly &#8211; communicate the capabilities and limitations of the system to prevent over-reliance. Anything that is produced by a generative model, should be made clear to the end user that it was indeed generated by AI.</li>
<li style="font-weight: 400;" aria-level="1">Apply conventional cybersecurity techniques &#8211; ensure that you have applied the established security techniques we have to mitigate security vulnerabilities that can arise from user input.</li>
<li style="font-weight: 400;" aria-level="1">Recognise and check for known attack patterns &#8211; build up a database of known malicious prompt patterns, and ensure that all user inputs are checked against this for similarity. </li>
<li style="font-weight: 400;" aria-level="1">Use content moderation to filter out unexpected or unwanted behaviour &#8211; even when utilising pre-aligned models, the non-deterministic output generated by LLMs should still go through content moderation checks to reduce negative outcomes. </li>
</ul>
More details can be found in our blog, <a href="https://datatonic.com/insights/generative-ai-large-language-models-responsibly/">Generative AI: Using Large Language Models Responsibly</a>.
<h4>Technology is advancing rapidly to enable new GenAI use cases…</h4>
Identifying relevant use cases and prioritising them is the first step to understanding what technology is required and where you’ll need to start. Below are some of the most impactful Generative AI use cases for telecommunications companies.
<h5>Generative AI in the Telecommunications Industry</h5>
Generative AI has several applications in the telecommunications industry, from creating personalised marketing messages to optimising network performance. For example, Generative AI can be used to generate text messages or email communications that are tailored to individual customers based on their usage habits. 
Additionally, Generative AI can be used to optimise network performance by generating synthetic data that simulates network behaviour, enabling engineers to test and optimise network configurations without disrupting real users. Let’s take a look at some types of use cases in more detail.
<h5>Transforming customer or employee experiences</h5>
Generative AI can help telcos improve customer experiences by enhancing Contact Centres with technology and freeing up support teams’ time to provide better and more personalised interactions. Businesses can apply foundational models, conversational AI, and search technologies to quickly and easily create multimodal chatbots and personalised self-serve customer experiences. 
This reduces the strain on current support teams, allowing them to focus on high-priority issues, and provides customers with faster and more personalised interactions, leading to lower waiting times and higher customer satisfaction, potentially reducing customer churn.
<h5>Improving network planning and operations</h5>
Generative AI can also be used to improve network planning and operations, through Network Capacity Planning. This includes processing customer experience and unstructured inputs to enhance capacity prediction models, which can be further augmented with historic usage data. Using GenAI to generate recommendations that feed into and help automate the network design cycle can also help to improve planning.
This has several benefits, for both the business and its customers. Firstly, this will increase prediction model reliability, leading to better and well-planned spending on new infrastructure. For the customer, this leads to improved network performance and enhanced customer satisfaction.
<h5>Employee Knowledge Search</h5>
In an industry with such large amounts of data, teams can spend hours navigating thousands of pages and tables of unstructured data in PDF and HTML formats, looking for the information they need. This reduces the time available for analysis and other important tasks.
Generative AI provides a solution to this, enabling users to quickly find the most relevant data and content via natural language search to boost employee productivity and knowledge sharing. Enterprise search makes this task easier by providing a summary of search results, as well as pointing to all related documents that discuss the same content, significantly boosting the efficiency of analysis. 
<h4>Conclusion</h4>
Integrating and ensuring the adoption of Generative AI, like any new solution, requires businesses to consider its people, its processes, and the integration of the right technology. By doing so, Telcos have the potential to drive business growth through enhanced customer experience, and improved network planning, alongside other efficiency gains.
To find out more about how your business can benefit from GenAI, visit our <a href="https://datatonic.com/solutions/generative-ai/">Generative AI</a> page, and read our Telco insights and solutions <a href="https://datatonic.com/solutions/telecommunications/">here</a>.

Adopting Generative AI for Telcos

Author: Yoni Hazan, Lead Analytics Consultant
When it comes to calculating the costs associated with a data team, the task is complex as there are several different factors that impact it from a financial perspective. Two major cost categories to consider in this calculation are personnel costs and the cost of data tooling. Understanding how to calculate these costs, and the return on investment associated with them, is critical to building an effective data stack.
In this article, we discuss these two areas, the key factors to consider when calculating the costs associated with both, and how investing in personnel and data tooling can position your business for long-term, scalable growth and success. 
<h4>Personnel Cost</h4>
The cost of a data team is the most important cost within your department as the team generates value regardless of the data tools used. The size of the team depends on the needs and size of the business. Below, is a general estimate of how data teams should be structured in relation to the size of the overall organisation.
<h5>Small business (10–50)</h5>
Your data team should be representative of one or two team members where the roles and responsibilities are divided between development, gathering requirements and prioritisation of the most valuable items that will help move the needle. These team members need to be flexible and adaptable to the changing needs of the team, working across architecture, data ingestion, modelling, and analysis.
<h5>Medium business (50–200)</h5>
Generally, medium-sized businesses will want to have a team of three to five members of which the roles of each team member will be more defined. Here’s an example team composition for an organisation of that size, with head counts depending on your data and business needs:
<ul>
<li style="font-weight: 400;" aria-level="1">One Team Lead or Manager directing the team, communicating with leadership across the organisation, and managing all individual team members.</li>
<li style="font-weight: 400;" aria-level="1">One Product Owner responsible for refining work and being responsible for ensuring that proposed work produces business value.</li>
<li style="font-weight: 400;" aria-level="1">One Data Engineer responsible for integrating data into a centralised data warehouse. More data engineers may be needed if there are a lot of complex data sources and fewer or none may be needed if third-party ingestion is sufficient (e.g., with <a href="http://www.fivetran.com/" target="_blank" rel="noopener">Fivetran</a>).</li>
<li style="font-weight: 400;" aria-level="1">One Analytics Engineer responsible for translating source-centric data into a business-centric data model that is ready for analysis or use. More analytics engineers may be needed if the logic or data is complex.</li>
<li style="font-weight: 400;" aria-level="1">One Analyst responsible for analysing data and identifying actionable insights within that data. More analysts may be needed if there are more complex analysis needs and fewer or no analysts may be needed if other functions are already self-serving their own analysis (e.g., Marketing could be sufficiently trained on the BI tool and able to handle their own analysis needs).</li>
</ul>
&nbsp;
<h5>Large business (200+)</h5>
For larger businesses, the size of the team should exceed five people to support the main needs of the company. This is due to the amount of data sources, data needs, as well as the need to maintain a robust data stack that can support the business needs. In larger businesses, it is essential to have a team lead who can identify where ROI is impacted the most, a business analyst who can gather requirements and provide visuals, analytics engineers, and a data engineer.
<h4>Data Tooling Costs</h4>
Above, we recommended rough guidelines for roles and headcount for different stages of your organisation’s growth. With that in mind, you can think of your tooling cost as your second major cost category and one that often works hand-in-hand with personnel costs. Teams must identify the most essential technologies and tools necessary for day-to-day work, and expand from there. 
Let’s explain that in more detail with some examples. With any data tools, the costs associated can be split by the upfront costs, and the ongoing costs of managing and maintaining platforms.
<h5>Upfront Costs</h5>
The upfront costs associated with tooling include initial software cost, servers and storage costs as well as implementation or migration costs, in the form of hiring a team of developers or consultants to set up and configure new tools.
When calculating the upfront costs of a tool, businesses should consider the following factors:
<ul>
<li style="font-weight: 400;" aria-level="1">The volume of data being stored and processed</li>
<li style="font-weight: 400;" aria-level="1">The complexity of the data, including the number of sources and data types</li>
<li style="font-weight: 400;" aria-level="1">The performance requirements for querying and processing data</li>
<li style="font-weight: 400;" aria-level="1">The data governance requirements, including security and compliance</li>
</ul>
&nbsp;
<h5>Ongoing Costs</h5>
The ongoing costs of data tooling include the cost of maintenance, keeping licensing up to date and operational and the staffing required, either through retaining a data team, explained above, or in working with consultants.
When calculating the ongoing costs of data tooling, businesses should consider the following factors:
<ul>
<li style="font-weight: 400;" aria-level="1">The forecasted growth of data over time</li>
<li style="font-weight: 400;" aria-level="1">As volume and complexity grow, the performance requirements may impact the need for additional tooling to maintain performance and scale.</li>
</ul>
&nbsp;
<h4>Potential Cost Savings</h4>
While the costs of these tools can be significant, there are potential cost savings that businesses can realise by implementing these tools, including:
<ul>
<li style="font-weight: 400;" aria-level="1">Increased efficiency &#8211; By centralising data storage and processing, businesses can increase efficiency and reduce the time and cost of data integration and transformation.</li>
<li style="font-weight: 400;" aria-level="1">Improved decision-making &#8211; By providing access to timely and accurate data, businesses can improve decision-making and reduce the cost of errors and missed opportunities.</li>
<li style="font-weight: 400;" aria-level="1">Scalability + Lower Maintenance Costs &#8211; Modern data tools can scale to handle increasing data volumes and complexity, reducing the need for additional investments.</li>
<li style="font-weight: 400;" aria-level="1">Automation + Time Saving &#8211; By automating data integration, transformation, and processing, businesses can reduce the cost of manual labour and increase productivity.</li>
</ul>
&nbsp;
A common issue organisations face when evaluating a modern data stack is identifying which tools to invest in relation to their unique business needs. Tools can be segmented into two categories, core tools and satellite tools. Below, we give a breakdown of these tool categories and some examples of tools Montreal Analytics recommends to its clients.
<h4>Core tools</h4>
Without these core tools, the data team will likely have very limited capacity. In some cases, you’ll be able to get the work done on your own through custom ingestion or analysis with free tools like Google Sheets, but in general, these categories require some spending.
Automated Ingestion
Automated Ingestion tools can be seen as the first element in your modern data stack, and the first core tool to consider when evaluating implementation, and the costs associated with this. Ingestion tools synchronise data from various sources and applications to a destination, like your data warehouse, using pre-built connectors.
Data Warehousing
Data Warehousing tools, such as <a href="https://datatonic.com/insights/forecasting-snowflake-credit-usage/" target="_blank" rel="noopener">Snowflake</a> or <a href="https://cloud.google.com/bigquery" target="_blank" rel="noopener">BigQuery</a>, serve as a centralised repository for all of an organisation’s data being pulled in from disparate sources from which the power of business-generated data can be harnessed. 
A business’ data warehouse will likely drive the majority of costs associated with your data tooling — impacted mainly by query time, as well as costs associated with running transformation jobs on a specific cadence to curate data and make it available for consumption to the end user.
BI Tooling
BI tools, such as <a href="https://www.looker.com/" target="_blank" rel="noopener">Looker</a> enable data teams to create reports and <a href="https://datatonic.com/insights/5-steps-looker-dashboard-best-practices/" target="_blank" rel="noopener">dashboards</a> that can be used to visualise data, identify opportunities within it and share real-time business analytics.
<h5>Satellite Tools</h5>
These tools are not core to most business needs, but they can provide either a lot of productivity gains or time savings/personnel cost savings.
Reverse ETL
Tools like <a href="http://getcensus.com/" target="_blank" rel="noopener">Census</a> or <a href="http://hightouch.com/" target="_blank" rel="noopener">Hightouch</a> sync customer data from your warehouse or BI tools into CRM, email marketing, advertising tools and more to enable other teams and departments to act on real-time data.
Data Observability
Tools like <a href="http://montecarlodata.com/" target="_blank" rel="noopener">Monte Carlo</a>, <a href="http://datafold.com/" target="_blank" rel="noopener">Datafold</a> or <a href="http://metaplane.com/" target="_blank" rel="noopener">Metaplane</a> enable data teams to monitor for missing or inaccurate data to ensure problems and data quality issues are identified and resolved in real-time.
Data Cataloguing
Data cataloguing tools like <a href="http://atlan.com/" target="_blank" rel="noopener">Atlan</a> act as a single source of truth and information for teams and an inventory of all data assets in an organisation, helping data teams discover and understand appropriate data for analytical or business purposes.
<h4>Conclusion</h4>
Calculating the costs associated with personnel and data tooling in a modern data stack is a complex task. However, it is critical to understand how to calculate these costs and the return on investment associated with them. Personnel cost is the most important cost within a data team, and the size of the team depends on the needs and size of the business. 
Data tooling costs also play a crucial role in the calculation of the overall cost. While representing a significant investment, by implementing modern data tools, businesses can realise potential cost savings such as increased efficiency, improved decision-making, scalability, and automation.
For more information on what you can do to manage costs and gain visibility over your spending, take a look at our FinOps whitepaper, <a href="https://datatonic.com/insights/four-steps-finops-maturity-whitepaper/" target="_blank" rel="noopener">4 Steps to Cloud FinOps Maturity</a>. Datatonic has a wealth of experience in data + AI. To discuss your challenges or work on a project together, contact us <a href="https://datatonic.com/contact/" target="_blank" rel="noopener">here</a>.

Understanding Your Data Stack Personnel and Tooling Costs

Authors: Fenella Slingsby-Smith, Business Analyst and Olivier Philippe, Strategy Specialist
In 2022, the global 5G market was $107 billion and it’s expected to more than triple in 5 years to <a href="https://www.marketsandmarkets.com/Market-Reports/5g-services-market-226908556.html">$331 billion by 2027</a>. 5G is much faster in unlocking benefits for customer experience and new revenue for providers. 
However, there are a few challenges for providers to consider before they can unlock this business value. One of the main costs introduced by 5G networks is the increase in energy consumption, and therefore costs. As a result, these businesses need to strike a balance between over-provisioning leading to excess energy consumption and under-provisioning leading to poor customer experience and therefore client churn. 
In this blog, we’ll look at a potential solution to this: using Reinforcement Learning to optimise network energy efficiency, without compromising service to the customers, to control the optimal set of activities for each station based on the information available at each point in time.
<h4>Introduction</h4>
Mobile networks have significantly evolved since their inception, starting with just voice communication and growing into sophisticated data networks that power our interconnected world. The latest generation, 5G, holds immense promise for providers and customers.
In this blog, we explore the following:
<ul>
<li style="font-weight: 400;" aria-level="1">The demand for 5G</li>
<li style="font-weight: 400;" aria-level="1">Mobile Network History &#8211; 1G to 5G</li>
<li style="font-weight: 400;" aria-level="1">Current Issues with 5G Adoption</li>
<li style="font-weight: 400;" aria-level="1">The Synergy of 5G and AI</li>
<li style="font-weight: 400;" aria-level="1">Energy efficacy solution</li>
<li style="font-weight: 400;" aria-level="1">Technical deep dive of solution</li>
</ul>
&nbsp;
<h4>The Demand for 5G</h4>
For providers, 5G represents a tremendous opportunity to meet the ever-increasing demand for data-driven services. The new network enables significantly higher data transfer speeds &#8211; up to 10 times faster than 4G, lower latency, and increased network capacity. This makes it possible to deliver seamless streaming experiences, support the Internet of Things (IoT), and enable groundbreaking innovations like autonomous vehicles and remote surgery.
Customers and businesses also stand to benefit greatly from 5G. Enhanced mobile broadband speeds and low latency will revolutionise various industries, such as healthcare, manufacturing, transportation, and entertainment. From real-time remote monitoring of patients to precision control of industrial processes, 5G will unlock a world of possibilities, enabling faster and more efficient operations.
<h4>Mobile Network History &#8211; 1G to 5G</h4>
The history of mobile networks can be traced back to the early 1980s when the first generation (1G) was introduced, enabling wireless voice communication. This was followed by 2G, which brought in digital technology, allowing for text messaging. With the advent of 3G, mobile networks took a significant leap forward by enabling faster data transmission and basic internet connectivity. 
4G, the previous generation, brought about substantial improvements in speed, capacity, and latency, paving the way for advanced applications and services. And now, we stand at the cusp of the fifth generation (5G), which promises unprecedented capabilities and transformative potential. 
<img decoding="async" loading="lazy" class="alignnone wp-image-22273" src="https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-300x179.png" alt="5G Telco" width="600" height="358" srcset="https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-300x179.png 300w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-1024x610.png 1024w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-768x458.png 768w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1.png 1386w" sizes="(max-width: 600px) 100vw, 600px" />
Figure 1: The increase in innovation of networks over time.
&nbsp;
<h4>Current Issues with 5G Adoption</h4>
While the anticipation for 5G is high, there are some challenges to its adoption:
Energy consumption: The adoption of 5G by telecommunications companies brings significant challenges in terms of energy consumption. The higher data transfer speeds, increased network capacity, and denser infrastructure requirements of 5G networks result in a surge in energy usage compared to previous generations. 
Telecommunication companies also need to deploy additional base stations and network equipment to support the extensive coverage and connectivity demands of 5G. These infrastructure expansions further contribute to elevated power consumption levels. Additionally, the advanced technologies and functionalities of 5G, such as massive MIMO (Multiple-Input Multiple-Output) and beamforming (a particular processing technique for signals that allow for directional transmission or reception), require continuous power supply, exacerbating energy demands. 
The substantial energy consumption not only impacts the operational costs of telecommunication companies but also raises environmental concerns due to increased carbon emissions. 
Cost of Implementation: Implementing 5G networks requires substantial investments by telecommunication providers. Upgrading existing infrastructure, installing new base stations, and deploying advanced equipment can be capital-intensive. The cost of spectrum licenses, technology acquisition, and network optimisation further add to the financial challenge. 
Infrastructure and Deployment Challenges + Delays: The rollout of 5G networks involves substantial infrastructure upgrades and the deployment of new equipment. This requires collaboration between telecommunication companies, government authorities, and regulatory bodies. 
<h4>The Synergy of 5G and AI</h4>
One of the key enablers for the success of 5G networks is the integration of artificial intelligence (AI). AI brings intelligent automation, predictive analytics, and enhanced network management capabilities to the telecommunications industry. It enables dynamic network optimisation, efficient resource allocation, and proactive issue resolution. 
AI-driven solutions empower telecommunication to gain valuable insights from the massive amounts of data generated by 5G networks. This data can be utilised to <a href="https://datatonic.com/insights/vodafone-5g-traffic-forecasting/">optimise network coverage</a>, identify bottlenecks, and personalise services based on customer preferences.
<h4>Energy Efficiency Solution</h4>
The transition to 5G requires denser infrastructure deployment, resulting in increased radius coverage overlap compared to previous generations. This shift places a heightened focus on the management and coordination of network infrastructure to achieve optimal performance and scalability of coverage. A large part of the infrastructure needed is the base stations. 
There are three types of base stations, macro, micro and pico. The term macrocell (macro) is used to describe the widest range of cell sizes. Macrocells are found in rural areas or along highways. Over a smaller cell area, a microcell (micro) is used in a densely populated urban area. Picocells are used for areas smaller than microcells, such as a large office, a shopping centre, or a train station.
Often, telecommunication companies either over-provision their network resources leading to inefficiencies or under-provision them, which can impact customer experience. Management must navigate the tradeoff between network performance and energy efficiency; this often leads to over-provisioning given the focus on customer experience, leading to excessive energy consumption throughout the industry. Notably, base stations alone account for a staggering <a href="https://beei.org/index.php/EEI/article/download/2705/1891">57% of energy consumption</a> across the industry. 
Machine Learning can be used to manage cost and environmental impact whilst maintaining high-quality service to users. This can be achieved by creating a Machine Learning model to adjust the number of active base stations depending on traffic and users&#8217; needs. 
<h4>Energy Efficiency: Technical Deep Dive</h4>
Machine Learning can optimise which pico base stations should be active or in sleep-modus at any given time; the increasing complexity and scale require an automated solution to optimise energy consumption. Incorporating ML into this automated solution can save significant energy costs, whilst retaining great service to users (see Figure 2).
<img decoding="async" loading="lazy" class="alignnone wp-image-22272" src="https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-2-300x134.png" alt="5G Telco" width="600" height="268" srcset="https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-2-300x134.png 300w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-2-1024x458.png 1024w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-2-768x343.png 768w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-2-1536x686.png 1536w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-2.png 1600w" sizes="(max-width: 600px) 100vw, 600px" />
Figure 2: On the left, we see the current infrastructure set-up leading to excessive energy consumption as all base stations are on. On the right we see the ML-optimised base station giving the same coverage to users whilst limiting energy consumption.
Reinforcement Learning is the state-of-the-art training method that can learn to make optimal decisions in a highly adaptable, dynamic environment. You can find out more about how this works <a href="https://datatonic.com/insights/reinforcement-learning-identifying-opportunities-use-cases/">here</a>. We can use Reinforcement Learning to train an agent to learn the optimal set of active base stations based on the information available to the agent at a certain point in time, which we call the state. 
Let’s dive a little bit deeper into the key components of the solution:
1. State: The state is defined as a set of locations of User Equipments (UEs) within a pre-defined user grid (brown area in Figure 3) in the DeepMIMO situation.
<img decoding="async" loading="lazy" class="alignnone wp-image-22271" src="https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-3-300x128.png" alt="" width="600" height="256" srcset="https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-3-300x128.png 300w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-3-1024x438.png 1024w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-3-768x328.png 768w, https://cms.datatonic.com/app/uploads/2023/08/Telco-Blog-1-3.png 1378w" sizes="(max-width: 600px) 100vw, 600px" />
Figure 3: (left) The methodology is inspired by the paper “<a href="https://arxiv.org/pdf/2103.11891.pdf">Increasing Energy Efficiency of Massive-MIMO Network via Base Stations Switching using Reinforcement Learning and Radio Environment Maps</a>” by M. Hoffman, P. Kryszkiewicz and A. Kliks.
(right) A public dataset,<a href="https://deepmimo.net/scenarios/o1-scenario/"> DeepMIMO Scenario 01 (Outdoor 1)</a> was adapted to obtain a similar scenario as in the increasing energy efficiency paper [1]. This is done by reducing the number of active base stations and the range of locations the User Equipments (UE) can find themselves in.
2. Actions: The actions are a binary vector representing the status of the base stations. In total, 6 base stations are considered where one macro and 5 pico base stations. A resulting binary vector for action 0, looks like 0: (0, 0, 0, 0, 0, 1), representing the 5 pico base stations to be in the sleeping modus and the macro station to be active. All base stations to be active correspond with action 31, represented as: 31: (1, 1, 1, 1, 1, 1).
3. Rewards: The reward depends on the energy efficiency and the number of served UEs. To compute the energy efficiency is computed based on the power consumption and the average UE bitrates.
4. Agent: The agent is the decision-making algorithm trying to optimise the energy efficiency, without compromising the number of served UEs. The agent will choose an action based on the current state. This action corresponds with a certain configuration of the base station&#8217;s modus. If the agent makes a good decision, it will get rewarded, if not, it will get penalised. 
5. Environment: The environment is a Heterogeneous network with 5 pico and one macro base station. As we don’t have real data available a simulated environment is built up based on the DeepMIMO data presented in Figure 2. The agent will interact with this simulated environment and learn from its experiences.
To train the model, we simulate this environment and evaluate the agent&#8217;s performance. This experience, consisting of the state and the reward, allows us to train our model by learning from its experiences. Repeating this process enough times leaves us with a model capable of controlling which micro base stations should be active or sleep-modus depending on the locations of its users, which can be deployed into the real world.
<h4>Conclusion</h4>
A key problem with 5G deployment is the technical complexity spanning across platforms and the cost to upgrade to 5G. With 6G on the horizon, optimising network coverage with ML to support these large volumes of data and high demand will become even more crucial.
By using Machine Learning, or more specifically Reinforcement Learning, we can optimise which base stations are active to provide optimal service and customer experience at lower costs for telecommunications companies. This solution further decreases costs by limiting excess energy consumption, resulting in significant value for both the business and society. 
Datatonic is <a href="https://datatonic.com/insights/datatonic-google-cloud-specialization-partner-award-machine-learning/">Google Cloud’s Machine Learning Partner of the Year</a> with a wealth of experience developing and deploying impactful Machine Learning models and MLOps Platform builds. Our experience includes <a href="https://datatonic.com/insights/vodafone-5g-traffic-forecasting/">forecasting 5G usage with Vodafone</a> and developing an <a href="https://datatonic.com/case-studies/bt-ai-accelerator-artificial-intelligence/">MLOps platform for BT</a> to reduce time to deployment from six months to six days. Need help developing an ML model, or deploying your Machine Learning models fast? <a href="https://datatonic.com/contact">Get in touch</a> to discuss your requirements!

Reducing Emissions and Operating Costs for 5G Providers 

Tracking Business KPIs with Looker

<h4>Transforming operations with Data, AI &amp; MLOps</h4>
Datatonic has developed a solution addressing requirements for cloud-based real-time anomaly detection for CSP’s. Datatonic is an industry leading data, AI and analytics consultancy who specialise in designing and building bespoke data, AI and analytics solutions on Google Cloud. 
Anomalies on any network can cause outages, potentially risking reputation damage for network providers, so it is important to swiftly identify, triage, and recover from potential service outages caused by anomalous behaviour. 
Using advanced AI/ML techniques, Datatonic enables CSP’s to monitor their networks, predict when anomalies will occur, and trigger actions or alerts to the desired locations to enable rapid (or automated) decision making before a service outage occurs.
&nbsp;
The unique benefits of this solution framework include
<ul>
<li style="font-weight: 400;" aria-level="1">Ingestion capabilities from multiple data sources to Google Cloud, enabling teams to effectively leverage their data using Google BigQuery</li>
</ul>
<ul>
<li style="font-weight: 400;" aria-level="1">End-to-end workflow to serve predictions from anomaly detection models in real-time, including corresponding data points and severity level information. </li>
<li style="font-weight: 400;" aria-level="1">Productionised Anomaly Detection models using Dataflow, BigQuery ML, VertexAI &amp; Vertex Pipelines that can scale up as needed to handle thousands of time series.</li>
<li style="font-weight: 400;" aria-level="1">Best practice MLOps templates to enable rapid time to value when developing new Machine Learning models and enables models to provide long-term business value with model monitoring and retraining</li>
</ul>
Datatonic enables CSP’s to efficiently develop, deploy, and productionise strategic Machine Learning use cases by leveraging their <a href="https://datatonic.com/insights/vertex-ai-mlops-turbo-templates/">Vertex AI Turbo Templates</a>. The Vertex AI Turbo Templates provide a template, or reference implementation, of the end-to-end ML lifecycle &#8211; helping CSP’s to deploy ML use cases such as anomaly detection models and other network automation use cases.
Read about our anomaly detection work with Inmarsat <a href="https://datatonic.com/case-studies/inmarsat-mlops-satellite-anomalies/">here</a>, and <a href="https://datatonic.com/contact/">get in touch</a> to find out more.

Datatonic Develops Real-time Anomaly Detection Solution on Google Cloud

Forecasting Snowflake Credit Usage

Authors: Nevile Reid, Head of Strategy + Advisory and Mike Cummings, Principal Analytics Engineer
The content in this blog has been adapted from our latest whitepaper, 4 Steps to Cloud FinOps Maturity. You can access the full paper <a href="http://datatonic.com/insights/four-steps-finops-maturity-whitepaper">here</a>. 
<h4>What is FinOps?</h4>
As businesses continue to adopt complex, multi-cloud architectures and operating models, Finance Operations has evolved to provide visibility and control over your cloud cost and investment.
In its most basic capacity, FinOps can provide a reference point for businesses looking to break down their cloud IT budgets into cost centres or provide financial reporting to understand simple ROI for digital spending. As it matures, it can provide businesses with more proactive capabilities &#8211; whether this is the ability to charge costs to business domains or project budgets, identify anomalies and drive financial forecasts, or inform the automation of key cloud infrastructure.
Whatever the use cases or operational scale, FinOps provides a critical vehicle for finance and data leaders to remain in control of their modern data stacks and their development.
<h4>The Cloud Cost Challenge</h4>
Businesses face a range of challenges when trying to build and develop their cloud IT estates, with many encountering issues with visibility of cloud costs, complex pricing models and long lead times in managing cost spikes. Even where tools for cost management exist, they often lack flexibility or can be overly simplistic, requiring teams to resort to a combination of cloud-native and third-party tools.
Due to the expansive strategic demand for data over the past 5-10 years, businesses have developed their data stacks at speed, leading to complex architectural arrangements and variations in tooling, working practices, and billing. This whirlwind of change often happens without effective cost management, as businesses prioritise the need for data, analytics, AI, and ML to remain competitive. This has created an unnecessary complexity for cost visibility, accountability, and proactive management.
Furthermore, in recent months, cost inflation in the wider economy has meant that reducing cloud costs and addressing inefficiencies with FinOps has rapidly risen in priority for CDOs, CFOs, and other data leaders. 
Investing in FinOps is the starting point for reducing unnecessary costs. If some of these risks are recognisable within your organisation, the reality is that intervention is needed, and organisations should invest money now to reduce costs in the long term.
<h4>Why is FinOps the solution?</h4>
FinOps enables businesses to quickly diagnose the causes of inefficiency and provides the visibility required to solve potential issues. Finance and data leaders can be empowered to intervene, drive accountability and control costs across the wider organisation.
Businesses need to take urgent steps to establish a FinOps capability, even in its most basic form, or increase the risk of long-term financial instability that scales relative to the growth of digital assets.
<h4>What does that solution look like?</h4>
FinOps as a practice involves several different areas of focus. These can be broken down into four main steps: 
1. Visibility
Quickly deploy a solution that enables you to access, process and visualise cost data &#8211; and even map it to cost centres or other breakdowns.
2. Insight
Once data is accessible, augment your analytical potential with AI-powered insights, including forecasting and anomaly detection, to make your teams proactive rather than reactive to cost changes.
3. Optimisation
Once your Cloud FinOps solution is producing valuable insights, it can be used as a driver of change to operational processes and guide further development of the cloud system &#8211; whether increasing accountability for cost, or as a strategic indicator for architectural improvements.
4. Operational 
To ensure full visibility and improvement over your financial processes, the FinOps solution can be further enhanced by ingesting data from multiple cloud providers, and even expand to SaaS tools or services.
By progressing through the various stages of FinOps, you can develop comprehensive maturity across your organisation.
For more information, read our whitepaper, <a href="http://datatonic.com/insights/four-steps-finops-maturity-whitepaper">4 Steps to Cloud FinOps Maturity</a>, which details how your organisation can quickly establish strong FinOps practices.
We’ll also have upcoming blogs on using FinOps to help with the transition to Data Mesh, FinOps and improving environmental efficiency, forecasting data warehouse costs, and more!

Events

Whitepapers

Insights

Datatonic at Big Data LDN

Related

Blue-Green Deployment with dbt and Snowflake

Generative AI Personalisation for Retail

Google Cloud Next: Our Top 10 Announcement Highlights