In part one of our MLOps Tools series, Aman compared TensorFlow Transform with BigQuery for data transformation. If you missed it, make sure you check it out here. In this part, we’ll be taking a deep-dive into orchestration tools on Google Cloud Platform (GCP): Kubeflow Pipelines vs. Cloud Composer.
Choosing the right orchestration tool is particularly important when evaluating the creation of an MLOps platform and will affect every step of the ML Journey that leads to model deployment:
When talking about IT systems and Cloud Architectures, the word orchestration refers to the automated configuration, coordination, and management of computer systems and software.
This word originates from Orchestra, which in classical music defines the study or practice of writing music for an orchestra.
I always found really accurate the usage of this word when talking about computing systems: if we think of our system as an orchestra, each one being a different instrument, the orchestrator would be for sure the conductor of the orchestra, fitting together all the different instruments in order to play the song.
Just as the conductor is a crucial part of the orchestra, the orchestrator plays a central role in any Cloud Architecture. Because of that, choosing the right orchestration tool in a Cloud Architecture or more specifically in a Cloud MLOps Architecture is really important as every orchestrator has different peculiarities and things that come out of the box.
During the last year, our MLOps team has developed a lot of experience in using the two main orchestrators available in GCP: Cloud Composer, built on the top of the open source framework Apache Airflow; and AI Platform Pipelines, based on Kubeflow Pipelines.
This blog summarises all of the insights we collected about these two products, in order to help you choose what orchestrator would fit better in your MLOps architecture.
Cloud Composer (see more here) is the GCP managed orchestration service built on top of Airflow. Airflow is an open source framework for orchestration of data engineering tasks, which centers around the concept of Directed Acyclic Graphs (DAGs). A DAG, similar to a flowchart, can be defined in Python to execute tasks with complex inter-dependencies between them.
On top of Airflow, Cloud Composer allows easier creation of the environment with no need to worry about the underlying infrastructure, good integration with other GCP tools, and robust monitoring through Cloud Operations monitoring and logging.
We recently suggested Cloud Composer as orchestrator for a fraud detection MLOps platform involving complex ingestion and preprocessing tasks. We also plan to use Composer to orchestrate the insertion of features in the feature store.
AI Platform Pipelines (see more here) allows the creation in a couple of simple steps of a Kubernetes engine Cluster with Kubeflow Pipelines standalone installed on it. Kubeflow Pipelines is a container-native workflow engine based on Argo for orchestrating portable, scalable machine learning jobs on Kubernetes. Belonging to the Kubeflow ecosystem, it can be either installed by default with Kubeflow or as an alternative installed as standalone.
We used Kubeflow Pipelines in a recent project to help Emotion AI pioneer, Realeyes, enhance their R&D capabilities with MLOps. Due to their big Data Scientists team and the fact that their core product is powered by Machine Learning, they are required to perform several experiments and iterations over a certain model whilst keeping MLOps best practices. Kubeflow Pipelines enabled them to do exactly this, making it a great tool of choice for their orchestration.
In this blog, we have seen how choosing the right orchestrator for an MLOps platform is a complex task, which needs to take into account the requirements of the platform itself.
Cloud Composer is particularly suggested in architectures requiring complex data pipelines for ML. This choice will enable data scientists and Machine Learning engineers to use Airflow’s capabilities to create complex DAGs, but also require manual implementation of ML functionalities such as metadata storage and metric visualization and evaluation.
When the architecture doesn’t involve complex data processing pipelines, or when Composer is not already used as orchestrator for other data transformations, AI Platform Pipelines is the suggested choice due to the built-in functionalities to orchestrate ML workflows, as well as the reduced monthly cost compared to Cloud Composer and the possibility to migrate the workflows to Serverless AI Platform Pipelines in the future.
We hope you enjoyed our MLOps tool comparison blog for Orchestration. To see our team’s evaluation on Run Cloud to Cloud Functions for Serving, check our part 3 of our series.
Want to know more about MLOps? Download our 2021 Guide to MLOps here.
Know exactly where and how to start your AI journey with Datatonic’s
three-week AI Innovation Jumpstart *.
* Duration dependent on data complexity and use case chosen for POC model
With your own data sets, convince your business of the value of migrating your data warehouse, data lake and/or streaming platform to the cloud in four weeks.
With your own data, see how Looker can modernise your BI needs
with Datatonic’s two-week Showcase.