As illustrated in this article, in order to increase the chances of your ML solution to make the cut from the lab into production, it is wise to employ a ML platform. To find the platform that best meets your organizational requirements, you need a framework which helps assess the capabilities of ML platforms. One example for this is the Accenture assessment framework that encompasses the end-to-end ML lifecycle.
To understand this framework, it is first necessary to define the functional capabilities a ML platform should cover to enable continuous development, integration, deployment, and monitoring of a ML solution.
To quote Ron Schmelzer: "There's no such thing as the machine learning platform". In a world where lines are drawn through an already complicated software and service environment, functional capabilities set a baseline for what is necessary to bring a ML solution to production. The result is a capability map (Figure 1) that covers the end-to-end ML lifecycle and comprises five functional areas: Data Ingestion & Storage, Experimentation Zone, Continuous Integration, Industrialization Zone and Data Presentation.
Figure 1: The functional areas and components which a ML platform should ideally cover
to enable an end-to-end ML lifecycle.
Each functional area (example indicated in purple) consists of components (example indicated in pink) which ensure the distinct requirements imposed on the respective functional area. Projects and use case implementations usually start by ingesting the collected data (Data Ingestion & Storage), followed by experimenting and developing solution algorithms (Experimentation Zone). ML models created in the process are then tested, integrated, and deployed to production (Continuous Integration). Additionally, the models are validated and continuously monitored (Industrialization Zone). The utilization of AI applications and ML solutions is possible through different types of endpoints, such as REST-API, Batch Service and Dashboards (Data Presentation).
In particular, the individual functional areas entail the following:
After having introduced the five functional areas of an ML platform and the components that ensure their necessary requirements, let us define the assessment indicators that will enable the evaluation of different ML platforms.
Scoring of a component
The ML platforms are assessed based on the coverage and maturity scores of each of their components. While the coverage score indicates the availability of the component on a certain ML platform, the maturity score indicates the component’s capabilities, robustness, and readiness for production.
Scoring of a functional area
As shown in Figure 2, the overall coverage and maturity score of a functional area is the cumulative average of the scores of the components, which comprise the respective functional area:
Figure 2: Scoring of component and functional area with respect to coverage and maturity.
The coverage score of a functional area provides information about the percentage of covered components, whereas the maturity score of a functional area indicates the overall capability and readiness of the underlying components.
Why do we need two distinct assessment indicators: coverage and maturity?
Let us imagine a Platform ‘A’ which has only a little coverage of components but high maturity in the covered components and another platform ‘B’ that covers every component but with little maturity. Without a distinction of coverage and maturity, both platforms might score the same, but from an operational perspective, this makes a huge difference. You would either need additional tools to enable the full ML lifecycle or lack the capability to fully industrialize your workflow.
In order to find the ML platform that best fulfils your requirements, you need to evaluate the components (and thereby the overarching functional areas) of different platforms. For the evaluation, the components’ coverage score (availability of the component) is just as important as its maturity score (capabilities, robustness, and readiness for production of the component).