The ML product owner
Perspective
Some (business) people new to Machine Learning think that the technology can perform magic tricks. However, even the greatest magicians simply demonstrate excellent craftsmanship that is replicable once you know what to do and when. So, even if some of the ML stuff seems to be magical, it all comes down to key principles that one needs to follow to successfully deploy an ML product (Ameisen, 2020).
​
To help you have a more realistic view of your ML pathway, we have defined five lead questions. They will reveal if a particular ML solution leads to a fruitful business case and if the solution is technically feasible in the long term. Click here for a print-out poster and evaluation sheet containing the feasibility assessment questions that you can fill out.
Please keep in mind that crucial insights can only happen once you iterate between two key perspectives: the business side and the ML technology side. On the one hand, without proper business input, chances are high that you will end up solving a problem that is not relevant to the business. On the other hand, without experienced ML developers involved, chances are high that you will start to conceptualize and develop something that will perform too badly, too slowly, or too expensively in production to justify a sustainable business case.
​
​
| What is the problem
| worth solving?
​
The problem can be relevant to your internal/external processes or it can be found in other markets where you see a potential to conquer the market with a better or cheaper solution than the incumbent ones. You can find inspiration by talking to your colleagues, customers, suppliers, partners and by reading our article on Machine Learning and Strategy (Erichsen, 2020).
​
You have to nail down the problem statement as a user story and provide more business context to the developers. For example, you can ask the following questions: What processes and overall context are relevant to the problem and later to the solution? What would be the solution’s value for the user and for the business? How much time or how many resources could be saved?
​
​
| How do humans solve the
| problem at hand?
​
In most ML projects, the answer to this question is the missing link between the business requirements and the technological options best suited to design an automated solution. From the business perspective, the answer to this question helps to understand the actual costs of the problem and the potential new revenue. It also helps you to determine which internal people/domain experts are required and how much of their time is needed for the co-development of the solution. From the technical perspective, this question also provides insights that help to determine the data and operational requirements, and the algorithmic approach that is most suitable (Bauer et al., 2012).
​
​
| What data is available
| for training?
​
There are many ML solutions out there in the market and open source ML models that are already pre-trained or available from research papers. In our experience, however, ML models often need to be trained with custom data to reach sufficient accuracy or regularly re-trained to cope with changes in the environment. In any case, training needs high-quality data and, depending on the use case and algorithmic approach, a lot of examples might be needed (Ng, 2021).
​
You should identify and describe how much data is legally available to your organization. That means, data that is compliant with the GDPR and other industry restrictions, that has consent of the data owner, and that can be authorized to leave the organization if required by external solution providers for training purposes.
​
Developers will ask you about format and volume, if the data is labeled so that the system can already make use of it to start learning, and how much data is available to be used for testing. Sometimes, the latter needs to be newly created by the humans working on the problem to have a ground truth reference for the algorithm.
​
| What system performance
| is expected?
​
There are three domains that determine the minimum thresholds that the solution must reach and that help us to measure the magnitude of success: accuracy, latency and costs. Be aware that there can be strong trade-offs between how good the system predictions are, how fast the results are delivered and how expensive it is to execute the ML models.
​
Accuracy: How good does the system need to be?
​
It is crucial that we define the minimum accuracy to be achieved, or the maximum error rate to be tolerated, so that the ML system is accepted by the business owner and the end users once it is in the production environment. Often, new ML development models do not perform well enough from the beginning. Accuracy will increase through training or after integrating multiple sources of information (Bauer et al,. 2015), but the slope of improvement will decrease, meaning that the improvement from 80% to 85% is far easier to achieve than the improvement from 90% to 95%.
​
The central idea of continuous learning approaches is to start with good enough accuracy and aim for later improvements in the production environment. These are prompted by different kinds of feedback – mostly from users or business operators. Nevertheless, you still need to define upfront the minimum accuracy for the model to be released in the production environment. When accuracy is the key value-driver of a potential solution, the primary goal is to reach the minimum accuracy first, and only then work on improvements to the operational costs if necessary.
​
​
Latency: How fast does the system need to deliver?
​
Some business problems require results to be delivered in real-time or with very little delay, such as when immediate actions or decisions need to be taken based on the classification, recommendation or prediction of the respective ML model (Warden & Situnayake, 2019).
​
The required latency is not only determined by hard business constraints, but also by the user experience (UX) or by the performance of a human reference (Dominey et al., 2010). For example, you could build a cloud service to process a video in real-time (that is, a second of video processed in less than a second), but make sure you understand the cost savings when processing such videos more slowly with smaller computing instances. If the customer does not require instantaneous access, simpler infrastructure can reduce implementation time, infrastructure complexity, and your dependency on seniors for maintenance.
​
Conversely, if a solution cannot achieve the same processing speed as the human reference, either because the models are not fast enough (even with powerful computing devices) or because the use case only allows limited computing power, it is a strong indicator that the technology is not ready for that particular use case.
​
​
Costs: How much should the business be willing to pay?
​
Of course, most things in life come with a price tag. Improvements in accuracy and latency per use case come with an increase in development and training efforts as well as in operating costs in the production environment. Getting a rough estimation of the operating costs as early as possible can save you a great deal of time, as it clarifies whether an ML project is actually worth the effort.
The maximum cost allowance of a potential solution can be drawn from comparisons with how humans are currently solving this task. For instance, if a worker is spending 5 hours on a task, the maximum cost allowance is 5 times the respective hourly rate. The same holds true if an external service provider is solving the problem. Here, the maximum cost allowance is the price the provider is charging - minus a premium for the switching costs.
When we talk about continuous learning or hybrid approaches, where humans are still involved in the process of solving the problem with ML support, it is a good rule of thumb to aim for at least 5x improvements with respect to the working time without ML.
​
​
| What is the scenario of
| use in production?
​
So far we have focused on describing the requirements of the brain of our application, this is, of the algorithmic approach that we will take. Now, we need to determine the requirements of its body, or in other words, its infrastructure and hardware requirements.
​
​
Where and how often will the user interact with the system?
​
Here is where you need a proper description of where, how and how often the users (or other systems) will interact with the ML system in production. These user stories allow ML engineers to determine the requirements and restrictions regarding infrastructure, hardware components, and energy supply (Malaska & Seidman, 2018).
​
For example, imagine an application where the user wants to detect if a tree has a harmful fungus. Similar applications for image classification make use of cloud services but this application has to work even when there is no network signal available in the forest. The solution will only help the user when it is working directly in front of a tree, because the user needs to mark the tree if it should be felled by other workers. Searching for connectivity elsewhere and coming back once the tree is classified with a proper diagnosis is not very practical. With these new insights, the product requirements now include on-device computation, and the technical experts understand better the questions they must ask to assess the feasibility of the solution.
​
​
Are there any constraints or specifics worth knowing?
​
The task of understanding the capabilities, constraints and other specifics relevant to different ML solutions can be a tricky one since a certain amount of ML knowledge is certainly helpful here. General considerations can come from connectivity constraints, available computation, battery consumption or privacy issues.
​
For example, some common constraints for audio use cases can be background noise, overlapping signals or speaker dialects (Davila-Chacon et al., 2018). On the other hand, for visual systems they can result from lighting conditions that change during the day or from visual noise such as dust, dirt splashes or humidity that could have a negative effect on the camera lenses (Davila-Chacon et al., 2011).
​
​
| Final evaluation!
​
With all these answers the product owner now can evaluate and lead the discussion around two central questions: Will the business side likely get a solution that is worth investing in, and can the ML engineers likely develop a solution that is capable of performing as requested?
​
Keep in mind that it is not the job of the product owner to answer all the questions in this section, but rather to make sure that these questions are answered by people who are knowledgeable about the business domain and the technical perspective. If you do not have experienced ML engineers in your company, hire them as freelancers or permanent employees —depending on your long-term goals.
​
​
​