Why Model Monitoring is Important?
Sources Neptune.ai
There are many different ways to deploy and monitor models, and the specific method used will depend on the type of model and the data it is trained on. However, a few key considerations should be kept in mind.
When deploying a model, it is important to consider how it will be used and what type of data will be applied. For example, if a model is going to be used for real-time predictions, it needs to be deployed in a way that can handle incoming data quickly and accurately. Similarly, if a model is going to be used for batch predictions, it needs to be deployed in a way that can handle large amounts of data efficiently.
Once a model is deployed, it is also important to monitor its performance to ensure that it is working as intended. This can be done in a number of ways, such as monitoring accuracy metrics or looking at the model's predictions over time. If a model is not performing as expected, it may need to be retrained or redeployed.
There are several cases of scenarios where the model is not performing well. And what information we need to check if we come across these problems.
Example of scenarios include :
Data related scenarios
Use of different data sources: The use of different data sources in research and production environments can lead to inherent differences in the same characteristics, ultimately leading to different predictions.
Features not accessible in production: The model expects to see the same features it saw during training in order to make predictions.
Unrepresentative training data: Feature distributions in the training data should be representative of the live data. Otherwise, your model may not perform well in production.
Data dependencies: Machine learning models are data dependent. Data changes can have a significant impact on model performance.
Model Related Scenarios
Changing environment: Machine learning models are often trained on historical data that does not take into account that populations and their behavior may not be the same as in the past (i.e. events like detecting financial distress can suddenly change people's behavior).
Security breaches: Unfortunately, someone may discover a bug in your model and be actively looking for ways to adjust your strategy to break it.
A formal investigation and checks are required to confirm these scenarios are happening in your production.
To get better insights into the model in live production, software tester engineers need to run several tests such as data test, features tests and data dependencies test and to look for more insight value in the production environment, bug findings reporting are required to prevent security breaches.
What are the metrics for monitoring ML system?
A metrics system in monitoring ML systems are measurements that we can observe, assess and track the performance of your machine learning system.
The measurement of the ML system that we should measure should cover from the usage and performance across a period of time.
Here are some metrics measurements we can use to monitor the ML system performance :
Latency
Memory/CPU usage
Disk Utilization
Statistical Measures over a period of time.
These measurements are handy to developers on getting better insights of the model performance.
Conclusion
Model monitoring is an integral part of the machine learning lifecycle and is growing as a core component of successful machine learning applications in production.
About Ever AI
Have a lot of data but don't know how to leverage the most out of it?
Need AI solutions for your business?
Have a Machine Learning model but don't know how to deploy? Sign up here, Ever AI Web Apps https://ever-ai.app/
Join our Telegram Channel for more information - https://t.me/aitechforeveryone
We provide a NO CODE End-to-end data science platform for you.
Visit https://www.ever-technologies.com/ever-ai for more info.
Would you like to understand the theory of AI better?
Contact us to have our trainers organise a workshop for you and your team.
留言