What are the business problems that we are trying to solve and how accurately can the model solve our problem?
The business problem that we are trying to solve is to predict the probability of a customer making a purchase on our website. We would like to be able to accurately predict whether a customer will make a purchase or not so that we can target them with ads accordingly. This will help us to make more money and to better target our ads. The model can accurately predict whether a customer will make a purchase or not with about 80% accuracy. So how do those 80% numbers come out? This is where evaluation metrics come into play.
What is a model without metric evaluation?
A model without metric evaluation is a model that has not been evaluated using any metrics. This means that the model's performance is unknown and it is not possible to say how well it will work on new data.
Why do we need this metric evaluation? Metric evaluation is important because it allows us to see how well a model is performing and to identify any areas where it may be lacking. Without metric evaluation, it would be difficult to improve the model or to know if it is working as well as it could be.
There are 4 types of evaluation metrics that can be used to see our model performance.
1. Accuracy
2. Precision
3. Recall
4. F1 Score
What is accuracy ?
Accuracy is the proportion of correct predictions out of all predictions.
When to use ?
Accuracy is important measures to consider when assessing the performance of a classifier.
Example :
You have a fraud detection system trained on imbalanced data. In normal transaction, the model is right 98% of the time. For the fraud class “Positive”, you are only right 50% of the time.
As the data is very sparse, do we still want accuracy as a metric of our model performance?
No. In this case, accuracy would not be a good metric of our model performance because the model could simply predict the majority class every time and still have a high accuracy.
What is precision in evaluation metrics?
Precision in evaluation metrics is a measure of how accurate a model is in predicting positive outcomes.
When to use?
Precision metrics are typically used to evaluate the performance of a classifier. The most common precision metric is the percentage of correctly classified instances, which is simply the accuracy of the classifier. Other precision metrics include the precision and recall of the classifier, which are typically used together to get a more complete picture of the classifier's performance.
Examples :
If most fraud detection is a positive result, but actually the detection is not a fraud. In this instance, precision helps when the costs of false positives are high.
What is Recall in evaluation metrics?
Recall is a metric that measures the ability of a model to correctly identify positive samples.
When to Use?
Recall in evaluation metrics typically used in classification tasks, precision and recall are used as a pair.
Examples :
If the model predicts the fraud a negative result, but the detection is actually a fraud. These events need recall metrics in assisting when the costs of false negatives are high.
What is F1 Score ?
F1 is the harmonic mean of precision and recall. A model with a high F1 score is said to be a balanced model.
When to use ?
As the F1 score is the overall measure of a model’s accuracy that combines precision and recall, let's take a look at the binary problem of 1 and 0. F1 scores are perfect when it’s 1 and a total failure when it is 0.
Example :
We're gonna use fraud detection as an example, if we say mostly “NO” for our training dataset. The precision here is almost or close to 0 and the accuracy is 98%.
Hence, the F1 score is also almost or close to0. Which indicates that the 98% of accuracy is worthless for our case.
In other words, good F1 score means that, you have:
Low false positives (identify false alarm)
Low false negatives (identify real threat)
The closer the F1 Score to 1, the better model’s accuracy.
Conclusion
As important to build the model from the start to finish, the evaluation metrics to assess the performance also plays a big role in solving the problem of business.
Keep in mind that, how wonderful the model that you have built if it's not accurately predicted on what needs to be predicted = worthless. Simplest machine learning model can keep up with advanced models if we do it right.
About Ever AI
Have a lot of data but don't know how to leverage the most out of it?
Need AI solutions for your business?
Have a Machine Learning model but don't know how to deploy? Sign up here, Ever AI Web Apps https://ever-ai.app/
We provide a NO CODE End-to-end data science platform for you.
Visit https://www.ever-technologies.com/ever-ai for more info.
Would you like to understand the theory of AI better?
Contact us to have our trainers organise a workshop for you and your team.