Error analysis is about digging deep into why something isn’t working - to learn from it. It might sound obvious, but it's shockingly underused, especially where it matters most: AI development.
Let's explore what it is through an example
Cats or Dogs ?
I'm skipping many details that may hurt Data Scientists for the sake of simplicity.
Say you have 200 images to classify as either cats or dogs. You build an AI and get 78% accuracy - not great. We need to do better. But how?
The typical response?
- "Let's try another model"
- "Let's tweak the (hyper)parameters and hope for the best."
Basically, this means blindly exploring different solutions to see what sticks. Then, we hope to learn something and slowly converge on a better solution.
But what if you could already learn what you want with this very first run?
Let's do error analysis!
You dig into your data and realize:
- Some puppies were classified as cats.
- Some images are completely dark - even you can't tell if it's a cat or a dog!
- Finally, some were actually mislabeled!
After removing these irrelevant samples (points 2 & 3), your model actually achieves 97% accuracy! The remaining 3% error comes from puppies being misclassified as cats.
The problem was not your model, but the data it was given.
Well, almost. There's still the issue of puppies being misclassified - this is a failure mode.
What does this actually mean?
In this case, we have 3 clear action items:
- Correct the mislabeled samples.
- Find a way to make to model better on puppy images (there are many!).
- Ensure proper lighting for production cameras 🤷♂️
- ... and plenty more!
Then, next iteration, do the same, you may find out new problems!
Basically, error analysis is what moves you past blindly tweaking solutions in hopes of improvement. Instead, it shifts the focus to understanding the root causes of failure and addressing them directly.
Top comments (0)