If your production line is running well, your defect rate is somewhere between 0.1% and 3%. That's good news for your quality metrics. It's a problem for your machine learning model.
A dataset collected from a healthy production line is 97-99.9% images of good parts. A supervised classifier trained on this dataset without any correction for the imbalance will quickly learn that calling everything "good" gives it 99% accuracy. The model is technically accurate by the number on the label while being completely useless for its actual purpose. This is the class imbalance problem, and it's the most common reason first-generation defect detection models disappoint in production.
This article covers the three most effective strategies for handling class imbalance in defect detection, and some honest notes on where each one falls short.
Understanding what the model is actually learning
Before getting into solutions, it's worth being precise about what happens mechanically when a model trains on imbalanced data. The loss function being minimized during training is summed across all examples. In a 99:1 dataset, 99 good-part images contribute to every gradient update while 1 defect image contributes to every 100th update. The model's internal representations and decision boundaries are overwhelmingly shaped by what a good part looks like, not what a defective part looks like.
When you then ask this model to classify a defective part, you're asking it to make a decision based on features it has barely been trained to recognize. Its confidence in the "defect" class is low because it has seen relatively few defect examples, and the decision boundary tilts toward "good" because that's the majority class that dominated training. Raising the classification threshold doesn't fix this; it just moves the operating point on a poorly calibrated probability surface.
Oversampling and undersampling
The most direct intervention is to rebalance the dataset. Undersampling means randomly removing good-part examples until the class ratio is more balanced. Oversampling means duplicating defect examples (or generating synthetic variants) until the defect class has more representation.
Random undersampling is fast and simple but discards information about good-part variation that could help the model generalize. If your good-part images cover a range of lighting conditions, fixture positions, and surface textures, undersampling reduces the model's exposure to that variation, which typically increases false positives on good parts that look slightly unusual.
Random oversampling of defect examples is easy to implement but doesn't add new information: you're showing the model the same defect images multiple times. This can lead to overfitting on the specific defect examples in your training set, meaning the model performs well on examples that look exactly like your training defects but poorly on novel defect manifestations (a crack that's slightly wider, a void in a different location on the part).
Synthetic oversampling techniques (SMOTE in feature space, or more relevantly for vision, augmentation-based approaches like random rotation, crop, brightness variation, and synthetic defect injection) do add information. Augmentation generates new images that the model hasn't seen; synthetic defect injection adds digitally rendered defect patterns to good-part images to produce labeled training examples. Both reduce overfitting on specific defect examples compared to direct duplication.
Cost-sensitive learning
Rather than rebalancing the dataset, you can modify the loss function to penalize misclassification of the minority class more heavily. A misclassified defect (false negative) contributes a higher loss penalty than a misclassified good part (false positive). This directly encodes the asymmetry of the business problem - missing a defect is more costly than a false alarm - into the training objective.
Cost-sensitive learning is the approach we prefer for most deployments because it avoids the information discarding problem of undersampling and the overfitting risk of naive oversampling. The cost ratio is a tunable hyperparameter that can be set to match your actual false-negative to false-positive cost ratio, rather than an arbitrary rebalancing target.
The limitation of cost-sensitive learning is that it requires more careful validation. Accuracy is not a useful metric for evaluating a cost-sensitive model (a model that calls everything "defect" has high recall but is useless). You need to validate on precision-recall curves and set the operating threshold based on the business-relevant tradeoff at your specific line. That's more sophisticated than reporting a single accuracy number, but it's a more honest characterization of model performance.
Anomaly detection as an alternative framing
When defect examples are extremely scarce (fewer than 50 labeled defect images), supervised classification may be impractical regardless of rebalancing strategy. The alternative is anomaly detection: train a model only on good-part images to learn the distribution of "normal," and flag any part whose image is a statistical outlier from that distribution.
Anomaly detection doesn't require any defect-labeled training data. The model learns what a good part looks like and calls anything that deviates significantly "anomalous." In practice, this produces a higher false positive rate than well-trained supervised models because normal part variation (from different lighting angles, material lot variation, fixturing differences) can trigger the anomaly detector. Calibrating the anomaly threshold to acceptable false positive rates while maintaining sensitivity requires careful tuning on a held-out good-part set that spans your production variation.
Anomaly detection is a good approach for new part numbers where you have zero or very few defect examples, or for catching truly novel defect types that fall outside your trained defect taxonomy. It's best used as a first-pass detector that routes uncertain cases to human review, rather than as a primary binary classifier. As defect examples accumulate, transitioning to supervised classification with those examples typically improves precision at equivalent recall.
Evaluation strategy for imbalanced problems
Regardless of which approach you use for handling imbalance, report precision and recall separately, not aggregate accuracy. In an inspection context, recall (what fraction of actual defects does the model detect?) is your primary KPI. Precision (what fraction of defect alarms are real defects?) determines your false positive rate and line stoppage frequency.
Both matter, but they trade off against each other through the classification threshold. The right threshold setting is the one that puts you at an acceptable operating point on your precision-recall curve, not the default 0.5 probability cutoff from a binary classifier. Set it based on your actual tolerance for missed defects versus false alarms - and re-evaluate it each time your production conditions change.