INT8 Quantization Without Losing the Accuracy You Paid For

Quantization is the cheapest speedup available on embedded industrial hardware and the easiest way to silently break a working model. The failure mode is almost never catastrophic. It is a two-point drop in recall on the one rare defect class the customer actually bought the system to catch.

On a recent inspection line, INT8 cut inference from 41 ms to 13 ms per frame and let one edge box cover four cameras instead of one. It also, in the first attempt, stopped detecting hairline cracks under specular light. Both facts came from the same conversion. Only one of them showed up in the aggregate metric.

Calibration data is the whole job

Calibration sets the dynamic range of each tensor, so it must mirror production: the same cameras, the same lenses, the same lighting extremes, the same motion blur at line speed. A thousand frames sampled evenly across a full 24-hour cycle beats fifty thousand frames captured on one clean afternoon.

• Sample across shifts, so night lighting and daylight ingress are both represented
• Include the dirty states: condensation, dust on the housing, a part sitting slightly off-centre
• Over-represent rare defect classes relative to their natural frequency
• Never calibrate on augmented or synthetic frames the sensor could not produce

Mixed precision is not a compromise

Keep the detection head and any layer with wide dynamic range in FP16. On most detector architectures that costs two to four milliseconds and removes the majority of the accuracy regression. Measure per-layer sensitivity rather than guessing: quantize one block at a time, evaluate, and record the delta.

Attention blocks, the final regression layers, and anything immediately after a normalization layer with large activation spread are the usual offenders. Excluding four layers out of two hundred typically recovers the accuracy while keeping ninety percent of the speedup.

Prove it per class, not in aggregate

If the delta is invisible in the aggregate but ugly per class, the aggregate was the wrong metric.

Ship nothing without a differential evaluation: run FP32 and INT8 over the identical held-out set and compare per-class precision and recall, plus the confidence distribution. A quantized model that keeps the same mAP but shifts its confidence distribution left will break every threshold the operators were trained on.

Add a golden-set regression test to CI. Fifty frames with known outcomes, run on every engine rebuild, failing the build on any per-class regression beyond a stated tolerance. Engine files get rebuilt whenever the driver, the hardware, or the TensorRT version changes, and each rebuild is a chance to quietly lose accuracy.

Budget for thermals, not just for throughput

An edge box in a control cabinet at 45 degrees ambient does not sustain bench numbers. Measure after an hour at full load, with the cabinet door closed, in the season the plant runs hottest. A model that hits 30 ms on a cold bench and 52 ms in a warm cabinet is a model that misses parts every afternoon.

• Lock clocks and power mode; do not benchmark with dynamic boost enabled
• Report sustained p99 latency after a 60-minute soak, never the first-minute average
• Leave 30 percent headroom for future classes, higher resolution, and a second model

Quantization done properly is boring: a measured speedup, a documented per-class delta, a regression test that fails loudly. That is the whole deliverable.