RTSP Streams That Don't Drop Frames

A detection model that scores 0.94 mAP on a benchmark is worth nothing on a packaging line if the stream feeding it stalls for eleven seconds every time a floor switch reboots. On every plant deployment I have worked on, the first two weeks went to transport and timing, not to the model. The model was already good enough. The pipe around it was not.

Industrial networks are hostile in specific, repeatable ways: PoE switches shared with PLC traffic, cameras with firmware that renegotiates RTP ports at random, cable runs next to variable-frequency drives, and a maintenance crew that will unplug something at 02:00 without telling anyone. Design for that environment and the vision part becomes routine.

Measure the transport before you touch the model

Before any optimization, instrument ingestion. For each camera, log the arrival timestamp of every frame, the decoder queue depth, the presentation timestamp gap, and the RTP packet-loss counter. Run it for a full production cycle, including shift change and cleaning.

The shape of the jitter tells you where the fault lives. Periodic spikes at a fixed interval are almost always keyframe intervals colliding with a bandwidth ceiling. Random multi-second gaps are network or firmware. A slow, monotonic rise in queue depth is your own consumer being too slow, and no amount of network tuning will fix it.

Bound every queue and drop the stale head

The single most common bug in production vision systems is an unbounded buffer between decode and inference. It hides the problem during commissioning and then, three weeks in, an operator reports that the reject arm fires on the wrong part. The model is fine. It is looking at a frame from four seconds ago.

Real-time systems must degrade by skipping, never by accumulating latency. A bounded queue of eight to ten frames that discards the oldest entry keeps end-to-end latency flat under load and makes back-pressure visible as a drop counter you can alert on.

If your latency grows under load instead of your drop count, you did not build a real-time system. You built a recording.

Decode where the hardware is strongest

On multi-camera installations, CPU decoding is usually the first wall. Hardware decoders on the GPU or SoC handle sixteen 1080p H.264 streams where software decode collapses at six. Keep the decoded frame in device memory and hand the inference engine a pointer instead of copying to host and back.

• Use hardware decode (NVDEC, V4L2 M2M) and keep frames on-device end to end
• Pin one decoder thread per camera; never multiplex cameras onto a shared thread
• Prefer TCP transport on noisy plant networks; UDP loses I-frames and artefacts cascade
• Set a short socket timeout, and treat a timeout as a state change rather than an exception to swallow

Reconnection is a state machine, not a retry loop

Treat each camera as a small state machine: connecting, streaming, degraded, failed. Transitions emit telemetry with a reason code. Back off exponentially, cap the backoff, and never let a reconnect storm from twelve cameras hit a switch at the same instant.

When a plant manager asks why camera 7 went quiet at 03:14 during the night shift, the answer should be a log line with a reason code, not an engineer reconstructing the evening from memory. That single discipline has resolved more supplier disputes than any accuracy metric I have shipped.

What to verify before you call ingestion done

• 24-hour soak across a real production cycle, including cleaning and shift change
• Deliberate failure injection: pull a cable, reboot a switch, power-cycle a camera
• p99 end-to-end latency measured from sensor exposure to business event, not just model time
• Drop counters exported as metrics and visible on the same dashboard as line throughput

Get those four right and the rest of the system stops being mysterious. Accuracy problems become accuracy problems again, instead of timing problems in disguise.