The day the traffic count hit zero

The brief sounded simple: watch a city's traffic cameras and react on its own. Count vehicles, spot congestion, and raise an alarm the moment there's an accident, a fire or a fight, without a person staring at a wall of screens.

The models were the easy part. YOLOv8 for incidents, ByteTrack to follow vehicles from frame to frame. With one or two cameras on a dev machine, everything worked.

Then we added more cameras.

Roads that were obviously busy reported almost no vehicles. Nothing crashed and no errors were logged. The numbers just quietly became wrong, which is the worst way for a system to fail.

The model wasn't the problem. Frames were never reaching it.

A tracker like ByteTrack needs a continuous run of frames to follow a car across the junction. When frames go missing, a car can appear once and vanish, and a car seen once is never counted.

So the question changed from "why is the model wrong?" to "where are the frames going?"

Changing the queue changed nothing, which pointed upstream, to how the frames were being read in the first place.

OpenCV's VideoCapture.read() is synchronous. Each call blocks until the next frame is decoded. One stream at 25 fps is fine. With many streams, every reader sits waiting on its own decode, falls behind, and the camera's buffer overflows. Frames are silently dropped, or arrive late and stale.

The bottleneck was never the broker. It was a blocking read loop at the very front of the pipeline.

GStreamer works the other way round. It is event-driven: you describe a pipeline (source → depay → decode → convert → sink), and it calls you when a frame is ready (appsink → new-sample). Decoding runs inside GStreamer's own threads, and the Python side just receives finished frames.

It still took tuning to hold up against real city cameras.

• rtspsrc over TCP with latency=0. Reliable delivery, and no jitter buffer quietly piling up old frames.
• A fixed frame rate and a queue between decode and the sink, so a slow consumer can't stall the decoder.
• Self-healing streams. Each pipeline watches its bus; when a camera drops off it is torn down and restarted after a pause. Flaky cameras were tracked and kept out of the main pool.

Fixing the reads exposed the next problem: sheer volume.

No queue or network link is built for that. So frames are compressed to JPEG at the source with TurboJPEG, before they ever touch the broker. Each frame shrinks from ~3 MB to 150–200 KB, and the fleet goes from ~7.5 GB/s to roughly 0.4–0.5 GB/s. The GPU side decodes them back for inference.

An event-driven producer/consumer system, split across servers.

• Producers (ingest server). One GStreamer process per camera, each decoding, compressing and publishing to its own RabbitMQ queue.
• Broker. A queue per camera, so the producing and consuming sides scale and fail independently.
• Consumers (GPU server). One consumer per camera runs YOLOv8 + ByteTrack, and reconnects with a growing wait, up to 30 seconds, if the broker drops.
• Storage and outputs. Detections, counts and incident frames in MongoDB; stream health and summaries in MySQL; a Dash dashboard per location and alerts with the captured frame.

With frames flowing reliably, the risk flipped from missing incidents to raising false ones. A detector that's right 95% of the time still cries wolf all day across a hundred cameras.

So alerts come from sequences, not single frames: an incident is raised only when 22 of the last 25 frames agree above the confidence threshold, then a cooldown suppresses repeats for the same event. That costs a fraction of a second and removes the one-frame glitches.