Encoder latency and jitter: where it comes from and how to measure it

How it works

The latency budget, stage by stage

End-to-end contribution delay is the sum of six stages, each with a different variance profile. Treating them separately is what makes a budget testable.

Capture and input conditioning. Frame arrival at the encoder’s input, colour conversion and scaling. On a hardware input this is fixed by design — one frame period plus a small fixed offset. On a software encoder taking frames from a capture card through the host kernel, this stage inherits the host’s interrupt and scheduling behaviour and becomes the first source of variance.

Look-ahead and rate control. The largest single contributor in most compressed workflows, and the most commonly misattributed. A three-frame look-ahead window at 50 fps is 60 ms of delay before a single bit is encoded. B-frame reordering adds further frames. This delay is deterministic — it is a design choice, not a symptom — which is why disabling look-ahead is the first lever in any low-latency configuration and why it costs 5–15% bitrate at equivalent perceptual quality.

Core encode. Transform, quantisation, motion estimation, entropy coding and in-loop filtering. On a fixed-function pipeline this is constant per frame. In software it varies with content: a scene cut, a crowd shot or a fast pan raises per-frame complexity and the encode time with it. This is why measured jitter rises during exactly the moments a live production cares about.

Packetisation and transport preparation. MTU fragmentation, FEC generation, encryption. Small per frame, but it scales with packet rate rather than bitrate, so it grows faster than expected at high channel counts.

Audio path and alignment. Frequently the dominant term in a low-latency budget and routinely left out of it. Transparent passthrough of already-embedded audio costs almost nothing, but any path that de-embeds, buffers and re-packetises audio essences separately from video adds tens to low hundreds of milliseconds, and compressed audio encoding adds more again. Because video must wait for audio alignment, the audio decision can set the floor for the whole chain regardless of how fast the video encoder is. Measure the budget with the audio mode you intend to run.

Network transit and protection buffer. For SRT or RIST this is dominated by the configured recovery buffer, not by propagation. It is a deliberate budget allocation.

Decode and output. Reference buffer fill, reordering and output timing alignment.

Why architecture shows up in the variance, not the mean

Three structural properties determine how much a platform’s delay moves under load.

Scheduling. A fixed-function pipeline has no operating-system scheduler between frames. Memory access is scheduled at design time, so cache misses and contention do not appear. A software encoder shares cores with the transport stack, the monitoring agent and any co-tenant, and every context switch is a candidate for a late frame.

Memory access pattern. Deterministic pipelines with on-chip buffers hold a constant per-frame memory profile. Software encoders at high stream counts develop random access patterns and cache pressure, and latency rises non-linearly as memory bandwidth approaches the device ceiling.

Device boundary crossings. GPU encode adds kernel launch overhead, PCIe transfer time and contention between compute and encoder blocks. Aggregate throughput can be very high while per-stream worst-case timing stays unpredictable, because those three costs vary with what else the device is doing.

The practical consequence: the same platform will produce very different jitter figures at 10% and 90% utilisation. Any measurement taken on an idle system is close to worthless for capacity planning.

Why it matters in broadcast

Switching deadlines are hard. A production mixer expects every source to present a frame within its switching window. A source that is late has no graceful degradation — the cut takes the previous frame or the wrong one. Because worst-case behaviour is what breaks the show, the p99 figure is the specification that matters and the mean is decoration.

Lip-sync tolerance is finite, cumulative and asymmetric. Perception thresholds are not symmetric: audio running ahead of video is detectable at around 45 ms, while audio running behind is tolerated to roughly 125 ms (ITU-R BT.1359). Specify the budget in the direction that binds, and remember that contribution delay variance consumes it before the production chain has spent any of its own.

Multi-source alignment compounds the problem. What sizes the alignment buffer is the spread between the earliest and latest feed, not the jitter of any one of them: four encoders each varying over a 20 ms window, independently, can present one feed at the bottom of its range and another at the top, so the buffer has to cover the full 20 ms spread plus margin — and more if the feeds differ in mean delay as well. State whether a quoted jitter figure is peak-to-peak or ±, because the two differ by a factor of two and the buffer is sized from the wrong one surprisingly often.

Acceptance testing is where this becomes commercial. Procurement that specifies “sub-100 ms encoder latency” and does not specify the measurement method, the load condition or the percentile has not specified anything enforceable.

Technical specifications and trade-offs

What to specify instead of a single number

A testable latency specification names five things:

Element	Example specification
Measurement point	Input frame boundary to first byte of encoded frame at the output interface
Load condition	All channels active at target codec, resolution and bitrate, sustained 60 minutes
Content profile	Representative source including scene cuts and high-motion sequences
Reported statistic	p50, p95 and p99 per-frame delay, plus maximum observed
Configuration	Look-ahead depth, GOP structure, latency mode, FEC and encryption state

Any figure quoted without all five is not comparable across vendors.

Three measurement methods

Timecode burn-in with a high-speed camera. The reference method for true glass-to-glass measurement. A pattern generator inserts a visible timecode or a millisecond counter; a camera at 240 fps or faster captures source and decoded output in the same frame; the difference is read off directly. Measures the whole chain including displays. Accurate to roughly one capture-frame period — about 4 ms at 240 fps — which is sufficient for chain-level budgets but too coarse to characterise encoder jitter alone.

RTP timestamp differencing. Capture packets at the encoder output and at the decoder input, compare RTP timestamps against arrival times, and derive per-frame transit and buffering behaviour. Isolates the transport stage cleanly and produces a full distribution rather than a spot value. Requires PTP-disciplined capture points or the clock offset contaminates the result.

Encoder-internal telemetry. Where the platform exposes per-frame queue latency, encoder occupancy and output timestamps through its API, this is the highest-resolution view of the encode stage and the only practical way to correlate a latency spike with the frame that caused it. It cannot substitute for an end-to-end measurement because it cannot see what the platform does not instrument.

Use at least two. A discrepancy between internal telemetry and RTP differencing is usually the transport buffer doing something the configuration did not describe.

Interpreting the distribution

• A tight distribution with a high mean is a configuration problem — look-ahead, GOP length or an oversized recovery buffer. It is fixable without changing hardware.

• A low mean with a long tail is a contention problem — scheduling, memory bandwidth or device sharing. It is not fixable by configuration alone and gets worse with channel count.

• A distribution that is tight at low load and long-tailed at high load tells you the headroom policy. Keep sustained utilisation below the point where the tail begins to extend; roughly 70% of peak throughput and 60% of peak packet rate is a workable starting rule, then measure.

• Periodic spikes at a fixed interval are almost always housekeeping — garbage collection, log rotation, telemetry flush or a scheduled key rotation.

Where the numbers usually go wrong

Measured latency that exceeds the datasheet figure by a large margin is most often explained by one of four things: look-ahead left at the default rather than the low-latency preset; a recovery buffer sized for a worse link than the one in use; the decoder’s output alignment buffer, which is frequently omitted from vendor figures; or measurement taken with a still test pattern, which is the least demanding content a rate controller can be given.

Related approaches and standards

• SMPTE ST 2110-21 defines sender traffic-shaping models and network compliance, and gives the vocabulary for describing how a sender’s packet timing behaves — directly relevant when the measured jitter originates in packetisation rather than encode.

• IEEE 1588 PTP and SMPTE ST 2059 provide the common time reference without which multi-point measurement is not meaningful.

• JPEG XS operates at sub-frame codec latency, which changes the shape of the budget entirely: transport and buffering, not encode, become the dominant terms.

• Low-delay AVC and HEVC profiles constrain reference structures to remove reordering delay; the bitrate cost is measurable and should be quantified during the same test.

• VMAF, SSIM and PSNR should be captured alongside latency during acceptance testing, because every latency reduction has a quality cost and a test that measures only one of them will produce a configuration nobody wants to run.

How Appear addresses this

Encoding on the X Platform runs in dedicated encoder silicon and FPGA rather than on a general-purpose CPU, so the delay does not move with host scheduling, co-tenant load or memory contention — the variance sources described above largely do not apply. The ECx module family — ECx110 for SDI I/O, ECx210 for ST 2110 and ST 2022-6 IP I/O — encodes AVC and HEVC with selectable latency modes down to ultra-low delay, so look-ahead is an explicit configuration choice rather than an inherited default.

Fixed pipeline depth is not the same as a guaranteed figure, and the distinction matters for an acceptance test. Measured latency on these modules still depends on the input path and on what the frame is carrying — ancillary data arrival time and type can move a ULL result, and the audio mode chosen can move it further than the video configuration does. Treat published figures as the specification and your own measurement under representative content as the acceptance criterion, which is the position this article argues for generally.

On instrumentation, be clear about what is available. The X Platform publishes Prometheus metrics per chassis and VX publishes per-step metrics continuously through its workflow model, so a software transport leg can be instrumented to the same standard as a hardware encode leg. Those metrics are flow-level rather than per-frame, so the encoder-internal method above is not the one to plan around: the timecode and RTP-differencing rigs carry the measurement, and the platform telemetry corroborates it.

For the platform-selection question this article deliberately leaves open — hardware, software, cloud or hybrid — see Comparing cloud/software and hardware encoding for live video contribution.