vsourcestream_web_logo

VSourceStream C++ library

v1.4.1

Table of contents

Overview

VSourceStream is a C++ library that receives video from network sources with minimum latency. It is made for applications built on the RapidPixel SDK. The VSourceStream class inherits the open source VSource interface class. So it works like the other video source libraries of the SDK, for example VSourceV4L2: an application written for the VSource interface uses VSourceStream through the same methods. Only the source string differs, and the fourcc of the Frame object selects the output format (see Output below).

Sources. The library receives RTSP streams (RTP over UDP unicast, UDP multicast, TCP interleaved or an HTTP tunnel) and RTSPS streams (RTSP over TLS, with RTP inside the TLS connection, plain UDP, SRTP or an HTTPS tunnel). It also receives SDP-less RTP and SRTP push streams, which a sender sends without RTSP signalling, and raw MPEG-TS datagrams (STANAG 4609 / MISB ST 1402). The video codec is H.264 (RFC 6184), H.265 / HEVC (RFC 7798) or JPEG (RFC 2435). The video is carried directly in RTP. H.264 and H.265 can also be carried in MPEG-TS, either inside RTP (RFC 2250) or in plain UDP datagrams.

Output. getFrame(…) returns the frames in arrival order, at the instant each frame is complete. The caller selects the output with the fourcc (the four-character format code) of the Frame object, as with the VSourceFile library. With H264, HEVC or JPEG the method returns the complete, decodable encoded frames: H.264 and H.265 frames in Annex-B format, JPEG frames as complete JFIF images. With a pixel format (RGB24, BGR24, YUYV, UYVY, GRAY, YUV24, NV12, NV21, YU12 or YV12; 1.3.0) the method returns decoded pictures in this format. A separate decode thread decodes every frame the instant it is complete. It uses the built-in decoders VDecoderH264, VDecoderH265 and VDecoderJPEG, or a codec given by setCodec(…). FormatConverter converts the pictures to the requested format. See Frame output and decoding.

Public interface. The public interface is the VSource interface plus two methods. getTelemetry(…) returns the telemetry of the frame that the last successful getFrame(…) returned: frame ID, key frame flag, 90 kHz timestamp, capture time on the sender’s clock, receive time on the local clock, the number of frames lost before this frame, the packet loss of the source (packets lost since initialization and the lost share of the last 10 seconds, 1.4.0), and the metadata bound to the frame. The metadata is the KLV (SMPTE 336M, for example MISB ST 0601; RFC 6597) and ONVIF metadata carried by the stream. With the -seidata option it also contains the SEI user data of the video, as units of type UNKNOWN. A frame without metadata has an empty metadata vector. Each metadata unit is bound to its video frame by its timestamp. setCodec(…) (1.3.0) replaces the built-in decoders with a custom video codec. Runtime statistics are not part of the public API: they are internal diagnostics for the test application and service tools (see Internal diagnostics).

Header files. The VSource.h file contains the VSourceParams class (video source params, with methods to serialize and deserialize them), the VSourceCommand enum (action commands) and the VSourceParam enum (param IDs). The VSourceStream.h file contains the VSourceStream class declaration and the telemetry data structures: the VSourceStreamTelemetry structure, the VSourceStreamMetadata structure and the VSourceStreamDataType enum.

Dependencies. VSourceStream depends on the VSource interface class (Apache 2.0 license), the VDecoderH264, VDecoderH265 and VDecoderJPEG libraries (software H.264, H.265 and JPEG decoders with NV12 output), the FormatConverter library (pixel format conversion) and the CryptoProvider library (cryptographic primitives and random values). All of them are included as git submodules. The system OpenSSL library, version 3.0 or newer, is optional: it provides TLS. Without OpenSSL the library is built without TLS, so rtsps:// sources can’t connect (see Build and connect to your project).

Compliance modes. Like the reference video server MServer, the library has four compliance modes. ONVIF (the default) is the interoperability mode. STRICT_FIPS and FIPS_ONVIF_BRIDGE apply the FIPS 140-3 restrictions and use the OpenSSL FIPS provider when it is installed. CRA_STRICT is the secure-by-default configuration of the EU Cyber Resilience Act. See Security and compliance modes.

Threads. Every VSourceStream object has one internal receive thread. It performs all network input / output, protocol handling and frame assembly. While decoded frames are requested, the object also runs a decode thread (see Frame output and decoding). No user code runs on these threads, except a custom codec on the decode thread.

Latency. The library is optimized for minimum latency (see Low latency design and adaptive jitter buffer and Measured results). It also adapts to the channel. Its jitter buffer follows the observed packet reordering. It repairs lost packets with RTCP NACK / RTX when the server offers this (see NACK, RTX and key frame requests). An RTSP session switches between UDP and TCP on network loss or on a growing TCP queue (see Adaptive transport).

Platforms. The library uses the C++17 standard and builds on Linux and Windows. Some low latency features work on Linux only (see Receive thread, kernel timestamps and scheduling).

Versions

Table 1 - Library versions.

Version Release date What’s new
1.0.0 26.09.2026 - First version.
1.1.0 28.09.2026 - TCP_QUICKACK re-armed after every read of an RTSP / RTSPS / HTTP tunnel connection (no delayed-ACK stall behind Nagle senders).
- Receive thread named “vsrtp-rx” with 1 ns timer slack; new -priority and -cpu options.
- Kernel receive timestamps for UDP (jitter, reorder lateness, frame arrival time, CUSTOM_1); kernel receive buffer drops counted (kernelDrops).
- Socket backlog read before reorder and hold deadlines expire.
- Idle-gap frame end for senders without a reliable RTP marker and for MPEG-TS frames without an end signal.
- Opt-in zero-copy frame callback on the receive thread (setFrameCallback()).
- Output latency budget (-maxlatency).
- RTCP random values drawn in batches.
- Adaptive jitter buffer: peak held for 10 s then decayed, jitter term removed, key-frame bypass of the reorder wait, default minimum wait 10 ms (3 ms with -nokeywait).
- AU-boundary loss rule: a loss between two frames no longer costs the next frame when that frame provably starts after the loss.
- Adaptive RTSP transport (UDP <-> TCP), on by default for rtsp:// and rtsps:// without a transport option (-autotransport / -noautotransport).
- Bounded key-frame wait (-keywait).
- NACK wait held for 10 s and given up after 8 unanswered gaps; key frame re-request paced by the observed response time.
- Channel quality statistics: lossPercent, reorderPercent, framesFrozenMsec, kernelDrops, transportSwitches, queueDelayUsec, metadataCarried, metadataDropped, callbackOverruns, framesLatencyDropped.
- Metadata binding reworked (EXACT, SINCE_PREVIOUS, CARRIED, UNTIMED): units are no longer lost because of frame timing; metadataFormat lists every format seen.
- New getFrameInfo() and getUserDataUnits() methods: timestamps, PTS, kernel arrival time, capture time (MISB ST 0604 SEI, RTP header extension, RTCP SR), per-unit format, binding, stream and validation flags.
- SEI user data units (-seidata), up to 2 RTSP metadata tracks, MPEG-TS KLV robustness (raw KLV, cells, several PES and PIDs), NAL-aligned MPEG-TS access units, RTP metadata without marker.
- Test application: 12 new groups, reference server MServer 2.1.0 (NACK / RTX, PLI / FIR).
1.2.0 30.09.2026 - Public API simplified: one additional method getTelemetry() replaces getUserData(), getFrameInfo(), getUserDataUnits(), getStats() and setFrameCallback(); frame callback removed; statistics moved to internal diagnostics.
- NACK retry after half the reorder wait (after the server’s expected answer when it answers later).
- RTCP sender report pairs whose clock readings are more than 100 ms apart are refused for the metadata clock mapping.
- Metadata read order learned from kernel arrival times.
- No capture time from an RTCP sender report with a relative (uptime) NTP time.
- Bounded telemetry buffer retention (at most 64 KiB kept beyond a unit).
- Tail-loss probe: a lost last packet of a frame is requested with a NACK after a short learned silence instead of when the next frame shows the gap.
- Tests with MServer 2.1.1.
1.3.0 02.10.2026 - Decoded output: the fourcc of the frame given to getFrame() chooses the output, as with VSourceFile: H264 / HEVC / JPEG - the encoded frames as received (fourcc corrected to the stream’s codec, also in VSourceParams::fourcc), RGB24, BGR24, YUYV, UYVY, GRAY, YUV24, NV12, NV21, YU12, YV12 - decoded pictures (a default Frame is YUV24).
- Built-in decoders VDecoderH264, VDecoderH265 and VDecoderJPEG of every object (2 decoding threads, limited range pictures expanded to full range only when they are limited range) and FormatConverter (2 threads); the settings are fixed.
- Decoding on a separate decode thread the instant a frame is complete, never in getFrame(); a second output buffer for the decoded pictures.
- Every decoded picture keeps the telemetry and metadata of its own frame, also with B-frames (display order); undecodable leading pictures of open GOP random access points dropped where decoding starts.
- New method setCodec(): a custom video codec replaces the built-in decoders.
- Test application: new group “decode”; built only on request (see Test application).
1.3.1 03.10.2026 - Test application: new group “codec” (setCodec()) and regression checks for every fix below; test codec VCodecLibav taken from ../VCodecLibav (else from MServer’s harness/VCodecLibav).
- Test application and example compiled with the warning flags of the library.
- Windows: builds with MSVC (the library’s own sources also with MinGW-w64; native threads, Winsock poll, CNG crypto backend); multicast reception and -priority / -cpu on Windows.
- Start: an RTP-Info seed is provisional, so the first key frame is no longer lost when the first packet differs from the announced sequence number.
- Transport: the UDP -> TCP fallback comes before the no-media watchdog (also with -reconnect below 3 s), a silent multicast session falls back to UDP unicast, the watchdog follows the video only, DESCRIBE retries are paced, the HTTP tunnel sends a Host header and keeps within its POST Content-Length, interleaved sessions close 50 ms after their TEARDOWN.
- Jitter buffer and repair: the reorder peak is held only by reordering of at least half its size; a fast server’s NACK retry after twice its answer delay; with RTX a reordered original no longer counts as a NACK answer.
- Metadata: KLV at a multiple of the frame rate is bound EXACT; carried units never come with an earlier frame; MPEG-TS KLV items split across PES keep their own bytes; marker-less ONVIF documents are delivered at once.
- Decoding: a codec that gives no picture for 32 frames is restarted; with B-frames decoding restarts at the key frame after damaged frames, so the telemetry binding stays exact; decoded pictures without a zero fill; buffer reuse for up to 16 caller Frame objects.
- Key frame requests: paced only between repetitions of one request; with -nokeywait the complete frame after a loss is delivered.
- The idle-gap AU end learns again after a misprediction; an H.264 / H.265 sender that stops setting the marker is detected.
- Source string: TLS options only with rtsps://, -iface 0.0.0.0 refused.
- Crypto: FIPS claims only with proof (SHA2-256 resolves, MD5 refused, DRBG probe), no library context freed while in use, HMAC keys wiped.
1.4.0 03.10.2026 - Telemetry: packet loss of the source in every VSourceStreamTelemetry: lostPackets (packets lost since initVSource(), never decreasing) and packetLossPercent (lost share of the last 10 s, in 1 s steps), also for raw MPEG-TS (by the continuity counters).
- Metadata types: VSourceStreamDataType NONE (a unit without data, the default; never delivered), KLV, ONVIF and UNKNOWN; the SEI user data of -seidata come as UNKNOWN (the SEI type is removed).
- Security: an SPS above the supported geometry in a running stream is refused: it is never cached nor forwarded, and the access unit that carries it is dropped. Before, the previous geometry stayed in force, so the SPS reached the application with a wrong geometry and reached the built-in decoders, which allocated pictures of its size. Decoded pictures above the maximum are dropped too.
- Linker hardening options declared PUBLIC (also for a shared library build).
- Diagnostics: packetsFiltered counts the datagrams the sender filter dropped (UDP media not from the RTSP server, SDP-less media not from the -source sender); the first dropped datagram of an SDP-less source is logged as a WARNING, as it already was over RTSP.
- Test application: the gst-rtsp-server launcher works in a terminal of an editor installed as a snap (VS Code); the C1 gst-rtsp-server KLV case runs with GStreamer 1.20, whose tsdemux has no synchronous KLV.
- Test application: the matrix group (every MServer delivery mode but HLS and WebRTC: codec x container x metadata x client path, the push leg’s containers and synonyms, payload bounds, authentication on every path, rtspEncryption strict, CRA_STRICT with MPEG-TS, a shared port) and the live group (MServer changes under a connected client, a session beyond the 60 s timeout, 7 concurrent clients); the compliance group follows MServer 2.1.3, whose FIPS profiles refuse to start without the validated module.
- The library was renamed VSourceRtp -> VSourceStream (the class, the data structures, the namespace, every header and the CMake project; the VSOURCE_RTP_* macros became VSOURCE_STREAM_*).
- Robustness: an uncaught exception on the receive or decode thread (for example std::bad_alloc) no longer calls std::terminate; the thread ends cleanly, is logged, and getFrame() times out.
- Build hardening: the built-in decoders (VDecoderH264/H265/JPEG) and FormatConverter compile with -fstack-protector-strong / _FORTIFY_SOURCE (MSVC /GS /guard:cf) and no longer use -march=native, so builds are reproducible.
- Fuzzing: libFuzzer targets for the attacker-controlled parsers (SPS, SDP, MPEG-TS / KLV) under test/fuzz, built with -D_VSOURCESTREAM_FUZZ=ON on a clang toolchain (EU CRA 2(h)/2(k)).
- Annex I Part II material added: SECURITY.md, SECURITY-ADVISORIES.md and an SBOM (sbom.cdx.json).
- VDecoderH264 (submodule): builds with GCC 14 and Clang 19 and newer, which have the generic bit-scan builtins (__builtin_clzg / __builtin_ctzg); no pthread.h in its non-MSVC Windows branch.
1.4.1 09.10.2026 Security review (EU CRA / FIPS 140-3 follow-up). The public interface is unchanged. The behaviour changes – all of them for invalid or hostile input, none for a valid stream – are marked “behaviour change”.
- MPEG-TS: splitting a PES at a codec boundary costs time linear in the bytes of the access unit (a PES of 4 MB of tiny units took 204 s, now 49 ms); the delivered frames are bit-identical.
- Parameter sets: the cache of VPS / SPS / PPS is bounded in total (64 KiB; a sender could force 4.7 MB into every key frame, 16 ms each) and the check of the sets a key frame references is linear. Behaviour change: a set that does not fit is refused and the key frames that need it wait.
- TLS (RTSPS, HTTPS tunnel): a server that flooded KeyUpdate messages without reading, or cut a post-handshake message in the middle, made the receive thread spin at 100 % CPU; the poll loop now waits for the event the TLS layer needs, and data the TLS layer read ahead during a pending write is read when the socket turns writable.
- RTSP: a hostile server flooding the connection cannot starve the other work of the receive thread (bounded read pass); unsolicited responses and server requests are logged at most four times per connection; an unknown option of the source string is named by its position, so no part of a password that holds a space reaches the log.
- RTCP and RTP sources: in SDP-less reception (rtp://, udp://, srtp://) RTCP reports go only to the address of the RTP packets of their SSRC (a forged sender report could aim receiver reports, NACK and PLI at any host; behaviour change: a sender whose sender reports come from another address than its RTP packets gets no receiver report, NACK or PLI); behaviour change: new RTP sources of every session are limited to a burst of 8 and one per second (a recycled SSRC cost 13 us per RTP + BYE pair); behaviour change: RTSPS with SRTP refuses the same key under two suites (one SRTCP key stream would protect two receivers).
- Decoder: an exception that ends the decode thread no longer leaves getFrame(), closeVSource() and the destructor waiting for ever; the decode thread yields to a waiting setCodec(), and the picture of the old codec is released before the new one is used.
- API: getFrame() answers a failing allocation of the output frame, getTelemetry() one of its unit buffers and initVSource() any failing allocation with FALSE; the destructors cannot throw; the log line of a thread that ends by an exception cannot throw again.
- Bitstream: an H.265 key frame is an IRAP picture of the base layer (behaviour change: an IRAP NAL unit of another layer no longer opens the key frame gate); an ONVIF metadata root name is no longer truncated; the Huffman tables that the DHT segments of an ONVIF JPEG header redefine are kept only when the whole header is accepted (behaviour change: a refused header leaves nothing for the frames after it).
- ONVIF metadata (behaviour change): after a packet loss, and for the rest of a unit dropped for its start, every unit needs an unambiguous start (the XML declaration or the root tag), so the rest of a document whose head was lost is no longer delivered as fragments at inner elements; an SEI whose payload type or size is coded with a very long run of 0xFF bytes can no longer wrap into a recovery point and open the key frame gate.
- Hygiene: the digest hash of the credentials is wiped after use; the log file comment no longer names a product.
- Tests: 73 new checks (run against the code before the fixes, the tests fail 29 checks and the codec group hangs until its watchdog stops it); the codec group no longer depends on the fairness of a mutex; strict compiler warnings fixed in the library, examples and tests (GCC 9 to 12, Clang 18); the file tree lists TestLiveGroup.cpp and TestMatrixGroup.cpp.

Library files

The repository structure is shown below:

CMakeLists.txt ------------------------- Main CMake file of the library.
3rdparty ------------------------------- Folder with third-party libraries.
    CMakeLists.txt --------------------- CMake file to include third-party libraries.
    CryptoProvider --------------------- Folder with CryptoProvider library source code (git submodule).
    FormatConverter -------------------- Folder with FormatConverter library source code (git submodule).
    VDecoderH264 ----------------------- Folder with VDecoderH264 library source code (git submodule).
    VDecoderH265 ----------------------- Folder with VDecoderH265 library source code (git submodule).
    VDecoderJPEG ----------------------- Folder with VDecoderJPEG library source code (git submodule).
    VSource ---------------------------- Folder with VSource library source code (git submodule).
example -------------------------------- Folder for example application files.
    CMakeLists.txt --------------------- CMake file for example application.
    main.cpp --------------------------- Source C++ file of example application.
src ------------------------------------ Folder with library source code.
    CMakeLists.txt --------------------- CMake file of the library.
    VSourceStream.h ----------------------- Main library header file.
    VSourceStream.cpp --------------------- C++ implementation file of the VSourceStream class.
    VSourceStreamVersion.h ---------------- Header file with library version.
    VSourceStreamVersion.h.in ------------- File for CMake to generate version header.
    impl ------------------------------- Folder with internal implementation files.
        VSourceStreamBitstream.h ---------- H.264 / H.265 / JPEG bitstream helpers: NAL units, SPS, key frames, SEI, JFIF.
        VSourceStreamBitstream.cpp -------- Implementation of the bitstream helpers.
        VSourceStreamCompliance.h --------- Compliance modes and resolved client policy.
        VSourceStreamCompliance.cpp ------- Implementation of the compliance modes and statements.
        VSourceStreamConfig.h ------------- Source string configuration and parser declaration.
        VSourceStreamConfig.cpp ----------- Source string parser and static compliance check.
        VSourceStreamCrypto.h ------------- CryptoProvider integration, library context, FIPS provider, DRBG, secure buffers.
        VSourceStreamCrypto.cpp ----------- Implementation of the crypto context.
        VSourceStreamDecoder.h ------------ Decode stage: decode thread, decoders or custom codec, format conversion, display order.
        VSourceStreamDecoder.cpp ---------- Implementation of the decode stage.
        VSourceStreamDepacketizer.h ------- RTP depacketizers: H.264, H.265, JPEG and KLV / ONVIF metadata, idle-gap frame end.
        VSourceStreamDepacketizer.cpp ----- Implementation of the depacketizers.
        VSourceStreamDiagnostics.h -------- Internal diagnostics: runtime statistics for the test application and service tools.
        VSourceStreamDiagnostics.cpp ------ Implementation of the internal diagnostics.
        VSourceStreamFrameBuilder.h ------- Key frame gate, parameter set injection, metadata binder, queue delay.
        VSourceStreamFrameBuilder.cpp ----- Implementation of the frame builder and metadata binder.
        VSourceStreamFrameQueue.h --------- FIFO output buffer: overflow policy, latency budget, metadata carry, buffer recycling.
        VSourceStreamFrameQueue.cpp ------- Implementation of the output buffer.
        VSourceStreamImpl.h --------------- Implementation class of VSourceStream (pimpl; its methods are in VSourceStream.cpp).
        VSourceStreamLog.h ---------------- Logger (console and VSourceStream.log file).
        VSourceStreamLog.cpp -------------- Implementation of the logger.
        VSourceStreamLossMeter.h ---------- Packet loss share over the last 10 s (telemetry, 1.4.0).
        VSourceStreamLossMeter.cpp -------- Implementation of the packet loss window.
        VSourceStreamNet.h ---------------- UDP sockets (kernel timestamps and drops), TCP / TLS connections, host names.
        VSourceStreamNet.cpp -------------- Implementation of the network layer.
        VSourceStreamOpenSsl.h ------------ RAII wrappers of OpenSSL objects.
        VSourceStreamPipeline.h ----------- Media pipeline: RTP sources, codec auto-detection, RTCP.
        VSourceStreamPipeline.cpp --------- Implementation of the media pipeline.
        VSourceStreamPoll.h --------------- poll() on Linux and Windows (WSAPoll).
        VSourceStreamReceiver.h ----------- Receive thread: event loop, scheduling, reconnection, statistics.
        VSourceStreamReceiver.cpp --------- Implementation of the receive thread.
        VSourceStreamRtcp.h --------------- RTCP compound parser and builders.
        VSourceStreamRtcp.cpp ------------- Implementation of the RTCP parser and builders.
        VSourceStreamRtp.h ---------------- RTP packet parser, header extensions, receiver statistics, adaptive reorder buffer, NACK.
        VSourceStreamRtp.cpp -------------- Implementation of the RTP core.
        VSourceStreamRtsp.h --------------- RTSP client: requests, authentication, HTTP(S) tunnel.
        VSourceStreamRtsp.cpp ------------- Implementation of the RTSP client.
        VSourceStreamSdp.h ---------------- SDP parser.
        VSourceStreamSdp.cpp -------------- Implementation of the SDP parser.
        VSourceStreamSession.h ------------ RTSP / RTSPS session (adaptive transport) and SDP-less (direct) session.
        VSourceStreamSession.cpp ---------- Implementation of the sessions.
        VSourceStreamSrtp.h --------------- SRTP / SRTCP receiver and SDES key parser.
        VSourceStreamSrtp.cpp ------------- Implementation of SRTP / SRTCP.
        VSourceStreamThread.h ------------- Native thread: pthread on Linux, _beginthreadex() on Windows.
        VSourceStreamTs.h ----------------- MPEG-TS demuxer: access units, KLV extraction.
        VSourceStreamTs.cpp --------------- Implementation of the MPEG-TS demuxer.
        VSourceStreamTypes.h -------------- Shared types and limits.
        VSourceStreamUtils.h -------------- Clocks, NTP conversion, strict parsers, base64, string helpers.
        VSourceStreamUtils.cpp ------------ Implementation of the helpers.
static --------------------------------- Folder with static files.
    vsourcestream_web_logo.png ------------ Library logo.
test ----------------------------------- Folder for test application files.
    CMakeLists.txt --------------------- CMake file for test application (embeds MServer and VCodecLibav).
    CertTool.h ------------------------- Test PKI generator declaration.
    CertTool.cpp ----------------------- Test PKI generator (CA and server certificates).
    CraftedMedia.h --------------------- Crafted test media declaration.
    CraftedMedia.cpp ------------------- Frame index and MISB ST 0604 SEI insertion, KLV / ONVIF test units.
    FfmpegProbe.h ---------------------- ffmpeg reference client declaration.
    FfmpegProbe.cpp -------------------- ffmpeg reference client (latency, time to first frame).
    InteropSupport.h ------------------- Interoperability test helpers declaration.
    InteropSupport.cpp ----------------- Tool discovery, reference streams, gst-rtsp-server launcher.
    RtspImpairRelay.h ------------------ RTSP-aware impairment relay declaration.
    RtspImpairRelay.cpp ---------------- RTSP relay rewriting Transport ports, impaired UDP media.
    ScriptedRtspServer.h --------------- Scripted RTSP server declaration.
    ScriptedRtspServer.cpp ------------- Scripted RTSP server (any SDP, crafted packets and timing).
    SrtpTestSender.h ------------------- SDP-less SRTP sender declaration.
    SrtpTestSender.cpp ----------------- SDP-less SRTP sender built from MServer internals.
    TcpRelay.h ------------------------- TCP relay declaration.
    TcpRelay.cpp ----------------------- TCP relay recording the requests of the client.
    TestAdaptGroup.cpp ----------------- Group "adapt".
    TestAllocations.h ------------------ Heap allocation counter declaration.
    TestAllocations.cpp ---------------- Heap allocation counter (replaced operator new / delete).
    TestAnalysis.h --------------------- Run analysis declaration.
    TestAnalysis.cpp ------------------- Matching of delivered frames against the sent clip.
    TestClips.h ------------------------ Test clips declaration.
    TestClips.cpp ---------------------- Test clip encoding and test patterns.
    TestCommon.h ----------------------- Test context, ports, standard runs and scripted RTSP helpers declaration.
    TestCommon.cpp --------------------- Test context, ports, standard runs and scripted RTSP helpers.
    TestComplianceGroup.cpp ------------ Group "compliance".
    TestDecodeGroup.cpp ---------------- Groups "decode" and "codec".
    TestDirectGroups.cpp --------------- Groups "direct", "mcastpush" and "srtpdirect".
    TestFeedbackGroup.cpp -------------- Group "feedback".
    TestFramework.h -------------------- Check reporter declaration.
    TestFramework.cpp ------------------ Check reporter, summary and time helpers.
    TestGroups.h ----------------------- Group registry and group entry points.
    TestImpairGroup.cpp ---------------- Group "impair".
    TestInteractive.cpp ---------------- Interactive mode (-i) and reference server mode (-server).
    TestInteropGroup.cpp --------------- Group "interop" and receiver mode (-receiver).
    TestLatencyGroup.cpp --------------- Group "latency".
    TestLiveGroup.cpp ------------------ Group "live": MServer changes under a connected client, a session beyond the 60 s timeout, concurrent clients.
    TestLowLatencyGroup.cpp ------------ Group "lowlat".
    TestMatrixGroup.cpp ---------------- Group "matrix": every MServer delivery mode (codec x container x metadata x client path), refusals, authentication, a shared port.
    TestReceiver.h --------------------- Receive helper declaration.
    TestReceiver.cpp ------------------- Receive helper running a VSourceStream object.
    TestRobustGroup.cpp ---------------- Group "robust".
    TestRtspCamera.cpp ----------------- Group "rtsp": scripted third-party camera checks.
    TestRtspGroups.cpp ----------------- Groups "rtsp", "multicast", "tunnel", "auth", "rtsps", "srtp", "klv", "ts".
    TestServerFixGroup.cpp ------------- Group "serverfix".
    TestServer.h ----------------------- Reference server declaration.
    TestServer.cpp --------------------- MServer reference server with a paced feeder.
    TestStatsGroup.cpp ----------------- Group "stats".
    TestTelemetryApiGroup.cpp ---------- Group "telemetryapi".
    TestTelemetryGroup.cpp ------------- Group "telemetry".
    TestUnit.cpp ----------------------- Group "unit": crypto, SRTP, parsers, SDP, RTSP, source strings, policy.
    TestUnitBuilder.cpp ---------------- Group "unit": frame builder checks.
    TestUnitInterop.cpp ---------------- Group "unit": codec detection and MPEG-TS end of access unit checks.
    TestUnitMedia.cpp ------------------ Group "unit": depacketizers, MPEG-TS, output buffer, reorder buffer, RTCP.
    TestV110Builder.cpp ---------------- Group "v110builder".
    TestV110Codec.cpp ------------------ Group "v110codec".
    TestV110Net.cpp -------------------- Group "v110net".
    TestV110Pipeline.cpp --------------- Group "v110pipeline".
    TestV110Public.cpp ----------------- Group "v110public".
    TestV110Rtp.cpp -------------------- Group "v110rtp".
    TestV110Session.cpp ---------------- Group "v110session".
    TestV110Ts.cpp --------------------- Group "v110ts".
    UdpImpairProxy.h ------------------- UDP impairment proxy declaration.
    UdpImpairProxy.cpp ----------------- UDP impairment proxy (loss, reorder, jitter, duplicates).
    main.cpp --------------------------- Source C++ file of test application.

The test application also needs the MServer and VCodecLibav source trees and the FFmpeg development libraries (see Test application).

Receive behaviour

This section describes the runtime behaviour of the library that the interface declaration does not show. Counters in bold, such as packetsLost or kernelDrops, are internal diagnostic statistics. They are not part of the public API (see Internal diagnostics). The public interface is the VSource interface plus getTelemetry(…) and setCodec(…).

Terms used below: a key frame is a frame that a decoder can decode without earlier frames (H.264 IDR, H.265 IRAP, every JPEG frame; the exact rule is in Frame delivery). A GOP (group of pictures) is a key frame and the frames that follow it up to the next key frame; these frames reference earlier frames of the GOP. Annex-B is the H.264 / H.265 byte stream format in which a start code precedes every NAL unit.

Supported sources

The URL scheme of the source string selects the source. Table 2 lists what each scheme receives.

Table 2 - Supported sources.

Scheme Transport Container Codecs Metadata Notes
rtsp:// UDP unicast (default; adaptive UDP / TCP, see Adaptive transport)
TCP interleaved (-tcp)
UDP multicast (-multicast)
RTSP over HTTP tunnel (-http)		 RTP
MPEG-TS in RTP		 H.264, H.265, JPEG
H.264, H.265 in MPEG-TS		 KLV track (SMPTE336M, RFC 6597)
ONVIF metadata track (VND.ONVIF.METADATA)
KLV in MPEG-TS		 Default port 554. Digest authentication (SHA-256, MD5). RTCP NACK / RTX and PLI / FIR when the server offers them.
rtsps:// TCP interleaved inside TLS (default)
UDP unicast / multicast (-udp / -multicast)
SRTP over UDP or TCP (-srtp)
RTSP over HTTPS tunnel (-http)		 RTP / SRTP
MPEG-TS in RTP		 as rtsp:// as rtsp:// Default port 322. TLS 1.2 / 1.3 with peer verification (needs a build with OpenSSL). Digest and Basic authentication. SRTP keys from the SDP (SDES a=crypto).
rtp:// UDP unicast or multicast group (push, no SDP) RTP
MPEG-TS in RTP
raw MPEG-TS		 H.264, H.265 (detected by content)
JPEG (payload type 26)
H.264, H.265 in MPEG-TS		 KLV / ONVIF on its own SSRC on the same port
KLV in MPEG-TS		 RTCP on port + 1; RTCP on the RTP port (rtcp-mux) is recognized too.
udp:// UDP unicast or multicast group raw MPEG-TS (STANAG 4609 / MISB ST 1402)
RTP accepted too		 as rtp:// as rtp:// Receives exactly like rtp://, reports protocol “UDP”.
srtp:// UDP unicast or multicast group (push, no SDP) SRTP
MPEG-TS in SRTP		 as rtp:// as rtp:// Pre-shared master key (-srtpkey). Every packet is authenticated; raw MPEG-TS is refused.

The library does not support:

  • HLS and WebRTC (MServer serves them to browsers);
  • SRT: an srt:// URL fails with “SRT is not supported” (MServer does not serve SRT either);
  • IPv6: IPv6 literals are refused, and host names are resolved to IPv4 addresses only;
  • audio: audio tracks and audio PIDs are ignored;
  • the H.264 interleaved packetization mode (packetization-mode=2);
  • multi-layer H.265, SVC and MVC;
  • MPEG-TS video other than H.264 and H.265;
  • SRTP keyed by MIKEY or DTLS-SRTP: the SRTP keys come only from the SDP (a=crypto) or from -srtpkey;
  • frames wider or higher than 8192 pixels, or with more than 8192 x 4352 pixels (H.264 / H.265 level 6.2).

Reception is tested with MServer, the reference server of the test application: its “matrix” and “live” groups take every delivery mode of MServer that is not HLS or WebRTC (RTSP over UDP, TCP, multicast and the HTTP tunnel; RTSPS interleaved, with plain UDP or multicast media, with SRTP over UDP or TCP, and through the HTTPS tunnel; the push leg in every container; H.264, H.265 and JPEG; KLV and ONVIF metadata; digest authentication and the compliance profiles) and MServer’s live changes under a connected client. It is also tested with independent senders: ffmpeg 6.1.1 (RTP, MPEG-TS and MPEG-TS in RTP with libx264, libx265 and MJPEG), the GStreamer 1.24.2 payloaders and MPEG-TS muxer, and gst-rtsp-server 1.24.2 (RTSP over UDP, TCP and HTTP, RTSPS with its GnuTLS TLS stack, digest authentication).

Source string

The source string is the source field of the VSourceParams object given to initVSource(…), or the initString parameter of openVSource(…). Format:

<url> [option ...]

The URL comes first. The options follow, separated by spaces or tabs. The string can hold at most 4096 bytes and 32 tokens. Option names and URL schemes are case-insensitive. An option value is the next token, so a value can’t contain spaces.

Initialization fails on an unknown option, a missing value, a value out of range, a repeated option or a refused combination. An ERROR log line then names the reason. The library never clamps or ignores a value.

Table 3 - URL formats.

URL Description
rtsp://[user[:password]@]host[:port]/path[?query] RTSP. host is a host name (letters, digits, ‘-‘ and ‘.’; resolved to one IPv4 address) or an IPv4 literal. Default port 554. User and password are percent-decoded (%XX); each can have at most 128 bytes. The user name (from the URL after decoding, or from -user) must not contain control characters, ‘:’, ‘”’ or a backslash. The password must not contain control characters. In the user-info, ‘/’, ‘?’ and ‘#’ must be percent-encoded; the last ‘@’ ends the user-info. Path and query must use RFC 3986 characters. Fragments (‘#’) are refused. The URL can have at most 1024 bytes, not counting the user-info.
rtsps://[user[:password]@]host[:port]/path[?query] RTSP over TLS, same syntax. Default port 322.
rtp://<ipv4>:<port> SDP-less RTP or MPEG-TS reception (see SDP-less reception and auto-detection). ipv4 is a local address (0.0.0.0 = any address) or a multicast group in 224.0.0.0/4, which the library joins. Port 1 - 65534 (RTCP uses port + 1). User-info, path, query and host names are not allowed (a lone ‘/’ is tolerated).
udp://<ipv4>:<port> Raw MPEG-TS datagrams; RTP is accepted too. Same syntax as rtp://.
srtp://<ipv4>:<port> SDP-less SRTP with a pre-shared key. Same syntax as rtp://. -srtpkey is required.

Table 4 - Source string options.

Option Valid with Default Description
-tcp rtsp://, rtsps:// rtsps:// Transport preference: RTP interleaved in the RTSP connection (inside TLS for rtsps://).
-udp rtsp://, rtsps:// rtsp:// Transport preference: RTP over UDP unicast (SRTP over UDP with -srtp).
-multicast, -udpm rtsp://, rtsps:// - Transport preference: UDP multicast. The library sends DESCRIBE for the URL as given. SETUP asks for RTP/AVP;multicast, and the reply gives the group and the ports.
-http rtsp://, rtsps:// - RTSP over an HTTP tunnel (an HTTPS tunnel for rtsps://). The media is interleaved in the tunnel.
-autotransport rtsp://, rtsps:// on when no transport option is given Adaptive transport (see Adaptive transport): a switch UDP -> TCP on network loss, and a TCP latency guard. A transport option (-tcp, -udp, -multicast, -http) turns it off. Given together with -tcp or -udp, -autotransport keeps it on, and that option selects the start transport.
-noautotransport rtsp://, rtsps:// - Turns the adaptive transport off. The firewall fallback UDP -> TCP still works.
-srtp rtsps:// - SRTP media (RTP/SAVP). The library sends DESCRIBE for the URL as given. If the SDP has no RTP/SAVP video with a usable a=crypto key, and the text after the first ‘?’ or ‘;’ contains neither “srtp” nor “savp” (case-sensitive, like MServer), the library sends DESCRIBE once more on the same connection. This time “?srtp” is appended to the URL (“&srtp” when the URL already has a query): this is MServer’s secure presentation.
-user NAME rtsp://, rtsps:// - User name, taken literally (not percent-decoded).
-password PASS rtsp://, rtsps:// - Password, taken literally. Requires -user.
-profile NAME all ONVIF Compliance mode: ONVIF, STRICT_FIPS, FIPS_ONVIF_BRIDGE or CRA_STRICT (case-insensitive). “NO” also selects ONVIF. Other names fail. See Security and compliance modes.
-cafile PATH rtsps:// system CA paths PEM CA bundle used for TLS peer verification. On Windows, OpenSSL has no system CA store by default (it does not read the Windows certificate store), so give -cafile or -fingerprint there.
-fingerprint FP rtsps:// - Pin the SHA-256 fingerprint of the server certificate: 32 hex pairs separated by ‘:’ (95 characters) or 64 hex digits, case-insensitive.
-noverify rtsps:// - Disables TLS peer verification. Allowed in the ONVIF profile only, and not together with -fingerprint or -cafile. The WARNING “TLS peer verification disabled (-noverify): the server is NOT authenticated” is logged once per initialization, not per reconnection.
-srtpkey BASE64 srtp:// required Master key followed by master salt, in strict standard base64 (RFC 4568 inline form), at most 128 characters. Decoded length: 30 bytes for the AES_CM_128 suites, 46 bytes for the AES_256_CM suites, 28 bytes for AEAD_AES_128_GCM, 44 bytes for AEAD_AES_256_GCM.
-srtpsuite NAME srtp:// AES_CM_128_HMAC_SHA1_80 SRTP suite (case-insensitive): AES_CM_128_HMAC_SHA1_80, AES_CM_128_HMAC_SHA1_32, AES_256_CM_HMAC_SHA1_80, AES_256_CM_HMAC_SHA1_32, AEAD_AES_128_GCM, AEAD_AES_256_GCM.
-iface IP rtp://, udp://, srtp://, rtsp:// and rtsps:// with -multicast system choice Unicast IPv4 address of the interface used for the multicast join (not 0.0.0.0, which names no interface).
-source IP rtp://, udp://, srtp:// any sender Accepts media only from this unicast IPv4 sender. A multicast group is then joined source-specific (SSM).
-timeout MS all 10000 Init / connect deadline, 100 - 60000 ms (see initVSource(…) for how long the call can block). Every reconnection attempt gets the same deadline.
-reconnect MS all (used by rtsp:// and rtsps://) 5000 No-media watchdog, 500 - 60000 ms: an RTSP session without video media for this time is reconnected (metadata packets alone do not keep it alive: a camera whose video stalled while its analytics stream goes on is reconnected).
-mindelay MS all 10 (3 with -nokeywait) Minimum reorder wait, 0 - 1000 ms (see Low latency design and adaptive jitter buffer). Why the default is 10 ms (1.0.0: 3 ms): the longer wait delays only the frames that are kept after a real loss. These are the next frame when the loss lies between two frames, the key frame after a skip (unless it is complete in the buffer) and the frames that -keywait lets through. In exchange, reordering of up to 10 ms (Wi-Fi, LTE, bonded links) no longer costs the rest of a GOP (measured over 124 channel traces). With -nokeywait every frame behind a gap waits, so the default is 3 ms there.
-maxdelay MS all 200 Maximum reorder wait, 1 - 5000 ms. TCP transports always use 0.
-buffer N all CUSTOM_2 (32) Output buffer depth, 1 - 512 frames. Overrides VSourceParams::custom2.
-nokeywait all - After packet loss, the non-key frames are delivered anyway (decoding artifacts until the next key frame). The key-frame bypass of the reorder wait is then off, and the default minimum wait is 3 ms.
-keywait MS all 0 (unbounded) Bounded key frame wait, 0 - 60000 ms (0 = unbounded). After packet loss, non-key frames are skipped for at most MS. Then they are delivered again until the next key frame (decoding artifacts are possible). The key frame is requested with RTCP PLI / FIR when the server supports it. See Bounded key frame wait.
-nometadata all - No metadata tracks are set up, and no KLV / ONVIF metadata is bound to frames: the telemetry has no metadata units. The ST 0604 SEI capture time of getTelemetry(…) is still parsed.
-seidata all - Also delivers the H.264 / H.265 SEI user_data_unregistered messages (x264 options, MISB ST 0604 time stamps, vendor data) as metadata units of type UNKNOWN (1.3.x: SEI). The SEI units of the frame’s own access unit are exact; they come after its exact KLV / ONVIF units. An access unit that is not delivered (skipped before a key frame or after a loss, corrupt, without a picture, oversized, dropped by the output buffer or by -maxlatency) gives its SEI units to the next delivered frame. These units are not exact, and they count toward the carry bound (see Metadata binding): after a long skip they can push out older carried KLV / ONVIF units. SEI NAL units above 64 KiB are not scanned. At most 16 user_data_unregistered messages (ST 0604 time stamps included) are taken per access unit. The SEI stays in the frame. Off by default: the telemetry then has KLV / ONVIF units only, as in 1.0.0. The ST 0604 capture time of getTelemetry(…) is parsed either way, within the same bounds.
-maxlatency MS all 0 (off) Output latency budget, 10 - 60000 ms (0 = off; 1 - 9 fail; see Output latency budget). When the oldest buffered frame is older than MS and a newer key frame is buffered, every frame before the newest buffered key frame is dropped. The drops count in CUSTOM_3, framesOverflow and framesLatencyDropped. The overflow policy does not change. So a consumer that stalled catches up at the next key frame instead of staying late. MS is also the budget of the TCP latency guard of the adaptive transport. Set it to at least a few frame intervals.
-priority N all inherited SCHED_FIFO priority 1 - 99 of the receive thread. Needs CAP_SYS_NICE or RLIMIT_RTPRIO. On Windows, any value gives THREAD_PRIORITY_TIME_CRITICAL (the highest priority below the realtime class). A failure gives one WARNING, and the source then works with the inherited policy. Without this option the receive thread inherits the policy and affinity of the thread that calls initVSource(…). For the full effect, the application must also raise the priority of its getFrame(…) thread. Measured with 1.1.0 under CPU contention: p99 4 - 10 ms -> 0.8 - 1.7 ms, only with both threads prioritised. See Receive thread, kernel timestamps and scheduling.
-cpu N all inherited Pins the receive thread to CPU 0 - 1023 (on Windows: 0 - 63, processor group 0). A failure gives one WARNING. Useful only on an otherwise idle CPU. See Receive thread, kernel timestamps and scheduling.

If both -mindelay and -maxdelay are given, min > max fails. A lone -mindelay above the default maximum (200 ms) raises the maximum to the same value. A lone -maxdelay below the default minimum (10 ms; 3 ms with -nokeywait) lowers the minimum to the same value. For example, -maxdelay 5 gives 5 - 5 ms, and -nokeywait -maxdelay 5 gives 3 - 5 ms.

Refused combinations (initialization fails):

  • more than one of -tcp, -udp, -multicast / -udpm and -http;
  • -tcp, -udp, -multicast, -http, -autotransport, -noautotransport, -srtp, -user, -password, -cafile, -fingerprint or -noverify with rtp://, udp:// or srtp://;
  • -autotransport together with -noautotransport, -multicast / -udpm or -http;
  • -keywait together with -nokeywait (also -keywait 0);
  • -seidata together with -nometadata;
  • -srtp with rtsp:// (the SRTP keys travel in the SDP, and only TLS protects the SDP);
  • -srtpkey or -srtpsuite without srtp://, srtp:// without -srtpkey, and a key whose decoded length does not match the suite;
  • -source with rtsp:// or rtsps://, and -iface with rtsp:// or rtsps:// without -multicast;
  • -noverify together with -fingerprint or -cafile;
  • credentials both in the URL user-info and in -user / -password, and -password without a user;
  • anything the compliance profile refuses, for example rtsp:// or -noverify in STRICT_FIPS and CRA_STRICT (see Security and compliance modes).

Examples:

rtsp://192.168.1.67:8900/Cam1 -tcp
rtsp://admin:S3cret%21@192.168.1.67:554/live
rtsp://192.168.1.67:554/live -user admin -password secret -udp -reconnect 3000
rtsp://192.168.1.67:554/live -multicast -iface 192.168.1.10
rtsp://192.168.1.67:554/live -http
rtsp://192.168.1.67:554/live -maxlatency 200 -priority 50 -cpu 3
rtsp://192.168.1.67:554/live -tcp -autotransport
rtsps://camera-1:322/live -cafile /etc/ssl/certs/ca-certificates.crt
rtsps://192.168.1.67:8555/live -srtp -udp -fingerprint <SHA-256 fingerprint of the server>
rtsps://camera-1:8555/live -profile CRA_STRICT -cafile /etc/ca.pem
rtp://0.0.0.0:8765
rtp://239.1.1.1:5004 -iface 192.168.1.10 -source 192.168.1.67
udp://0.0.0.0:5600 -keywait 2000 -seidata
srtp://0.0.0.0:5004 -srtpkey <base64 master key and salt> -srtpsuite AEAD_AES_128_GCM

Transport selection and fallback

By default, rtsp:// uses RTP over UDP unicast, and rtsps:// uses RTP interleaved inside the TLS connection. Without a transport option, the adaptive transport may switch between UDP and TCP at run time. The transport options are only preferences. The server may refuse or change the transport, and the client follows the server as long as the compliance profile allows it.

The RTSP sequence is:

  1. Open the control connection (plain TCP, TLS, or the HTTP / HTTPS tunnel).
  2. OPTIONS.
  3. DESCRIBE with “Accept: application/sdp”.
  4. SETUP of the video track, then SETUP of each metadata track: up to 2 tracks, in SDP order, on the same connection and in the same transport family (see Metadata binding).
  5. PLAY of the aggregate control URL with “Range: npt=0.000-“.

A DESCRIBE answered with 404 or 503 is retried every 200 ms until the -timeout deadline. (MServer answers 404 until its encoder has produced the parameter sets.) A connection that the server closes right after it was opened is re-opened at the same pace, never in a tight loop. Redirects (301, 302, 303, 307) are followed up to 3 times. A redirect is never followed when it changes TLS, or when it points to another host or port while credentials are configured. Initialization then fails instead, so the credentials reach only the configured host.

  • 461 Unsupported Transport. After this answer, the client sends SETUP with the next allowed transport: TCP -> UDP, UDP -> TCP, MULTICAST -> UDP -> TCP. The HTTP tunnel has no fallback. The client never tries a transport that the compliance profile forbids. The reconnections of this source then start with the transport that worked, so the refused one is not asked for again.
  • The Transport header of the reply wins. The client follows a server that switches to TCP or multicast. It refuses an unrequested UDP unicast transport, a transport that the profile forbids, and SRTP without a key. It accepts a downgrade from SRTP to plain RTP only where the profile allows plain media.
  • Firewall fallback. If no RTP video packet arrives over UDP within 3 s after PLAY (or within -reconnect, if that is shorter, so the fallback always comes before the no-media reconnection), the library logs a WARNING (“No RTP over UDP within 3000 ms after PLAY (firewall?): reconnecting over TCP”). It then tears the session down and sets it up again over TCP. A multicast session that receives nothing (no multicast route, IGMP snooping without a querier) falls back the same way, to UDP unicast (“No RTP over multicast within 3000 ms after PLAY (no multicast route?): reconnecting over UDP unicast”), or to TCP when the profile forbids UDP. All later reconnections of this source stay on TCP until the next initVSource(…) or RESTART. (With the adaptive transport on, the TCP latency guard can still make its single UDP trial later; see Adaptive transport.) When the fallback happens, initVSource(…) has already returned TRUE, because it waits at most 3 s for the first frame after PLAY. The frames arrive once the TCP session runs.
  • Adaptive transport (1.1.0). It is on by default for rtsp:// and rtsps:// when the source string has no transport option. The session may then switch between UDP and TCP at run time, on network loss or on a growing TCP queue. See Adaptive transport.
  • UDP unicast. Before SETUP, the client binds a random even port pair in 49152 - 65534. It accepts media only from the IP address of the RTSP server. RTCP goes to the RTCP port of the server.
  • Multicast. The group and the ports come from the SETUP reply, else from the SDP. The sockets are bound to the group (on Windows, which refuses that, to the port on all addresses) and join it on the -iface interface. A source= parameter in the reply gives a source-specific join. Without -iface, a failed join is retried on the local address of the RTSP control connection. A group that this host can’t join is handled like 461: WARNING, TEARDOWN, a new control connection, then UDP, then TCP.
  • SRTP (-srtp). SETUP asks for RTP/SAVP/TCP (the default) or for RTP/SAVP over UDP (-udp). The keys of a track come from the track’s own a=crypto attribute, else from the session-level a=crypto, else from the a=crypto of the video track. (MServer puts a=crypto on the video m-line only.) Tracks with the same suite and key share one SRTP receiver. Two tracks that offer the same key with different suites are refused: the session fails with “Two tracks offer the same SRTP key with different suites”, because both receivers would protect their SRTCP with the same key stream.
  • HTTP / HTTPS tunnel (-http). The RTSP requests go base64-encoded over an HTTP POST leg. The responses and the media come over an HTTP GET leg. An x-sessioncookie pairs the two legs, and both heads carry a Host header (for reverse proxies and virtual hosts). The POST leg announces a Content-Length of 32767 bytes; the client re-opens it before a request would go beyond that body, so a proxy that honours the length loses no request. RTCP is never sent through the tunnel, so NACK, PLI and FIR are not used there.
  • Keep-alive. The client sends GET_PARAMETER (if the server lists it in Public) or OPTIONS every min(session timeout / 2, 25 s), but never more often than every 0.5 s. The session timeout comes from the Session header; it is 60 s when the header gives none. A timeout outside 1 - 3600 s, or one that is not an integer, is ignored with one WARNING per connection (“RTSP session timeout outside 1..3600 s ignored, 60 s assumed”). The session id is kept.
  • Tolerance of camera replies. Some cameras send malformed ttl= or ssrc= hints in the Transport header of a SETUP reply, for example a sign-extended 64-bit ssrc=FFFFFFFFC6C3D3B9. The client drops such a hint with a WARNING (“SETUP reply of the video track: Malformed Transport ssrc ignored”), and the transport stays usable. The other Transport parameters (profile, ports, interleaved channels) are still checked strictly. Some SDP media sections are kept as unusable, and never selected, instead of failing the whole SDP: a section whose format is not a payload type (TP-Link’s “m=application/TP-LINK 0 RTP/AVP smart/1/90000”), and a section other than video with an invalid port. An invalid video port still fails. A relative a=control URL is appended to the Content-Base (else to the Content-Location, else to the request URL), and the query is kept. For example, …/realmonitor?channel=1&subtype=1/ and trackID=0 give …/realmonitor?channel=1&subtype=1/trackID=0.

The transport in use is logged in the “RTSP session established” INFO line (see Logging) and reported by the internal diagnostics as transport: “UDP”, “TCP”, “MULTICAST”, “HTTP” or “HTTPS”.

Adaptive transport

A camera link can change while the source runs. Loss on Wi-Fi or LTE makes RTP over UDP lose whole GOPs, because every lost packet costs the frames up to the next key frame. A bottleneck has another effect on an RTSP / TCP connection: TCP never loses a packet, but it delivers later and later, and the queue grows to seconds. The adaptive transport (1.1.0) watches both conditions. It switches the RTSP session between RTP over UDP unicast and RTP interleaved in the RTSP (or TLS) connection. Table 5 shows when it is active.

Table 5 - Adaptive transport activation.

Source string Start transport Adaptive transport
rtsp:// without a transport option UDP on
rtsps:// without a transport option TCP inside TLS on, but only TCP flush reconnects: UDP would take the media out of the TLS protection
rtsps:// -srtp without a transport option SRTP over TCP inside TLS on (SRTP over UDP may be tried)
-tcp or -udp as given off; on with -autotransport too (the option then selects the start transport)
-multicast or -http as given off (-autotransport is refused)
-noautotransport as without the option off (the firewall fallback UDP -> TCP stays)
rtp://, udp://, srtp:// - not applicable (-autotransport is refused)

The rules are evaluated once per second while the session is established, counted from PLAY:

  1. UDP -> TCP (loss rule). The session switches to TCP when both of these conditions hold over the last 5 s (a window of at least 20 frames):
    • The frames lost to packet loss reach 20 % of all frames. Lost frames are the incomplete frames and the frames skipped while waiting for a key frame. Skipped frames count only after the session has produced a frame or has had 20 incomplete frames, so the initial key frame wait of a clean join is no loss.
    • The loss is network loss: the local kernel drops of the window (kernelDrops) are below 50 % of the RTP packets that the window declared lost. A packet is declared lost when it never arrived, or when it arrived only after the reorder wait gave it up: a retransmission or a reordered packet that comes too late costs the same frames.

    TCP can’t help a receiver that overflows its own socket buffer, and frames lost without any declared packet loss never cause a switch. The session reconnects over TCP. UDP then counts as measured bad until the next initVSource(…) or RESTART.

  2. TCP latency guard. The guard acts when the queueing delay (queueDelayUsec, see below) stays above the budget for 1 s. The budget is -maxlatency when given, else 1000 ms. The delay is sampled after every read round. One sample at or below the budget restarts the 1 s count, so a periodic key frame spike never adds up. The session then tries UDP once, if the compliance profile allows UDP and UDP was neither measured bad nor tried before. Otherwise the session reconnects over TCP. This flushes the queues of the server and of the network: the new session starts at the live edge with a key frame.

Blocks and limits:

  • After every action, no further action is taken for 60 s. This block doubles with every action, up to 10 min. The block, the “UDP measured bad” state and the trial state survive reconnections. Only the next initVSource(…) or RESTART resets them.
  • UDP is tried at most once per initialization, whatever the outcome of the trial (loss rule, firewall fallback, a failure). So the transports can’t oscillate. A UDP trial that fails the loss rule returns to TCP at once, without the block.
  • A transport that the compliance profile forbids is never the target; the TCP flush reconnect still happens. Multicast and the HTTP tunnel are never switched.
  • A switch reconnects without backoff. Like every reconnection, it resumes with a key frame. frameId continues, and the metadata units that are still waiting are counted in metadataUnbound.
  • The firewall fallback (no RTP over UDP within 3 s after PLAY, see Transport selection and fallback) works independently of the adaptive transport.

Every action is logged as a WARNING with its reason and counted in transportSwitches. Two examples: “Adaptive transport: 25 of 100 frames lost over UDP in the last 5 s (41 packets lost or late, 0 kernel drops): switching to TCP (next adaptive action not before 60 s)” and “Adaptive transport: TCP queueing delay 1450 ms above the 300 ms budget for 1001 ms: trying UDP once (next adaptive action not before 60 s)”. The internal diagnostics show the evidence (see Internal diagnostics). transport is the current transport. lossPercent, packetsLate and framesFrozenMsec show the loss rule; packetsLate counts the repairs and the reordered packets that came after the reorder wait. queueDelayUsec shows the latency guard. kernelDrops shows the local drops, which are no reason to switch.

Queueing delay. queueDelayUsec needs no clock synchronisation. The transit time of a frame is its arrival time minus its RTP (or PTS) time. The delay is the smallest transit time of the last 16 frames (at most 2 s old) minus the smallest transit time of the last 30 s. Because the smallest values are used, B-frame reordering and single late key frames do not count. The delay grows when a queue builds up anywhere on the path (TCP backlog, bottleneck, server queues). The measurement restarts when the sender’s timestamps step back by more than 1 s (or by 16 frame intervals, if that is longer), or when the transit time jumps by more than 10 s (sender restart, timestamp jump).

Limitations:

  • A sender whose RTP clock runs more than about 3 % slower than real time looks like a growing queue. Examples are timestamps from a frame counter while the frame rate drops, or a wrong clock rate. Such a sender gets a TCP flush reconnect whenever the block expires. Use -noautotransport (or a larger -maxlatency) with such a camera.

The “adapt” and “feedback” groups of the test application verify the rules end to end through an RTSP-aware impairment relay; see Table 14 in Measured results.

SDP-less reception and auto-detection

rtp://, udp:// and srtp:// receive push streams without signalling. The library binds the given port (media) and port + 1 (RTCP) on the given address. It requests 8 MiB receive buffers for both. A multicast group is bound with SO_REUSEADDR, so several receivers can share it. The library joins the group on the -iface interface, source-specific with -source.

initVSource(…) returns TRUE as soon as the sockets are bound, even if no sender exists yet. It waits for the first frame for at most min(-timeout, 3 s). So without a sender, width, height and fourcc stay 0 / “” until the first frame arrives; every getFrame(…) updates them. Until media flows, getFrame(…) just times out. (The internal diagnostics report media flow as connected.)

The library classifies every datagram on the media port by its content: raw MPEG-TS (a multiple of 188 bytes with sync bytes; refused for srtp://), RTCP (rtcp-mux) or RTP. The RTCP port accepts RTCP only. With -source, both ports drop the packets of other senders. For each RTP stream (SSRC), the payload is detected as follows:

  • Payload type 26 is JPEG. Payload type 33, or MPEG-TS content, is MPEG-TS: the video codec (H.264 or H.265) comes from the PMT, and the KLV streams come from the PMT and the PES.
  • A payload that starts with the KLV universal label (06 0E 2B 34) or with an XML start (‘<’) is metadata (KLV or ONVIF). An ONVIF stream whose documents start with a UTF-8 BOM is not recognized on rtp://, udp:// and srtp:// (over RTSP the BOM is accepted). Payload type 98 (MServer’s metadata type) is only a hint: video on payload type 98 is recognized by a plausible SPS.
  • H.264 and H.265 on other dynamic payload types are told apart by decisive evidence: parameter sets (SPS / PPS for H.264, VPS / SPS / PPS for H.265), or fragments of key frames and parameter sets. Ambiguous packets are buffered (up to 64) and replayed into the chosen depacketizer. MServer’s payload type map (96 = H.264, 97 = H.265) is used only after 5 s without decisive evidence, and later evidence corrects it.

Rules of SDP-less reception:

  • The stream must carry its parameter sets in-band: the first delivered frame is the first key frame whose parameter sets were seen.
  • A payload type change on the same SSRC (MServer’s codec switch between H.264, H.265 and JPEG) switches the codec. The new codec starts with a key frame.
  • A new RTP source must pass the RFC 3550 probation (two packets in sequence). Metadata packets are accepted at once.
  • Up to 8 SSRCs are tracked. One of them is the active video source. Another SSRC takes over when the active one has been silent for 1 s (or at once after its RTCP BYE), and it starts at its next key frame. So two senders on one port are never interleaved. A raw MPEG-TS stream competes in the same way. An SSRC that is silent for 30 s is forgotten. New SSRCs are taken from a budget of 8 that refills by one per second, so a sender can’t recycle SSRCs at packet rate; the packets of an SSRC that finds the budget empty are dropped, with one WARNING per session.
  • While it is not yet known whether the stream carries metadata, the first frame is held for at most 3 ms. So the first KLV unit binds even when the receiver started before the sender.
  • RTCP receiver reports go to the address that the sender’s SR (RTCP sender report) came from, if that is the address of the RTP packets of the same SSRC (an SR from another address is used for timing only, so an SR sent from a third address can’t redirect the reports). No report is sent before the first such SR.
  • Missing media never ends the session: it keeps listening. Socket errors reopen the sockets with backoff.
  • srtp://: every RTP and RTCP packet (on both ports) is authenticated before use. A receiver that joins a running stream recovers the SRTP rollover counter with a bounded, rate-limited search. A wrong key delivers nothing and counts in srtpAuthFailures.

Every UDP socket requests an 8 MiB receive buffer (also for RTSP over UDP and multicast). Without the CAP_NET_ADMIN capability, the Linux kernel caps this request at net.core.rmem_max. If less than 4 MiB is granted, one WARNING per session names the granted size and the remedy (sysctl -w net.core.rmem_max=8388608 or CAP_NET_ADMIN). A small buffer may overflow on the bursts of large key frames.

Frame delivery

FIFO output buffer. Frames are delivered in order, oldest first, from an output buffer of CUSTOM_2 frames. The depth is 1 - 512 frames, 32 by default. It is set by -buffer, by VSourceParams::custom2 at initialization, or by setParam(…) at any time (applied immediately). Decoded pictures (1.3.0) come from a second buffer of the same depth (see Frame output and decoding).

A caller that waits inside getFrame(…) receives each frame the instant it is complete, so in steady state the buffer adds no latency. Here VSourceStream differs on purpose from VSourceV4L2, which always delivers the newest captured frame and drops older ones. Raw camera frames are independent, but encoded H.264 / H.265 frames are not: every inter frame references earlier frames, and skipping a single frame makes the rest of the GOP undecodable. So a caller that is briefly slow must lose nothing. The buffer absorbs stalls of up to CUSTOM_2 frames, and the frames are delivered afterwards, in order.

Call getFrame(…) in a dedicated loop, at a rate not lower than the frame rate of the stream. CUSTOM_1 shows how far the caller lags behind. It is the age of the delivered frame, which also equals the wall clock at delivery minus receiveTimeUsec of getTelemetry(…). The internal diagnostics also report the number of buffered frames (bufferedFrames).

Telemetry. With every frame, getFrame(…) takes a snapshot of the frame’s telemetry: frame ID, key frame flag, 90 kHz timestamp, capture and receive time, the frames lost before it, and its metadata units. getTelemetry(…) returns this snapshot until the next successful getFrame(…). See Frame telemetry and capture time and Metadata binding. Since 1.2.0 there is no frame callback: every frame is delivered through the output buffer and getFrame(…).

Overflow policy. The caller may stall longer than the buffer holds (CUSTOM_2 frames, or 256 MiB of frames and metadata). The buffer then drops the oldest frame. It drops more frames from its head until the head is a key frame, so that the remaining frames stay decodable. If the buffer becomes empty this way and the incoming frame is not a key frame, the incoming frames are dropped until the next key frame. The library requests that key frame with RTCP PLI / FIR when the server negotiated it (see RTCP). Every dropped frame counts in framesOverflow and in CUSTOM_3. The first frame after the gap is a key frame. JPEG frames are all key frames, so only the oldest frame is dropped. Note that a depth of 1 is not a “newest frame” mode for H.264 / H.265: any overflow costs the frames up to the next key frame.

Output latency budget. With -maxlatency MS (1.1.0), the buffer also bounds the age of the buffered frames. A consumer that stalled then catches up at the next key frame instead of staying late. See Output latency budget.

Key frame gate. The first delivered frame is always a key frame that carries every parameter set it references, also with -nokeywait. The same holds after a reconnection and after a codec switch. A key frame is:

  • for H.264: an IDR picture, a picture with a recovery point SEI, or a picture made only of I / SI slices with SPS and PPS before it;
  • for H.265: an IRAP picture (IDR, CRA, BLA) of the base layer or a picture with a recovery point SEI; an IRAP of another layer does not make the base layer decodable.

The gate opens only when the PPS referenced by the slices, its SPS and (for H.265) its VPS are available, in the frame or in the parameter set cache.

A frame with a missing packet is never delivered. There is one exception: for a sender without a reliable RTP marker, a loss of the last packets of a frame can’t be detected before the frame is delivered (see the idle gap in Low latency design and adaptive jitter buffer). A loss between two complete frames does not cost the next frame when that frame provably starts with its first received packet and its timestamp follows the previous frame. This is the AU-boundary loss rule (see Low latency design and adaptive jitter buffer).

After packet loss, and after a sender restart, the following non-key frames are skipped until the next key frame. They count in framesSkipped, and their duration counts in framesFrozenMsec. -nokeywait turns this skipping off. With -keywait MS the frames are skipped for at most MS; then the non-key frames are delivered again until the next key frame (see Bounded key frame wait). A key frame request (PLI / FIR) is sent when it was negotiated. The request is repeated after 1.5 x the observed key frame response time (150 - 1000 ms; 500 ms until a response time is measured; 1.0.0: always 500 ms). It is repeated at once when the requested key frame arrives damaged.

The leading pictures of the random access point that reopens the gate are dropped as well (counted as skipped, at most 32). After an H.265 CRA / BLA, these are the RASL pictures up to the first trailing picture; RADL pictures pass. After an H.264 non-IDR key frame (an open GOP I picture or a recovery point), these are the pictures whose timestamp is below the key frame’s. The leading pictures of later random access points, while the gate is open, are delivered.

Effect of packet loss. Before the reorder buffer declares a missing packet lost, it waits for the packet for at most its current adaptive wait (see Low latency design and adaptive jitter buffer). When the server offers NACK, the library also requests a retransmission of the packet (see NACK, RTX and key frame requests). A packet that is really lost costs its frame. With the default key frame wait, it also costs the rest of the GOP. With a GOP of 25 frames at 25 fps this is up to 1 s of video, unless the server answers the PLI / FIR request earlier with a key frame. -keywait bounds this freeze. With -nokeywait, only the frame with the missing packet is dropped, and the following frames are delivered. The decoder then shows artifacts until the next key frame. The first frame of the stream is still a key frame, and the key frame request is still sent.

Parameter sets. The library collects VPS / SPS / PPS from the SDP (sprop-parameter-sets, sprop-vps, sprop-sps, sprop-pps) and from the stream, per id and per codec. Every key frame gets every cached parameter set whose id it does not carry itself. The sets are inserted in the order VPS, SPS, PPS, after a leading access unit delimiter (which is kept) and before the first slice. So every delivered key frame is self-contained: a decoder can start at any of them. The cache holds at most 64 KiB of sets in all (real streams need a few hundred bytes); a set that would not fit is not cached, and the key frames that need it wait.

Output format. This is the output when the Frame given to getFrame(…) has the fourcc H264, HEVC or JPEG. Any of the three is corrected to the codec of the stream. For an uncompressed fourcc the frames are decoded (1.3.0; see Frame output and decoding).

  • H264 and HEVC frames are Annex-B byte streams with a 4-byte start code (00 00 00 01) before every NAL unit.
  • JPEG frames are complete JFIF images. The library rebuilds them from the RFC 2435 headers (quantization tables, standard Huffman tables, restart interval) or from the ONVIF JPEG header extension, which carries images above 2040 pixels.
  • Frame::fourcc is Fourcc::H264, Fourcc::HEVC or Fourcc::JPEG. Width and height come from the SPS (with the cropping / conformance window) or from the JPEG header. Frame::size is the payload size in bytes. Frame::sourceId is not touched.

The frame buffer follows the Frame convention for encoded formats: width * height * 4 bytes. The buffer is reused while it is the buffer that the library allocated for this Frame object and it is large enough. Otherwise it is allocated once (see Frame buffer in Frame output and decoding). Some frames are dropped and counted in framesIncomplete and CUSTOM_3: a frame larger than 32 MiB, a payload larger than width * height * 4 bytes, and a frame wider or higher than 8192 pixels or with more than 8192 x 4352 pixels. Such a frame is never truncated, and the caller’s frame is never allocated for it.

frameId. Frame::frameId increments by 1 for every delivered frame, starting at 1. It is never reset: it continues over reconnections and over re-initialization of the same object. It is a 32-bit counter converted to int. After 2^31 deliveries (about 414 days at 60 fps) it continues with negative values. After 2^32 deliveries it passes 0 again, and 0 is then a valid ID (when getTelemetry(…) returns TRUE). So compare frame IDs for equality only. The frameId field of VSourceStreamTelemetry holds the same value. frameId does not reveal dropped frames; CUSTOM_3 does.

cycleTimeMks. Every getFrame(…) call resets the CYCLE_TIME_MKS param to 0 at its start. On success, the call sets it to the interval since the previous successful call, in microseconds. For the first frame after initVSource(…), it is the interval since initVSource(…) returned. So it measures the delivery interval that the caller sees, not the frame interval of the sender.

CUSTOM_1, CUSTOM_2, CUSTOM_3.

  • CUSTOM_1 (read only) is the age of the last delivered frame, in microseconds: the delivery time minus the arrival time of its last packet (steady clock). It includes the reorder wait, the metadata hold and the time spent in the output buffer. Since 1.1.0, the arrival time of a UDP packet is its kernel receive time (Linux), so the age also includes the kernel queueing and the thread wake-up. For TCP, TLS and the HTTP tunnel, the arrival time is the read time.
  • CUSTOM_2 (read / write) is the output buffer depth, in frames.
  • CUSTOM_3 (read only) is the number of frames lost between the previous delivered frame and the last delivered frame: overflow drops, -maxlatency drops, incomplete frames (including frames that getFrame(…) dropped at delivery) and skipped frames. It is also lostFrames of the frame’s telemetry (getTelemetry(…)).

fps. The frame rate comes from the SPS VUI timing information, else from the SDP a=framerate attribute. Otherwise the library measures it as the median interval of the last 32 frames; the RTP timestamps are sorted, so B-frames are handled. The value is rounded to 2 decimals.

Frame output and decoding

Output selection. The fourcc of the Frame object given to getFrame(…) selects what the method delivers (1.3.0), as with the VSourceFile library (Table 6).

Table 6 - Output of getFrame(…) by the fourcc of the frame.

Frame::fourcc set by the caller Delivered frame Frame::size
H264, HEVC or JPEG The stream’s own encoded frame as received, whichever of the three is asked for: Frame::fourcc is set to the codec of the stream (see Output format in Frame delivery). Frame size
NV12 Decoded picture as it comes out of the decoder (4:2:0, Y plane and interleaved UV plane). width * height * 3 / 2
NV21, YU12, YV12 Decoded picture converted to the 4:2:0 format. width * height * 3 / 2
YUYV, UYVY Decoded picture converted to the packed 4:2:2 format. width * height * 2
RGB24, BGR24, YUV24 Decoded picture converted to 3 bytes per pixel. width * height * 3
GRAY Luma of the decoded picture. width * height
any other value Nothing: getFrame(…) returns FALSE at once (logged as ERROR at most once per second). -

A default-constructed Frame has the fourcc YUV24, so it asks for decoded pictures. To receive the encoded frames, set the fourcc to H264 (or HEVC / JPEG) before the call. The library writes the delivered format into Frame::fourcc. So a reused Frame keeps asking for the same output, except after a picture that was delivered as NV12 instead (see below). To fix the output, set the fourcc before every call. VSourceParams::fourcc of getParams(…) is the name of the format of the last delivered frame (“H264”, “HEVC”, “JPEG”, “NV12”, “YUV24”, …).

Decoding. The first getFrame(…) with an uncompressed fourcc starts the decode thread of the object (named “vsrtp-dec” on Linux). From then on, every frame is decoded the instant it is complete, whether a getFrame(…) waits or not. The decode thread:

  1. takes the frame from the output buffer of the encoded frames;
  2. decodes it to NV12 with the built-in decoder of its codec (VDecoderH264, VDecoderH265 or VDecoderJPEG) or with the custom codec of setCodec(…); every VSourceStream object has its own three decoders and its own converter;
  3. converts the picture to the requested format with FormatConverter (not for NV12);
  4. puts the picture into the second output buffer (CUSTOM_2 pictures).

getFrame(…) only copies a picture that is ready. Decoding never runs in the thread of the caller. A caller that is slow or stalls does not stop the decoding: the pictures wait in the buffer. The settings are fixed; they can’t be changed through the library:

  • every built-in decoder uses 2 threads (VCodecParam::CUSTOM_2 = 2) and decodes full NV12 pictures (H.264 / H.265: VCodecParam::CUSTOM_1 = 0);
  • the colour range is automatic (VCodecParam::CUSTOM_3 = 1). A limited range picture is expanded to full range (luma 16 - 235 and chroma 16 - 240 to 0 - 255). A full range picture is delivered as decoded. An H.264 / H.265 picture counts as limited range when the bitstream signals or infers limited range and the luma samples of the picture confirm it. For JPEG, the luma samples decide;
  • the converter uses 2 threads;
  • a picture that the converter refuses (for example one of odd width or height) is delivered as NV12, with Frame::fourcc NV12;
  • VDecoderH265 decodes one NAL unit per call, so it gets the NAL units of a frame one by one. (With a whole access unit per call, its output would fall behind by the parameter sets and SEI of every key frame.);
  • the data given to a decoder is followed by 64 zero bytes that its size does not count (VDecoderH264 reads up to 15 bytes beyond its input with aligned 16-byte loads).

Telemetry of a decoded frame. A decoded frame keeps everything of its own encoded frame: the telemetry of getTelemetry(…) (frame identity, key frame flag, timestamp, capture and receive time, metadata units), CUSTOM_1 (its age, which includes its decoding time) and CUSTOM_3. H.264 and H.265 decoders output the pictures in display order. When the stream has B-frames, this order differs from the order of reception: a picture comes out after the frames that are shown before it. The library gives each picture the frame with the smallest 90 kHz timestamp among the frames that went into the decoder and whose picture has not come out yet. (RTP and MPEG-TS timestamps are presentation times.) So every picture has the telemetry and metadata of its own frame, also with B-frames, and the timestamps of the decoded frames increase. The decoded pictures of a stream with B-frames come later than its encoded frames, by the reorder delay of the stream. Measured with libx264 and libx265 at 25 fps with 2 B-frames: 40 - 80 ms median and 165 ms p95 from reception to delivery.

Restarts and switches. Decoding starts, or restarts, at a key frame: at the first uncompressed request, after compressed requests, after initVSource(…) and RESTART, on a codec switch of the stream, and when setCodec(…) changes the codec. The frames before this key frame are dropped. So are the leading pictures after it that refer to pictures before it, by the rule of the key frame gate: H.265 RASL pictures after a CRA / BLA, and H.264 pictures with a lower timestamp after a non-IDR key frame (see Key frame gate in Frame delivery). The same applies to the key frame that follows frames dropped by the output buffer of the encoded frames. A getFrame(…) that asks for another kind of output switches at once:

  • encoded to decoded: decoding starts at the next key frame (the frames up to it are lost);
  • decoded to encoded: the decoded pictures not yet delivered are dropped, and the encoded frames continue at a key frame;
  • one uncompressed format to another: the new format applies from the next picture on, without a key frame wait; the pictures of the old format that are still buffered are dropped.

Every dropped frame counts in CUSTOM_3 and in lostFrames of the telemetry of the next delivered frame. Its metadata units are carried to the next delivered frame (not exact).

Overload and overflow. The output buffer of the encoded frames feeds the decode thread. When the decoder can’t keep up (or its thread is stalled), the overflow policy drops the encoded frames beyond CUSTOM_2, up to the next key frame (requested with PLI / FIR). So the decoder continues with a decodable frame and never gives a picture of a broken reference chain. When the caller is slow, the buffer of the decoded pictures overflows. Every picture is complete in itself, so only the oldest picture is dropped, and decoding goes on. -maxlatency bounds the age of the buffered pictures in the same way; the newest buffered picture is kept.

Frame buffer. getFrame(…) reuses the buffer of the caller’s Frame object when two conditions hold. First, it is the buffer that the library allocated for this Frame object at an earlier call: the data pointer, width, height and frameId are as the library left them. (A Frame that the caller creates anew, even at the same address, is never taken for the previous one.) Second, the buffer is large enough. Otherwise getFrame(…) releases the buffer and allocates once, in the shape of the delivered frame. The size follows the Frame convention: width * height * 4 bytes for H264 / HEVC / JPEG, and the exact picture size for the uncompressed formats. A smaller picture, another format or another codec then costs no allocation. The library keeps track of up to 16 Frame objects per source (for example, an application ring of frames); beyond that, the least recently used one is forgotten and gets a new buffer at its next call. In the test application, all 10 uncompressed formats are delivered in one buffer when the first format is RGB24 (the largest).

Threads and resources. The decode thread runs once uncompressed frames have been requested since the last initialization. It parks without CPU use while encoded frames are requested. It ends with closeVSource(), initVSource(…), RESTART and the destructor. It inherits the scheduling policy and CPU affinity of the thread whose getFrame(…) started it. The worker threads that the decoders and the converter start from it inherit its name “vsrtp-dec” (3 - 4 threads of that name were measured per decoding object). The decoders keep their idle worker threads until the object is destroyed.

Measured results. Measured by the “decode” group of the test application: 640x480 at 25 fps from MServer 2.1.1 over RTSP / TCP, Release build, one run. From reception (receiveTimeUsec) to delivery of the NV12 picture, p50 / p95: 1.1 / 2.5 ms for H.264, 3.9 / 7.3 ms for H.265 and 2.9 / 4.8 ms for JPEG. For 1280x720 H.264 to YUV24: 3.7 ms of decoding and conversion per picture, and 0.21 ms (median) for a getFrame(…) of a ready picture.

Every delivered picture is bit-exact with the picture of a test-owned decoder of the same kind and settings, and of a test-owned FormatConverter for the other formats. This holds for H.264, H.265 and JPEG (all 10 output formats with H.264), for RTP and MPEG-TS, for limited range content (all pictures expanded) and for full range content (none expanded). B-frame streams from ffmpeg (libx264 / libx265, closed and open GOP, decoding started mid-stream, late joins) are delivered in display order, with the timestamp and key frame flag of each picture’s own frame.

Output latency budget

The FIFO output buffer never drops a frame while it has room (see Frame delivery). So a consumer that stalled, for example for 1.5 s on a slow disk, receives the buffered 1.5 s of video afterwards. It stays 1.5 s late until the buffer has drained, and the buffer drains only as fast as the consumer is faster than the stream. The output latency budget -maxlatency MS (1.1.0; 10 - 60000 ms; 0 = off, the default) bounds the age of the buffered frames:

  • In every getFrame(…), and whenever a frame is buffered, the library compares the age of the oldest buffered frame with the budget. The age is now minus the arrival of the frame’s last packet, as for CUSTOM_1. When the oldest frame is older than the budget and a newer key frame is buffered, every frame before the newest buffered key frame is dropped. The consumer then continues, decodable, at the latest key frame.
  • Without a newer buffered key frame nothing is dropped, because dropping non-key frames would break the GOP. So the budget acts at the next key frame. Right after a stall, the latency is about the budget plus up to one GOP. After that, the consumer is back at the live edge.
  • Dropped frames count in framesLatencyDropped, framesOverflow and CUSTOM_3, and in lostFrames of getTelemetry(…). Their metadata units are carried to the next delivered frame (not exact). The overflow policy of the buffer (CUSTOM_2 frames, 256 MiB) does not change.
  • The budget is also the threshold of the TCP latency guard of the adaptive transport, instead of 1000 ms.

Choose as budget the largest latency that the application accepts, and at least a few frame intervals (for example 200 ms at 25 fps). A consumer that keeps up is never affected, because its oldest buffered frame is never older than a frame interval or two. Measured in the “lowlat” group (two runs): after a 1.5 s stall of the consumer (buffer depth 100), the first frame delivered after the stall was 659 ms, 660 ms old with -maxlatency 200, against 1460 ms, 1461 ms without the budget. 20 frames were dropped, and the next frames were back at the live edge.

Bounded key frame wait

By default, a packet that is really lost costs its frame and every non-key frame up to the next key frame (the key frame gate, see Frame delivery). The decoder never gets a frame with a broken reference, but the picture freezes for up to one GOP. With a 4 s GOP, this is up to 4 s, unless the server answers the key frame request earlier. -nokeywait delivers the frames after a loss at once, with decoding artifacts until the next key frame. -keywait MS (1.1.0; 1 - 60000 ms; 0 = unbounded, the default) is the compromise between the two:

  • After a loss, non-key frames are skipped for at most MS. The time counts from the arrival of the frame that closed the gate to the arrival of the skipped frame. After that, the non-key frames are delivered again until the next key frame (decoding artifacts are possible).
  • When the bound expires, a key frame request (RTCP PLI / FIR) is sent if the server supports it. The WARNING “No key frame MS ms after a loss: delivering non-key frames until the next key frame” is logged (rate-limited).
  • The bound applies to the key frame gate after packet loss and after a sender restart. The first frame of a stream (after a connect, a reconnection or a codec switch) is still always a key frame. The overflow policy of the output buffer still drops frames up to the next key frame.
  • -keywait keeps the key frame wait mode: the key-frame bypass of the reorder wait and the 10 ms default minimum wait stay (see Low latency design and adaptive jitter buffer). It can’t be combined with -nokeywait.
  • framesFrozenMsec counts the time during which frames were skipped. Measured in the “adapt” group (two runs), with one lost packet 23 frames (920 ms) before the next key frame: 920 ms, 923 ms frozen by default, and 240 ms, 239 ms with -keywait 220. The freeze ends with the first frame after the bound, which comes 6 frame intervals of the 25 fps test stream after the damaged frame. A bound that ends exactly on a frame (for example -keywait 200) lets that frame pass or not, depending on microseconds of arrival time: repeated runs gave 200 ms and 240 ms.

Low latency design and adaptive jitter buffer

Minimum latency is the primary design goal:

  • There is no playout buffer and no pacing. A frame goes to the output buffer the instant its last packet arrives. The library detects the last packet by the marker bit, by a timestamp change, or by the MPEG-TS end of access unit detection. A caller blocked in getFrame(…) wakes at once. Only the reordering / loss handling and the metadata binding may hold data. Both hold times are adaptive, bounded and minimal.
  • The single receive thread sleeps in ppoll() until the next packet or the next timer (microsecond resolution, at most 100 ms). It never busy-waits. UDP packets carry their kernel receive time. Every TCP read re-arms TCP_QUICKACK. The socket backlog is read before a deadline expires. The thread can run with SCHED_FIFO on a CPU of its own. See Receive thread, kernel timestamps and scheduling; most of these features are Linux only.
  • In steady state, the library’s own code makes no heap allocation per packet or per frame. (The crypto backend may allocate, for example in the HMAC of AES-CM SRTP.) Buffers are recycled between the receive thread and getFrame(…), which copies each frame once into the caller’s Frame object. getTelemetry(…) reuses the buffers of the caller’s structure.
  • UDP sockets request 8 MiB receive buffers, so that the bursts of large key frames fit into the kernel buffer. Drops are counted (see Receive thread, kernel timestamps and scheduling).
  • The library detects the end of an MPEG-TS access unit by these signals, from the lowest to the highest latency: (1) the PES_packet_length is reached; (2) a datagram ends, and the last video packet carried stuffing, or another PID followed the video payload, or the datagram is shorter than the largest datagram of the stream (the last two rules apply only after 8 PES ends confirmed them); (3) the next payload unit start on the video PID; (4) an RTP timestamp change. When the PES packets do not carry whole access units (GStreamer NAL-aligned), the access units are split by the codec rules: AUD, first slice of a new picture, parameter sets or SEI after a VCL NAL unit, PTS change. One case gets no early signal: an access unit whose payload fills its last TS packet exactly (no stuffing), when that packet is also the last one of a full datagram. Such an access unit ends with the next frame’s first packet, one frame interval later. It is rare (for ffmpeg’s muxer with 564-byte datagrams, 1 of 100 frames of the test clip), and it is kept that way on purpose: an idle-gap timer on every full datagram would expose every access unit of such a stream to a cut while it is still in transfer.
  • Idle-gap frame end. Some senders have no reliable RTP marker bit, and some MPEG-TS frames have no end signal. For them, a frame is released after a learned idle gap after its last packet. The gap is twice the largest packet gap inside the last 16 confirmed frames, within 1 ms - half a frame interval. The rule is active after 8 confirmed frames. For H.264 / H.265 over RTP, a raised gap is held for 10 s (or for twice the key frame interval, if that is longer). Then it decays by x0.8 every 500 ms, but never below twice the largest recent gap and key-frame gap, so a one-off pause ages out. For MPEG-TS, the gap never shrinks. A confirmed frame that the rule would have split turns the rule off until 8 new confirmations; meanwhile the gap is one frame interval. Measured: the frames of marker-less RTP, of RTP with the marker on every packet and of unsignalled or NAL-aligned MPEG-TS end about 1.2 ms after their last packet (p95 below 1.6 ms), instead of one frame interval (40 ms at 25 fps). For such senders, a loss of the last packets of a frame can’t be detected before that frame is delivered: the loss shows only with the next packet. That frame may then be delivered without its tail; this is the one exception to “a frame with a missing packet is never delivered”. The frames after it are skipped until the next key frame. The same happens when the sender pauses inside a frame for longer than the learned gap. The rule then learns again (8 confirmations) with at least twice that gap, so one stall of the sender does not cost the rest of the session. After the fourth such pause, or when the gap was at half a frame interval already, the rule stays off for that stream. A marker on a packet that leaves a fragmented NAL unit unfinished (an FU without its end bit) never ends a frame.

Adaptive reorder (jitter) buffer. The reorder buffer of every RTP stream is the only jitter buffer:

  • Packets in sequence pass straight through, without a copy. Packets are held only behind a gap. They are held until the missing packet arrives, or until the gap start plus the current wait. At that time the missing packets are declared lost, and delivery continues.
  • The wait adapts: wait = clamp(max(minimum, 1.25 x estimate), -mindelay, -maxdelay). The default range is 10 - 200 ms (3 - 200 ms with -nokeywait). The estimate is the peak of the observed lateness, first held and then decaying. Since 1.1.0 there is no jitter term. (In 1.0.0 the jitter term pinned the wait at the maximum after queue drains and never helped.)
  • Default minimum 10 ms (1.1.0; 1.0.0: 3 ms). In the key frame wait mode, the frames behind a real loss are skipped anyway. So the longer wait delays only the frames that are kept after a loss: the next frame when the loss lies between two frames, the key frame after a skip (unless it is complete in the buffer), and the frames that -keywait lets through. In exchange, reordering by up to 10 ms (Wi-Fi, LTE, bonded links) no longer costs the rest of a GOP (measured over 124 channel traces). With -nokeywait every frame behind a gap waits, so the minimum is 3 ms there. -mindelay overrides both defaults.
  • Growth on impairment. A packet that fills a gap in time raises the estimate to its lateness: estimate = max(estimate, lateness). A packet that arrives after it was declared lost is dropped and counted in packetsLate. It raises the estimate to 1.5 x its lateness at once, so the next gap waits long enough.
  • Hold, then decay. After a lateness sample of at least half the peak, the peak is held for 10 s. Then it is multiplied by 0.8 every 500 ms. So recurring bursts of reordering (Wi-Fi, LTE) find the wait still in place, and after the network has recovered, the wait returns to the minimum within seconds. Smaller reordering does not hold a past peak: while it goes on, the wait settles at about twice its size.
  • Key-frame bypass (only in the default key frame wait mode, not with -nokeywait). The packets held behind a gap may contain a complete key frame whose predecessor is missing. The gap is then declared lost at once, and the key frame is delivered without waiting: the frames before it would be skipped anyway. A complete key frame here is a contiguous, marker-terminated run of packets that starts with a key-frame start, also when an access unit delimiter leads that start. A key-frame start is, for H.264, a single IDR or SPS NAL unit, a STAP-A that contains one, or an IDR FU-A start; for H.265, the start of a single, aggregated or fragmented IRAP picture, VPS or SPS; for JPEG, fragment offset 0.
  • TCP zero wait. RTP interleaved in TCP / TLS or in a tunnel can’t be reordered. The wait is 0, and a gap is an immediate loss.
  • NACK / RTX. When the server offers NACK (a=rtcp-fb nack), missing packets are requested at once, and the wait is at least the estimated retransmission delay. This delay is held for 10 s, like the reordering peak, and capped by -maxdelay. It is no longer added after 8 unanswered gaps in a row. See NACK, RTX and key frame requests.
  • Bounds. At most 2048 packets or 8 MiB are held. Beyond that, the oldest gap is declared lost at once.
  • The internal diagnostics report the current wait as reorderWaitUsec, and the events as packetsReordered, packetsLate, packetsLost and packetsDuplicated. reorderPercent is the share of reordered packets in the last receiver report interval.

AU-boundary loss rule (1.1.0; AU = access unit, the data of one frame). A loss can lie strictly between a completed frame (its end was seen) and the first received packet of the next frame. This loss does not damage the next frame when:

  • that first received packet provably starts the frame. For H.264 it is an AUD, SPS, PPS or SEI, or a slice with first_mb_in_slice 0 (single NAL unit or FU-A start). For H.265 it is an AUD, VPS, SPS, PPS or prefix SEI, or a slice with first_slice_segment_in_pic_flag 1. For JPEG it has fragment offset 0;
  • and, for a non-key frame, its RTP timestamp follows the previous frame: the step is at most 1.5 x the median frame interval, on a stream whose timestamps never went back. With B-frames a lost reference can’t be excluded, so a B-frame stream keeps no non-key frame this way. The library learns the reordering at the first backward timestamp step. So in the first mini-GOP after a connect or a codec switch, a B-frame stream is still treated as a stream without reordering;
  • and, for a key frame, every parameter set it references is available, and no cached set changed since the last key frame. A key frame that brings a changed parameter set, or that follows one seen in the stream, is not decoded with old sets. A change whose new sets were all lost can’t be detected. Such a key frame is kept with the cached sets, and a key frame request (PLI / FIR, when negotiated, paced like every request) is sent at once. So the sender’s next key frame corrects a lost change.

Then the frame and the rest of the GOP are delivered instead of being skipped until the next key frame. For example, when the lost packet carried only the leading SEI of a P frame, that frame (without the SEI) and the rest of its GOP are delivered. (In 1.0.0 the rest of the GOP was skipped.) A whole lost frame still costs the frames up to the next key frame, because its timestamp step is too large. A key frame kept this way opens the gate like any key frame; its undecodable leading pictures are dropped.

Metadata hold bounds. A frame is held for its metadata only while the stream carries or declares metadata: a metadata track, a metadata SSRC, or a PMT with a KLV stream. With -nometadata a frame is never held.

  • Since 1.1.0, a frame waits for its unit only when this pays off. The library learns this per metadata stream. The wait pays off when the frame has fewer exact units of the stream than the stream usually sends, the metadata is dense (at least every second frame gets a unit), the units usually arrive after their frame, and their average delay is below half a frame interval. Whether a unit arrives after its frame is judged by the arrival times of the unit and of the frame’s last packet, not by the order in which the receive thread reads the video socket and the metadata socket. (1.1.0 used the read order: a unit that arrived microseconds after its frame, but was read first in every other round, kept the frames from being held and came carried.) The hold is clamp(3 x average delay, 250 us, 30 ms); for MPEG-TS it is at most 0.5 ms. At most 2 frames are held, and the next frame releases a held one. Sparse, leading and off-grid metadata (units that never match a frame) and metadata on an unrelated RTP clock cost no hold.
  • Before anything is learned, only the first frame of a stream (after every connect / reconnect) may wait for its unit: once, and for at most 10 ms (1.0.0: 30 ms). After its first unit, it waits at most 250 us for the units of the other metadata tracks. With several tracks, a frame waits (learned per track) until every dense trailing track has delivered its unit.
  • MPEG-TS frames are released as soon as the demuxer signals that no further unit can belong to them. Normally this is at the end of the datagram that completes the access unit, or when a KLV PES bound to the access unit completes. When the video payload ends a datagram and the PMT declares KLV, the release waits for the end of the next datagram. The hold deadline remains a fallback.
  • A unit that arrives later than its frame’s hold, after its frame went to the output buffer, comes with the next delivered frame (carried, not exact; 1.0.0 still attached it to a buffered frame). See Metadata binding.

Receive thread, kernel timestamps and scheduling

Every VSourceStream object has one receive thread. It performs all network input / output, protocol handling and frame assembly. No user code runs on it. (Decoded frames, 1.3.0, come from a separate decode thread; see Frame output and decoding.) The thread sleeps in ppoll() until the next packet or the next timer, and it never busy-waits. Version 1.1.0 removes the delays that the operating system adds around it, as described below.

Platform support. Most features of this section are Linux only. On Windows, the receive thread gets no name and no timer slack. -priority sets THREAD_PRIORITY_TIME_CRITICAL, and -cpu pins the thread to a CPU of processor group 0 (a failure gives one WARNING). The thread sleeps in WSAPoll(), whose timeout has a resolution of 1 ms. A UDP packet gets the poll time as its arrival time instead of a kernel receive time stamp. Kernel drops are not counted, and TCP_QUICKACK is not available.

Thread name and timer slack. The receive thread is named “vsrtp-rx” (visible in top -H, ps -L, perf and gdb). It runs with a timer slack of 1 ns (PR_SET_TIMERSLACK). So the timers that release data (reorder deadline, metadata hold, idle-gap frame end) expire on time instead of up to 50 us late.

Scheduling (-priority, -cpu). initVSource(…) creates the receive thread. The thread inherits the scheduling policy, priority and CPU affinity of the thread that calls initVSource(…). Under CPU contention, other work preempts a normal-priority thread, and the tail latency grows to milliseconds. Two options change that:

  • -priority N (1 - 99) runs the receive thread with SCHED_FIFO priority N. This needs CAP_SYS_NICE or an RLIMIT_RTPRIO of at least N: for example ulimit -r 50, an rtprio entry in /etc/security/limits.conf, or setcap cap_sys_nice+ep on the executable. On success, the INFO line “Receive thread runs with SCHED_FIFO priority N” is logged. On failure, one WARNING is logged: “Cannot run the receive thread with SCHED_FIFO priority N: … (needs CAP_SYS_NICE or RLIMIT_RTPRIO >= N); it keeps the inherited scheduling”. The source then works with the inherited policy.
  • -cpu N (0 - 1023) pins the receive thread to CPU N. On success, the INFO line “Receive thread pinned to CPU N” is logged. A failure (a CPU that does not exist, or one outside the cpuset of the process) logs one WARNING: “Cannot pin the receive thread to CPU N: …; it keeps the inherited affinity”. The thread then keeps the inherited affinity. Pinning helps only on a CPU that no other busy thread uses, for example a CPU isolated with isolcpus. On a shared CPU, pinning makes the tail latency worse.
  • Raise the thread that calls getFrame(…) too. A frame that the receive thread releases on time still waits until the consumer thread is scheduled. So under CPU contention, only both threads together give the full effect. Measured with 1.1.0: p99 4 - 10 ms -> 0.8 - 1.7 ms, only with both threads prioritised. Give the consumer thread SCHED_FIFO at the same priority as the receive thread, or at a slightly lower one.

An application whose initializing thread already runs with the wanted policy does not need -priority: the receive thread inherits the policy. Example of a consumer thread that raises its own priority and opens the source with -priority 50:

// Consumer thread: SCHED_FIFO just below the receive thread (-priority 50).
sched_param parameters{};
parameters.sched_priority = 49;
if (pthread_setschedparam(pthread_self(), SCHED_FIFO, &parameters) != 0)
{
    std::cout << "Consumer thread keeps its normal priority" << std::endl;
}
std::string initString = "rtsp://192.168.1.67:554/live -priority 50";
source.openVSource(initString);
cr::video::Frame frame;
while (source.getFrame(frame, 1000))
{
    // Hand the frame to the decoder.
}

Kernel receive timestamps (UDP). Every UDP socket enables SO_TIMESTAMPNS: the kernel stamps each datagram when it arrives. The library converts the stamp to the steady clock (CLOCK_MONOTONIC). A stamp that would lie in the future, or more than 1 s in the past (after a wall clock step), is replaced by the poll time. The kernel time is the arrival time of the packet for the RFC 3550 jitter (jitterUsec), for the reorder lateness and the gap deadlines, for the idle-gap frame end, for the frame receive time (receiveTimeUsec of getTelemetry(…), converted back to the wall clock at delivery) and for CUSTOM_1.

So a stall of the receive thread neither inflates the reorder wait nor hides itself: CUSTOM_1 (the age of the delivered frame) includes the kernel queueing and the thread wake-up. A large CUSTOM_1 with a small bufferedFrames means that the receive thread was late (CPU contention: raise its priority). A large CUSTOM_1 with a large bufferedFrames means that the consumer is late. RTSP over TCP, TLS and the HTTP tunnel have no per-packet kernel stamp: their arrival time is the time the data was read. Measured in the “lowlat” group, with the receive thread stopped for 300 ms by a thread-directed signal whose handler sleeps: the 7 frames sent during the stall carried their kernel receive time (receiveTimeUsec), within 5 ms of their send time. CUSTOM_1 of the first frame that waited included the wait (260.7 ms, 261.0 ms).

Kernel receive buffer drops. Every UDP socket also enables SO_RXQ_OVFL. The kernel then reports how many datagrams it dropped because the receive buffer of the socket was full: the receive thread fell behind, or net.core.rmem_max limits the 8 MiB buffer request. The drops are counted in kernelDrops. At most once per 10 s per session, they are reported by the WARNING “Kernel receive buffer overflow: N datagrams dropped (total M); the receive thread falls behind or the buffer is too small. Raise the receive thread priority (-priority N, -cpu N; also for the thread calling getFrame()) or the buffer limit (sysctl -w net.core.rmem_max=8388608)”. The dropped RTP packets of the stream also show as packetsLost, because they never reached the library. But the adaptive transport does not take such loss for network loss.

Socket backlog before deadlines. At most 256 datagrams are read per socket and poll round. A timer may be due after a round that read data while a socket still holds datagrams. This happens after a stall, for example a preemption of the receive thread followed by a key frame burst. The timers then wait until the backlog is read, for at most 8 rounds (2048 datagrams per socket), so a permanent flood can’t starve them. So a packet that arrived in time is never declared lost because it was still queued in the kernel. With RTSP over UDP, the metadata socket is read before the video socket in every round, so the metadata units that are already queued still find their frames.

TCP_QUICKACK. Linux may delay a TCP acknowledgement by 40 ms or more. Some cameras and servers send with Nagle’s algorithm (TCP_NODELAY off). Such a server holds the last, small segment of a frame until the previous segment is acknowledged. Without a countermeasure, the tail of a frame then waits for the delayed ACK of the receiver. The library re-arms TCP_QUICKACK after every successful read of its TCP connections, so every segment is acknowledged at once. This covers plain TCP, every socket read of a TLS connection (including the handshake), RTSP control and interleaved media, and both legs of the HTTP / HTTPS tunnel. Measured against a Nagle sender with 1.1.0: p95 33 ms without TCP_QUICKACK, 0.12 ms with it. In the “lowlat” group, no frame of a Nagle RTSP / TCP sender, of RTSPS or of an HTTP tunnel behind a Nagle relay took longer than 30 ms (p95 0.22 - 2.01 ms, see Table 11). The library’s own connections use TCP_NODELAY.

RTCP

  • Identity. The receiver SSRC and the CNAME (16 random bytes, base64) are drawn from the DRBG, the cryptographic random generator of CryptoProvider.
  • Random values in batches (1.1.0). The random values that randomize the report interval are drawn 64 at a time, in one DRBG call, at initialization and from the report timer. So no DRBG call runs on the packet path inside a frame burst.
  • Receiver reports. The library sends one RR + SDES CNAME compound packet per track, with RFC 3550 report blocks (fraction lost, cumulative lost, extended highest sequence number, jitter, LSR, DLSR). It sends one every 1 s, randomized by +-50 %. The first one goes out at the first RTP packet or SR. Destinations: the RTCP port of the server (RTSP over UDP), the interleaved RTCP channel (TCP), group:port + 1 with TTL 1 (multicast), and the source address of the sender’s SR, if it is the address of the RTP packets of that SSRC (SDP-less modes). For SRTP media, RTCP is protected with SRTCP. RTCP is never sent through the HTTP tunnel. Counted in rtcpRrSent.
  • Sender reports map RTP timestamps to the wall clock of the sender. This mapping gives the capture time of getTelemetry(…) when the frame carries no better time (see Frame telemetry and capture time). It also gives the internal latencyUsec, and it binds the metadata of a stream that uses another RTP clock. Counted in rtcpSrReceived.
  • BYE. A BYE from the server ends an RTSP session, and the client reconnects. In SDP-less modes, a BYE retires the SSRC, and another SSRC takes over at once. On close, the client sends BYE after TEARDOWN.
  • Feedback (only when the SDP negotiates it with a=rtcp-fb; never through the HTTP tunnel). The library sends a generic NACK (RFC 4585) for missing packets, a PLI for key frame requests after loss or overflow, and a FIR (RFC 5104) for a decoder start or when only FIR is negotiated. See NACK, RTX and key frame requests. Counted in nackSent and keyFrameRequests.
  • RTCP on the RTP port (rtcp-mux) is recognized by its packet type.

NACK, RTX and key frame requests

RTCP feedback (RFC 4585) lets the receiver repair losses of RTP over UDP. A generic NACK (negative acknowledgement) asks the server to send lost packets again. The server sends them as RTX retransmissions (RFC 4588). When repair is not possible, a PLI (picture loss indication) or a FIR (full intra request, RFC 5104) asks the encoder for a new key frame. The library uses only what the SDP of the server negotiates (a=rtcp-fb), and it never sends feedback through the HTTP tunnel. Over TCP nothing is lost, so no NACK is sent there; key frame requests go over the interleaved RTCP channel. SDP-less sources (rtp://, udp://, srtp://) have no negotiation and send no feedback.

NACK and RTX (a=rtcp-fb:<pt> nack):

  • Request and retry. Every missing packet is requested with a generic NACK as soon as its gap is seen. If the packet has not arrived, it is requested once more. The time of this retry depends on the server. For a server that answers within half the reorder wait, the retry goes out after twice its measured answer delay (at least 1 ms, at most half the wait). So both answers come within the wait, and a lost retransmission is repaired about three answer delays after the loss (at a 3 ms answer delay and a 200 ms wait: after about 9 ms instead of about 104 ms with a retry after half the wait). For a server that answers later, the retry goes out at the expected answer time: 1.25 x the measured answer delay, but no later than 1 ms before the wait ends. So the retry goes out only when the first answer is really missing; a retry before the answer would cost one more NACK and a duplicate retransmission per loss. The answer to the retry then comes after the wait, as a late answer that raises the wait, so later retries repair in time. 1.1.0 retried after max(wait / 2, 20 ms). With a fast server, whose reorder wait stays at the 10 ms minimum, that retry was never sent, and a lost retransmission stayed lost.
  • Tail-loss probe. A lost LAST packet of a frame (its marker packet) leaves no gap until the next frame starts, one frame interval later (40 ms at 25 fps). So the library probes. When a frame is open without its marker packet, and no packet has arrived for a learned silence after its last packet, one NACK asks for the 4 sequence numbers after that packet. (A server ignores the ones it has not sent yet.) The frame then completes one answer later. The silence is twice the second largest packet gap inside the last 128 frames that ended with their marker packet, within 1 ms - a quarter frame interval. It is learned from the first 8 such frames; before that, no probe is sent. A probe is false when its packet then comes as an original (it was only late; also after its retransmission). A false probe doubles the silence, up to 64 x and at most one frame interval. The silence is halved again after every 5 s without a false probe. There is at most one probe per frame. A probe that stays unanswered falls back to the NACK of the gap that the next frame shows. The probe works only for RTP video over UDP with NACK negotiated, from a sender whose marker bit ends every frame (not for marker-less senders, MPEG-TS, metadata or TCP), and not after the give-up described below. A retransmission beyond the highest received sequence number that no probe asked for is dropped and counted as a duplicate. Probes are counted in tailProbes, tailProbeRepairs and tailProbeFalse (and in nackSent).
  • RTX packets (a=rtpmap:<pt> rtx/<rate> with a=fmtp:<pt> apt=<video pt>, multiplexed by SSRC on the video port) are unwrapped into the original packet, with the original sequence number, and fill the gap. They give no jitter sample. An RTX of a packet that arrived anyway is a duplicate. reorderPercent does not count repairs, and packetsLost counts only the packets that were never repaired.
  • Wait for the retransmission. The reorder wait covers the retransmission delay: it is at least the estimated time from the gap to the arrival of its RTX (initially 50 ms). A larger sample raises the estimate at once and holds it for 10 s. During the hold, smaller samples are ignored, and a sample of at least half the estimate restarts the hold. After the hold, the estimate follows the samples down in 1/8 steps. The estimate is capped by -maxdelay.
  • Give-up. Some servers negotiate NACK but do not answer. When 8 gaps in a row expire without any retransmission, the retransmission delay is no longer added to the wait, and the reorder wait falls back to its reordering estimate. NACKs are still sent, and the first repaired gap adds the delay again. With RTX negotiated, only an RTX packet counts as a retransmission: a requested packet that arrives as the original was only reordered, so it neither repairs a gap here nor gives a retransmission delay sample. (Without RTX, a retransmission is an exact copy of the original, so every arrival of a requested packet counts.)
  • In the default key frame wait mode, a longer wait costs no frame that would otherwise be delivered. The frames behind an unrepaired loss are skipped anyway, and a complete key frame behind the gap is released at once (key-frame bypass).

Key frame requests (a=rtcp-fb:<pt> nack pli, a=rtcp-fb:<pt> ccm fir):

  • A key frame is requested: when a frame is lost and the key frame gate closes; when a key frame after a loss is kept with the cached parameter sets (the AU-boundary loss rule; the loss may have carried a parameter set change); when the output buffer overflows; when -keywait expires; and for a decoder start (a stream joined in the middle of a GOP, after a connect or a codec switch).
  • PLI is used whenever the server negotiated it, because RFC 5104 forbids FIR as a reaction to losses. FIR is used only when PLI is not negotiated, or for a decoder start.
  • The answers of the server pace the requests. A request is repeated after 1.5 x the observed key frame response time (from the request to the key frame arrival), within 150 - 1000 ms (500 ms until a response time is measured; 1.0.0: always 500 ms). A damaged key frame that arrives while a request is outstanding counts as the answer, and the request is repeated at once. The pacing only spaces out the repetitions of one request: once a key frame answers it (and after a codec switch), the request after a new loss goes out at once. Every PLI / FIR counts in keyFrameRequests, and every NACK packet in nackSent.

With MServer 2.1. MServer is the reference server of the test application (2.1.4 since VSourceStream 1.4.1, 2.1.3 since 1.4.0, 2.1.2 since 1.3.1, 2.1.1 since 1.2.0, 2.1.0 before). It offers NACK / RTX and PLI / FIR for RTP over UDP. Its H.264 SDP carries a=rtcp-fb:96 nack, a=rtcp-fb:96 nack pli, a=rtcp-fb:96 ccm fir, a=rtpmap:99 rtx/90000 and a=fmtp:99 apt=96;rtx-time=3000. (H.265: video payload type 97, RTX payload type 100.)

MServer 2.1.0 answered NACKs with retransmissions 40 - 80 ms after the loss, because it read RTCP over UDP only every 50 ms; this was the cost of a repaired frame. 2.1.1 reads the RTCP sockets of UDP sessions in the poll set of its reactor and answers at once. In pass-through mode (pre-encoded video) MServer can’t produce a key frame on request. In raw-encoder mode it answers PLI / FIR, but ignores requests within 200 ms of the last one. For JPEG, 2.1.0 announced NACK but no RTX payload type. So its retransmissions were ignored, and each loss cost only its own frame (JPEG frames are independent). 2.1.1 announces the JPEG RTX payload type 99 (m=video <port> RTP/AVP 26 99, a=rtpmap:99 rtx/90000, a=fmtp:99 apt=26;rtx-time=3000), so JPEG losses are repaired like H.264 losses.

Measured with VSourceStream 1.2.0 in the “feedback” group against MServer 2.1.1, in two runs. The loss is on the server-to-client leg, and the retransmissions cross the same lossy leg:

  • The retransmission arrived 0.29 - 0.60 ms (p50) after the loss.
  • At 0.5 %, 2 % and 5 % packet loss, every lost H.264 packet was repaired: 0 packets lost after repair, 0 frames lost, 0 ms frozen. The frame latency grew by at most 0.12 ms (mean) and 0.20 ms (p95) at 0.5 % and 2 %, and by 1.31 ms, 0.97 ms (p95) at 5 %. (The tail-loss probe requests a lost last packet of a frame after about 1 ms instead of with the next frame; before the probe, this cost +39 ms.)
  • With the last packet of every fifth frame lost, the p95 grew by 0.88 ms, 1.28 ms.
  • JPEG at 2 %: all 62 and 61 lost packets were retransmitted in time, and 0 frames were lost.
  • Without RTX, the same loss cost 22 - 107 frames and up to 6.3 s of frozen video at 0.5 % and 2 %, and nearly all of the stream at 5 % (13 and 6 of 152 frames delivered).

See Table 13. The adaptive transport stays on UDP while RTX repairs the loss.

Metadata binding

KLV (SMPTE 336M, for example the MISB ST 0601 UAS Datalink Local Set) and ONVIF metadata travel next to the video in several ways. The library binds every metadata unit to a video frame by its timestamp, and returns the unit with that frame. The metadata field of getTelemetry(…) holds one VSourceStreamMetadata per unit: type, stream index, exact and valid flags, and bytes. Metadata is received from these sources:

  • RTSP metadata tracks: m=application sections with rtpmap SMPTE336M (KLV, RFC 6597) or VND.ONVIF.METADATA (ONVIF XML). Up to 2 of them are set up after the video track, in SDP order (1.1.0; 1.0.0: only the first one), for example a KLV track and an ONVIF track. A section is set up only if it has its own control URL: not empty, not “*”, and not the URL of the video, of the aggregate or of another metadata track. A section without its own URL is skipped with the WARNING “Metadata track skipped: no own control URL”, and the search goes on. With -srtp, the section must be RTP/SAVP with a usable key; otherwise it is skipped with the WARNING “Metadata track skipped: no SRTP offer”. Each track has its own stream index (0, 1 in SETUP order). No metadata track is set up when the video is MPEG-TS, because its KLV comes from the TS. A KLV unit is complete at the RTP marker bit or when its BER length is satisfied. An ONVIF document is complete at the marker bit or at a timestamp change (1.1.0: for senders without a marker, and for senders that set the marker on every packet). For a sender without a marker, a document is also complete as soon as its root element is closed, so an event document is delivered at once, not with the next document. Each unit can have at most 1 MiB.
  • MPEG-TS: the KLV elementary streams of the program, in the layouts of Table 7. A KLV PES, and a KLV item reassembled across PES or metadata AU cells, can have at most 64 KiB. A larger PES or reassembly is dropped. So a KLV item longer than 64 KiB is never delivered whole: it is dropped, or it comes as truncated pieces with valid FALSE.
  • SDP-less RTP: KLV or ONVIF units on their own SSRC, on the same port as the video (MServer’s rtp-klv push stream). Every metadata SSRC takes the next free stream index when it first appears.
  • SEI (only with -seidata, 1.1.0): the user_data_unregistered messages of the H.264 SEI and of the H.265 prefix / suffix SEI (x264 options, MISB ST 0604 time stamps, vendor data). They become units of type UNKNOWN (1.4.0; 1.3.x: SEI), the type of user data whose format the library does not interpret: the 16-byte UUID that names the format, followed by the payload, with the emulation prevention bytes removed, and with stream 0. The SEI units of the frame’s own access unit are exact. The SEI units of access units that were not delivered (skipped before a key frame or after a loss, corrupt, without a picture, oversized, dropped by the output buffer or by -maxlatency) are carried to the next delivered frame, not exact. SEI NAL units above 64 KiB are not scanned, and at most 16 messages are taken per access unit. The SEI also stays in the frame.

At most 4 metadata streams are bound per source. The units of a fifth stream count in metadataDropped.

Table 7 - MPEG-TS KLV layouts.

Layout Signalled by Binding
Asynchronous KLV stream type 0x06 with the “KLVA” registration descriptor, PES stream id 0xBD to the video access unit started last before the KLV PES (by position); a KLV PES that carries a PTS further than min(450 ticks, 1/4 frame step) from that access unit binds by its PTS (B-frames)
Synchronous KLV stream type 0x15 with the “KLVA” metadata descriptor, PES stream id 0xFC, metadata access unit cells by PTS, after the 5-byte cell headers are removed; 1.1.0: several cells per PES and fragmented cells (cell_fragment_indication) are reassembled
Raw KLV in a metadata PES (1.1.0) a stream type 0x15 PES that starts with the KLV key 06 0E 2B 34 instead of a cell list (ffmpeg -c copy writes it) by PTS
KLV muxed before its video (1.1.0) an asynchronous KLV PES that starts before the video PES of its access unit (seen on 3 consecutive PES) to the next video access unit
KLV spanning several PES (1.1.0) a KLV item longer than its PES reassembled (at most 64 KiB; a longer item is never delivered whole: it is dropped or comes as truncated pieces with valid FALSE); the KLV structure decides where items end
Several KLV PIDs (1.1.0) up to 4 KLV elementary streams in the PMT each PID its own stream index, in PMT order; a new PID takes the lowest free index
Undeclared KLV (1.1.0) stream type 0x15 without descriptor, 0x06 / 0x15 with a program-level “KLVA” registration accepted when the PES packets start with the KLV key (an elementary stream with another format descriptor - AC-3, AAC, DTS, DVB subtitles, teletext - is never taken)

Binding (1.1.0: a unit is never lost because of frame timing). The library maps the timestamp of an RTP metadata unit to the video clock. When both streams use one clock (MServer), the mapping is direct. Otherwise it goes through the RTCP sender reports of both streams. Before the first sender reports, it goes through the RTSP RTP-Info of both tracks, with the metadata clock rate of the rtpmap (for example SMPTE336M/1000). Two sender reports may describe two states of the sender clock: their NTP times, compared with their arrival times, differ by more than 100 ms. This happens after a wall clock step between them, or with gst-rtsp-server, which moves its NTP time by about 1.8 s a few seconds after PLAY, one stream after the other. Such sender reports are not combined: the mapping in use stays until the next sender report. MPEG-TS units use the PTS. Every unit is bound in one of four ways. Only the first way makes it exact (exact TRUE: the unit describes exactly this frame):

  • exact: its timestamp equals the timestamp of the frame within +-min(450 ticks of 90 kHz, 1/4 frame step). The SEI units of the frame’s own access unit are exact too; they are the only exact SEI units.
  • since the previous frame: its timestamp lies between the exact windows of the previous frame and of this frame (metadata at a higher rate than the video, or off the frame grid). The “previous frame” is the frame with the highest timestamp so far, because with B-frames the frames arrive in decoding order.
  • carried: the unit missed its own frame. It arrived after that frame went to the output buffer, even when the application has not read that frame yet (1.0.0 still attached it to a buffered frame). Or its frame was dropped: by the key frame wait after a loss or before the first key frame after a connect, reconnect or mid-GOP join, as incomplete, by an overflow or by -maxlatency. With -seidata, the SEI units of such access units are carried too. Or its clock can’t be related to the video clock. A carried unit comes with the next delivered frame whose timestamp is not before its own; a unit without a timestamp comes with the next delivered frame. At most 64 units / 1 MiB are carried, SEI units included. Beyond that, the oldest units are dropped and counted in metadataDropped.
  • untimed: the unit has no timestamp (for example MPEG-TS synchronous KLV without PTS). It is bound to the frame being assembled, or to the next one.

Exact units describe the delivered frame. Units since the previous frame describe the time since the previous frame. Carried and untimed units belong to another moment. The exact flag tells only the first kind apart. So place every unit that is not exact by the time inside it, not by the frame it came with: MISB ST 0601 Tag 2 of a KLV UAS Datalink Local Set, ONVIF UtcTime, or the MISB ST 0604 time of an SEI unit. Also use its position: the units that are not exact come in the order of their own timestamps (see Returned data), but those timestamps are not returned. A unit without such an embedded time (for example KLV that is not an ST 0601 local set, or SEI user data other than ST 0604) can therefore only be ordered, not placed. For such units, a unit since the previous frame and a carried unit can’t be told apart. A frame waits for its own unit only when that pays off; the hold bounds are described in Low latency design and adaptive jitter buffer.

Returned data. getTelemetry(…) returns the units bound to the frame that the last successful getFrame(…) call returned. A frame gets at most 256 units and 4 MiB; further units come with the next frame, within the carry bound above. The units are returned as received: KLV as the KLV unit (one or more KLV items), ONVIF as the XML document, MPEG-TS KLV as the KLV bytes of the PES, and the SEI user data (UNKNOWN) as UUID and user data. The order is: the exact units first (KLV / ONVIF in arrival order, then SEI in bitstream order); then the other units in the order of their own timestamps (arrival order for equal timestamps); and the units without a usable timestamp last. So when the frame has an exact KLV / ONVIF unit, the first element of metadata is the frame’s first exact KLV / ONVIF unit. (Without -seidata, every unit is KLV / ONVIF.) Each VSourceStreamMetadata unit gives:

  • type: KLV, ONVIF or UNKNOWN (the SEI user data of -seidata) (VSourceStreamDataType); a delivered unit never has the type NONE, and a frame without metadata has an empty metadata vector;
  • stream: the stream index, so that the units of two KLV tracks of one source can be told apart. RTSP metadata tracks are numbered in SETUP order, SDP-less metadata SSRCs in the order of first appearance, and MPEG-TS KLV PIDs in PMT order; the UNKNOWN (SEI) units have 0. The full rule is in VSourceStreamMetadata structure;
  • exact: see above;
  • valid: the unit passed the structure checks and, when present, the MISB ST 0601 checksum. For KLV: the universal label is 06 0E 2B 34, the BER lengths are consistent, the items fill the unit exactly, and every MISB ST 0601 local set ends with a correct Tag 1 checksum. ONVIF and UNKNOWN (SEI) units are always valid (see below). A unit that fails these checks is delivered anyway, with valid FALSE; ST 0601 lets the application discard it. Two kinds of units are dropped instead: a unit broken by packet loss, and a unit of an RTP metadata track or SSRC that does not start like a unit. KLV must start with the key prefix 06 0E 2B 34. ONVIF must start as an XML document: ‘<’ as its first byte, or after a UTF-8 BOM and optional XML white space. After XML white space without a BOM, for a unit whose first packet follows a packet loss, and for a unit that goes on with the RTP timestamp of a unit dropped for its start, only <?xml or the MetadataStream start tag is accepted (with any namespace prefix, for example <tt:MetadataStream). At most 1024 bytes of BOM and white space may come before the ‘<’. So every delivered ONVIF unit is valid;
  • data: the bytes of the unit.

The internal diagnostics (see Internal diagnostics) report the formats seen as metadataFormat (“KLV”, “ONVIF”, “SEI”, joined by ‘+’). They count the units in metadataUnits, metadataCarried, metadataDropped and metadataUnbound (units still waiting when the stream restarts at a reconnect, or when the source closes). -nometadata disables KLV / ONVIF metadata completely. Metadata-only streams are not supported.

Frame telemetry and capture time

getTelemetry(…) (1.2.0) returns the VSourceStreamTelemetry of the frame that the last successful getFrame(…) returned. getFrame(…) takes the telemetry together with the frame, so both describe the same frame (frameId equals Frame::frameId). The telemetry places the frame in three time bases:

  • Stream timeline: timestamp is the 90 kHz media timestamp of the frame: the RTP timestamp, or for MPEG-TS (raw or in RTP) the low 32 bits of the PES PTS. keyFrame marks a key frame: an H.264 IDR, recovery point or intra picture with SPS / PPS before it; an H.265 IRAP or recovery point; every JPEG frame.
  • Local wall clock: receiveTimeUsec is the time at which the last packet of the frame arrived (CLOCK_REALTIME, UNIX epoch microseconds). For UDP it is the kernel receive time on Linux (see Receive thread, kernel timestamps and scheduling); for TCP, TLS and the HTTP tunnel it is the read time. The receive path keeps this time on the monotonic clock, and getFrame(…) converts it with the wall clock offset of the delivery moment. So it equals the kernel stamp within about 2 us, unless the wall clock was stepped or slewed in between (NTP slewing: at most 0.5 us per ms from arrival to delivery). The wall clock at delivery minus receiveTimeUsec is the local age of the frame (CUSTOM_1).
  • Sender wall clock: captureTimeUsec is the capture time of the frame on the sender’s wall clock (UNIX epoch microseconds), from the best source available (Table 8). It is -1 when unknown.

lostFrames (= CUSTOM_3 of the frame) counts the frames lost before this frame since the previous successful getFrame(…). metadata holds the metadata units bound to the frame (see Metadata binding); it is empty when the frame has none.

Packet loss (1.4.0). lostPackets counts the media packets the source lost since initVSource(…), across automatic reconnections; it never decreases. For RTP (also MPEG-TS in RTP) these are the RTP packets of all streams of the source (video and metadata) that had not arrived when the reorder wait and the NACK retransmissions were over. A packet that arrives later still counts, because it came too late for its frame, and a duplicate never counts. (The RFC 3550 cumulative loss of the RTCP receiver reports is different: late packets and duplicates reduce it.) SRTP packets that fail authentication and datagrams dropped by the local kernel (full receive buffer) count as lost too. Raw MPEG-TS has no sequence numbers, so there the TS packets missing by the continuity counters of the video, KLV and table PIDs are counted. This is a lower bound: 16 or more packets missing on one PID count modulo 16. Over TCP nothing is lost on the way, so only the packets the sender itself left out count.

packetLossPercent is the lost share of the last 10 seconds: lost / (received + lost), counted per second over the current second and the 9 seconds before it. The window is two RTCP report intervals (RFC 3550: at least 5 s, the interval of the “fraction lost” a sender sees in the receiver reports). It is long enough for a stable value at low packet rates: a 500 kbit/s stream sends about 50 packets per second, so one lost packet in 10 s is 0.2 %. And it is short enough for a decision within seconds: the value rises with the first lost packets and is back to 0 at most 10 s after the loss ended. Both values are taken when the frame is delivered. The value is meant for the application’s own logic, for example:

if (source.getFrame(frame, 1000) && source.getTelemetry(telemetry) &&
    telemetry.packetLossPercent > 5.0f)
{
    // More than 5 % of the packets of the last 10 s were lost: warn the
    // operator, ask the camera for a lower bitrate, or use TCP.
}

Table 8 - Capture time sources of captureTimeUsec, highest priority first.

Source Carried by Available with
MISB ST 0604 SEI MISB ST 0604 precision time stamp in a user_data_unregistered SEI of the frame (H.264 UUID “MISPmicrosectime”, H.265 UUID A8687DD4-D759-3758-A5CE-F0338B6545F1); the SEI stays in the frame. H.264 / H.265 over every source, raw MPEG-TS included; parsed always (also without -seidata and with -nometadata).
RTP header extension RTP header extension of the frame’s first packet: the ONVIF replay extension (profile 0xABAC, NTP timestamp) or the RFC 8285 abs-capture-time element (64-bit NTP time, its clock offset ignored), negotiated with the SDP attribute a=extmap and the URI http://www.webrtc.org/experiments/rtp-hdrext/abs-capture-time. ONVIF extension: every RTP source. abs-capture-time: RTSP (the SDP negotiates its id), also MPEG-TS in RTP.
RTCP sender report Sender wall clock of the frame’s RTP timestamp through the RTCP sender report mapping. RTP sources after the first sender report with a wall clock NTP time (not raw MPEG-TS; not a relative NTP time, see below).
none - captureTimeUsec is -1.

An NTP time of 0, or one before 1970, counts as no time. So does the relative NTP time of an RTCP sender report: RFC 3550 6.4.1 lets a sender without a wall clock send, for example, its uptime. The seconds of such a time have the most significant bit clear, which a wall clock NTP time can’t have before the NTP era wrap on 2036-02-07. So while the local clock is before that date, the frame gets no capture time from such a sender report. (The sender report still relates the clocks of the metadata and the video streams.) Every other time comes from the sender and is passed through without a plausibility check: a wrong sender clock gives a wrong captureTimeUsec.

With synchronized clocks (PTP or NTP on sender and receiver), receiveTimeUsec minus captureTimeUsec is the sender-to-receiver latency of the frame. The wall clock time minus captureTimeUsec is its glass-to-delivery latency. The capture times of the frame and of its units (for example KLV Tag 2) relate video and telemetry directly. Example:

cr::video::Frame frame;
cr::video::VSourceStreamTelemetry telemetry;
if (source.getFrame(frame, 1000) && source.getTelemetry(telemetry) &&
    telemetry.captureTimeUsec >= 0)
{
    // Meaningful when the clocks of sender and receiver are synchronized.
    const int64_t nowUsec = std::chrono::duration_cast<std::chrono::microseconds>(
        std::chrono::system_clock::now().time_since_epoch()).count();
    std::cout << "Frame " << telemetry.frameId << " captured " <<
    (nowUsec - telemetry.captureTimeUsec) / 1000.0 << " ms ago, received " <<
    (telemetry.receiveTimeUsec - telemetry.captureTimeUsec) / 1000.0 <<
    " ms after capture" << std::endl;
}

Measured results

The results below were measured by the test application over the loopback interface (127.0.0.1) of the development machine: Intel Core i7-13700H (20 logical CPUs), x86-64 Linux 6.18 under WSL2, GCC 13.3 Release build, OpenSSL 3.0.13, ffmpeg 6.1.1, GStreamer 1.24.2, and the reference server MServer 2.1.1 running in the test process. They come from the two full test runs of the 1.2.0 final verification, with 4423 checks each. Run 1 passed all checks. Run 2 passed all but one kernel receive time stamp check of v110net, which a step of the host’s wall clock hit; this check has since been made step-aware. Each run used a private network namespace. Where a cell holds two values separated by a comma, they are run 1 and run 2.

Loopback adds no network delay, loss or bandwidth limit of its own. The UDP impairment proxy and the RTSP-aware relay of the test add the loss, reordering, jitter and bottlenecks. The numbers show the processing latency of the receivers and orders of magnitude. They are not guarantees for other hardware and networks.

Video frame latency. A scripted RTSP server (for raw MPEG-TS, a crafted sender) sends H.264 640x480 at 25 fps. Each frame carries a frame index SEI. The latency is the time from sending the last packet of a frame to its delivery, that is, until getFrame(…) returns the frame or the reference client writes it to its output pipe. The reference clients are ffmpeg (-fflags nobuffer -flags low_delay, -c copy to an Annex-B pipe) and GStreamer (rtspsrc latency=0 ! rtph264depay ! fdsink). ffmpeg completes a frame only when the next frame starts. This costs one frame interval (40 ms), and two frame intervals with raw MPEG-TS.

Table 9 - Video frame latency, mean / p95, ms (“latency” group).

Path VSourceStream getFrame(…) ffmpeg GStreamer
RTSP / TCP interleaved 0.18 / 0.40, 0.22 / 0.54 40.36 / 41.05, 40.37 / 41.14 0.51 / 0.94, 0.55 / 1.01
RTSP / UDP 0.18 / 0.39, 0.17 / 0.32 40.34 / 41.16, 40.39 / 41.04 0.43 / 0.99, 0.47 / 0.97
Raw MPEG-TS over UDP 0.18 / 0.35, 0.20 / 0.39 80.45 / 81.50, 80.53 / 81.56 -

Latency with metadata. On the paths that carry KLV, the latency is the time from MServer sending a frame (the send time is carried in its KLV unit) to getFrame(…) returning the frame with that unit. ffmpeg reads the KLV data stream with -fflags nobuffer -flags low_delay -probesize 32 -analyzeduration 0. VSourceStream was lower in all 8 mean and p95 comparisons. The test requires VSourceStream to be at most 1 ms above ffmpeg.

Table 10 - Latency of frames with synchronous KLV compared with ffmpeg, mean / p95, ms (“latency” group).

Path VSourceStream ffmpeg
Raw MPEG-TS over UDP (1280x720) 0.54 / 1.02, 0.68 / 1.46 0.89 / 1.93, 0.78 / 1.98
MPEG-TS in RTP over RTSP / TCP (640x480) 0.80 / 1.44, 0.75 / 1.30 0.90 / 1.77, 1.00 / 1.62

Time to first frame over RTSP / TCP (the server forces a key frame on PLAY; mean of 3 attempts each): 25.9 ms, 30.6 ms for VSourceStream, from openVSource(…) to the first frame returned by getFrame(…). For ffmpeg: 1193.6 ms, 1203.8 ms, as the wall time of the whole process ffmpeg -rtsp_transport tcp -i <url> -frames:v 1 -f null -, including the process start-up and its default stream probing.

Table 11 - Low latency features (“lowlat” group).

Case Result
RTSP / TCP sender with Nagle’s algorithm, loopback MSS (TCP_QUICKACK) latency mean / p95 0.13 / 0.24 ms, 0.23 / 0.47 ms; no frame above 30 ms (1.1.0 measurement without TCP_QUICKACK: p95 33 ms)
The same, 1280x720, MSS 1448 0.12 / 0.22 ms, 0.18 / 0.37 ms
RTSPS behind a Nagle relay 0.82 / 1.68 ms, 1.10 / 2.01 ms
RTSP over HTTP tunnel behind a Nagle relay 0.85 / 1.80 ms, 1.01 / 1.76 ms
Frame end by the learned idle gap (latency p50 / p95 from frame 30): RTP without marker 1.18 / 1.41 ms, 1.20 / 1.36 ms (1.0.0: one frame interval, 40 ms)
The same: RTP with the marker on every packet 1.16 / 1.41 ms, 1.17 / 1.33 ms
The same: MPEG-TS access unit ending exactly on a datagram 1.24 / 1.47 ms, 1.26 / 1.49 ms
The same: GStreamer-style NAL-aligned MPEG-TS 1.19 / 1.46 ms, 1.20 / 1.48 ms
Consumer stalled for 1.5 s, buffer depth 100, -maxlatency 200 first frame after the stall 659 ms, 660 ms old against 1460 ms, 1461 ms without the budget; 20 frames dropped (framesLatencyDropped), then back at the live edge
Receive thread stalled for 300 ms (UDP; the test stops the library’s receive thread with a thread-directed signal whose handler sleeps; 1.1.0 used a slow frame callback) receiveTimeUsec of the 7 frames sent during the stall within 5 ms of their send time (kernel receive time); CUSTOM_1 of the first waiting frame includes the wait (260.7 ms, 261.0 ms)
30000 datagrams blasted at the socket during a 1 s stall kernelDrops 22963 (both runs), shown as packetsLost 244 (both runs) and framesFrozenMsec 1799, 1801; delivery resumes

Impairment. The “impair” group sends H.264 over rtp:// through a UDP impairment proxy, in five phases. rtp:// has no key frame request path, so every frame lost to a dropped packet costs the rest of its GOP. The latency of phase 2 includes the jitter that the proxy adds to every packet.

Table 12 - Impairment phases (“impair” group).

Phase Impairment Latency mean / p95, ms Reorder wait, result
1 (3 s) none 0.61 / 1.47, 0.80 / 1.54 10 ms, the default minimum
2 (8 s) 5 % of the packets reordered by 15 - 25 ms, up to 20 ms order-preserving jitter, 0.5 % loss 20.10 / 24.56, 20.23 / 24.96 grew to 29.0 ms, 29.4 ms; 124, 118 packets reordered, 14, 14 lost; 0 late packets after the adaptation (1.0.0: 3 - 4); 84 of 164 frames lost (both runs), all explained by the proxy drops
3 (17 s) none 0.67 / 1.48, 0.83 / 1.52 held at its peak (29018 us, 29368 us) 8 s after the impairment, then decayed back to 10 ms
4 (4 s) 2 % duplicated packets 0.73 / 1.66, 0.80 / 1.52 17, 17 duplicates discarded, no frame lost
5 (4 bursts, 3 s apart) 400 ms bursts with 20 % of the packets reordered by 25 - 35 ms 3.39 / 31.97, 3.50 / 31.56 the first burst raised the wait to 62.2 ms, 63.7 ms, held between the bursts: late packets only in the first burst (3, 3), none and no frame loss in bursts 2 - 4

NACK / RTX repair. The “feedback” group receives H.264 over RTSP / UDP from MServer 2.1.1 through an RTSP-aware relay. The relay drops packets on the server-to-client leg, and the retransmissions cross the same lossy leg. The control run drops the retransmissions too.

Table 13 - Repair by NACK / RTX against MServer 2.1.1, H.264 over UDP (“feedback” group).

Loss Packets dropped Lost after repair Frames lost / frozen Loss to retransmission p50, ms Latency added by the repair, p50 / p95 / mean, ms Control without RTX
0.5 % 7, 7 0, 0 0 / 0 ms 0.30, 0.60 -0.17 / -0.25 / -0.06, -0.33 / 0.02 / -0.12 5, 5 packets dropped: 36, 22 frames lost, 1.4 s, 0.9 s frozen
2 % 26, 26 0, 0 0 / 0 ms 0.30, 0.52 -0.29 / -0.16 / 0.12, -0.03 / 0.20 / 0.09 101, 107 frames lost, 3.3 s, 6.3 s frozen
5 % 66, 64 0, 0 0 / 0 ms 0.29, 0.50 0.04 / 1.31 / 0.61, -0.09 / 0.97 / 0.16 63, 66 packets lost, 13, 6 frames delivered (repaired: 152)
20 % of the frames lose their last packet 19, 19 (the marker packets) 0, 0 0 / 0 ms 1.38, 1.64 p95 +0.88, +1.28 18, 18 repaired by a tail-loss probe (every loss after the first 12 frames), 0 false probes; without the probe (negative control build) p95 +39.8 ms

The added latency is measured against the loss-free run of the same path, so it can be slightly negative. MServer 2.1.1 reads RTCP over UDP as soon as it arrives, so a retransmission comes about 0.3 - 0.6 ms after the loss. (2.1.0: 40 - 80 ms, which added 39 - 80 ms to the p95 of repaired frames in 1.1.0.)

At 5 % loss, about one frame in 20 loses its last packet. This loss would show only with the first packet of the next frame, 40 ms later. The tail-loss probe asks for the packet after about 1 ms of silence. So the packet is repaired about 1.5 ms after it was sent, and the p95 grows by 1.31 ms, 0.97 ms. (Before the probe, in this round: +14 .. +39 ms, depending on whether such a frame fell on the 95th percentile; the negative control build without the probe: +38.6, +39.1 ms.) The last row of Table 13 loses the last packet of every fifth frame: without the probe, this costs one frame interval on the p95.

Lost retransmissions are requested again within half the reorder wait (after twice the measured answer delay), and MServer 2.1.1 answers within it. (1.1.0 never sent this retry with a 10 ms wait: in an A/B verification at 5 % loss, 2 of 3 runs of the 1.1.0 retry rule lost 2 and 9 packets.) packetsLost reports only the unrepaired packets: 0 on the repaired runs, and 5, 5 (0.5 %), 27, 32 (2 %) and 63, 66 (5 %) on the control runs.

Other cases:

  • H.265 at 2 % loss: 19, 19 packets dropped, 0 lost after repair, 0 frames lost.
  • JPEG at 2 % loss (MServer 2.1.1 announces the RTX payload type 99 with apt=26): 62, 61 packets dropped, every one retransmitted in time, 0, 0 lost, 0, 0 frames lost. A JPEG frame of several packets that lost its last packet is repaired by the tail-loss probe too. Before the probe, 2 - 4 frames were lost; with MServer 2.1.0, all were lost.
  • TCP: no NACK, no retransmission.
  • The first UDP subscriber of a freshly started raw-encoder server (4 s GOP) got its first frame, a key frame, 35 ms, 37 ms after initVSource(…). (2.1.0: up to the next natural key frame.)
  • Key frame requests to a raw-encoder server (4 s GOP, 1 % loss, retransmissions dropped): 23, 21 PLI sent and 18, 17 answered (MServer’s 200 ms rate limit); request to key frame p50 37.4 ms, 34.3 ms; 16, 15 freezes of 144 ms, 135 ms on average (at most 357 ms, 364 ms), instead of up to the 4000 ms GOP.
  • At 5 % loss that RTX repairs, the adaptive transport stays on UDP.

Table 14 - Adaptivity (“adapt” group, RTSP-aware impairment relay, MServer 2.1.1).

Case Result
Default rtsp://, 10 % UDP loss that RTX can’t repair switched to TCP after 2008 ms, 2004 ms, 1 switch; 100 frames in the next 4 s; framesFrozenMsec 1040 ms, 1081 ms, lossPercent 10.2 %, 12.1 %
Default rtsp://, 25 % UDP loss from PLAY on switched to TCP 2816 ms, 2908 ms after PLAY
Default rtsp://, 10 % UDP loss repaired by MServer’s RTX in time stays on UDP: 200 frames, 0 lost, 158, 162 RTX
The same, the RTX delivered 400 ms late (after the reorder wait) switched to TCP after 3007 ms, 3005 ms (packetsLate 79, 79; packetsLost 11, 11)
Scripted server: the last packet of one frame per second only 400 ms late, or lost and repaired by RTX 300 ms late switched to TCP after 5029 ms, 5030 ms and 5046 ms, 5020 ms with packetsLost 0 (packetsLate 5); the same loss repaired in time: stays on UDP, 201 frames, 0 lost
-udp, 10 % UDP loss no switch
30000 datagrams blasted at the receiver during a 1.5 s receive thread stall (a thread-directed signal; 1.1.0: a slow frame callback) kernelDrops 23058 (both runs), packetsLost 0: no switch
TCP bottleneck (275.9 kB/s stream, 138.0 kB/s limit), -tcp -autotransport -maxlatency 300 queueDelayUsec up to 812 ms, 813 ms; one UDP trial after 4011 ms, 4007 ms; on UDP the queue is gone (queueDelayUsec 51 us, 45 us)
The same over rtsps:// without -srtp TCP flush reconnect instead of UDP (the media stays inside TLS)
The same with -tcp alone no guard: queueDelayUsec 2.22 s, 2.23 s and growing
Complete key frame behind a reorder gap (wait 75 ms, 75 ms) released after 0.09 ms, 0.19 ms (key-frame bypass) against 75.4 ms, 75.1 ms with -nokeywait
Lost packet 23 frames before the next key frame 920 ms, 923 ms frozen by default, 240 ms, 239 ms with -keywait 220 (the first frame after the bound, 6 frame intervals after the damaged one)
Lost packet carrying only the leading SEI of a P frame that frame and the rest of its GOP delivered (AU-boundary loss rule; 1.0.0: rest of the GOP lost)
NACK unanswered for 8 gaps, then answered again reorder wait 50 -> 10 -> 39.0 ms, 39.0 ms

1080p stress. The stress runs of the “interop” group pass a 1920x1080 H.264 clip at 25 fps through MServer (GOP 25, 11.98 Mbit/s on the wire, key frames of 455 KiB on average and up to 693 KiB). Each run lasts 10 s per transport. The receiver runs in a separate process and measures its own CPU time. The latency is the receive time minus the send time, on CLOCK_MONOTONIC.

Table 15 - 1080p stress (“interop” group).

Transport Frames (byte-exact) Lost packets Latency mean / p95 / max, ms CPU of the receiving process, % of one core
RTSP UDP 252 (252), 252 (252) 0, 0 0.86 / 1.74 / 9.57, 0.79 / 1.22 / 8.28 0.94, 0.84 at 11.96, 12.00 Mbit/s
RTSP TCP 252 (252), 252 (252) 0, 0 0.82 / 1.93 / 4.38, 1.05 / 2.09 / 6.75 0.72, 0.90 at 11.93, 11.99 Mbit/s
rtp:// push 252 (252), 252 (252) 0, 0 0.89 / 2.04 / 6.77, 1.09 / 1.98 / 9.37 0.93, 1.11 at 12.42, 12.45 Mbit/s

The maximum latency comes from the large key frames, which MServer sends in bursts. With the independent senders of the “interop” group (ffmpeg, the GStreamer payloaders and MPEG-TS muxer, gst-rtsp-server; 34 receptions in 33 cases), the frame age at delivery (delivery minus arrival of the last packet) stayed at 0.37 ms p95 or less.

Telemetry. The “telemetry” group sends KLV, ONVIF and SEI telemetry in many layouts (scripted RTSP server, crafted push senders, MServer, ffmpeg, gst-rtsp-server). It counts how the delivered units were bound (see Metadata binding). The values are identical in both runs unless two values are given. 1.1.0 reported four kinds of binding publicly (exact, since the previous frame, carried, untimed). Since 1.2.0 the telemetry reports only exact. So Table 16 gives the units that were exact on their frame and the units that were not.

Table 16 - Metadata binding matrix, units exact / not exact (“telemetry” group).

Case Units exact / not exact
RTP KLV on its frame’s timestamp 100 / 0
RTP KLV at 50 Hz on 25 Hz video 100 / 99 (the units between two frames come with the next one)
RTP KLV at 10 Hz off the frame grid 0 / 25
RTP KLV 1 tick off the frame’s timestamp 100 / 0
RTP KLV at 1 Hz (sparse: no hold) 0 / 4
RTP KLV later than the next frame 0 / 98
RTP KLV of every 10th frame 10 ms late (after the hold), followed by a unit between two frames 90 / 20 (the next frame carries both, the late unit first: time order, re-sorted from the stored order on 10 of 10 frames)
RTP KLV of frames dropped by the key frame gate 80 / 20
RTP KLV on an unrelated RTP clock, mapped by RTCP SR 99 / 1
RTP KLV with wrong checksums / broken BER lengths 90 / 0, all delivered, the broken ones with valid FALSE
RTP KLV without marker; with the marker on every packet 100 / 0 each
Two RTSP metadata tracks, RTP-Info offset, 1 kHz KLV clock 200 / 0
gst-rtsp-server KLV track, provisional RTP-Info offset before its first RTCP SR 117 / 34, 130 / 21 (77.5 %, 86.1 % on their own frame, the rest carried to the next frame: gst-rtsp-server sends some units after their frame, how many depends on the host’s timing: the test binary of the previous verification gave 92.7 %, 98.0 % in its runs and 85.4 %, 77.0 % on the day of these runs; 1.1.0: 90.7 %, 96.7 %)
MPEG-TS asynchronous KLV muxed before its video 97 / 3
MPEG-TS synchronous KLV at 50 Hz on 25 Hz video 100 / 99
MPEG-TS synchronous KLV at 1 Hz 2 / 2
MPEG-TS two KLV PIDs 120 / 0
MPEG-TS several / fragmented metadata AU cells 200 / 0
MPEG-TS KLV spanning 3 PES 100 / 0
MPEG-TS raw KLV in a PES with PTS (ffmpeg layout) 100 / 0
MPEG-TS synchronous KLV 2 ms after its video 0 / 99, 3 / 96 (MPEG-TS holds are capped at 0.5 ms: the unit comes with the next frame)
ffmpeg -c copy restream of MPEG-TS KLV 9 / 240, 1 / 248 (ffmpeg sends the KLV about 40 ms after its frame)

The capture time of H.264 and H.265 frames with an ST 0604 SEI came from the SEI. The capture time of frames with an abs-capture-time or ONVIF replay header extension came from the extension, also for MPEG-TS in RTP. MServer 2.1.1 adds the MISB ST 0604 time stamp SEI to every H.264 / H.265 frame that carries an ST 0601 KLV unit with Tag 2. In every KLV run of the test application, the SEI time equals the frame’s KLV Tag 2 and captureTimeUsec. Frame-to-metadata latency (from the MServer send time to the delivery of the frame with its unit; p50 / p95; runs 1 and 2): RTSP KLV track over UDP 0.44 / 1.25 ms, 0.74 / 1.45 ms; ONVIF track 0.55 / 1.48 ms, 0.61 / 1.38 ms; MPEG-TS synchronous KLV in RTP 0.58 / 1.39 ms, 0.70 / 1.42 ms; raw MPEG-TS asynchronous KLV 0.58 / 1.31 ms, 0.80 / 1.47 ms.

Security and compliance modes

The -profile option selects the compliance mode (default ONVIF). The modes mirror the modes of MServer on the client side. Table 17 shows what each mode allows.

Table 17 - Compliance modes.

Rule ONVIF STRICT_FIPS FIPS_ONVIF_BRIDGE CRA_STRICT
rtsp:// (plaintext RTSP) yes no no no
rtp://, udp:// (plaintext media) yes no no no
srtp:// yes no yes yes
rtsps:// with RTP inside TLS yes yes yes yes
rtsps:// with plain RTP over UDP / multicast yes no no no
rtsps:// with SRTP (-srtp) yes no yes yes
SRTP suites (srtp:// and -srtp) all none AES_CM_128_HMAC_SHA1_80 / _32 all (AEAD_AES_128_GCM / AEAD_AES_256_GCM only without an active FIPS provider)
RTSP over HTTP tunnel yes no no no
RTSP over HTTPS tunnel yes yes yes yes
TLS peer verification on by default, -noverify allowed required required required
Digest MD5 only without active FIPS provider no no no
Digest SHA-256 yes yes, qop=auth required yes yes
Basic authentication over TLS only no no over TLS only
TLS cipher suites OpenSSL defaults FIPS lists FIPS lists OpenSSL defaults
FIPS provider requested no yes yes no
fullModuleCoverage FALSE TRUE with active FIPS provider FALSE FALSE

A source or option that the mode refuses makes initVSource(…) fail with an ERROR that names the rule, for example “Refused by the compliance profile: Profile CRA_STRICT refuses rtsp:// (plaintext RTSP control); use rtsps://”. Rules that depend on the server (transport fallback, authentication scheme, SRTP suite of the SDP) are enforced while connecting. When the server offers only refused mechanisms, the connection fails; there is no downgrade. Every initialization that passes the compliance check logs the compliance statement of the mode at INFO level, after the “Opening …” line:

Table 18 - Compliance statements.

Mode Statement
ONVIF ONVIF interoperability mode. MD5 digest (not FIPS-approved) and AES-GCM SRTP (its nonces are built outside the validated module) are enabled, so no FIPS claim applies to this configuration.
STRICT_FIPS, FIPS provider active Strict FIPS 140-3 mode. All cryptography is performed by the validated module; media travels as RTP interleaved inside TLS. MD5 digest and SRTP are refused.
STRICT_FIPS, no FIPS provider Strict FIPS 140-3 CONFIGURATION without the validated module (standard crypto provider). The algorithm restrictions are enforced – MD5 digest and SRTP are refused, TLS with peer verification is mandatory – but the validated module is NOT active, so NO FIPS 140-3 claim applies to this deployment.
FIPS_ONVIF_BRIDGE, FIPS provider active FIPS/ONVIF bridge. Only FIPS-approved primitives are used, and SRTP is limited to AES_CM_128_HMAC_SHA1_80 / _32. LIMITED CONFORMANCE: the SRTP key derivation (SP 800-135 s5.3) is not implemented by the validated module.
FIPS_ONVIF_BRIDGE, no FIPS provider FIPS/ONVIF bridge CONFIGURATION without the validated module (standard crypto provider). Only FIPS-approved primitives are selected and SRTP is limited to AES_CM_128_HMAC_SHA1_80 / _32, but the validated module is NOT active, so NO FIPS 140-3 claim applies to this deployment.
CRA_STRICT EU CRA secure-by-default mode. TLS with peer verification is mandatory, MD5 digest is disabled, no plaintext RTSP connection is opened, and media is refused unless it is protected – SRTP, or RTP interleaved inside the TLS connection. SRTP remains available, so the media key derivation is outside the validated module and coverage is not claimed to be complete.

TLS verification options. The TLS client requires TLS 1.2 or newer and disables compression, renegotiation and session tickets. Peer verification works as follows:

  • no option: the certificate chain is verified against the system default CA paths, and the server name is checked (SNI and host name for host names, the IP address for IPv4 literals);
  • -cafile PATH: the chain is verified against this PEM bundle instead, with the same name check;
  • -fingerprint FP: only the SHA-256 fingerprint of the server certificate is checked, in constant time, after the handshake. The chain and the name are not evaluated, and a -cafile is not used. A mismatch fails the connection;
  • -noverify (ONVIF only): no verification. The WARNING “TLS peer verification disabled (-noverify): the server is NOT authenticated” is logged once per initVSource(…), not on reconnections, and tlsPeerVerified is FALSE.

In STRICT_FIPS and FIPS_ONVIF_BRIDGE, the TLS client offers only TLS_AES_256_GCM_SHA384 and TLS_AES_128_GCM_SHA256 (TLS 1.3), ECDHE-ECDSA / ECDHE-RSA with AES-128 / AES-256 GCM (TLS 1.2), and the groups P-256, P-384 and P-521. It refuses TLS 1.2 without the Extended Master Secret extension. The internal diagnostics report the negotiated version and cipher and the verification result as tlsVersion, tlsCipher and tlsPeerVerified. The “RTSP session established” INFO line of a TLS connection ends with “, TLS peer verified” or “, TLS peer NOT verified”.

SRTP suites. The library supports AES_CM_128_HMAC_SHA1_80 and AES_CM_128_HMAC_SHA1_32 (RFC 3711 / RFC 4568), AES_256_CM_HMAC_SHA1_80 and AES_256_CM_HMAC_SHA1_32 (RFC 6188), and AEAD_AES_128_GCM and AEAD_AES_256_GCM (RFC 7714). The mode limits the choice (Table 17). With rtsps:// -srtp, the keys come from the SDP a=crypto attributes (SDES with an inline key; the lifetime is ignored; MKI and multiple keys are refused; the same key with different suites in two tracks is refused too). With srtp://, the key comes from -srtpkey. The receiver keeps a 128-packet replay window for SRTP and SRTCP. It derives the rollover counter per RFC 3711 and commits it only after authentication. It handles up to 8 SSRCs per key, and it protects its own RTCP with SRTCP. In the FIPS modes, an SRTP known-answer self test runs at every initialization that uses SRTP. The internal diagnostics report the suite in use as srtpSuite, and the rejected packets as srtpAuthFailures and srtpReplayDrops.

FIPS provider behaviour. The library performs its cryptographic operations through the CryptoProvider library (3rdparty/CryptoProvider): ciphers, SRTP protection, HMAC, SHA digests, secure erase and random values (CSPRNG / DRBG). The MD5 of the digest authentication is computed by the library itself. CryptoProvider selects its backend at build time: OpenSSL (libcrypto) on Linux when CMake finds OpenSSL, Windows CNG (bcrypt) on Windows, and a standalone implementation on Linux without OpenSSL. TLS (libssl) runs in the library context of the OpenSSL backend of CryptoProvider, so TLS uses the same provider as the other primitives.

For STRICT_FIPS and FIPS_ONVIF_BRIDGE, the library initializes CryptoProvider in FIPS mode:

  • OpenSSL backend: CryptoProvider loads the system OpenSSL configuration and the “fips” and “base” providers, and it enables the FIPS default properties. This needs an OpenSSL installation with the FIPS provider (fips.so) installed and configured. The library then proves the enforcement on this library context: SHA2-256 can be fetched, and MD5 can’t.
  • Windows CNG backend: CryptoProvider claims FIPS only when the Windows system FIPS policy is on. In a build with TLS, the claim also needs the proof on the OpenSSL library context described above, so it is not given.
  • Standalone backend: CryptoProvider never claims FIPS.

When the validated module is not active, the FIPS claim is withdrawn: the restrictions of the mode stay enforced, fipsActive is FALSE, and complianceMode carries the suffix “ (unvalidated provider)”, the form MServer used up to 2.1.2 (for example “STRICT_FIPS (unvalidated provider)”). VSourceStream is a client: it keeps the connection policy of the mode and only withdraws the claim. MServer 2.1.3, a server, refuses to start a FIPS profile without the module, and the compliance group of the test application expects that. In every mode, the random generator (DRBG) must answer a probe before the context is used.

The crypto context is process-wide: all VSourceStream objects share it. A context created for a non-FIPS mode is upgraded to the FIPS module by the first initialization in a FIPS mode. CryptoProvider keeps the replaced library context alive, so the objects that still use it stay valid. Once a FIPS mode has initialized the context, the decision is final for the process. fullModuleCoverage is TRUE only for STRICT_FIPS with the FIPS provider active.

EU CRA engineering measures.

  • Defaults: TLS peer verification is on by default in every mode, there are no default credentials, and the strict modes refuse every unprotected path. The default mode, however, is ONVIF, which allows plaintext sources, MD5 digest and -noverify for interoperability: a deployment that must be secure by default (EU CRA Annex I Part I 2(b)) has to select -profile CRA_STRICT (or a FIPS mode) in every source string.
  • Bounded parsers: every limit on untrusted input is a constant. Input above a limit is rejected, never clamped. Examples: the source string 4096 bytes / 32 tokens, SDP 64 KiB, RTSP response head 16 KiB and 64 headers, RTSP body 64 KiB, frame 32 MiB, frame geometry 8192 x 8192 and 8192 x 4352 pixels (1.4.0: an SPS above it is never cached nor given to a decoder, and the access unit that carries it is dropped), output buffer 256 MiB, reorder buffer 2048 packets / 8 MiB, RTP metadata unit 1 MiB, MPEG-TS KLV PES and reassembled KLV item 64 KiB (a longer KLV item is never delivered whole: it is dropped or comes as truncated pieces with valid FALSE), SEI 64 KiB per NAL unit and 16 messages per access unit, 4 metadata streams, metadata 256 units / 4 MiB per frame, carried metadata 64 units / 1 MiB, parameter set cache 64 KiB in total, 8 RTP sources (a burst of 8 new ones, then one per second). Retained memory is bounded in the same way. For example, a metadata unit buffer keeps at most 64 KiB beyond the unit it holds. So the telemetry structure of getTelemetry(…) keeps at most the metadata of the current frame plus 256 x 64 KiB, and the source keeps at most 16 x 64 KiB of spare unit buffers.
  • Zeroization: passwords, SRTP master and session keys, SDP bodies with a=crypto lines, and the buffers that held an Authorization header are wiped with cr::crypto::secureErase of CryptoProvider when they are released (with the OpenSSL backend, this calls OPENSSL_cleanse).
  • No secrets in logs: passwords, keys, a=crypto lines and Authorization headers are never logged. URLs are logged with the password replaced by “***”. Non-printable bytes are replaced, and lines are cut at 1024 bytes.
  • DRBG: every random value (SSRC, CNAME, digest cnonce, local UDP ports, tunnel session cookie) comes from the CSPRNG / DRBG of CryptoProvider (cr::crypto::Random). Fingerprints and authentication tags are compared in constant time (cr::crypto::constantTimeEquals).
  • Hardening: the library is compiled with -fstack-protector-strong and, in Release, with -D_FORTIFY_SOURCE=2 (MSVC: /GS and /guard:cf). On Linux, it passes -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack to every executable or shared object that links it (MSVC: /guard:cf). It uses no -march=native and no -ffast-math (reproducible builds). See Build and connect to your project.
  • Source address filtering: RTSP media over UDP is accepted only from the address of the RTSP server. SDP-less media is accepted only from the -source sender, when given. SDP-less RTCP reports go only to the address that the RTP packets of their SSRC come from.
  • Other measures: no version banner (the User-Agent is “VSourceStream”); credentials are never sent to another host after a redirect; the library never raises SIGPIPE; on Linux, the log file is written only if it is a regular file owned by the user (no symbolic links).

Note: the source string given by the caller stays in VSourceParams::source as given, including a password or an SRTP key, and getParams(…) returns it. Treat it as a secret in the application.

Thread safety

All public methods are thread safe. getParams(…), isVSourceOpen(), getParam(…), setParam(…) and getTelemetry(…) never block behind a waiting getFrame(…). closeVSource() and initVSource(…) / RESTART interrupt a waiting getFrame(…).

initVSource(…) (also through openVSource(…)), RESTART and closeVSource() are serialized: one runs at a time, and the others wait for it. A waiting closeVSource() or initVSource(…) aborts a running initVSource(…) within about 20 ms; that initVSource(…) then returns FALSE. While one of them runs in another thread:

  • getFrame(frame, 0) returns FALSE at once;
  • getFrame(frame, N) waits at most N ms in total;
  • getFrame(frame, -1) waits for it to finish.

getFrame(…) also returns FALSE at once while a closeVSource() / initVSource(…) / RESTART of another thread waits to start. So a getFrame(…) loop can’t delay them.

Several threads may call getFrame(…): every frame is delivered to exactly one of them. getTelemetry(…) refers to the frame of the last successful getFrame(…) of any thread. getFrame(…) takes the snapshot together with the frame, so the telemetry always describes exactly one delivered frame, until the next successful getFrame(…) replaces it. Call getTelemetry(…) from the thread that called getFrame(…), right after it. With several getFrame(…) threads, compare frameId with Frame::frameId.

A delivery updates CUSTOM_1, CUSTOM_3 and the telemetry in one step. So while no getFrame(…) is delivering and getTelemetry(…) returns TRUE, CUSTOM_3 equals lostFrames of the telemetry, also with several getFrame(…) threads. After closeVSource() or a failed initVSource(…), CUSTOM_3 keeps its last value, while the telemetry is reset.

Blocking bounds: initVSource(…) normally blocks for at most about -timeout, plus the close of an open source. If the session reports its failure late, it waits up to 1 s longer. closeVSource() blocks for up to about 500 ms (the wait for the TEARDOWN answer; with media over TCP or the HTTP tunnel at most 50 ms after the TEARDOWN was sent, because its answer may queue behind the media and the close ends the session anyway). A host name resolution (rtsp:// / rtsps:// host names) can’t be interrupted and may extend both. Use an IPv4 literal for a hard bound.

setCodec(…) may be called from any thread at any time. It waits for a decoding call of the previous codec that is in progress.

Every VSourceStream object uses one internal receive thread (see Receive thread, kernel timestamps and scheduling). initVSource(…) starts it, and closeVSource() stops it. Once decoded frames have been requested, the object also uses the decode thread, with the worker threads of the decoders and the converter (see Frame output and decoding). No user code runs on these threads, except a custom codec on the decode thread. The constructor starts no thread.

Error recovery and reconnection

initVSource(…) returns FALSE when:

  • the source string is invalid, or the compliance mode refuses the source;
  • the crypto provider can’t be initialized;
  • the server is unreachable;
  • TLS verification fails, or TLS is not available (a build without OpenSSL);
  • authentication fails;
  • the server answers with an RTSP error, other than 404 / 503 on DESCRIBE (these are retried until the deadline);
  • a redirect is refused;
  • the -timeout deadline expires;
  • or another thread aborts the initialization.

The reason is logged at ERROR level.

After a successful initialization, the receive thread reconnects automatically:

  • RTSP / RTSPS: on a lost control connection, a keep-alive answered with 454 or not answered, a server TEARDOWN or REDIRECT, an RTCP BYE, no video media for -reconnect ms (default 5000), no RTP over UDP within 3 s after PLAY (or within -reconnect, if shorter; the session then reconnects over TCP, a multicast session over UDP unicast, see Transport selection and fallback), and an adaptive transport switch. A failed session whose control connection is still alive is first torn down with TEARDOWN (a wait of at most 500 ms; with media over TCP or the HTTP tunnel at most 50 ms after the TEARDOWN was sent), so the server does not keep streaming to it. The TEARDOWN is skipped after a server TEARDOWN or a keep-alive answered with 454, because the server has already ended the session then.
  • rtp://, udp://, srtp://: on socket errors only. A stream without media keeps listening.

Reconnection attempts are delayed by a backoff. The backoff starts at 250 ms and doubles up to 4 s. It is reset to 250 ms after a productive session (media received, or established for at least 4 s). A transport switch retries at once. Every reconnection is counted in reconnects and logged as a WARNING with its reason (“Reconnecting in 250 ms (…)”). An adaptive transport switch is the exception: the session logs its WARNING with the reason, and the reconnection itself follows at once with the INFO line “Adaptive transport: reconnecting now (…)”. During reconnection, the source stays open (IS_OPEN = 1), and getFrame(…) keeps waiting. connected is FALSE while no media packet has arrived for 2 s. Delivery resumes with a key frame, and frameId continues.

The library follows these stream changes without reconnection:

  • A codec change (MServer switches between H.264, H.265 and JPEG on the same SSRC) starts with a key frame of the new codec.
  • A resolution change follows the SPS: width, height and getParams(…) are updated with the next frame.
  • A sender restart (a sequence jump on the same SSRC) resets the stream. The stream resumes at the next key frame (unless -nokeywait is given).
  • A new SSRC takes over as described in SDP-less reception and auto-detection.

The bounded parsers drop malformed packets.

After closeVSource(), the source is never reopened automatically. Use initVSource(…), openVSource(…) or the RESTART command to open it again.

Logging

The LOG_LEVEL param sets the log level at any time. VSourceParams::logLevel sets it at initialization; an invalid value selects 0.

Table 19 - Log levels.

Level Output
0 Logging disabled (default).
1 File VSourceStream.log in the working directory of the process (appended).
2 Console (stdout).
3 File and console.

Line format: [VSourceStream][<source file>][<line>] <ERROR | WARNING | INFO>: <text>. On the console, ERROR lines are red and WARNING lines are yellow. In the file, every line starts with the local time “YYYY-MM-DD HH:MM:SS.uuuuuu”. The file is opened and closed for every line, so it may be rotated or deleted at any time. On Linux, the library writes the file only if it is a regular file with one link, owned by the user of the process.

  • INFO lines report the opening, the established session, the stream geometry, new RTP sources, the applied -priority / -cpu and the closing.
  • WARNING lines report reconnections, transport fallbacks, adaptive transport actions, skipped tracks, ignored malformed hints, a small UDP receive buffer, kernel receive buffer overflows, an expired -keywait bound, a failed -priority / -cpu, and -noverify.
  • ERROR lines report the reason of a failed initialization, and frames that can’t be delivered.

Repetitive lines are rate-limited, so a broken stream can’t flood the log. Secrets are never logged.

Example (console, level 2, colours not shown; the source line numbers depend on the library build). It was taken with the example application against VSourceStreamTest -server rtsp-klv. The ERROR line comes from a second attempt with -profile CRA_STRICT:

[VSourceStream][VSourceStream.cpp][1491] INFO: Opening rtsp://127.0.0.1:18500/live, compliance ONVIF, CryptoProvider 1.0.0 [no FIPS claim]
[VSourceStream][VSourceStream.cpp][1495] INFO: ONVIF interoperability mode. MD5 digest (not FIPS-approved) and AES-GCM SRTP (its nonces are built outside the validated module) are enabled, so no FIPS claim applies to this configuration.
[VSourceStream][VSourceStreamFrameBuilder.cpp][1230] INFO: Video geometry 640x480 (H264)
[VSourceStream][VSourceStreamSession.cpp][1000] INFO: RTSP session established: rtsp://127.0.0.1:18500/live, H264 over TCP, with metadata track
[VSourceStream][VSourceStreamPipeline.cpp][2650] INFO: New RTP source 0xBE96C622 on track 0
[VSourceStream][VSourceStreamPipeline.cpp][3224] INFO: Video source 0xBE96C622 (H264) active on track 0
[VSourceStream][VSourceStream.cpp][1612] INFO: Opened rtsp://127.0.0.1:18500/live: H264 640x480
[VSourceStream][VSourceStreamPipeline.cpp][2650] INFO: New RTP source 0xE5014283 on track 1
[VSourceStream][VSourceStreamPipeline.cpp][2161] INFO: Metadata RTP source 0xE5014283
[VSourceStream][VSourceStream.cpp][1441] ERROR: Refused by the compliance profile: Profile CRA_STRICT refuses rtsp:// (plaintext RTSP control); use rtsps://

Internal diagnostics

Runtime statistics are not part of the public API since 1.2.0 (1.0.0 and 1.1.0 had a public statistics structure). They are internal diagnostics for the test application and service tools. They are held in the structure cr::vsourcestream::impl::DiagnosticStats of the internal header src/impl/VSourceStreamDiagnostics.h, which documents every field. This interface may change without notice, so an application should not depend on it. The counters that this document names in bold (for example packetsLost, kernelDrops or transport) are fields of that structure (Table 20).

All counters are cumulative since the last successful initVSource(…), and they survive automatic reconnections. After closeVSource(), and after a failed initVSource(…), the values of the last session stay readable (connected FALSE). Reading them never blocks behind a waiting getFrame(…). While initVSource(…) / RESTART / closeVSource() runs in another thread, a read waits for at most about 5 ms and then returns the values of the last session.

The public interface reports the same events in other ways: CUSTOM_1 (the age of the delivered frame), CUSTOM_3 and lostFrames of getTelemetry(…) (the frames lost before the delivered frame), the telemetry itself (receive and capture time, metadata units), and the WARNING lines of the log (reconnections, transport fallbacks and switches, kernel receive buffer overflows; see Logging).

Table 20 - Internal diagnostic counters.

Topic Counters
Session protocol (“RTSP”, “RTSPS”, “RTP”, “SRTP”, “UDP”), transport (“UDP”, “TCP”, “MULTICAST”, “HTTP”, “HTTPS”), container (“RTP”, “MPEG-TS”), codec (“H264”, “HEVC”, “JPEG”), metadataFormat (formats seen), connected (session established and a media packet within the last 2 s), reconnects, transportSwitches (adaptive transport actions)
Security complianceMode, fipsActive, fullModuleCoverage, tlsVersion, tlsCipher, tlsPeerVerified, srtpSuite, srtpAuthFailures, srtpReplayDrops
Packets packetsReceived, packetsLost (never arrived or never repaired), packetsReordered (out of order but in time), packetsLate (arrived after they were declared lost), packetsDuplicated, packetsFiltered (1.4.0: datagrams dropped by the sender filter – RTSP media over UDP not from the server’s address, SDP-less media not from the -source sender), bytesReceived, kernelDrops (local receive buffer overflows), jitterUsec (RFC 3550), reorderWaitUsec (current adaptive reorder wait), lossPercent, reorderPercent (last receiver report interval)
Frames framesReceived, framesDelivered, framesIncomplete, framesSkipped (key frame wait), framesOverflow (output buffer), framesLatencyDropped (-maxlatency), framesFrozenMsec, bufferedFrames, latencyUsec (sender wall clock by RTCP SR to delivery), queueDelayUsec (see Adaptive transport)
Metadata metadataUnits, metadataCarried, metadataDropped, metadataUnbound (see Metadata binding)
RTCP rtcpSrReceived, rtcpRrSent, nackSent, tailProbes, tailProbeRepairs, tailProbeFalse (see NACK, RTX and key frame requests), keyFrameRequests
Decoding (1.3.0) framesDecoded, decodeFailures (frames that gave no picture: dropped before a key frame or as leading pictures, or not decodable), decodeTimeUsec (smoothed time to decode and convert a picture), decodedOutput (decoded frames requested), customCodec (setCodec(…)), decoderThreads (2), decoderColorRange (1, automatic), converterThreads (2), decodeThreadId (kernel thread id of the decode thread, 0 if none); bufferedFrames and framesOverflow count both output buffers (see Frame output and decoding)

VSourceStream class description

VSourceStream class declaration

VSourceStream class declared in VSourceStream.h file. Class declaration:

namespace cr
{
namespace video
{
/// Network video source: RTSP, RTSPS, RTP, SRTP and MPEG-TS receiver.
class VSourceStream : public VSource
{
public:

    /// Class constructor.
    VSourceStream();

    /// Class destructor. Calls closeVSource().
    ~VSourceStream() override;

    /// Copy constructor deleted.
    VSourceStream(const VSourceStream&) = delete;

    /// Copy assignment deleted.
    VSourceStream& operator=(const VSourceStream&) = delete;

    /// Get string of current library version.
    static std::string getVersion();

    /// Open video source by source string.
    bool openVSource(std::string& initString) override;

    /// Init video source.
    bool initVSource(VSourceParams& params) override;

    /// Get open status.
    bool isVSourceOpen() override;

    /// Close video source.
    void closeVSource() override;

    /// Get next video frame (FIFO, oldest first): encoded or decoded by frame.fourcc.
    bool getFrame(Frame& frame, int32_t timeoutMsec = 0) override;

    /// Set video source param.
    bool setParam(VSourceParam id, float value) override;

    /// Get video source param value.
    float getParam(VSourceParam id) override;

    /// Get video source params structure.
    void getParams(VSourceParams& params) override;

    /// Execute command.
    bool executeCommand(VSourceCommand id) override;

    /// Decode and execute command.
    bool decodeAndExecuteCommand(uint8_t* data, int size) override;

    /// Get the telemetry of the frame returned by the last successful getFrame().
    bool getTelemetry(VSourceStreamTelemetry& telemetry);

    /// Set a custom video codec for decoding (nullptr: the built-in decoders).
    void setCodec(VCodec* codec);
};
}
}

getVersion method

The getVersion() method returns a string with the current version of the VSourceStream class, in the “Major.Minor.Patch” format. Method declaration:

static std::string getVersion();

The method can be used without a VSourceStream class instance:

std::cout << "VSourceStream v: " << cr::video::VSourceStream::getVersion() << std::endl;

Console output:

VSourceStream v: 1.4.1

openVSource method

The openVSource(…) method initializes the video source with a source string. It copies the current params of the object (for example LOG_LEVEL and CUSTOM_2 set earlier by setParam(…)), sets their source field to the given string, and calls initVSource(…). So it blocks and fails in the same way. Instead of openVSource(…), the user can call initVSource(…). Method declaration:

bool openVSource(std::string& initString) override;
Parameter Value
initString Source string “url [option …]”, for example rtsp://192.168.1.67:8900/Cam1 -tcp or rtp://0.0.0.0:8765. See Source string for the URL formats, the options and their defaults. The string is not modified.

Returns: TRUE if the video source is open or FALSE if not (see initVSource(…)).

initVSource method

The initVSource(…) method initializes the video source with a set of params. It works in these steps:

  1. It closes the source if it is open.
  2. It parses params.source. Any error fails the call.
  3. It resolves the compliance mode, initializes the crypto provider and applies the compliance policy.
  4. It starts the receive thread. It waits until the session is established (RTSP: PLAY answered; rtp:// / udp:// / srtp://: sockets bound / joined), until the session fails, or until the -timeout deadline expires.
  5. It waits for the first frame for at most the remaining deadline, and at most 3000 ms, so that width, height and fourcc are known. It returns TRUE either way.

The telemetry is reset: getTelemetry(…) returns FALSE until the first frame of the new session is delivered.

Blocking: normally at most about -timeout, plus the close of an open source. If the session reports its failure late, the call waits up to 1 s longer. A host name resolution can’t be interrupted and may extend this; use an IPv4 literal for a hard bound. A closeVSource() or initVSource(…) / RESTART from another thread aborts the wait within about 20 ms; this call then returns FALSE. Instead of initVSource(…), the user can call openVSource(…). Method declaration:

bool initVSource(VSourceParams& params) override;
Parameter Value
params VSourceParams class object. Only three fields are used as input:
source - source string, see Source string.
logLevel - log level 0 - 3 (an invalid value selects 0), see Logging.
custom2 - initial output buffer depth 1 - 512 frames (-buffer overrides it); any other value (for example the default 0) keeps the current depth (default 32).
All other fields do not affect the initialization (but see the note below Table 26). On success the method updates fourcc (“H264”, “HEVC”, “JPEG”, or “” if no frame arrived yet), width, height and fps (0 if not known yet) and isOpen (TRUE). source is unchanged.

Returns: TRUE if the video source is initialized or FALSE if not (invalid source string, refused by the compliance policy, crypto provider not available, server unreachable, TLS verification failure or no TLS in the build, authentication failure, RTSP error, timeout, aborted by another thread, no memory or no thread: no exception leaves the call, and what it had started is released).

isVSourceOpen method

The isVSourceOpen() method returns the initialization status of the video source. The status stays TRUE during automatic reconnections, whether media is flowing or not. (The internal diagnostics report media flow as connected.) The status is also available as isOpen of the VSourceParams class and as the IS_OPEN param. Method declaration:

bool isVSourceOpen() override;

Returns: TRUE while the video source is initialized and not closed or FALSE if not.

closeVSource method

The closeVSource() method closes the video source. It stops the receive thread (RTSP TEARDOWN, RTCP BYE) and the decode thread. It interrupts a waiting getFrame(…). It drops the buffered frames and pictures with their metadata, and it resets the telemetry (getTelemetry(…) then returns FALSE). No automatic reconnection happens afterwards. An initVSource(…) or RESTART running in another thread is aborted; it returns FALSE. The internal diagnostics of the closed session stay readable (connected FALSE). The method blocks for up to about 500 ms (the wait for the TEARDOWN answer; with media over TCP or the HTTP tunnel at most 50 ms after the TEARDOWN was sent), plus a host name resolution in progress in the receive thread. Calling it on a closed source does nothing. The destructor calls it. Method declaration:

void closeVSource() override;

getFrame method

The getFrame(…) method returns the next video frame. Frames are delivered in order, oldest first, from the output buffer (see Frame delivery). They are encoded as received, or decoded in the pixel format that the fourcc of the frame selects (1.3.0; see Frame output and decoding). The method returns FALSE at once, whatever timeoutMsec is, if the source is not open: before initVSource(…), after closeVSource(), or after a failed initialization. It also returns FALSE at once while a closeVSource() / initVSource(…) / RESTART of another thread waits to start. A successful call also takes the telemetry of the frame, which getTelemetry(…) returns until the next successful call. Method declaration:

bool getFrame(Frame& frame, int32_t timeoutMsec = 0) override;
Parameter Value
frame Video frame, input and output (see the Frame class description).
Input: its fourcc selects the output (Table 6). Fourcc::H264, Fourcc::HEVC or Fourcc::JPEG: the encoded frame of the stream. Fourcc::RGB24, BGR24, YUYV, UYVY, GRAY, YUV24, NV12, NV21, YU12 or YV12: the decoded picture in this format (a default Frame is YUV24). Any other fourcc: FALSE at once.
Output on success: fourcc is the delivered format (encoded: the codec of the stream, whichever of H264 / HEVC / JPEG was asked for; decoded: the requested format, or NV12 for a picture that the converter refuses). width / height come from the SPS, the JPEG header or the decoder. size is the payload size in bytes (encoded: the frame size; decoded: the picture size of the format). frameId is +1 per delivered frame (never reset; a 32-bit counter converted to int, see Frame delivery). sourceId is not touched.
Buffer: the frame buffer is reused without a new allocation when it is the buffer that the library allocated for this Frame object at an earlier call (data pointer, width, height and frameId as the library left them) and it is large enough. Otherwise it is released and allocated once, in the shape of the delivered frame: width * height * 4 bytes for H264 / HEVC / JPEG, and the exact picture size for the uncompressed formats. A payload larger than width * height * 4 bytes, or a geometry above the supported maximum, is dropped, never truncated; so is a frame whose buffer cannot be allocated (no memory: FALSE, no exception, the Frame object is left empty).
timeoutMsec Time to wait for a new frame, milliseconds:
- timeoutMsec == -1 - the method waits endlessly until a new frame arrives. closeVSource() and initVSource(…) / RESTART interrupt the wait. If one of them is running in another thread, the method waits for it.
- timeoutMsec == 0 - the method only checks whether a new frame exists. It never waits, not even for a running initVSource(…) / closeVSource().
- timeoutMsec > 0 - the method waits for a new frame for the given time. This is the total time, including a wait for a running initVSource(…) / RESTART / closeVSource() of another thread.

Returns: TRUE if a frame that was never delivered before was copied, or FALSE if not (timeout expired, source not open, wait interrupted, frame dropped at delivery, unsupported fourcc). A call that returns FALSE does not change the telemetry.

Every call resets the CYCLE_TIME_MKS param to 0 at its start. On success, the call sets it to the interval since the previous successful call; for the first frame after initVSource(…), it is the interval since initVSource(…) returned. The CUSTOM_1 param gives the age of the delivered frame, and the CUSTOM_3 param gives the number of frames lost before it (see VSourceParam enum). getTelemetry(…) gives its telemetry: frame ID, key frame flag, timestamp, capture and receive time, lost frames and metadata units.

setParam method

The setParam(…) method sets a new value of a video source param. Only two params are writable, before and after initialization:

  • LOG_LEVEL: 0 - disable, 1 - file VSourceStream.log, 2 - console, 3 - both. Applied immediately.
  • CUSTOM_2: output buffer depth, 1 - 512 frames. Applied immediately. If more frames are buffered, the overflow policy drops the oldest ones.

The values must be exact integers in range. All other params are not supported: WIDTH, HEIGHT and FPS are detected from the stream; gain, exposure, focus and region of interest don’t exist for a network source; read only params can’t be set. For these params the method returns FALSE and logs “Not supported parameter.”. Method declaration:

bool setParam(VSourceParam id, float value) override;
Parameter Description
id Video source parameter ID according to VSourceParam enum.
value Video source parameter value.

Returns: TRUE if the parameter was set or FALSE if not (not supported or out of range).

getParam method

The getParam(…) method returns the value of a video source param. It can be called at any time, and it never blocks behind a waiting getFrame(…). Method declaration:

float getParam(VSourceParam id) override;
Parameter Description
id Video source parameter ID according to VSourceParam enum. Supported: LOG_LEVEL; WIDTH, HEIGHT, FPS (detected, 0 until known); CYCLE_TIME_MKS; IS_OPEN (1 / 0); CUSTOM_1, CUSTOM_2, CUSTOM_3 (see VSourceParam enum).

Returns: parameter value or -1 if the parameter is not supported (gain, exposure, focus, region of interest).

getParams method

The getParams(…) method returns a copy of the video source params. The fields custom1, custom2 and custom3 hold the CUSTOM_1, CUSTOM_2 and CUSTOM_3 values described in VSourceParam enum. fourcc is the format of the last delivered frame: “H264”, “HEVC”, “JPEG”, or the name of a decoded format such as “YUV24” (before the first delivery: the codec of the stream). width, height and fps describe the stream; every delivered frame updates them. source is the source string as given. Method declaration:

void getParams(VSourceParams& params) override;
Parameter Description
params VSourceParams class object.

executeCommand method

The executeCommand(…) method executes a video source command. The only command is RESTART. It closes the source and initializes it again with the current params (source string, log level, output buffer depth). It blocks like initVSource(…), and it resets the telemetry in the same way. It also works after closeVSource(), and it clears the closed-by-user state. Method declaration:

bool executeCommand(VSourceCommand id) override;
Parameter Description
id Video source action command ID according to VSourceCommand enum.

Returns: TRUE if the command was executed successfully or FALSE if not (unknown command, or the re-initialization failed, for example before any initialization or when there is no source string).

decodeAndExecuteCommand method

The decodeAndExecuteCommand(…) method decodes a command and executes it on the video source side. It decodes the commands encoded by the encodeCommand(…) and encodeSetParamCommand(…) methods of the VSource interface class. When a command is decoded, the method calls setParam(…) or executeCommand(…). Before decoding, it rejects a SET_PARAM packet (first byte 0x01) whose size is not 11 bytes, a packet shorter than 7 bytes, and a null pointer. The method is thread safe: it can be called from any thread. Method declaration:

bool decodeAndExecuteCommand(uint8_t* data, int size) override;
Parameter Description
data Pointer to input command.
size Size of command. Must be 11 bytes for SET_PARAM and at least 7 bytes for COMMAND.

Returns: TRUE if the command was decoded (SET_PARAM or COMMAND) and executed or FALSE if not.

getTelemetry method

The getTelemetry(…) method (1.2.0) returns the telemetry of the frame that the last successful getFrame(…) call returned:

  • its identity: frame ID and key frame flag;
  • its timing: 90 kHz timestamp, capture time on the sender’s wall clock, receive time on the local wall clock;
  • the number of frames lost before it;
  • the packet loss of the source when it was delivered (1.4.0): the packets lost since initialization and the lost share of the last 10 seconds (see Frame telemetry and capture time);
  • the metadata units bound to it: KLV, ONVIF and, with -seidata, the SEI user data as UNKNOWN. A frame without metadata has none (an empty metadata vector).

See VSourceStreamTelemetry structure, Metadata binding and Frame telemetry and capture time. getFrame(…) takes this snapshot together with the frame it returns. So both always describe the same frame (frameId equals Frame::frameId), whatever the receive thread does meanwhile (new, dropped or replaced frames, reconnections). The next successful getFrame(…) of any thread replaces the snapshot. So call the method from the thread that called getFrame(…), right after it. The method is thread safe, and it never blocks behind a waiting getFrame(…).

Memory. The buffers of the output structure are reused, so in steady state no memory is allocated (for units of at most 64 KiB, or of unchanged sizes). Pass the same structure to every call. metadata keeps its capacity, and the data of each unit is assigned in place. When a frame has fewer units, the source keeps the data buffers of the removed elements: up to 16 buffers of at most 64 KiB each, which closeVSource() and initVSource(…) free. It hands them to the elements of a later frame with more units. Retained memory is bounded (EU CRA 2(h)): a unit buffer above 64 KiB is released before it takes a smaller unit. So the structure keeps at most the metadata of the current frame (at most 4 MiB) plus 64 KiB per unit (256 units: 16 MiB). The source keeps at most 16 x 64 KiB of spare buffers, besides the metadata of the last delivered frame. Method declaration:

bool getTelemetry(VSourceStreamTelemetry& telemetry);
Parameter Description
telemetry Output, VSourceStreamTelemetry structure (reset to its defaults when FALSE is returned).

Returns: TRUE if a frame was delivered since the last initVSource(…) / openVSource(…) and telemetry describes it. FALSE if not: before the first frame, after closeVSource(), after a failed initialization (RESTART resets the telemetry like initVSource(…)), or when there is no memory for the metadata units (no exception is thrown). telemetry is then reset to its defaults: frameId 0, keyFrame FALSE, timestamp 0, captureTimeUsec -1, receiveTimeUsec 0, lostFrames 0, lostPackets 0, packetLossPercent 0, no metadata.

Example:

cr::video::Frame frame;
cr::video::VSourceStreamTelemetry telemetry;
if (source.getFrame(frame, 1000) && source.getTelemetry(telemetry))
{
    std::cout << "Frame " << telemetry.frameId << (telemetry.keyFrame ? " key" : "") <<
    " timestamp " << telemetry.timestamp << " lost before " << telemetry.lostFrames <<
    ", packets lost " << telemetry.lostPackets << " (" << telemetry.packetLossPercent <<
    " % in 10 s), " << telemetry.metadata.size() << " metadata units" << std::endl;
    for (const cr::video::VSourceStreamMetadata& unit : telemetry.metadata)
    {
        if (unit.type == cr::video::VSourceStreamDataType::KLV && unit.exact && unit.valid)
        {
            // unit.data holds a checked KLV unit that describes exactly this frame.
            std::cout << "KLV stream " << unit.stream << ": " << unit.data.size() <<
            " bytes" << std::endl;
        }
    }
}

setCodec method

The setCodec(…) method (1.3.0) sets a custom video codec. This codec decodes the received frames instead of the built-in decoders (VDecoderH264, VDecoderH265, VDecoderJPEG) when getFrame(…) is asked for an uncompressed fourcc (see Frame output and decoding). Method declaration:

void setCodec(VCodec* codec);
Parameter Value
codec Pointer to an object of a class that implements the VCodec interface, or nullptr to return to the built-in decoders (the default).

Input of the codec. The codec gets every encoded frame of the stream, in decoding order, as src of VCodec::transcode(…). H.264 / H.265 frames are access units in Annex-B format with 4-byte start codes (src.fourcc H264 or HEVC). JPEG frames are complete JFIF images (src.fourcc JPEG). src.data is followed by 64 zero bytes that src.size does not count, so a codec may read a little beyond its input. dst.fourcc is set to NV12 before every call.

Output of the codec. A call that returns TRUE delivers a picture in dst. The picture may be in any uncompressed format that FormatConverter takes. It is converted to the requested format when the formats differ, or delivered as it is when the converter refuses it. Pictures are expected in display order, as the built-in decoders give them.

Telemetry of the pictures. A picture gets the telemetry and metadata of the oldest frame (by timestamp) given to the codec that has no picture yet. So a codec may delay its output by a constant number of frames (B-frames, a hardware pipeline), and every picture still carries its own frame’s telemetry. The rule needs one picture for every frame. A codec that gives no picture for 32 frames before its first one is restarted (RESET, decoding starts again at the next key frame, with a WARNING). A codec that stops giving pictures for a while and then goes on by itself, without a RESET, binds the pictures after the pause to older frames, until decoding restarts. A codec that returns broken pictures (smaller than their geometry) only loses those frames.

Calling rules. The codec is called on the decode thread of the source only, never concurrently. It gets VCodecCommand::RESET when decoding restarts: at the first uncompressed request, after compressed requests, after initVSource(…) / RESTART, on a codec switch of the stream, and on a codec change. setCodec(…) can be called at any time, from any thread. It waits for a transcode(…) call in progress, so the previous codec is no longer used when setCodec(…) returns. Decoding continues with the new codec at the next key frame. The library does not change the settings of the codec (threads, colour range). The codec object must stay valid until setCodec(nullptr) has returned or the source is destroyed. The codec must not throw: an exception ends the decode thread, decoding stays off until closeVSource() / initVSource(…) and getFrame(…) for an uncompressed fourcc times out (asking for encoded frames still works). The codec must not call back into the source (closeVSource(), setCodec(…), initVSource(…)) from transcode(…): the decode thread holds the codec while it runs, and the call would wait for it. The internal diagnostics report customCodec.

encodeSetParamCommand method of VSource interface class

The encodeSetParamCommand(…) static method encodes a command that changes a param of a remote video source. To control a video source remotely, the developer designs a protocol, encodes the command according to it, and delivers the command over the communication channel. To simplify this, the VSource class contains static methods that encode control commands. The VSource class provides two types of commands: a param change command (SET_PARAM) and an action command (COMMAND). encodeSetParamCommand(…) encodes a SET_PARAM command: byte 0 is 0x01, bytes 1 and 2 are the major and minor version of the VSource library, bytes 3 - 6 are the param ID (int32), and bytes 7 - 10 are the value (float). Method declaration:

static void encodeSetParamCommand(uint8_t* data, int& size, VSourceParam id, float value);
Parameter Description
data Pointer to data buffer for encoded command. Must have size >= 11 bytes.
size Size of encoded data. Will be 11 bytes.
id Parameter ID according to VSourceParam enum.
value Parameter value.

encodeSetParamCommand(…) is static, so it is used without a VSource class instance. It is used on the client side (the control system). Command encoding example:

// Buffer for encoded data.
uint8_t data[11];
// Size of encoded data.
int size = 0;
// Encode command: output buffer depth 64 frames.
cr::video::VSource::encodeSetParamCommand(data, size, cr::video::VSourceParam::CUSTOM_2, 64.0f);

encodeCommand method of VSource interface class

The encodeCommand(…) static method encodes an action command for a remote video source. It is the second of the two command types described in encodeSetParamCommand(…). encodeCommand(…) encodes a COMMAND (action command): byte 0 is 0x00, bytes 1 and 2 are the major and minor version of the VSource library, and bytes 3 - 6 are the command ID (int32). Method declaration:

static void encodeCommand(uint8_t* data, int& size, VSourceCommand id);
Parameter Description
data Pointer to data buffer for encoded command. Must have size >= 7 bytes.
size Size of encoded data. Will be 7 bytes.
id Command ID according to VSourceCommand enum.

encodeCommand(…) is static, so it is used without a VSource class instance. It is used on the client side (the control system). Command encoding example:

// Buffer for encoded data.
uint8_t data[7];
// Size of encoded data.
int size = 0;
// Encode command.
cr::video::VSource::encodeCommand(data, size, cr::video::VSourceCommand::RESTART);

decodeCommand method of VSource interface class

The decodeCommand(…) static method decodes a command on the video source side (the edge device). The version bytes of the command must match the version of the VSource library on the decoding side. Method declaration:

static int decodeCommand(uint8_t* data, int size, VSourceParam& paramId, VSourceCommand& commandId, float& value);
Parameter Description
data Pointer to input command.
size Size of command. Should be 11 bytes for SET_PARAM and 7 bytes for COMMAND.
paramId Parameter ID according to VSourceParam enum. After decoding a SET_PARAM command, the method returns the parameter ID here.
commandId Command ID according to VSourceCommand enum. After decoding a COMMAND, the method returns the command ID here.
value Parameter value (after decoding a SET_PARAM command).

Returns: 0 if a COMMAND was decoded, 1 if a SET_PARAM command was decoded, or -1 on errors (size below 7 bytes, version mismatch, unknown type byte). Note: for a SET_PARAM command whose size is not 11 bytes, the method returns 0 (VSource 1.8.5), as if a COMMAND was decoded. decodeAndExecuteCommand(…) of VSourceStream rejects such a packet before decoding.

Data structures

The VSource.h file of the VSource interface class defines the IDs of the parameters (VSourceParam enum) and the IDs of the action commands (VSourceCommand enum). The VSourceStream.h file defines the telemetry of a delivered frame (VSourceStreamTelemetry structure), one metadata unit of the telemetry (VSourceStreamMetadata structure) and the type of a metadata unit (VSourceStreamDataType enum).

VSourceCommand enum

Enum declaration:

enum class VSourceCommand
{
    /// Restart.
    RESTART = 1
};

Table 21 - Video source commands description.

Command Description
RESTART Closes the source and initializes it again with the current params (source string, log level, output buffer depth). Blocks like initVSource(…). Also clears the closed-by-user state that closeVSource() sets and that blocks automatic reconnection.

VSourceParam enum

Enum declaration:

enum class VSourceParam
{
    /// [read/write] Logging mode. Values: 0 - Disable, 1 - Only file,
    /// 2 - Only terminal, 3 - File and terminal.
    LOG_LEVEL = 1,
    /// [read/write] Frame width. User can set frame width before initialization
    /// or after. Some video source classes may set width automatically.
    WIDTH,
    /// [read/write] Frame height. User can set frame height before
    /// initialization or after. Some video source classes may set height
    /// automatically.
    HEIGHT,
    /// [read/write] Gain mode. Value depends on implementation but it is
    /// recommended to keep default values: 0 - Manual control, 1 - Auto.
    GAIN_MODE,
    /// [read/write] Gain value. Value: 0(min for particular video source class)
    /// - 65535(max for particular video source class).
    GAIN,
    /// [read/write] Exposure mode. Value depends on implementation but it is
    /// recommended to keep default values: 0 - Manual control, 1 - Auto.
    EXPOSURE_MODE,
    /// [read/write] Exposure value. Value: 0(min for particular video source
    /// class) - 65535(max for particular video source class).
    EXPOSURE,
    /// [read/write] Focus mode. Value depends on implementation but it is
    /// recommended to keep default values: 0 - Manual control, 1 - Auto.
    FOCUS_MODE,
    /// [read/write] Focus position. Value: 0(full near) - 65535(full far).
    FOCUS_POS,
    /// [read only] Video capture cycle time. **VSource** class sets this value
    /// automatically. This parameter means time interval between two captured
    /// video frame.
    CYCLE_TIME_MKS,
    /// [read/write] FPS. User can set frame FPS before initialization or after.
    /// Some video source classes may set FPS automatically.
    FPS,
    /// [read only] Open flag. 0 - not open, 1 - open.
    IS_OPEN,
    /// Region of interest upper left corner x coordinate.
    ROI_X,
    /// Region of interest upper left corner y coordinate.
    ROI_Y,
    /// Region of interest width.
    ROI_WIDTH,
    /// Region of interest height.
    ROI_HEIGHT,
    /// [read/write] Custom parameter. Depends on implementation.
    CUSTOM_1,
    /// [read/write] Custom parameter. Depends on implementation.
    CUSTOM_2,
    /// [read/write] Custom parameter. Depends on implementation.
    CUSTOM_3
};

Table 22 - Video source params description. Some params are not supported in VSourceStream implementation.

Parameter Access Description
LOG_LEVEL read / write Logging mode. Specifies the log output mode. Values:
0 - Disable,
1 - Only file (VSourceStream.log),
2 - Only terminal (console),
3 - File and terminal.
Applied immediately, see Logging.
WIDTH read only Frame width of the last delivered frame (or of the first frame known at initialization), pixels, from the SPS or the JPEG header. 0 until known. Can’t be set.
HEIGHT read only Frame height of the last delivered frame (or of the first frame known at initialization), pixels. 0 until known. Can’t be set.
GAIN_MODE not supported setParam(…) returns FALSE, getParam(…) returns -1.
GAIN not supported setParam(…) returns FALSE, getParam(…) returns -1.
EXPOSURE_MODE not supported setParam(…) returns FALSE, getParam(…) returns -1.
EXPOSURE not supported setParam(…) returns FALSE, getParam(…) returns -1.
FOCUS_MODE not supported setParam(…) returns FALSE, getParam(…) returns -1.
FOCUS_POS not supported setParam(…) returns FALSE, getParam(…) returns -1.
CYCLE_TIME_MKS read only Interval between the two last successful getFrame(…) calls, microseconds; 0 after a call that delivered no frame. See Frame delivery.
FPS read only Frame rate of the stream: SPS VUI timing, else SDP a=framerate, else measured. 0 until known. Can’t be set.
IS_OPEN read only Open flag. 0 - not open, 1 - open (also during automatic reconnections).
ROI_X not supported setParam(…) returns FALSE, getParam(…) returns -1.
ROI_Y not supported setParam(…) returns FALSE, getParam(…) returns -1.
ROI_WIDTH not supported setParam(…) returns FALSE, getParam(…) returns -1.
ROI_HEIGHT not supported setParam(…) returns FALSE, getParam(…) returns -1.
CUSTOM_1 read only Age of the last delivered frame, microseconds: delivery time minus the arrival time of its last packet (1.1.0: the kernel receive time for UDP, so the age includes kernel queueing and the thread wake-up). See Frame delivery.
CUSTOM_2 read / write Output buffer depth, frames. Values: 1 - 512, default 32. Applied immediately. See Frame delivery.
CUSTOM_3 read only Number of frames lost before the last delivered frame since the previous delivered one: overflow drops, -maxlatency drops, incomplete frames (including frames dropped at delivery by getFrame(…)) and skipped frames; also lostFrames of the frame’s telemetry (getTelemetry(…)).

Note: after a failed initVSource(…), and while one is running, WIDTH, HEIGHT and FPS hold the values passed in params (for a default VSourceParams: 1920, 1080 and 0); they describe the stream only after a successful initialization.

VSourceStreamTelemetry structure

The VSourceStreamTelemetry structure (1.2.0) is declared in the VSourceStream.h file. The getTelemetry(…) method returns it: it is the telemetry of the frame that the last successful getFrame(…) returned. Structure declaration:

/**
 * @brief Telemetry of the frame returned by the last successful getFrame()
 * (VSourceStream::getTelemetry()).
 */
struct VSourceStreamTelemetry
{
    /// Frame::frameId of that frame: +1 per delivered frame, never reset.
    /// A 32-bit counter converted to int: after 2^31 deliveries it goes on
    /// with negative values and after 2^32 it passes 0 again (a valid id
    /// when getTelemetry() returns TRUE); compare it with Frame::frameId
    /// for equality only.
    int frameId{0};
    /// TRUE for a key frame: H.264 IDR, picture with a recovery point SEI
    /// or intra picture preceded by SPS / PPS; H.265 IRAP or recovery
    /// point; every JPEG frame.
    bool keyFrame{false};
    /// Media timestamp of the frame, 90 kHz: its RTP timestamp; MPEG-TS
    /// (raw or in RTP): the PES PTS, low 32 bits.
    uint32_t timestamp{0};
    /// Capture time of the frame on the sender's wall clock, UNIX epoch
    /// microseconds, -1 if unknown. Source, best first: the frame's MISB ST
    /// 0604 precision time stamp SEI (H.264 UUID "MISPmicrosectime", H.265
    /// UUID A8687DD4-D759-3758-A5CE-F0338B6545F1), the RTP header extension
    /// of the frame's first packet (ONVIF replay extension, or
    /// abs-capture-time negotiated with SDP a=extmap), the RTCP sender
    /// report mapping of the frame's RTP timestamp (after the first sender
    /// report; never for raw MPEG-TS; none from a sender report with a
    /// relative NTP time, RFC 3550 6.4.1: seconds with the most significant
    /// bit clear while the local clock is before 2036). The sender's value
    /// is passed through without a plausibility check (only a time of 0 or
    /// before 1970 counts as none): a sender clock that is wrong gives a
    /// wrong capture time.
    int64_t captureTimeUsec{-1};
    /// Receive time of the frame's last packet on the local wall clock
    /// (CLOCK_REALTIME), UNIX epoch microseconds. UDP: the kernel receive
    /// time stamp (SO_TIMESTAMPNS; the poll time when the kernel gives
    /// none); TCP / TLS / HTTP tunnel: the time the data was read from the
    /// socket. The receive path keeps this time on the monotonic clock and
    /// getFrame() converts it with the wall clock offset of the delivery
    /// moment: it equals the kernel stamp within about 2 us unless the
    /// wall clock was stepped or slewed in between (NTP slewing: at most
    /// 0.5 us per ms from arrival to delivery). With synchronised clocks
    /// receiveTimeUsec - captureTimeUsec is the sender-to-receiver latency.
    int64_t receiveTimeUsec{0};
    /// Frames lost before this frame since the previous successful
    /// getFrame() (incomplete, skipped until a key frame, dropped by the
    /// output buffer or -maxlatency, dropped at delivery): the CUSTOM_3
    /// value of this frame.
    int lostFrames{0};
    /// Media packets lost since initVSource() when the frame was delivered
    /// (1.4.0; cumulative, never decreases, also across automatic
    /// reconnections). RTP (also MPEG-TS in RTP): the RTP packets of all
    /// streams of the source (video and metadata) the receiver gave up
    /// waiting for, after the reorder wait and the NACK retransmissions
    /// (when the server offers them): a packet that arrives after that
    /// still counts (it came too late for its frame), a duplicate never
    /// does, and an SRTP packet that fails authentication counts as lost.
    /// Raw MPEG-TS (udp://, rtp:// with TS datagrams): the TS packets
    /// missing by the continuity counters of the video, KLV and table PIDs
    /// (a lower bound: 16 or more packets missing on one PID count modulo
    /// 16). UDP counts the datagrams the local kernel dropped too (full
    /// receive buffer); over TCP only the packets the sender itself left
    /// out.
    uint64_t lostPackets{0};
    /// Share of the media packets lost over the last 10 s before the frame
    /// was delivered (1.4.0), percent 0..100: lost / (received + lost) of
    /// the packets counted as for lostPackets (received: RTP packets that
    /// arrived, or raw TS packets of the checked PIDs), in 1 s steps (the
    /// window is the current second and the 9 before it). 0 when nothing
    /// was lost in the window. Meant for decisions (alarm, transport or
    /// stream change): it rises with the first lost packets and falls back
    /// to 0 within 10 s after the loss ended.
    float packetLossPercent{0.0f};
    /// Metadata units bound to the frame (see the class description of
    /// VSourceStream): exact units first, then the others in time order.
    /// Empty if none.
    std::vector<VSourceStreamMetadata> metadata;
};

Table 23 - VSourceStreamTelemetry structure fields description.

Field Type Description
frameId int Frame::frameId of that frame: +1 per delivered frame, never reset. A 32-bit counter converted to int: after 2^31 deliveries it goes on with negative values and after 2^32 it passes 0 again (a valid ID when getTelemetry(…) returns TRUE); compare it with Frame::frameId for equality only.
keyFrame bool TRUE for a key frame: H.264 IDR, picture with a recovery point SEI or intra picture preceded by SPS / PPS; H.265 IRAP or recovery point; every JPEG frame.
timestamp uint32_t Media timestamp of the frame, 90 kHz: its RTP timestamp; MPEG-TS (raw or in RTP): the PES PTS, low 32 bits.
captureTimeUsec int64_t Capture time of the frame on the sender’s wall clock, UNIX epoch microseconds, -1 if unknown. Source, best first (Table 8): the frame’s MISB ST 0604 precision time stamp SEI (H.264 UUID “MISPmicrosectime”, H.265 UUID A8687DD4-D759-3758-A5CE-F0338B6545F1), the RTP header extension of the frame’s first packet (ONVIF replay extension, or abs-capture-time negotiated with SDP a=extmap), the RTCP sender report mapping of the frame’s RTP timestamp (after the first sender report; never for raw MPEG-TS; none from a sender report with a relative NTP time, RFC 3550 6.4.1: seconds with the most significant bit clear while the local clock is before 2036). The sender’s value is passed through without a plausibility check (only a time of 0 or before 1970 counts as none): a sender clock that is wrong gives a wrong capture time. See Frame telemetry and capture time.
receiveTimeUsec int64_t Receive time of the frame’s last packet on the local wall clock (CLOCK_REALTIME), UNIX epoch microseconds. UDP: the kernel receive time stamp (SO_TIMESTAMPNS; the poll time when the kernel gives none); TCP / TLS / HTTP tunnel: the time the data was read from the socket. The receive path keeps this time on the monotonic clock and getFrame(…) converts it with the wall clock offset of the delivery moment: it equals the kernel stamp within about 2 us unless the wall clock was stepped or slewed in between (NTP slewing: at most 0.5 us per ms from arrival to delivery). With synchronised clocks receiveTimeUsec - captureTimeUsec is the sender-to-receiver latency.
lostFrames int Frames lost before this frame since the previous successful getFrame(…) (incomplete, skipped until a key frame, dropped by the output buffer or -maxlatency, dropped at delivery): the CUSTOM_3 value of this frame.
lostPackets uint64_t Media packets lost since initVSource(…) when the frame was delivered (1.4.0; cumulative, never decreases, also across automatic reconnections). RTP (also MPEG-TS in RTP): the RTP packets of all streams of the source (video and metadata) the receiver gave up waiting for, after the reorder wait and the NACK retransmissions (when the server offers them): a packet that arrives after that still counts (it came too late for its frame), a duplicate never does, and an SRTP packet that fails authentication counts as lost. Raw MPEG-TS (udp://, rtp:// with TS datagrams): the TS packets missing by the continuity counters of the video, KLV and table PIDs (a lower bound: 16 or more packets missing on one PID count modulo 16). UDP counts the datagrams the local kernel dropped too (full receive buffer); over TCP only the packets the sender itself left out. See Frame telemetry and capture time.
packetLossPercent float Share of the media packets lost over the last 10 s before the frame was delivered (1.4.0), percent 0 - 100: lost / (received + lost) of the packets counted as for lostPackets (received: RTP packets that arrived, or raw TS packets of the checked PIDs), in 1 s steps (the window is the current second and the 9 before it). 0 when nothing was lost in the window. Meant for decisions (alarm, transport or stream change): it rises with the first lost packets and falls back to 0 within 10 s after the loss ended.
metadata std::vector<VSourceStreamMetadata> Metadata units bound to the frame (VSourceStreamMetadata, see Metadata binding): exact units first, then the others in time order. Empty if the frame has no metadata.

VSourceStreamMetadata structure

The VSourceStreamMetadata structure (1.2.0) is declared in the VSourceStream.h file. It is one metadata unit bound to the frame: an element of the metadata field of VSourceStreamTelemetry (see Metadata binding). Structure declaration:

/**
 * @brief One metadata unit received with the frame
 * (VSourceStreamTelemetry::metadata).
 */
struct VSourceStreamMetadata
{
    /// Data type: KLV, ONVIF or UNKNOWN for a delivered unit (NONE only in
    /// a default-constructed one).
    VSourceStreamDataType type{VSourceStreamDataType::NONE};
    /// Metadata stream index 0..3, unique among the metadata streams of the
    /// source: RTSP metadata track in SETUP order; SDP-less metadata SSRCs
    /// and MPEG-TS KLV PIDs take the next free index when they first appear
    /// (the KLV PIDs of a program in PMT order, so a plain MPEG-TS source
    /// numbers its KLV PIDs 0, 1, ... in PMT order); UNKNOWN (SEI): 0.
    /// Tells apart two KLV tracks of one source.
    int stream{0};
    /// TRUE if the unit's timestamp equals the frame's (within +-min(450
    /// ticks of 90 kHz, 1/4 frame step), after the clock mapping): it
    /// describes exactly this frame; an UNKNOWN (SEI) unit is exact only
    /// when it comes from the frame's own access unit. FALSE if it belongs
    /// to another moment: a unit between the previous frame and this one
    /// (metadata at a higher rate than the video, or off the frame grid), a
    /// late unit carried to this frame (it arrived after its own frame was
    /// handed to the output buffer, or its frame was dropped: also the
    /// UNKNOWN units of an access unit that was not delivered), a unit without
    /// timestamp (e.g. MPEG-TS KLV without PTS) or one whose clock can't be
    /// related to the video clock. Place a unit that is not exact by the
    /// time inside its data (MISB ST 0601 Tag 2, ONVIF UtcTime, MISB ST
    /// 0604) and by its position (those units come in time order); a unit
    /// without such a time can only be ordered.
    bool exact{true};
    /// TRUE if the unit passed the structure checks and, when present, the
    /// MISB ST 0601 checksum. KLV: every item has a 16-byte key starting
    /// with 06 0E 2B 34 and a BER length that fits, the items fill the unit
    /// exactly, and if the unit carries MISB ST 0601 (an item with the UAS
    /// local set key 06 0E 2B 34 02 0B 01 01 0E 01 03 01 01 00 00 00),
    /// every such local set parses and ends with a correct Tag 1 checksum
    /// (mandatory in ST 0601). ONVIF: always TRUE, since only a unit that
    /// starts as an XML document is delivered: '<' as its first byte, or
    /// after a UTF-8 BOM and optional XML white space; after XML white
    /// space without a BOM, for a unit whose first packet follows a packet
    /// loss, and for a unit that goes on with the RTP timestamp of a unit
    /// dropped for its start, only "<?xml" or the MetadataStream start tag
    /// (any namespace prefix, e.g. <tt:MetadataStream); at most 1024 bytes of
    /// BOM and white space before the '<'. UNKNOWN: always TRUE (the 16-byte
    /// UUID is present). A unit that fails the checks is delivered with
    /// valid FALSE; a unit broken by packet loss, and a unit of an RTP
    /// metadata track or SSRC that does not start like a unit (KLV: 06 0E
    /// 2B 34; ONVIF: see above), is dropped.
    bool valid{true};
    /// Unit bytes: one KLV unit (one or more KLV items), one XML document
    /// or, UNKNOWN, one SEI payload (UUID + user data). Never empty in a
    /// delivered unit.
    std::vector<uint8_t> data;
};

Table 24 - VSourceStreamMetadata structure fields description.

Field Type Description
type VSourceStreamDataType Data type: KLV, ONVIF or UNKNOWN for a delivered unit (NONE only in a default-constructed one), see VSourceStreamDataType enum.
stream int Metadata stream index 0 - 3, unique among the metadata streams of the source: RTSP metadata track in SETUP order; SDP-less metadata SSRCs and MPEG-TS KLV PIDs take the next free index when they first appear (the KLV PIDs of a program in PMT order, so a plain MPEG-TS source numbers its KLV PIDs 0, 1, … in PMT order); UNKNOWN (SEI): 0. Tells apart two KLV tracks of one source.
exact bool TRUE if the unit’s timestamp equals the frame’s (within +-min(450 ticks of 90 kHz, 1/4 frame step), after the clock mapping): it describes exactly this frame; an UNKNOWN (SEI) unit is exact only when it comes from the frame’s own access unit. FALSE if it belongs to another moment: a unit between the previous frame and this one (metadata at a higher rate than the video, or off the frame grid), a late unit carried to this frame (it arrived after its own frame was handed to the output buffer, or its frame was dropped: also the UNKNOWN units of an access unit that was not delivered), a unit without timestamp (for example MPEG-TS KLV without PTS) or one whose clock can’t be related to the video clock. Place a unit that is not exact by the time inside its data (MISB ST 0601 Tag 2, ONVIF UtcTime, MISB ST 0604) and by its position (those units come in time order); a unit without such a time can only be ordered.
valid bool TRUE if the unit passed the structure checks and, when present, the MISB ST 0601 checksum. KLV: every item has a 16-byte key starting with 06 0E 2B 34 and a BER length that fits, the items fill the unit exactly, and if the unit carries MISB ST 0601 (an item with the UAS local set key 06 0E 2B 34 02 0B 01 01 0E 01 03 01 01 00 00 00), every such local set parses and ends with a correct Tag 1 checksum (mandatory in ST 0601). ONVIF: always TRUE, since only a unit that starts as an XML document is delivered: ‘<’ as its first byte, or after a UTF-8 BOM and optional XML white space; after XML white space without a BOM, for a unit whose first packet follows a packet loss, and for a unit that goes on with the RTP timestamp of a unit dropped for its start, only <?xml or the MetadataStream start tag (any namespace prefix, for example <tt:MetadataStream); at most 1024 bytes of BOM and white space before the ‘<’. UNKNOWN: always TRUE (the 16-byte UUID is present). A unit that fails the checks is delivered with valid FALSE; a unit broken by packet loss, and a unit of an RTP metadata track or SSRC that does not start like a unit (KLV: 06 0E 2B 34; ONVIF: see above), is dropped.
data std::vector<uint8_t> Unit bytes: one KLV unit (one or more KLV items), one XML document or, UNKNOWN, one SEI payload (UUID + user data). Never empty in a delivered unit.

VSourceStreamDataType enum

The type of a metadata unit (the type field of VSourceStreamMetadata). Enum declaration:

/**
 * @brief Type of a metadata unit (VSourceStreamMetadata::type).
 */
enum class VSourceStreamDataType
{
    /// No metadata: the type of a unit that holds none (data empty), the
    /// default of VSourceStreamMetadata. A delivered unit never has it; a
    /// frame without metadata has an empty VSourceStreamTelemetry::metadata.
    NONE = 0,
    /// KLV (SMPTE 336M; MISB ST 0601 and others): RTP metadata track (RFC
    /// 6597), SDP-less KLV SSRC, MPEG-TS KLV PID (sync or async).
    KLV = 1,
    /// ONVIF metadata stream XML document (vnd.onvif.metadata,
    /// tt:MetadataStream).
    ONVIF = 2,
    /// User data of a format the source does not interpret (1.4.0; 1.3.x:
    /// SEI): the H.264 / H.265 SEI user_data_unregistered payloads, only
    /// with the -seidata option: the 16-byte UUID that names the format,
    /// followed by the user data (emulation prevention bytes removed; the
    /// SEI also stays in the frame). The units of the frame's own access
    /// unit are exact; those of access units that were not delivered
    /// (skipped before a key frame or after a loss, corrupt, without a
    /// picture, oversized, dropped by the output buffer or -maxlatency) come
    /// with the next delivered frame, not exact.
    UNKNOWN = 3
};

Table 25 - VSourceStreamDataType enum values description.

Value Description
NONE 0. No metadata: the type of a unit that holds none (data empty), the default of VSourceStreamMetadata. A delivered unit never has it; a frame without metadata has an empty metadata vector of VSourceStreamTelemetry.
KLV 1. KLV (SMPTE 336M; MISB ST 0601 and others): RTP metadata track (RFC 6597), SDP-less KLV SSRC, MPEG-TS KLV PID (sync or async).
ONVIF 2. ONVIF metadata stream XML document (vnd.onvif.metadata, tt:MetadataStream).
UNKNOWN 3. User data of a format the source does not interpret (1.4.0; 1.3.x: SEI): the H.264 / H.265 SEI user_data_unregistered payloads, only with the -seidata option: the 16-byte UUID that names the format, followed by the user data (emulation prevention bytes removed; the SEI also stays in the frame). The units of the frame’s own access unit are exact; those of access units that were not delivered (skipped before a key frame or after a loss, corrupt, without a picture, oversized, dropped by the output buffer or -maxlatency) come with the next delivered frame, not exact.

VSourceParams class description

VSourceParams class declaration

The VSourceParams class is used to initialize the video source (initVSource(…) method) and to get all current params (getParams(…) method). VSourceParams can also be written to and read from JSON files (JSON_READABLE macro, see the ConfigReader class description). It also provides methods to encode and decode the params. Class declaration:

class VSourceParams
{
public:
    /// Logging mode. Values: 0 - Disable, 1 - Only file,
    /// 2 - Only terminal, 3 - File and terminal.
    int logLevel{0};
    /// Video source: file, video stream, video device, camera num, etc.
    std::string source{"/dev/video0"};
    /// FOURCC: RGB24, BGR24, YUYV, UYVY, GRAY, YUV24, NV12, NV21, YU12, YV12.
    /// Value says to video source class which pixel format preferable for
    /// output video frame. Particular video source class can ignore this params
    /// during initialization. Parameters should be set before initialization.
    std::string fourcc{"YUYV"};
    /// Frame width. User can set frame width before initialization
    /// or after. Some video source classes may set width automatically.
    int width{1920};
    /// Frame height. User can set frame height before
    /// initialization or after. Some video source classes may set height
    /// automatically.
    int height{1080};
    /// Gain mode. Value depends on implementation but it is
    /// recommended to keep default values: 0 - Manual control, 1 - Auto.
    int gainMode{1};
    /// Gain value. Value: 0(min for particular video source class)
    /// - 65535(max for particular video source class).
    int gain{0};
    /// Exposure mode. Value depends on implementation but it is
    /// recommended to keep default values: 0 - Manual control, 1 - Auto.
    int exposureMode{1};
    /// Exposure value. Value: 0(min for particular video source
    /// class) - 65535(max for particular video source class).
    int exposure{1};
    /// Focus mode. Value depends on implementation but it is
    /// recommended to keep default values: 0 - Manual control, 1 - Auto.
    int focusMode{1};
    /// Focus position. Value: 0(full near) - 65535(full far).
    int focusPos{0};
    /// Video capture cycle time. **VSource** class sets this value
    /// automatically. This parameter means time interval between two captured
    /// video frame.
    int cycleTimeMks{0};
    /// FPS. User can set frame FPS before initialization or after.
    /// Some video source classes may set FPS automatically.
    float fps{0};
    /// Open flag. 0 - not open, 1 - open.
    bool isOpen{false};
    /// Region of interest upper left corner x coordinate.
    int roiX{0};
    /// Region of interest upper left corner y coordinate.
    int roiY{0};
    /// Region of interest width.
    int roiWidth{0};
    /// Region of interest height.
    int roiHeight{0};
    /// Custom parameter. Depends on implementation.
    float custom1{0.0f};
    /// Custom parameter. Depends on implementation.
    float custom2{0.0f};
    /// Custom parameter. Depends on implementation.
    float custom3{0.0f};

    JSON_READABLE(VSourceParams, logLevel, source, fourcc, width, height,
                  gainMode, exposureMode, focusMode, fps, roiX, roiY, roiWidth,
                  roiHeight, custom1, custom2, custom3);

    /// Copy operator.
    VSourceParams& operator= (const VSourceParams& src);

    /// Encode (serialize) params.
    bool encode(uint8_t* data, int bufferSize, int& size,
                VSourceParamsMask* mask = nullptr);

    /// Decode (deserialize) params.
    bool decode(uint8_t* data, int dataSize);
};

Table 26 - VSourceParams class fields description.

Field Type Description
logLevel int Logging mode. Specifies the log output mode. Values:
0 - Disable,
1 - Only file (VSourceStream.log),
2 - Only terminal (console),
3 - File and terminal.
Input of initVSource(…) (an invalid value selects 0).
source string Source string “url [option …]”, see Source string. Input of initVSource(…), returned unchanged by getParams(…).
fourcc string Output only: format of the last delivered frame: “H264”, “HEVC” or “JPEG” for encoded frames (before the first delivery the codec of the stream, “” until known), or the name of the decoded format, “RGB24”, “BGR24”, “YUYV”, “UYVY”, “GRAY”, “YUV24”, “NV12”, “NV21”, “YU12” or “YV12” (1.3.0). Ignored as input: the output is chosen by the fourcc of the frame given to getFrame(…).
width int Output only: frame width, pixels, 0 until known. Ignored as input.
height int Output only: frame height, pixels, 0 until known. Ignored as input.
gainMode int Not supported. Can have any value.
gain int Not supported. Can have any value.
exposureMode int Not supported. Can have any value.
exposure int Not supported. Can have any value.
focusMode int Not supported. Can have any value.
focusPos int Not supported. Can have any value.
cycleTimeMks int Output only: interval between the two last successful getFrame(…) calls, microseconds (CYCLE_TIME_MKS param).
fps float Output only: frame rate of the stream (SPS VUI timing, else SDP a=framerate, else measured), 0 until known. Ignored as input.
isOpen bool Output only: open flag. false - not open, true - open.
roiX int Not supported. Can have any value.
roiY int Not supported. Can have any value.
roiWidth int Not supported. Can have any value.
roiHeight int Not supported. Can have any value.
custom1 float Output only: age of the last delivered frame, microseconds (CUSTOM_1 param).
custom2 float Input of initVSource(…): initial output buffer depth 1 - 512 frames (-buffer overrides it, other values keep the current depth). Output: current depth (CUSTOM_2 param).
custom3 float Output only: frames lost before the last delivered frame (CUSTOM_3 param).

Note: the VSourceParams fields listed in Table 26 reflect the params that the methods setParam(…) and getParam(…) set and get.

Note: after a failed initVSource(…), and while one is running, getParams(…) returns fourcc, width, height and fps as they were passed in params (for a default VSourceParams: “YUYV”, 1920, 1080 and 0); they describe the stream only after a successful initialization.

Serialize video source params

The VSourceParams class provides the encode(…) method to serialize the video source params (the fields of the VSourceParams class, see Table 26). Serialization is needed to send the video source params over communication channels. The method doesn’t encode the fields source and fourcc. It can exclude particular params from serialization. For this, the method inserts a binary mask of 3 bytes after the header byte 0x02 and the major and minor version of the VSource library. Each bit of the mask stands for one param, and the decode(…) method recognizes the mask. Method declaration:

bool encode(uint8_t* data, int bufferSize, int& size, VSourceParamsMask* mask = nullptr);
Parameter Value
data Pointer to data buffer.
bufferSize Data buffer size. It must be at least 7 bytes; otherwise the method returns FALSE. If the buffer is smaller than required, it is filled with fewer parameters.
size Size of encoded data. 79 bytes with all parameters.
mask Parameters mask: a pointer to a VSourceParamsMask structure. VSourceParamsMask (declared in the VSource.h file) has a flag for each field (parameter) of the VSourceParams class. To exclude parameters from serialization, pass a pointer to a mask in which the flags of these parameters are false.

Returns: TRUE if params encoded or FALSE if not.

VSourceParamsMask structure declaration:

struct VSourceParamsMask
{
    bool logLevel{true};
    bool width{true};
    bool height{true};
    bool gainMode{true};
    bool gain{true};
    bool exposureMode{true};
    bool exposure{true};
    bool focusMode{true};
    bool focusPos{true};
    bool cycleTimeMks{true};
    bool fps{true};
    bool isOpen{true};
    bool roiX{true};
    bool roiY{true};
    bool roiWidth{true};
    bool roiHeight{true};
    bool custom1{true};
    bool custom2{true};
    bool custom3{true};
};

Example without parameters mask:

// Prepare params.
cr::video::VSourceParams in;
in.source = "rtsp://192.168.1.67:8900/Cam1 -tcp";
in.logLevel = 2;

// Encode data.
uint8_t data[1024];
int size = 0;
in.encode(data, 1024, size);
std::cout << "Encoded data size: " << size << " bytes" << std::endl; // 79 bytes

Example with parameters mask:

// Prepare params.
cr::video::VSourceParams in;
in.source = "rtsp://192.168.1.67:8900/Cam1 -tcp";
in.logLevel = 2;

// Prepare params mask.
cr::video::VSourceParamsMask mask;
mask.logLevel = false; // Exclude logLevel. Others by default.

// Encode data.
uint8_t data[1024];
int size = 0;
in.encode(data, 1024, size, &mask);
std::cout << "Encoded data size: " << size << " bytes" << std::endl; // 75 bytes

Deserialize video source params

The VSourceParams class provides the decode(…) method to deserialize the video source params (the fields of the VSourceParams class, see Table 26). Deserialization is needed to receive the video source params over communication channels. The method doesn’t decode the fields source and fourcc; it sets them to empty strings. It recognizes automatically which parameters encode(…) serialized. Parameters that are not in the data are set to 0. The version bytes of the data must match the version of the VSource library on the decoding side. Method declaration:

bool decode(uint8_t* data, int dataSize);
Parameter Value
data Pointer to encoded data buffer. Data size should be at least 6 bytes (header and mask) plus the parameters the mask announces.
dataSize Size of data.

Returns: TRUE if data decoded (deserialized) or FALSE if not.

Example:

// Encode data.
cr::video::VSourceParams in;
uint8_t data[1024];
int size = 0;
in.encode(data, 1024, size);

std::cout << "Encoded data size: " << size << " bytes" << std::endl;

// Decode data.
cr::video::VSourceParams out;
if (!out.decode(data, size))
{
    std::cout << "Can't decode data" << std::endl;
}

Read params from JSON file and write to JSON file

The VSource library depends on the ConfigReader library, which provides methods to read params from a JSON file and to write params to a JSON file. Example of writing params to a JSON file and reading them back:

// Write params to file.
cr::video::VSourceParams in;
in.logLevel = 2;
in.source = "rtsp://192.168.1.67:8900/Cam1 -tcp";
in.custom2 = 64.0f;
cr::utils::ConfigReader inConfig;
inConfig.set(in, "vSourceParams");
inConfig.writeToFile("TestVSourceParams.json");

// Read params from file.
cr::utils::ConfigReader outConfig;
if (!outConfig.readFromFile("TestVSourceParams.json"))
{
    std::cout << "Can't open config file" << std::endl;
    return false;
}
cr::video::VSourceParams out;
if (!outConfig.get(out, "vSourceParams"))
{
    std::cout << "Can't read params from config file" << std::endl;
    return false;
}

TestVSourceParams.json will look like this (VSource 1.8.5 writes the fields in the order of the JSON_READABLE declaration):

{
    "vSourceParams": {
        "logLevel": 2,
        "source": "rtsp://192.168.1.67:8900/Cam1 -tcp",
        "fourcc": "YUYV",
        "width": 1920,
        "height": 1080,
        "gainMode": 1,
        "exposureMode": 1,
        "focusMode": 1,
        "fps": 0.0,
        "roiX": 0,
        "roiY": 0,
        "roiWidth": 0,
        "roiHeight": 0,
        "custom1": 0.0,
        "custom2": 64.0,
        "custom3": 0.0
    }
}

Build and connect to your project

Requirements:

  • Operating system: Linux or Windows. On Linux the library uses POSIX threads (CMake links pthread). On Windows it uses Winsock 2 and the native Windows threads, without a pthread library (CMake links ws2_32).
  • Tools: CMake 3.13 or newer and a C++17 compiler: GCC or Clang on Linux; MSVC on Windows. The library’s own sources and the decoders also compile with a MinGW-w64 cross compiler on Linux (checked with zig / Clang), but the built-in H.264 decoder VDecoderH264 is made for MSVC on Windows (its Windows threading layer replaces the pthread functions), so a MinGW-w64 build is not a supported configuration.
  • OpenSSL (optional): the OpenSSL 3.0 or newer development package (Debian / Ubuntu: sudo apt install libssl-dev, RHEL / Fedora: sudo dnf install openssl-devel, Alpine: apk add openssl-dev). If OpenSSL is installed where CMake does not look, set -DOPENSSL_ROOT_DIR=/path/to/openssl. When CMake finds OpenSSL, it prints “VSourceStream: OpenSSL (TLS enabled)", and the library supports TLS (rtsps://, including the HTTPS tunnel). Without OpenSSL, CMake prints "VSourceStream: OpenSSL not found (TLS disabled)", and an rtsps:// source can't connect. On Linux, OpenSSL is also the crypto backend of CryptoProvider, and it brings the OpenSSL FIPS provider (see [Security and compliance modes](#security-and-compliance-modes)).
  • Git submodules in 3rdparty: VSource, CryptoProvider (cryptographic primitives: the OpenSSL backend on Linux when OpenSSL is found, Windows CNG on Windows, a standalone implementation otherwise) and, since 1.3.0, the decoders VDecoderH264, VDecoderH265 and VDecoderJPEG and the converter FormatConverter.

Typical commands to build the VSourceStream library:

cd VSourceStream
git submodule update --init --recursive
mkdir build
cd build
cmake ..
make

The build type defaults to Release. The build creates the VSourceStream library (static by default) and the example application VSourceStreamExample in build/bin. The test application VSourceStreamTest is built only on request. Configure with cmake .. -D_SUBMODULE_CACHE_OVERWRITE=OFF -D_VSOURCESTREAM_TEST=ON; without the first option, the defaults overwrite the second. The test application is then built in build/bin when CMake finds the MServer and VCodecLibav source trees and the FFmpeg development libraries. Otherwise CMake prints one of these messages and skips the test application (see Test application):

  • “VSourceStreamTest: MServer not found at …; test application skipped”;
  • “VSourceStreamTest: VCodecLibav not found; test application skipped”;
  • “VSourceStreamTest: FFmpeg development libraries not found; test application skipped”.

Compiler settings of the library:

  • Release optimization is -O2 -funroll-loops (MSVC: /O2). It is deliberately not -Ofast or -march=native: reproducible builds are an EU CRA obligation, -march=native breaks cross compilation, and -ffast-math is inappropriate for timestamp arithmetic. To tune for a specific device (GCC / Clang), set the CMake cache variable VSOURCE_STREAM_TARGET_ARCH, which adds -march=<value>, for example cmake .. -DVSOURCE_STREAM_TARGET_ARCH=armv8-a.
  • Warnings and hardening (all configurations). GCC / Clang: -Wall -Wextra -Wpedantic -Wshadow, -fstack-protector-strong -fPIC, and in Release -D_FORTIFY_SOURCE=2. MSVC: /W4 /GS /guard:cf, with the definitions WIN32_LEAN_AND_MEAN, NOMINMAX, _CRT_SECURE_NO_WARNINGS and _WINSOCK_DEPRECATED_NO_WARNINGS. The example and the test application are compiled with the same warning flags (/W4 with MSVC).
  • A static library can’t apply linker hardening to itself. So the library declares its hardening link options PUBLIC: they reach every executable or shared object that links it (and the library itself when it is built as a shared library): -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack on Linux, and /guard:cf with MSVC (none with MinGW-w64).

To connect the VSourceStream library to your CMake project as source code, do the following. For example, your repository has this structure:

CMakeLists.txt
src
    CMakeLists.txt
    yourLib.h
    yourLib.cpp

Create the folder 3rdparty in your repository and add VSourceStream there: as a git submodule (git submodule add <VSourceStream repository URL> 3rdparty/VSourceStream and git submodule update --init --recursive), or as a copy of the repository folder, including its 3rdparty folder (VSource, CryptoProvider, the decoders and FormatConverter). The new structure of your repository:

CMakeLists.txt
src
    CMakeLists.txt
    yourLib.h
    yourLib.cpp
3rdparty
    VSourceStream

Create a CMakeLists.txt file in the 3rdparty folder. It should contain:

cmake_minimum_required(VERSION 3.13)

################################################################################
## 3RD-PARTY
## dependencies for the project
################################################################################
project(3rdparty LANGUAGES CXX)

################################################################################
## SETTINGS
## basic 3rd-party settings before use
################################################################################
# To inherit the top-level architecture when the project is used as a submodule.
SET(PARENT ${PARENT}_YOUR_PROJECT_3RDPARTY)
# Disable self-overwriting of parameters inside included subdirectories.
SET(${PARENT}_SUBMODULE_CACHE_OVERWRITE OFF CACHE BOOL "" FORCE)

################################################################################
## CONFIGURATION
## 3rd-party submodules configuration
################################################################################
SET(${PARENT}_SUBMODULE_VSOURCESTREAM                      ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_VSOURCESTREAM)
    SET(${PARENT}_VSOURCESTREAM                            ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_VSOURCESTREAM_TEST                       OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_VSOURCESTREAM_EXAMPLE                    OFF CACHE BOOL "" FORCE)
endif()

################################################################################
## INCLUDING SUBDIRECTORIES
## Adding subdirectories according to the 3rd-party configuration
################################################################################
if (${PARENT}_SUBMODULE_VSOURCESTREAM)
    add_subdirectory(VSourceStream)
endif()

The file 3rdparty/CMakeLists.txt adds the folder VSourceStream to your project and excludes the test application and the example from compiling. (When VSourceStream is included as a sub-repository, the test application and the example are excluded by default: ${PARENT}_VSOURCESTREAM_TEST and ${PARENT}_VSOURCESTREAM_EXAMPLE default to OFF, and ${PARENT}_VSOURCESTREAM defaults to ON.) The new structure of your repository:

CMakeLists.txt
src
    CMakeLists.txt
    yourLib.h
    yourLib.cpp
3rdparty
    CMakeLists.txt
    VSourceStream

Next, include the 3rdparty folder in the main CMakeLists.txt file of your repository. Add this line at the end of your main CMakeLists.txt:

add_subdirectory(3rdparty)

Then link the VSourceStream library in your src/CMakeLists.txt file:

target_link_libraries(${PROJECT_NAME} VSourceStream)

Linking VSourceStream also links its dependencies: VSource, CryptoProvider, the decoders, FormatConverter, pthread on Linux or ws2_32 on Windows, and OpenSSL when CMake found it. It also adds the hardening link options to your target (see above). Done!

Simple example

The example shows how to:

  • initialize the video source with a source string and the generic VSource interface methods;
  • select the output with the fourcc of the frame: the encoded frames as received, or decoded pictures in a pixel format (see Frame output and decoding);
  • receive frames in a loop that stops after 30 s without a frame;
  • get the telemetry of every frame with getTelemetry(…): frame ID, key frame flag, timestamp, capture and receive time, lost frames, lost packets and the lost share of the last 10 s, and the metadata units bound to the frame (KLV / ONVIF, and the SEI user data as UNKNOWN with -seidata) with their type, stream index, size and exact / valid flags.

The example reads the source string and the output format from the console. Enter keeps the defaults: “rtsp://127.0.0.1:8554/live -tcp” and H264 (the encoded frames). The log goes to the console. Example:

#include <cstdint>
#include <iostream>
#include <string>
#include <utility>
#include "VSourceStream.h"

namespace
{
/// getFrame() waits of 1 s in a row without a frame after which the example
/// stops. The source reconnects by itself meanwhile (the defaults -reconnect
/// 5 s, -timeout 10 s and a backoff of at most 4 s leave room for a complete
/// reconnection), but a server that is gone for good or a stream that never
/// decodes in the requested format must not keep the example waiting forever.
constexpr int MAX_WAITS_WITHOUT_FRAME{30};

/// Our fourcc names: compressed formats and the pixel formats getFrame() decodes to.
const std::pair<const char*, cr::video::Fourcc> g_fourccNames[] = {
    {"H264", cr::video::Fourcc::H264}, {"HEVC", cr::video::Fourcc::HEVC},
    {"JPEG", cr::video::Fourcc::JPEG}, {"RGB24", cr::video::Fourcc::RGB24},
    {"BGR24", cr::video::Fourcc::BGR24}, {"YUYV", cr::video::Fourcc::YUYV},
    {"UYVY", cr::video::Fourcc::UYVY}, {"GRAY", cr::video::Fourcc::GRAY},
    {"YUV24", cr::video::Fourcc::YUV24}, {"NV12", cr::video::Fourcc::NV12},
    {"NV21", cr::video::Fourcc::NV21}, {"YU12", cr::video::Fourcc::YU12},
    {"YV12", cr::video::Fourcc::YV12}};

/// Get our fourcc as text ("H264", "BGR24", ...).
std::string fourccText(cr::video::Fourcc fourcc)
{
    for (const auto& entry : g_fourccNames)
    {
        if (entry.second == fourcc)
        {
            return entry.first;
        }
    }
    return "?";
}

/// Get our fourcc by its text.
bool fourccByText(const std::string& text, cr::video::Fourcc& fourcc)
{
    for (const auto& entry : g_fourccNames)
    {
        if (text == entry.first)
        {
            fourcc = entry.second;
            return true;
        }
    }
    return false;
}

/// Get our metadata type as text.
std::string typeText(cr::video::VSourceStreamDataType type)
{
    switch (type)
    {
    case cr::video::VSourceStreamDataType::KLV:
    {
        return "KLV";
    }
    case cr::video::VSourceStreamDataType::ONVIF:
    {
        return "ONVIF";
    }
    case cr::video::VSourceStreamDataType::UNKNOWN:
    {
        return "UNKNOWN";
    }
    case cr::video::VSourceStreamDataType::NONE:
    default:
    {
        break;
    }
    }
    return "NONE";
}
}

/// Entry point.
int main()
{
    // Read the source string, e.g. "rtp://0.0.0.0:5004".
    std::cout << "VSourceStream v" << cr::video::VSourceStream::getVersion() << std::endl;
    std::string initString = "rtsp://127.0.0.1:8554/live -tcp";
    std::cout << "Source [" << initString << "]: ";
    std::string line;
    if (std::getline(std::cin, line) && !line.empty())
    {
        initString = line;
    }

    // Read the output format: H264 / HEVC / JPEG - frames as received,
    // a pixel format (BGR24, NV12, ...) - decoded frames.
    cr::video::Fourcc output = cr::video::Fourcc::H264;
    std::cout << "Output (H264 = as received, or BGR24, RGB24, NV12, GRAY...) [H264]: ";
    if (std::getline(std::cin, line) && !line.empty() && !fourccByText(line, output))
    {
        std::cout << "Unknown output format " << line << std::endl;
        return -1;
    }

    // Init video source (log to console).
    cr::video::VSourceStream source;
    if (!source.setParam(cr::video::VSourceParam::LOG_LEVEL, 2))
    {
        // Not fatal: the source works without its log.
        std::cout << "Can't set the log level" << std::endl;
    }
    if (!source.openVSource(initString))
    {
        std::cout << "Can't open the video source" << std::endl;
        return -1;
    }

    // Main loop, bounded by MAX_WAITS_WITHOUT_FRAME.
    cr::video::Frame frame;
    cr::video::VSourceStreamTelemetry telemetry;
    int waitsWithoutFrame = 0;
    while (waitsWithoutFrame < MAX_WAITS_WITHOUT_FRAME)
    {
        // Wait new frame 1 sec. The fourcc of the frame chooses the output.
        frame.fourcc = output;
        if (!source.getFrame(frame, 1000))
        {
            ++waitsWithoutFrame;
            std::cout << "No frame" << std::endl;
            continue;
        }
        waitsWithoutFrame = 0;

        // Telemetry of this frame: same thread, right after getFrame().
        if (!source.getTelemetry(telemetry))
        {
            std::cout << "No telemetry for frame " << frame.frameId << std::endl;
            continue;
        }

        // Show information.
        std::cout << "Frame " << telemetry.frameId << " " << frame.width << "x" <<
            frame.height << " " << fourccText(frame.fourcc) << " " << frame.size << " bytes" <<
            (telemetry.keyFrame ? " key" : "") << ", timestamp " << telemetry.timestamp <<
            ", capture " << telemetry.captureTimeUsec << " us, receive " <<
            telemetry.receiveTimeUsec << " us";
        if (telemetry.captureTimeUsec >= 0)
        {
            std::cout << " (receive - capture " <<
                telemetry.receiveTimeUsec - telemetry.captureTimeUsec << " us)";
        }
        std::cout << ", lost " << telemetry.lostFrames << " frames, " <<
            telemetry.lostPackets << " packets (" << telemetry.packetLossPercent <<
            " % in 10 s)" << std::endl;

        // Metadata units bound to the frame.
        for (const cr::video::VSourceStreamMetadata& unit : telemetry.metadata)
        {
            std::cout << "  " << typeText(unit.type) << " stream " << unit.stream << " " <<
                unit.data.size() << " bytes, exact " << (unit.exact ? "yes" : "no") <<
                ", valid " << (unit.valid ? "yes" : "no") << std::endl;
        }
    }

    // The destructor closes the source.
    std::cout << "No frame for " << MAX_WAITS_WITHOUT_FRAME << " s, stopped" << std::endl;
    return -1;
}

Console output against the reference server of the test application (VSourceStreamTest -server rtsp-klv: H.264 with a KLV track), with the default output H264 and without the log lines:

VSourceStream v1.4.1
Source [rtsp://127.0.0.1:8554/live -tcp]: rtsp://127.0.0.1:18500/live -tcp
Output (H264 = as received, or BGR24, RGB24, NV12, GRAY...) [H264]:
Frame 1 640x480 H264 56256 bytes key, timestamp 1239584462, capture 1790999184698912 us, receive 1790999184699944 us (receive - capture 1032 us), lost 0 frames, 0 packets (0 % in 10 s)
  KLV stream 0 44 bytes, exact yes, valid yes
Frame 2 640x480 H264 7471 bytes, timestamp 1239588008, capture 1790999184738788 us, receive 1790999184739164 us (receive - capture 376 us), lost 0 frames, 0 packets (0 % in 10 s)
  KLV stream 0 44 bytes, exact yes, valid yes
Frame 3 640x480 H264 7231 bytes, timestamp 1239591654, capture 1790999184778945 us, receive 1790999184779722 us (receive - capture 777 us), lost 0 frames, 0 packets (0 % in 10 s)
  KLV stream 0 44 bytes, exact yes, valid yes
Frame 4 640x480 H264 7189 bytes, timestamp 1239595217, capture 1790999184818813 us, receive 1790999184819346 us (receive - capture 533 us), lost 0 frames, 0 packets (0 % in 10 s)
  KLV stream 0 44 bytes, exact yes, valid yes

The same with the output BGR24 (decoded pictures; the log line “Decoding started (built-in decoders, 2 threads)” is omitted):

VSourceStream v1.4.1
Source [rtsp://127.0.0.1:8554/live -tcp]: rtsp://127.0.0.1:18500/live -tcp
Output (H264 = as received, or BGR24, RGB24, NV12, GRAY...) [H264]: BGR24
Frame 1 640x480 BGR24 921600 bytes key, timestamp 1239854433, capture 1790999187698821 us, receive 1790999187699566 us (receive - capture 745 us), lost 0 frames, 0 packets (0 % in 10 s)
  KLV stream 0 44 bytes, exact yes, valid yes
Frame 2 640x480 BGR24 921600 bytes, timestamp 1239858025, capture 1790999187738836 us, receive 1790999187739401 us (receive - capture 565 us), lost 0 frames, 0 packets (0 % in 10 s)
  KLV stream 0 44 bytes, exact yes, valid yes
Frame 3 640x480 BGR24 921600 bytes, timestamp 1239861592, capture 1790999187778723 us, receive 1790999187779224 us (receive - capture 501 us), lost 0 frames, 0 packets (0 % in 10 s)
  KLV stream 0 44 bytes, exact yes, valid yes

The capture time is the MISB ST 0604 time stamp SEI that MServer 2.1.1 adds with the KLV Tag 2 time of the frame. Sender and receiver share the wall clock here, so receive - capture is the delivery path. The receive time is the arrival of the encoded frame, because a decoded frame has the telemetry of its own encoded frame.

Test application

The test application VSourceStreamTest checks how the VSourceStream library works. It contains an automated test suite of 36 groups, an interactive console mode and a reference server mode. The reference server is the MServer video server library (RTSP, RTSPS, multicast, HTTP tunnels, SRTP, MPEG-TS, KLV, push streams), embedded in the test process. The test encodes its clips with FFmpeg through the VCodecLibav test codec (JPEG: the libavcodec MJPEG encoder). MServer streams these clips in pass-through mode. Every delivered frame is compared byte-exact with what was sent (JPEG: pixel-exact after decoding) and decoded again.

Requirements of the test application:

  • Linux. The test application uses Linux sockets, processes and kernel features (kernel time stamps, thread signals, /proc), so it builds and runs on Linux only. The library and the example build on Windows too.
  • The MServer source tree, version 2.1.4 (the “compliance” group expects the FIPS profiles of 2.1.3 and later, which refuse to start without the validated module; the “feedback” group needs the NACK / RTX and PLI / FIR of 2.1.1 or newer). MServer builds on Linux and Windows. CMake takes the tree from ../MServer (next to the VSourceStream repository), or from the path in the CMake cache variable VSOURCE_STREAM_MSERVER_DIR (cmake .. -DVSOURCE_STREAM_MSERVER_DIR=/path/to/MServer).
  • The VCodecLibav source tree (the test codec). CMake takes it from ../VCodecLibav, or from the path in the CMake cache variable VSOURCE_STREAM_VCODEC_LIBAV_DIR. When this path has no CMakeLists.txt, CMake falls back to the copy in MServer’s harness/VCodecLibav. CMake prints the tree it uses: “VSourceStreamTest: test codec VCodecLibav from ".
  • Both trees are added with add_subdirectory(… EXCLUDE_FROM_ALL), and their build files go to the build directory (build/mserver and build/vcodeclibav). The build of the test application does not write into these source trees.
  • The FFmpeg development libraries libavcodec, libavformat, libavutil, libavfilter and libswscale (Debian / Ubuntu: sudo apt install libavcodec-dev libavformat-dev libavutil-dev libavfilter-dev libswscale-dev).
  • If one of these requirements is missing, CMake prints a STATUS message and skips the test application. The library and the example are built anyway.
  • The test application is built only on request: cmake .. -D_SUBMODULE_CACHE_OVERWRITE=OFF -D_VSOURCESTREAM_TEST=ON (see Build and connect to your project). It is compiled with the warning flags of the library (-Wall -Wextra -Wpedantic -Wshadow; /W4 with MSVC).
  • nlohmann_json.hpp. VSource 1.8.5 brings ConfigReader 2.0 (without nlohmann JSON) under the same target name as ConfigReader 1.x, and ConfigReader 2.0 wins in this combined build. VStreamer 3.2 (part of MServer) takes nlohmann JSON from ConfigReader 1.x and declares its JSON functions with nlohmann types. So when MServer’s tree contains the ConfigReader 1.x header 3rdparty/VStreamer/3rdparty/ConfigReader/src/nlohmann_json.hpp, the CMake file of the test application force-includes this header (-include) into the VStreamer, MServer and VSourceStreamTest targets. VStreamer 3.3 uses ConfigReader 2.0 too: the header is absent, and nothing is added.
  • Optional tools, found at run time: /usr/bin/ffmpeg (reference client of the “latency” group, sender of the “interop” group and, with libx264 / libx265, of the B-frame cases of the “decode” group), gst-launch-1.0 / gst-inspect-1.0 (GStreamer senders), python3 with libgstrtspserver-1.0 (gst-rtsp-server cases). Cases whose tool is missing are skipped and reported with a “skip” or “info” line.
  • Network: the tests use 127.0.0.1 and the ports 18500 - 19519, so only one test run may run on a host at a time. The multicast cases also join multicast groups on a local interface; the “mcastpush” group uses the non-loopback address of the host. Multicast cases that need a multicast route are skipped on a host without one.

Command line (printed by ./VSourceStreamTest -h):

Usage: VSourceStreamTest [-v] [group ...]   run all or the given groups
       VSourceStreamTest -list               list the groups
       VSourceStreamTest -i                  interactive console mode
       VSourceStreamTest -server <variant>   reference server only (-server list)
       VSourceStreamTest -receiver <source> <msec> <file>   receive into a file (interop stress helper)
Environment: VSRTP_BUDGET_SCALE=<1..100> multiplies every group's time budget (sanitizer builds)
  • Without arguments, all groups run in the order of Table 27. Group names select groups (./VSourceStreamTest rtsp srtp). An unknown name prints “Unknown group: " and the usage, and the application exits with 1.
  • -list prints the groups with their description.
  • -v prints verbose output and sets the log level of every source under test to 2 (console).
  • -i starts the interactive mode, and -server the reference server mode. -receiver runs the receiver child process of the interop stress runs: it receives for the given time and writes the frames, the statistics and its own CPU time to the file.

Table 27 - Test groups.

Group Budget, s What it verifies
unit 180 Internal classes against MServer internals: SRTP known-answer tests, MServer SRTP / SRTCP protection to VSourceStream and back (AES-CM and GCM, rollover, rollover counter recovery of a receiver that joins after the wrap, replay, tampering, wrong key); digest (RFC 7616 vectors); base64 and strict parsers; SDP, Transport, RTP-Info and Session parsing; the RTSP client against a scripted server; source strings and the compliance matrix; H.264 / H.265 / JPEG depacketizers against MServer’s packetizer (byte-exact); MPEG-TS demuxer; output buffer policy; reorder buffer; RTCP; SPS parsing; frame builder (geometry limits, leading pictures, parameter sets per id); codec detection on payload type 98 and MPEG-TS end of access unit detection; 1.4.0: the 10 s packet loss window of the telemetry; 1.4.1: the log of unsolicited RTSP messages is bounded and no part of a password reaches an error text.
rtsp 180 RTSP over UDP and TCP for H.264, H.265 and JPEG: frames byte-exact and decodable, first frame a key frame, frameId +1 per frame, parameter sets in key frames, statistics, RTCP SR / RR, latencyUsec plausible; a scripted third-party camera (URL with a query, SDP with a vendor section, 64-bit ssrc= and an out-of-range session timeout in the replies).
multicast 90 RTSP with UDP multicast (-multicast, with and without -iface).
tunnel 90 RTSP over HTTP and RTSPS over HTTPS tunnels.
auth 120 Digest authentication with URL credentials and -user / -password, wrong and missing credentials refused; SHA-256 chosen over MD5, the password never on the wire, User-Agent “VSourceStream” without version (checked through a TCP relay).
rtsps 150 RTSPS with -noverify, -fingerprint and -cafile (IP address and host name), refusals: wrong fingerprint, self-signed certificate without an option, unrelated CA; tlsPeerVerified.
srtp 120 RTSPS + SRTP (RTP/SAVP) over UDP and TCP: frames byte-exact, no authentication failures or replay drops.
klv 120 KLV metadata track over UDP and TCP and ONVIF metadata track over TCP: every unit bound to its frame (exact) and byte-exact in the metadata of getTelemetry(…); a unit that trails its frame by more than the hold may come with a later frame only when its frame waited the hold.
ts 120 MPEG-TS in RTP over RTSP (H.264 and H.265) with synchronous and asynchronous KLV, over UDP and TCP.
direct 200 SDP-less rtp:// and udp:// push (RTP, RTP + KLV, MPEG-TS in RTP, raw MPEG-TS; H.264, H.265, JPEG) with the receiver started before and after the sender; auto-detection; RTCP.
mcastpush 90 Push to a multicast group received by rtp://: RTP (receiver first, no -iface) and MPEG-TS in RTP with KLV (mid-GOP join, -iface on the egress address); skipped without a multicast route.
srtpdirect 120 srtp:// with pre-shared keys (AES_CM_128_HMAC_SHA1_80, AEAD_AES_128_GCM) across the sequence number wrap, SRTCP, a wrong key delivers nothing and counts authentication failures.
compliance 180 Client-side refusals (CRA_STRICT with rtsp://, rtp:// and -noverify; STRICT_FIPS with udp://, srtp:// and -srtp; unknown profile); CRA_STRICT against MServer in the same mode (TLS interleaved, SRTP over TCP and UDP, plain UDP and a missing trust anchor refused), complianceMode and fullModuleCoverage; STRICT_FIPS and FIPS_ONVIF_BRIDGE against MServer: on a host with the validated module the servers start and stream (complianceMode without a suffix), on a host without it (the usual case) the servers refuse to start (MServer 2.1.3 fails closed) and the client still refuses -srtp under STRICT_FIPS.
robust 300 Server restart, media pause of 7 s (watchdog), codec change H.264 -> JPEG -> H.265 (RTSP and push), resolution change, consumer stall within and beyond the output buffer (overflow policy, CUSTOM_1, CUSTOM_2, CUSTOM_3), closeVSource() during getFrame(frame, -1), RESTART, RESTART / initVSource() racing closeVSource(), bounded readers during a blocking initVSource(), frame geometry (an SPS claiming 16384x16384 never delivered or allocated, caller’s frame buffer reused; 1.4.0: SPS of 16384x16384 and 8192x8192 in a running stream never delivered nor decoded, encoded and decoded output, the stream plays on).
impair 150 Loss, reordering, jitter and duplicates through a UDP proxy: reorder wait at the 10 ms minimum when clean, grows under impairment, still at its peak 8 s after the impairment and back at the minimum later (hold-then-decay), at most 2 late packets after the adaptation, a bursty reordering phase, lost frames explained by the proxy drops plus one GOP per late packet the receiver reported (packetsLate; at least 1, at most 2 GOPs), duplicates discarded without frame loss, latency back to the clean level; 1.4.0: the packet loss of the telemetry per phase (none while clean, a share while impaired, back to 0 once the 10 s window is clean, duplicates not counted, lostPackets never decreasing and equal to the proxy drops plus at most the late packets).
latency 240 Video frame latency (frames identified by a frame index SEI) against ffmpeg and GStreamer for RTSP / TCP, RTSP / UDP and raw MPEG-TS over UDP: VSourceStream mean and p95 at most 1 ms above the reference. Latency against ffmpeg for raw MPEG-TS over UDP (1280x720) and MPEG-TS in RTP over RTSP / TCP (640x480), both with synchronous KLV: VSourceStream mean and p95 at most 1 ms above ffmpeg’s (the ffmpeg client is started up to 3 times until it delivers KLV); mean time to first frame (RTSP / TCP, 3 attempts) lower than ffmpeg’s.
stats 90 getVersion(), behaviour before initialization, parameters (CUSTOM_2, LOG_LEVEL, unsupported and read only params), getters (getTelemetry(…) too) never block behind getFrame(…), internal statistics fields, commands (unknown command refused; decodeAndExecuteCommand(…): a SET_PARAM of size 7 and an empty packet refused, an encoded SET_PARAM CUSTOM_2 = 48 applied without a restart).
interop 480 Independent senders: ffmpeg (RTP, MPEG-TS, MPEG-TS in RTP; H.264, H.265, MJPEG; 1080p; payload type 98; mid-GOP join; sender restart with and without BYE; multicast; MPEG-TS access unit ends detected at once, except an exact-fit access unit at the end of a full datagram, which ends with the next frame), GStreamer payloaders and MPEG-TS muxer, gst-rtsp-server (RTSP over UDP / TCP / HTTP, H.265, JPEG, MPEG-TS, digest MD5, wrong password, RTSPS with -cafile and -fingerprint), MServer 1080p stress runs with the receiver in a separate process.
v110rtp 300 1.1.0 reorder buffer: hold-then-decay (the hold restarted only by a sample of at least half the peak), no jitter term, key-frame bypass, NACK give-up (with RTX a reordered original is no answer), NACK retry after twice the answer delay of a fast server (within half the wait) or after the answer of a slower server (one NACK per loss at 6 and 16 ms answer delay, a retry repair after one late retry answer, the retry still before the end of the wait when packets are processed after their arrival, a server answering just within half the wait retried after half the wait, premature retries of a slowing server stop); the 1.2.0 tail-loss probe (off unless enabled, silence learned from 8 frames, a lost marker packet repaired about 1.5 ms after it was sent, one probe per frame with the next frame’s NACK as fallback, at most one false probe for a paced sender and three for a slowing one, a retransmission for a packet that was only late counts false, none without NACK, on TCP or after the give-up, a retransmission nobody asked for dropped); RTP header extension parsing; the relative RTCP NTP time rule.
v110codec 300 1.1.0 depacketizers: AU-start proof, idle-gap frame end (learned again after a misprediction), marker reliability (a lost first slice of a field is no proof against the marker; a sender that stops setting it is detected), metadata unit ends (marker-less ONVIF at its closed root element), SEI parsing (1.4.1: a payload type or size coded by a long run of 0xFF bytes does not wrap), KLV validation, key-start packets, the size caps (frame, NAL units per access unit, metadata unit, JPEG extension, 24-bit JPEG offset), and the rest of an ONVIF document whose head was lost (never delivered in pieces).
v110ts 300 1.1.0 MPEG-TS: KLV robustness (short items split across PES keep their own bytes), NAL-aligned access units, idle-gap end; 1.4.0: TS packets missing by the continuity counters (the raw TS part of lostPackets); 1.4.1: a 4 MB backlog of tiny access units is split in linear time.
v110builder 300 1.1.0 frame builder, metadata binder and output buffer: AU-boundary loss rule, -keywait, key frame request pacing, queue delay, frozen time, carry (never to a frame shown before the unit), SEI units, -maxlatency; the telemetry unit order (exact, time order, untimed last) and the delivered formats (KLV, ONVIF, SEI); no RTCP SR capture time from a relative NTP time; 1.4.1: the parameter set cache is bounded in total and a tiny key frame stays small, the check of the referenced sets is linear.
v110pipeline 300 1.1.0 pipeline wiring on a live MediaPipeline: key-frame bypass behind a gap, idle-gap deadline, clock mappings (RTP-Info anchor, 1 kHz metadata clock, late or wrong anchors, sender reports from two states of the sender clock), stream indexes of two metadata tracks, RTP header-extension capture time (also MPEG-TS in RTP, whose frame timestamp is the PES PTS), retransmissions and key frames on a payload type the SDP does not announce (neither reset nor replace the video source), the RTCP destination of SDP-less reception (only from the address of the RTP packets), the creation rate of sources (a burst of 8, then one per second).
v110session 300 1.1.0 SDP a=extmap: parsing and id lookup (RFC 8285 session / media scope, directions, case-sensitive URI), limits, malformed entries (the adaptive transport and metadata tracks of the sessions are covered end to end by adapt, telemetry and feedback).
v110net 300 1.1.0 network layer: kernel receive timestamps (clock conversion, SO_TIMESTAMPNS), kernel drops (SO_RXQ_OVFL, WARNING text), TCP_QUICKACK re-armed after every read of plain TCP and TLS against a Nagle sender; 1.4.1: a TLS 1.3 peer that floods KeyUpdate messages without reading, or cuts a post-handshake message in the middle, does not make the poll loop spin (a bounded number of rounds per second).
v110public 300 1.1.0 source string and public API: the new options (ranges, defaults, minimum-wait and adaptive-transport resolution, refused combinations and their error texts), getTelemetry(…) before any frame, the internal statistics fields; 1.4.0: the values of VSourceStreamDataType and the empty default unit.
lowlat 180 1.1.0 latency end to end: TCP_QUICKACK against Nagle senders (scripted RTSP server, RTSPS / HTTP tunnel relay), receive thread scheduling options (frames delivered with SCHED_FIFO and -cpu 0), kernel arrival timestamps and drops with a stalled receive thread, frame ends of marker-less RTP and unsignalled / NAL-aligned MPEG-TS (crafted and GStreamer), -maxlatency.
adapt 200 1.1.0 adaptivity end to end through an RTSP-aware impairment relay: adaptive transport (UDP loss -> TCP, also when the repairs or reordered packets come after the reorder wait, while timely RTX repairs stay on UDP; kernel drops are no reason; TCP latency guard -> UDP trial), key-frame bypass, default minimum wait, AU-boundary loss rule, -keywait, NACK give-up, channel statistics.
telemetry 240 1.1.0 telemetry end to end: getTelemetry(…) on every carriage of the reference server (same frame as getFrame(…), receive time, lostFrames = CUSTOM_3, MServer 2.1.1’s MISB ST 0604 SEI time = KLV Tag 2 = captureTimeUsec), the binding matrix (50 Hz, sparse, off grid, +-1 tick, late, gate-dropped, mapped clocks, marker-less; units that are not exact in time order, also where it differs from the stored order: a late unit and a unit between two frames on one frame), the RTCP SR capture time fallback (ONVIF track, no ST 0604 SEI, within 20 ms of the receive time on loopback), MPEG-TS KLV robustness (before its video, cells, fragments, several PES, two PIDs, raw KLV, ffmpeg -c copy), ST 0604 SEI, RTP header-extension capture time, two metadata tracks, the provisional RTP-Info offset (scripted and gst-rtsp-server), validation flags.
feedback 240 RTCP feedback against MServer 2.1.1 through the RTSP-aware impairment relay: NACK / RTX repair at 0.5 / 2 / 5 % loss (lost after repair only what the relay dropped with every retransmission, frozen time, added latency (p95 at most 10 ms at 0.5 / 2 %, 20 ms at 5 %), loss to retransmission at most 5 ms p50: MServer 2.1.1 reads RTCP over UDP at once), the tail-loss probe (20 % of the frames lose their last packet: every one after the first 12 frames repaired by a probe, nothing lost, p95 at most 10 ms more; a clean run sends no probe), H.265 RTX, JPEG RTX (the payload type announced with apt=26, repairs), TCP without NACK, the first subscriber of a fresh raw-encoder server gets a key frame within 1 s, key frame requests to a raw-encoder server when RTX cannot repair (the key frame is the server’s next frame), no adaptive switch while RTX repairs.
telemetryapi 240 1.2.0 getTelemetry(…) end to end: FALSE and reset before the first frame and after initVSource(…), openVSource(…), RESTART and closeVSource(…), a getFrame(…) timeout keeps the telemetry; for every frame the telemetry of the frame getFrame(…) returned (frameId, lostFrames = CUSTOM_3 and the sender’s frame counter gap across output buffer drops and a reconnect, key frame flag, receive time against CUSTOM_1, capture time = MServer 2.1.1’s ST 0604 SEI = KLV Tag 2 for H.264 / H.265 over RTSP KLV tracks and MPEG-TS); crafted streams: timestamp across the 32-bit RTP wrap and the 33-bit PTS wrap, MPEG-TS in RTP (the PES PTS, not the RTP timestamp), KLV at 1x / 2x / 0.5x the video rate (exact flags), two KLV streams, wrong ST 0601 checksums (valid FALSE), ONVIF and SEI units, buffer reuse of the caller’s structure; bounded buffer retention (a 300 KiB unit, the spare pool, closeVSource(), no allocation in steady state), no capture time from a relative RTCP SR NTP time, CUSTOM_3 = lostFrames after close / re-init and with two getFrame(…) threads; a second thread reading parameters, statistics and the telemetry; 1.4.1: a failing allocation in getFrame(…) and getTelemetry(…) is answered with FALSE.
serverfix 180 MServer 2.1.1 fixes seen from the client: a configured but disabled push leg sends neither RTP nor RTCP, 60 RTSPS viewers vanishing while the server writes (SIGPIPE at its default action), RTSP requests with bodies followed by pipelined requests, an HTTP tunnel POST leg cut into 5-byte segments, ;srtp and ;multicast URLs with a KLV track, the MPEG-TS PMT version on a KLV PID / codec change followed by the library; 2 skips with reason (33-bit PTS continuity: covered by MServer’s own harness and the telemetryapi group; the MServer test program’s SDP).
decode 300 1.3.0 decoded output: H.264, H.265 and JPEG from MServer (RTSP / TCP with a KLV track, H.265 also over MPEG-TS): every delivered picture bit-exact with the picture of a test-owned decoder with the library’s settings (2 threads, automatic colour range) and, for the other formats, of a test-owned FormatConverter, each with its own frame’s KLV unit, key frame flag and frameId, no frame of decoder delay (reception to delivery p95 below 20 ms); limited range content expanded, full range content not; all 10 uncompressed formats in one session (switch without key frame wait, VSourceParams::fourcc, one output buffer); a Frame created anew for every call (same address, built for a smaller format than asked for) never taken for the previous buffer; compressed requests corrected to the stream’s codec, nothing decoded; switches between encoded and decoded output and between formats (restart at a key frame, the decode thread parked and reused, CUSTOM_3 = the frames missing); live codec changes H.264 -> JPEG -> H.265 and resolution changes of a raw-encoder server while decoding (decoding goes on with every codec, the geometry followed, every picture the server’s pattern frame of its own frame); decoding off the getFrame(…) thread (1280x720: pictures decoded during a consumer stall, getFrame(…) of a ready picture a fraction of the decode time); decoder stall (encoded frames dropped up to a key frame, no broken picture) and output overflow (only the oldest pictures dropped, decoding undisturbed); RESTART, closeVSource() releasing a waiting getFrame(…), re-initialization, two instances and the destructor (the decode thread by its kernel id); an unsupported fourcc refused at once; ffmpeg senders with B-frames (libx264 closed and open GOP, libx265 open GOP; from the start, decoding started mid-stream, late join): encoded frames byte-exact in decoding order, pictures bit-exact in display order with their own timestamps and key frame flags; no decode thread left once every source is destroyed. setCodec(…) is covered by the codec group.
codec 240 1.3.0 setCodec(…) with a custom codec (a VCodecLibav software decoder wrapped by a probe): every picture bit-exact with the codec’s picture of its own clip frame (H.264, H.265, JPEG; NV12 and converted formats, a codec output in another format, an output delay); the calling contract (decode thread only, never concurrent, whole frames with 4-byte start codes and 64 zeroed padding bytes, dst.fourcc NV12, RESET at restarts); codec changes while decoding; setCodec(…) waiting for a call in progress; failing and broken codecs, a codec stuck until it is reset (restarted after 33 calls without a picture, then every picture exact); RESTART and the destructor; 1.4.1: a codec that throws ends decoding without leaving getFrame(…), closeVSource(…) or the destructor waiting.
matrix 600 Every MServer delivery mode VSourceStream can receive (all but HLS and WebRTC), each cell byte-exact with its metadata and its statistics naming the path: the KLV track and a synchronous-KLV transport stream on all 11 client paths (RTSP over UDP, TCP, multicast and the HTTP tunnel; RTSPS interleaved, plain UDP and multicast media under a TLS control channel, the HTTPS tunnel; SRTP over UDP, over TCP and through the HTTPS tunnel); HEVC and JPEG with KLV on the tunnelled, TLS-interleaved, multicast and SRTP paths; ONVIF metadata documents with every codec and on multicast, SRTP and the HTTPS tunnel; asynchronous-KLV transport streams on multicast, inside TLS, in SRTP and tunnelled; the validate and restamp KLV modes (restamped units on every frame, Tag 2 within 1 s of the receive time); a transport stream whose caller passes no KLV; -nometadata on a KLV stream; payload sizes 576 (every frame fragmented, 3 TS packets per datagram) and 9000; digest authentication through both tunnels, on multicast, in SRTP and on plain UDP under TLS, no credentials through the tunnel and URL credentials together with -password refused; rtspEncryption strict (no plaintext listener; TLS-interleaved, plain UDP, SRTP and HTTPS served); CRA_STRICT with a transport stream and with a KLV track inside TLS and SRTP, plain UDP refused by the client policy; two streams on one shared port and an unknown path; the push leg: RTP + KLV SSRC with HEVC and JPEG, ONVIF documents on PT 98 without an SDP, raw MPEG-TS with HEVC (synchronous and asynchronous), the stream type synonyms (mpegts, mpegts-rtp, mpegts-rtp-klv), a raw transport stream received on an rtp:// URL, token-bucket pacing, payload sizes 576 and 9000, the -source filter accepting and rejecting the sender, and RTP + KLV, raw MPEG-TS and JPEG pushed to a multicast group (skipped without a multicast route).
live 420 MServer changes under a connected client: the KLV track added live (the client is not disturbed and gets no unit it never set up; after RESTART every frame carries its unit) and removed (video goes on, the units stop); the stream disabled for 7 s (nothing published, the no-media watchdog reconnects) and enabled again (resumes with a key frame); the password rotated (the old client’s session dropped and its reconnects refused, a client with the new password plays, the old password refused); the stream renamed (the old path answers 404, the new path plays); live codec changes H.264 -> H.265 on an MPEG-TS leg with JPEG refused there, and H.264 -> JPEG -> H.265 over SRTP / TCP, without a reconnect; 7 clients at once on one KLV stream (UDP, TCP, multicast, HTTP tunnel, TLS interleaved, HTTPS tunnel, SRTP), each byte-exact with its metadata and the server holding every session; a 66 s session beyond MServer’s 60 s session timeout (a TCP client judged byte-exact with no loss, a UDP client alongside, the server holding both at 63 s).

Every check prints one line: ok : <group>: <check>, FAIL : <group>: <check> (<detail>), skip : ... or info : .... The run ends with a summary table. Every group has a time budget (Table 27). A group that exceeds its budget aborts the process (watchdog, exit code 125), so a hang can never block a CI run. The environment variable VSRTP_BUDGET_SCALE (1 - 100) multiplies every budget, for sanitizer builds that run the same checks several times slower. The exit code is 0 if every check passed, else the number of failures (at most 125).

With 1.4.1, a full run of all 36 groups took about 30 minutes (1805 s), of which the matrix group took about 4.8 minutes, the interop group about 3.4 minutes, the live and decode groups about 2.4 and 2.2 minutes. It executed 7241 checks against MServer 2.1.4 with no failure, 5 of them skipped with a reason: 3 because that host had no right to SCHED_FIFO and a small net.core.rmem_max, and 2 by design (a timestamp wrap that a test cannot wait 13 hours for, and a fix in MServerTest that the test application does not use). The host had GStreamer 1.20 with its plugins and gst-rtsp-server, so the GStreamer cases ran. (With 1.4.0: 7168 checks against MServer 2.1.3, 1776 s. With 1.3.1, on a host without the GStreamer plugins and gst-rtsp-server: 4521 checks, 12 skipped, 1268 s. With 1.3.0: 4717 checks, 2 skipped, 1257 s and 1254 s.) (When the test’s own sender paused inside a frame on a loaded host, the lowlat group repeats a crafted run, which adds one check.) A group run alone may show a few more checks: its clip-encoding checks, which a full run does only once. For example, stats shows 38 instead of 37, and decode 298 instead of 294. The groups print the measured values of Measured results as “info” lines. To run the application, use these commands:

cd build/bin
./VSourceStreamTest -list
./VSourceStreamTest
./VSourceStreamTest stats

Output of ./VSourceStreamTest stats (shortened):

VSourceStream v1.4.1 test suite
=== group stats ===
ok   : stats: getVersion() 1.4.1 is Major.Minor.Patch
ok   : stats: unopened source: IS_OPEN 0, getFrame FALSE
ok   : stats: initVSource with an invalid source string returns FALSE
ok   : stats: getTelemetry without a frame: FALSE (reset)
ok   : stats: encode clip H264:640:480 (37 ms)
ok   : stats: stats: start reference server
ok   : stats: initVSource
...
ok   : stats: decodeAndExecuteCommand: SET_PARAM of size 7 refused
ok   : stats: decodeAndExecuteCommand: empty refused
ok   : stats: decodeAndExecuteCommand: encoded SET_PARAM CUSTOM_2 = 48 applied
ok   : stats: decodeAndExecuteCommand: no restart, frames keep flowing
ok   : stats: closeVSource: IS_OPEN 0

==================== summary ====================
group           checks  failures  skipped   seconds
stats               38         0        0       5.4
TOTAL               38         0        0       5.4
RESULT: PASS

In interactive mode (./VSourceStreamTest -i), the application asks for a source string and a log level, and opens the source with initVSource(…). It then prints one line per frame: frame ID, geometry, fourcc, size, cycle time, fps, age (CUSTOM_1), lost frames (CUSTOM_3), and the number and total size of the metadata units of the frame’s telemetry (getTelemetry(…)). Ctrl+C stops it. Example against ./VSourceStreamTest -server rtsp-klv, started in another console (log lines omitted):

========================================
VSourceStream v1.4.1 test
========================================
Set source (e.g. rtsp://127.0.0.1:8554/live -tcp): rtsp://127.0.0.1:18500/live -tcp
Set log level (0 - disable, 1 - file, 2 - console, 3 - both): 2
Video source init!
fourcc H264 640x480 fps 25
[1] 640x480 H264 56256 bytes | 5.690 ms | 175.75 fps | age 5995 us | lost 0 | metadata 1 units 44 bytes
[2] 640x480 H264 7471 bytes | 34.100 ms | 29.33 fps | age 67 us | lost 0 | metadata 1 units 44 bytes
[3] 640x480 H264 7231 bytes | 40.503 ms | 24.69 fps | age 111 us | lost 0 | metadata 1 units 44 bytes
[4] 640x480 H264 7189 bytes | 39.530 ms | 25.30 fps | age 198 us | lost 0 | metadata 1 units 44 bytes

The fps column is computed from the cycle time of the frame. In reference server mode (./VSourceStreamTest -server <variant>), the application only runs MServer with a test clip: RTSP port 18500, RTSPS port 18501, push streams to 127.0.0.1 port 18520. It prints ready-to-use source strings and stops on Enter. ./VSourceStreamTest -server list prints the variants:

Server variants (VSourceStreamTest -server <variant>):
  rtsp-h264 - RTSP, H.264 pass-through clip
  rtsp-h265 - RTSP, H.265 pass-through clip
  rtsp-jpeg - RTSP, JPEG pass-through clip
  rtsp-klv - RTSP, H.264 + RFC 6597 KLV track
  rtsp-onvif - RTSP, H.264 + ONVIF metadata track
  rtsp-auth - RTSP with digest authentication
  rtsps - RTSP + RTSPS (self-signed) + SRTP presentation
  multicast - RTSP with multicast pool 239.50.0.0/16
  ts-sync - RTSP, MPEG-TS in RTP, sync KLV
  ts-async - RTSP, MPEG-TS in RTP, async KLV
  push-rtp - push RTP to 127.0.0.1
  push-rtp-klv - push RTP + KLV to 127.0.0.1
  push-ts-rtp - push MPEG-TS in RTP (sync KLV) to 127.0.0.1
  push-ts-udp - push raw MPEG-TS (sync KLV) to 127.0.0.1
  raw-h264 - RTSP, raw-encoder mode (key frame on PLAY)
  cra-strict - RTSPS only, CRA_STRICT, test CA certificate

Table of contents