RapidPixel SDK
v2.0.0
Overview
RapidPixel SDK is a complete set of C++17 libraries and sample applications for building camera software. The libraries cover the whole chain of such software, from video capture to streaming and device control: video capture, filtering, stabilization, tracking and object detection, encoding and decoding, streaming, control of cameras and lenses, communication, and service functions such as logging. At the same time, the SDK is general-purpose software and is not tied to any sector or application: the libraries provide generic algorithms, data structures, interfaces, protocol implementations and device controllers based on open public data, and the developer of a program decides what the program does with them. Compared with open-source libraries such as OpenCV, FFmpeg or GStreamer, the RapidPixel SDK offers building blocks of much larger scale that combine multiple algorithms to solve a specific problem (object detection, video server etc.).
The CMake project of the SDK builds available modules into one static library on Linux or with MSVC on Windows, and supports source integration, a portable SDK bundle and OS installation. Modules can also be used through their own projects. Availability, acceleration and achievable frame rate depend on the operating system, hardware, input size and installed dependencies; a hardware-specific backend does not work on every CPU. Start with the build and installation guide and check the documentation of the modules selected for your application. The image below shows the functionality covered by the RapidPixel SDK (the parts of the SDK are green):
Functions
RapidPixel SDK includes libraries:
- Video Processing Pipeline
- Video Tracking
- Video Stabilization
- Object Detection (motion detection, change detection, neural networks)
- Video Streaming (RTSP, RTP, MPEG-TS, WebRTC, HLS)
- Video Filtering (digital zoom, defog, color manipulation etc.)
- Video Capture (various interfaces)
- Video Encoding and Decoding
- Communication
- Laser Range Finder Controllers
- Lens Controllers
- Camera Controllers
- Block Camera Controllers (lens and camera combined)
- Service libraries (files, buffers, video output, joysticks etc.)
- Examples and Templates (various combinations of libraries for many cases)
- Tutorials
How to start
Using libraries with external dependencies
A program which uses the RapidPixel SDK links one library of the SDK (libRapidPixel.a on Linux, RapidPixel.lib on Windows), which contains all libraries of the SDK build. The program links an external library only if it uses a library of the SDK which needs it: a program which uses only libraries without external dependencies links nothing else. The build of the SDK writes the exact list of the external libraries of every library of the build into lib/<configuration>/RapidPixelDependencies.md, and the CMake targets RapidPixel::<library> link them automatically, see Getting started.
External dependencies
Most libraries of the RapidPixel SDK need no other library: a program which uses only them links only the library of the SDK. The other libraries use a third-party library, a library of the operating system or the OpenMP runtime of the compiler. An optional library is used only if it is installed when the SDK is built, so the exact list depends on the build: the build writes it into lib/<configuration>/RapidPixelDependencies.md (see Using libraries with external dependencies).
Third-party libraries:
The third-party libraries are not a part of the SDK: the repositories of the SDK contain no code of them. They are installed on the build computer (ONNX Runtime is downloaded as a package, see DnnOnnxDetector), the libraries of the SDK include their header files and the programs link them. The column License gives the license of each library. The MIT, BSD and Apache 2.0 licenses allow the use in a proprietary program if the copyright and license notices of the library are kept. A program linked with an LGPL library must allow the user to replace this library (for example with dynamic linking). A build of FFmpeg with GPL components (for example the packages of Ubuntu) is licensed under the GPL. The libraries of hardware manufacturers are used under the license of the manufacturer.
| Third-party library | Required by | Used if installed by | License |
|---|---|---|---|
| ONNX Runtime (package downloaded from the releases page) | DnnOnnxDetector | - | MIT. The official packages contain third-party components (file ThirdPartyNotices.txt of the package) and collect telemetry, which DnnOnnxDetector switches off |
| OpenVINO execution provider plugin of ONNX Runtime | - | DnnOnnxDetector (loaded at run time) | MIT, contains the OpenVINO runtime (Apache 2.0) |
| OpenVINO™ | DnnOpenVinoDetector | - | Apache 2.0 |
| OpenCV (version >= 4.5) | Fmd | - | Apache 2.0 |
| FFmpeg (libavcodec, libavformat, libavutil, libavfilter, libswscale) | VCodecLibav | - | LGPL 2.1 or later, GPL 2 or later if built with GPL components, not redistributable if built with nonfree components. The components of the build have their own licenses, see the note below |
| GStreamer (with GLib and GObject) | VCodecGst | - | LGPL 2.1 or later. The plugins and the libraries which they use have their own licenses, see the note below |
| Intel Libva (VAAPI) (libva, libva-drm) | VCodecLibva | - | MIT |
| libcamera (libcamera, libcamera-base) | VSourceLibCamera | - | LGPL 2.1 or later |
| OpenSSL 3 (libssl, libcrypto) | MServer (Linux), OConnector, License (Linux, through CryptoProvider: without OpenSSL the library does not work) | CryptoProvider (Linux; on Windows if its OpenSSL backend is selected) and the libraries which use it: SnmpServer, VSourceStream (also the TLS transport of RTSPS on Linux and Windows) | Apache 2.0 |
| NVIDIA Jetson Multimedia API and VPI | VCodecJetPack | Denoiser and Fmd (VPI), VCodecV4L2 (Jetson extensions of the Jetson Multimedia API) | License of NVIDIA JetPack |
| Libraries of NXP i.MX (Hantro H1 encoder, G2D) | VCodecImxVpu | ImageResizer (G2D) and the libraries which use it: DigitalZoom, DnnOnnxDetector, DnnOpenVinoDetector, MServer | License of the NXP BSP |
Licenses of the FFmpeg and GStreamer components. The license of FFmpeg and GStreamer in the table is the license of their core libraries only. The license of a particular FFmpeg build depends on the components it is built with: the encoders libx264 and libx265 and the other components enabled with --enable-gpl are GPL, components enabled with --enable-nonfree (for example libfdk-aac) make the build not redistributable, and the external libraries linked into the build (codecs, hardware SDKs) have their own licenses. GStreamer loads its elements from plugins at run time, and every plugin and the library which it wraps has its own license (for example gst-plugins-ugly contains GPL elements such as x264enc, gst-plugins-bad and vendor plugins contain elements under other licenses). The SDK does not choose the FFmpeg build or the GStreamer plugins: the user of VCodecLibav and VCodecGst has to check and control the licenses of the FFmpeg build, of the GStreamer plugins used in the pipelines and of their components, and the patent licenses of the codecs (for example H264 and HEVC), for every product and distribution. See FFmpeg license and legal considerations and GStreamer licensing.
OpenCV is needed only by one library of the SDK: Fmd (the library and its example, demo and benchmark applications). The other libraries, their test applications, examples and benchmarks, the example applications and the tutorials of the SDK do not use OpenCV: they generate or read the video frames and draw on them with the libraries of the SDK (for example SimpleUI, FormatConverter and VSourceFile).
Libraries of the operating system and of the compiler:
| Library | Used by | License |
|---|---|---|
| V4L2 API (Linux kernel interface, no library to link) | VSourceV4L2, VCodecV4L2 (also links libdl), VOutputV4L2 | - (system calls of the kernel) |
| Microsoft Media Foundation (WMF) (Windows) | VSourceWmf, VCodecWmf | Part of Windows |
X11 (libX11, libXext) and libsystemd (Linux, used if installed) | SimpleUI (window) and PtzEmulator (through SimpleUI), SimpleFileDialog (dialog of the desktop) | X11: MIT; libsystemd: LGPL 2.1 or later |
| EGL and OpenGL ES (used if installed) | VStabiliserCv (acceleration) | Library of the graphics driver |
OpenMP runtime of the compiler (libgomp with GCC) | Ged, Gmd, CvTracker; if the compiler supports OpenMP: FormatConverter, ImageResizer, Dehazer, Denoiser, MotionMagnificator, VStabiliserCv and the libraries which use FormatConverter or ImageResizer: DigitalZoom, DnnOnnxDetector, DnnOpenVinoDetector, MServer, VideoRecorderMp4, VPipeline, VSourceFile, VSourceStream | Runtime of the compiler (libgomp: GPL 3 with the GCC Runtime Library Exception) |
Windows socket libraries (ws2_32, iphlpapi; Windows only) | TcpClient, TcpServer, UdpSocket, UdpDataChannel, License, MServer, RtpPusher, SnmpServer, VSourceStream and the libraries which use them | Part of Windows |
Windows CNG (Cryptography API: Next Generation, bcrypt; Windows only) | CryptoProvider (default backend on Windows) and the libraries which use it: License, MServer, SnmpServer, VSourceStream | Part of Windows |
Other libraries of Windows: winmm (timer), ole32 (COM); Windows only | MServer and RtpPusher (winmm), SimpleFileDialog (ole32) | Part of Windows |
On Linux the following libraries (65 in a full build) need no other library. On Windows the network libraries among them need the Windows socket libraries of the table above (RtpPusher also winmm):
- Video Tracking: VTracker.
- Video Stabilization: VStabiliser, PanTiltSynchronizer.
- Object Detection: ObjectDetector, MvExtractorH264.
- Video Streaming: VStreamer, RtpPusher, NetworkHealthChecker.
- Video Filtering: VFilter, ImageFlip, ColorProcessor, BadPixelProcessor.
- Video Capture: VSource, VSourceV4L2.
- Video Encoding and Decoding: VCodec, VDecoderH264, VDecoderH265, VDecoderJPEG, VEncoderJPEG.
- Communication: SerialPort, CrsfParser, SbusParser, TcpClient, TcpServer, UdpSocket, UdpDataChannel, NtpUdpChannel, PelcoDParser, BoberManager, BoberClient, KlvEncoder.
- Laser Range Finder Controllers: Lrf, LrfUltisenseParser, LrfUltisense.
- Lens Controllers: Lens, AFEngine, Flc, Olc, Rpl, Slc, SunnyLens, Ylc, Klc.
- Camera Controllers: Camera, FlirBosonCamera, HitachiCamera, IRayFtCamera, KowaCamera, OwlCamera.
- Block Camera Controllers: FujiSxCamera, G5Camera, IRayBlockCamera, PanTilt, ViscaParser, ViscaCamera.
- Service libraries: Frame, ConfigReader, Logger, VOverlay, ChildProcess, FileFinderByName, Joystick, VOutputFb, VOutputV4L2, VOutputPipe.
SDK structure
RapidPixel SDK is a CMake project which includes a set of libraries and examples. Each library can be used separately, together with the libraries it depends on (they are listed in its documentation). Each library includes detailed documentation, examples and test applications. No external dependencies in the tables below means that the library uses no third-party library; the libraries of the operating system and the OpenMP runtime of the compiler which some libraries use are listed in External dependencies. SDK structure by categories:
Video Processing Pipeline
| Library | Description |
|---|---|
| VPipeline (Linux and Windows) | Provides a fully functional video processing pipeline with interfaces to connect different video filters, video trackers, object detectors, and video stabilization modules. The motivation of the VPipeline library is to provide a simple interface to a complex video processing pipeline as well as camera and lens control regardless of hardware. The library serves two main goals: provides a video processing pipeline and provides a remote control interface for the pipeline (encoding/decoding commands and telemetry). No external dependencies. |
Video Tracking
| Library | Description |
|---|---|
| VTracker (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different video trackers. The VTracker interface class does nothing; it just provides an interface, defines data structures, and provides methods to encode/decode commands and encode/decode params. No external dependencies. |
| CvTracker (Linux and Windows) | Video tracker implementation. Provides tracking of objects in video with additional functions and modes. Compatible with low-power processors. No external dependencies. |
Video Stabilization
| Library | Description |
|---|---|
| VStabiliser (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different video stabilization algorithms. The VStabiliser interface class does nothing; it just provides an interface, defines data structures, and provides methods to encode/decode commands and encode/decode params. No external dependencies. |
| VStabiliserCv (Linux and Windows) | Video stabilizer implementation. Provides 2D and 3D digital video stabilization (horizontal offset, vertical offset, and rotation). Optional acceleration: OpenGL ES / EGL output warp with CPU fallback. No external dependencies (the EGL and OpenGL ES libraries of the graphics driver are used if they are installed). |
| PanTiltSynchronizer (Linux and Windows) | Measures the latency between a video stream and a stream of Pan/Tilt measurements with an ID, from any angle source: compares the motion of the picture with the change of the pan and the tilt, and reports the latency and the ID of the measurement whose arrival time, corrected by the measured latency, is nearest to the arrival time of the last frame. Uses the Frame library. No external dependencies. |
Object Detection
| Library | Description |
|---|---|
| ObjectDetector (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different object detectors (motion detectors, change detectors, neural networks, etc.). The ObjectDetector interface class does nothing; it just provides an interface, defines data structures, and provides methods to encode/decode commands and encode/decode params. No external dependencies. |
| Gmd (Linux and Windows) | Motion object detector implementation. Provides automatic detection of moving objects in videos. The library processes video of different camera types (daylight, SWIR, MWIR and LWIR) and detects small-size and low-contrast moving objects against complex backgrounds. No external dependencies. |
| Ged (Linux and Windows) | Video change detector implementation. Provides automatic detection of subtle changes in videos. The library processes video of different camera types (daylight, SWIR, MWIR and LWIR) and detects small-size and low-contrast objects against complex backgrounds. No external dependencies. |
| DnnOpenVinoDetector (Linux and Windows) | DNN object detector implementation. Provides automatic detection of objects in videos with neural networks based on the OpenVINO™ runtime. Tested with the Ultralytics YOLOv8 neural network model. External dependencies: OpenVINO™ (Apache 2.0 license). |
| DnnOnnxDetector (Linux and Windows) | DNN object detector implementation. Provides automatic detection of objects in videos with neural networks based on the ONNX Runtime library. Tested with the Ultralytics YOLOv8 neural network model. The network runs on the CPU, or on Intel CPUs and GPUs with the OpenVINO execution provider plugin of ONNX Runtime. The library switches off the telemetry of ONNX Runtime. External dependencies: ONNX Runtime (MIT license, not included in the SDK: the package is downloaded separately), OpenVINO execution provider plugin of ONNX Runtime (optional, MIT license, loaded at run time). |
| MvExtractorH264 (Linux and Windows) | The library for motion vector (optical flow) extraction from H264 compressed video. No external dependencies. |
| Fmd (Linux and Windows) | Motion object detector implementation. Provides automatic detection of moving objects on videos captured by a moving camera. The library processes video of different camera types (daylight, SWIR, MWIR and LWIR) and detects small-size objects against complex moving backgrounds. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license); optional: CUDA optical flow if OpenCV is built with CUDA, NVIDIA VPI 3 (linked if installed, proprietary license). |
Video Streaming
| Library | Description |
|---|---|
| VStreamer (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different video streamers. The VStreamer interface class does nothing; it just provides an interface, defines data structures, and provides methods to encode/decode commands and encode/decode params. No external dependencies. |
| RtpPusher (Linux and Windows) | RTP (H264, H265 and JPEG) and MPEG-TS video streamer compatible with standard RTP / MPEG-TS clients (VLC, FFmpeg, GStreamer, etc.). No external dependencies. |
| MServer (Linux and Windows) | RTSP, RTSPS, RTSP over HTTP / HTTPS, RTP, SRTP, MPEG-TS, WebRTC (WHEP) and HLS video server with KLV support. Depends on CryptoProvider. External dependencies: OpenSSL 3 on Linux (linked directly and through CryptoProvider, required, Apache 2.0 license); Windows CNG on Windows (through CryptoProvider, part of Windows). On Windows the functions which need TLS or DTLS (RTSPS, HTTPS, SRTP, WebRTC) are not available. |
| NetworkHealthChecker (Linux and Windows) | The library for estimating network load. Used by video streamers for automatic bandwidth control. No external dependencies. |
Video Filtering
| Library | Description |
|---|---|
| VFilter (Linux and Windows) | Provides an interface as well as defines data structures and rules for various video filter implementations. The VFilter interface represents a video filter module that has video frames as input and output. The VFilter interface class does nothing; it just provides an interface and methods to encode/decode commands and encode/decode params. No external dependencies. |
| MotionMagnificator (Linux and Windows) | Video filter implementation. Computes the amplitude of the motion of every pixel of digital video and gives it as a grayscale image or shows it in the intensity channel of the frames, so that motion which is not visible to the naked eye becomes visible. No external dependencies. |
| Dehazer (Linux and Windows) | Video filter implementation. Provides video dehazing (defog): improves the local contrast of video frames to reduce the effect of haze and fog. No external dependencies. |
| DigitalZoom (Linux and Windows) | Video filter implementation. Provides digital zoom functions with bicubic interpolation. No external dependencies. |
| ImageFlip (Linux and Windows) | Video filter implementation. Provides simple image flip function: up-down, left-right, both. No external dependencies. |
| Denoiser (Linux and Windows) | Video filter implementation. Provides noise cancellation based on a bilateral filter. External dependencies: optional OpenMP runtime of the compiler (CPU parallelism); optional NVIDIA VPI backend on Linux (linked if installed, proprietary license). |
| ColorProcessor (Linux and Windows) | Color filter: palette, contrast, brightness, saturation and white balance. No external dependencies. |
| BadPixelProcessor (Linux and Windows) | Bad pixel detection and filtering for LWIR, MWIR and SWIR thermal cameras. No external dependencies. |
Video Capture
| Library | Description |
|---|---|
| VSource (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different video source classes (video capture classes). The VSource interface class does nothing; it just provides an interface and methods to encode/decode commands and encode/decode params. No external dependencies. |
| VSourceV4L2 (Linux only) | Video source implementation. Provides video capture and video source (V4L2 compatible video sources) control function based on V4L2 API. External dependencies: V4L2 API (default subsystem in Linux). |
| VSourceLibCamera (Linux only) | Video source implementation. Provides video capture and video source control function based on libcamera. External dependencies: libcamera (libcamera and libcamera-base, linked, LGPL 2.1 or later license). |
| VSourceFile (Linux and Windows) | Video source implementation. Provides video frame capture from H264, HEVC, JPEG, MPEG-4 Part 2 (MP4V) and uncompressed BGR24 video files: raw elementary streams (.h264, *.hevc, *.mjpeg) as well as MP4 (.mp4, .mov, *.m4v) and AVI (.avi) containers. H264, HEVC and JPEG frames are decoded to the requested pixel format; MP4V frames are delivered encoded. No external dependencies. |
| VSourceWmf (Windows only) | Video source implementation. Provides video capture and video source control based on Microsoft Media Foundation (WMF) on Windows. External dependencies: Microsoft Media Foundation (WMF). |
| VSourceStream (Linux and Windows) | Video source implementation. Provides RTSP, RTSPS, RTP, SRTP and MPEG-TS video capture with software decoding and minimum latency. Depends on CryptoProvider. External dependencies: OpenSSL 3 (optional, Apache 2.0 license): on Linux the cryptographic functions of CryptoProvider (without OpenSSL its built-in implementation is used), on Linux and Windows the TLS transport of RTSPS (without OpenSSL RTSPS is not available); Windows CNG on Windows (through CryptoProvider, part of Windows). |
Video Encoding and Decoding
| Library | Description |
|---|---|
| VCodec (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different video codecs (video encoding and decoding). The VCodec interface class doesn’t do anything; it just provides an interface. No external dependencies. |
| VCodecV4L2 (Linux only) | Video codec implementation. Provides hardware-accelerated video encoding/decoding (H264, HEVC and JPEG codecs) based on V4L2 API on Linux only. External dependencies: V4L2 API (default subsystem in Linux); on NVIDIA Jetson the headers of the Jetson Multimedia API are used if they are installed and the NvBufSurface libraries of the Jetson BSP are loaded at run time. |
| VCodecJetPack (NVIDIA Linux only) | Video codec implementation. Provides hardware-accelerated video encoding and decoding (H264 and HEVC codecs) for NVIDIA Jetson platforms through the V4L2 interface of the Jetson Multimedia API. External dependencies: header v4l2_nv_extensions.h of the Jetson Multimedia API for the build; libv4l2 with the NVIDIA plugins of the Jetson BSP, loaded at run time. |
| VCodecLibav (Linux and Windows) | Video codec implementation. Provides hardware and software-accelerated video encoding/decoding functions (H264, HEVC (H265) and JPEG codecs) for Linux and Windows OS based on FFmpeg. External dependencies: FFmpeg (linked; LGPL 2.1 or later, GPL 2 or later if FFmpeg is built with GPL components, not redistributable if built with --enable-nonfree). The components of FFmpeg (for example the encoders libx264 and libx265, GPL) have their own licenses: the user of the library has to check and control the licenses of the FFmpeg build and of its components, see FFmpeg license. |
| VCodecGst (Linux only) | Video codec implementation. Provides video encoding and decoding functions for H264, HEVC (H265), and JPEG codecs for Linux OS based on GStreamer. External dependencies: GStreamer (linked, LGPL 2.1 or later license). The plugins of GStreamer and the libraries which they use (for example x264enc of gst-plugins-ugly, GPL) have their own licenses: the user of the library has to check and control the licenses of the plugins used in the pipelines, see GStreamer licensing. |
| VCodecLibva (Linux only) | Video codec implementation. Provides hardware video encoding and decoding for H264, HEVC (H265), and JPEG codecs on Linux OS based on Intel Libva (VAAPI). External dependencies: Intel Libva (VAAPI) (linked, MIT license). |
| VCodecImxVpu (Linux only) | Video codec implementation. Provides H264 video encoding for the IMX8 platform based on the H1 hardware encoder. External dependencies: Hantro H1 encoder library and headers (libhantro_h1, h264encapi.h) of the NXP BSP. |
| VCodecWmf (Windows only) | Video codec implementation. Provides hardware-accelerated and software video encoding and decoding for H264, HEVC (H.265), and JPEG formats based on Microsoft Media Foundation (WMF) and Windows Imaging Component (WIC) on Windows 11 and later. External dependencies: Microsoft Media Foundation (WMF). |
| VDecoderH264 (Linux and Windows) | Video decoder implementation. Provides software video decoding for H264 format. No external dependencies. |
| VDecoderH265 (Linux and Windows) | Video decoder implementation. Provides software video decoding for H265 format. No external dependencies. |
| VDecoderJPEG (Linux and Windows) | Video decoder implementation. Provides software video decoding for JPEG format. No external dependencies. |
| VEncoderJPEG (Linux and Windows) | Video encoder implementation. Provides software video encoding for JPEG format. No external dependencies. |
Communication
| Library | Description |
|---|---|
| UdpDataChannel (Linux and Windows) | Provides reliable point-to-point communication between two applications based on UDP. The library includes two primary classes: UdpDataClient (connects to server) and UdpDataServer (waits for connection from client). Both classes provide two-way communication. No external dependencies. |
| NtpUdpChannel (Linux and Windows) | Provides a mechanism to simplify sending and receiving data over the UDP transport layer via low bandwidth networks. The library streams data taking into account channel bandwidth and controls data loss. No external dependencies. |
| SerialPort (Linux and Windows) | Provides a simple interface to work with serial ports. No external dependencies. |
| TcpClient (Linux and Windows) | Provides methods to work with TCP socket as client (connect, close, send data and receive data). No external dependencies. |
| TcpServer (Linux and Windows) | Provides methods to work with TCP socket as server (wait for connection, close, send data and receive data). No external dependencies. |
| UdpSocket (Linux and Windows) | Provides methods to work with UDP ports (open, close, send data and receive data). No external dependencies. |
| SbusParser (Linux and Windows) | Provides methods for encoding and decoding SBUS protocol data. No external dependencies. |
| CrsfParser (Linux and Windows) | Provides a mechanism for decoding and encoding packets of the CRSF (Crossfire) radio control protocol. The library is used for CRSF packets translation and extracts only CRSF_FRAMETYPE_RC_CHANNELS_PACKED packets. For other packet types the library just detects them. No external dependencies. |
| PelcoDParser (Linux and Windows) | Provides methods for encoding and decoding PelcoD responses and commands. No external dependencies. |
| BoberManager (Linux and Windows) | UDP message manager C++ library designed to handle communication between multiple clients over a network. It uses UDP packets and string-based protocol. No external dependencies. |
| BoberClient (Linux and Windows) | Communication client C++ library for user applications, enabling interaction with any other service through BoberManager. It uses UDP packets and string-based protocol. No external dependencies. |
| KlvEncoder (Linux and Windows) | The library to encode KLV metadata for video streaming. No external dependencies. |
| SnmpServer (Linux and Windows) | SNMP agent library: SNMPv2c and SNMPv3 (USM and VACM) command responder and notification originator (traps and informs). Scalar objects and tables are registered by the application with values or handlers; the library can generate the SMIv2 MIB module of the registered objects. Depends on CryptoProvider (SNMPv3 only). External dependencies: OpenSSL 3 on Linux (through CryptoProvider, optional, with FIPS provider support, Apache 2.0 license); Windows CNG on Windows (through CryptoProvider, part of Windows). |
| OConnector (Linux only) | Library for building ONVIF® services: implements the ONVIF® Profile S, T and M specifications, WS-Discovery and the HTTP / HTTPS transport and hands every client request to the application as a typed callback. Video streams are not carried: the library returns the addresses of the streams. Depends on CryptoProvider. External dependencies: OpenSSL 3 (linked directly and through CryptoProvider, required, Apache 2.0 license). |
Laser Range Finder Controllers
| Library | Description |
|---|---|
| Lrf (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different laser range finders. The Lrf interface class does nothing; it just provides an interface and methods to encode/decode commands and encode/decode params. No external dependencies. |
| LrfUltisenseParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for Safran Laser Range Finders. No external dependencies. |
| LrfUltisense (Linux and Windows) | Software controller for Safran Laser Range Finders. No external dependencies. |
Lens Controllers
| Library | Description |
|---|---|
| Lens (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different lens controllers. The Lens interface class does nothing; it just provides an interface and methods to encode/decode commands and encode/decode params. No external dependencies. |
| AFEngine (Linux and Windows) | Provides auto focus functions for different lens controllers. Supports ROI, auto ROI detection and other parameters. Provides a hardware-agnostic interface to be implemented in different lens controllers. No external dependencies. |
| Slc (Linux and Windows) | Software controller for Stingray SWIR lenses. Includes auto-focus functions based on the AFEngine library. No external dependencies. |
| Olc (Linux and Windows) | Software controller for Ophir infrared lenses. Includes auto-focus functions based on the AFEngine library. No external dependencies. |
| Flc (Linux and Windows) | Software controller for Fujinon lenses. No external dependencies. |
| Ylc (Linux and Windows) | Software controller for Yamano lenses. Includes auto-focus functions based on the AFEngine library. No external dependencies. |
| SunnyLens (Linux and Windows) | Software controller for Sunny continuous zoom lenses. Includes auto-focus functions based on the AFEngine library. No external dependencies. |
| Rpl (Linux and Windows) | Software controller for RP thermal / SWIR lenses. Includes auto-focus functions based on the AFEngine library. No external dependencies. |
| Klc (Linux and Windows) | Software controller for KOWA lenses. No external dependencies. |
Camera Controllers
| Library | Description |
|---|---|
| Camera (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different camera controllers. The Camera interface class does nothing; it just provides an interface and methods to encode/decode commands and encode/decode params. No external dependencies. |
| OwlCamera (Linux and Windows) | Software controller for Owl 640 SWIR cameras. No external dependencies. |
| KowaCamera (Linux and Windows) | Software controller for KOWA cameras. No external dependencies. |
| HitachiCamera (Linux and Windows) | Software controller for Hitachi HD-SDI Series cameras. No external dependencies. |
| IRayFtCamera (Linux and Windows) | Software controller for IRay FT thermal cameras. No external dependencies. |
| FlirBosonCamera (Linux only) | Software controller for FLIR Boson cameras. No external dependencies. |
Block Camera Controllers
| Library | Description |
|---|---|
| FujiSxCamera (Linux and Windows) | Software controller for Fujinon Sx series cameras (block cameras which combine lens and camera (image sensor) and provide one control interface). No external dependencies. |
| ViscaCamera (Linux and Windows) | Software controller for VISCA Protocol compatible cameras. Includes camera, lens and pan-tilt control interfaces. No external dependencies. |
| ViscaParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for VISCA Protocol compatible cameras. No external dependencies. |
| PanTilt (Linux and Windows) | Provides a standard interface as well as defines data structures and rules for different pan-tilt controllers. The PanTilt interface class does nothing; it just provides an interface and methods to encode/decode commands and encode/decode params. No external dependencies. |
| G5Camera (Linux and Windows) | Software controller for G5 Infrared thermal cameras (block cameras which combine lens and camera (image sensor) and provide one control interface). No external dependencies. |
| IRayBlockCamera (Linux and Windows) | Software controller for InfiRay FT block cameras (block cameras which combine lens and camera (image sensor) and provide one control interface). No external dependencies. |
Service libraries
| Library | Description |
|---|---|
| FileFinderByName (Linux and Windows) | Provides methods to find files by name part(s). It finds data files or device files. No external dependencies. |
| FormatConverter (Linux and Windows) | Provides methods to convert images between pixel formats. FormatConverter supports conversion between all uncompressed pixel formats listed in the Fourcc enum (RGB24, BGR24, YUYV, UYVY, GRAY, YUV24, NV12, NV21, YU12, YV12) of the Frame library. No external dependencies. |
| Logger (Linux and Windows) | Provides logging functions: printing in terminal and/or printing in file. No external dependencies. |
| ChildProcess (Linux only) | Provides methods to run / stop an external process from your application on Linux. No external dependencies. |
| VOutputV4L2 (Linux only) | Provides an interface to write video frames to V4L2 video devices (for example, V4L2 loopback devices) in Linux. External dependencies: V4L2 API (default subsystem in Linux). |
| VOutputFb (Linux only) | Provides an interface to write video frames to the Linux framebuffer. No external dependencies. |
| VOutputPipe (Linux only) | Provides an interface to write video frames to named pipe (FIFO) in Linux OS. Third-party applications such as ffmpeg and GStreamer read the pipe as an ordinary raw video stream, so the user’s application can be a video source for them. No external dependencies. |
| Joystick (Linux and Windows) | Provides a simple interface to work with various types of joysticks. No external dependencies. |
| SimpleFileDialog (Linux and Windows) | Provides a simple dialog to choose a file in Windows and Linux. No external dependencies (on Linux libsystemd is used for the file dialog of the desktop if it is installed). |
| SimpleUI (Linux and Windows) | Provides functions to draw lines, rectangles, circles, triangles and text on video frames (BGR24, RGB24 and YUV24) and to show the frames in a window with sliders, keyboard and mouse input. No external dependencies (on Linux the window uses X11 if it is installed, otherwise the window is headless). |
| VideoRecorderMp4 (Linux and Windows) | Provides methods for recording H264, H265 (HEVC) and MJPEG video to MP4 files with control over file and folder size. Includes software JPEG encoder. No external dependencies. |
| Frame (Linux and Windows) | Provides video frame data structures and pixel format descriptions. No external dependencies. |
| ConfigReader (Linux and Windows) | Provides methods to read/write config files or strings in JSON format. The library has a simple interface to read application configuration parameters from file/string or write configuration parameters to file/string. No external dependencies. |
| License (Linux and Windows) | Provides application licensing with ECDSA P-256 signatures verified using a compiled-in public key. Licenses from the previous AES-based implementation must be reissued. Depends on CryptoProvider. External dependencies: OpenSSL 3 on Linux (through CryptoProvider, required: without OpenSSL the library does not work; optional FIPS provider, Apache 2.0 license); Windows CNG on Windows (through CryptoProvider, part of Windows). |
| VOverlay (Linux and Windows) | Provides a standard interface for overlaying information on video for different implementations. This is the standard for overlaying information in video streamer interfaces. No external dependencies. |
| ImageResizer (Linux and Windows) | Implements video resizing with bicubic interpolation: SIMD-optimized software implementation, or the G2D hardware of NXP i.MX processors. External dependencies: optional NXP G2D library (libg2d of the NXP i.MX BSP, linked if installed, license of the NXP BSP). |
| CryptoProvider (Linux and Windows) | C++17 cryptographic library that provides a unified, OpenSSL-like interface across Windows and Linux platforms with automatic backend selection. Used by License, MServer, OConnector, SnmpServer and VSourceStream. External dependencies: Windows CNG (Cryptography API: Next Generation, bcrypt, part of Windows) on Windows; OpenSSL 3 on Linux (optional, Apache 2.0 license: without OpenSSL the built-in implementation without ECDSA is used; the OpenSSL backend can also be selected on Windows). |
| PtzEmulator (Linux and Windows) | Software emulator of a PTZ camera: implements the Camera, Lens, PanTilt and VSource interfaces in one object and renders video frames of a virtual sphere for the current pan, tilt, zoom and focus. Uses the SimpleUI library for drawing. No external dependencies (on Linux X11 is linked through SimpleUI if it is installed). |
Examples and Templates
| Application | Description |
|---|---|
| WebControlPanelTemplate (Linux only) | The template for embedded WEB control page devices. The template includes the files to build an embedded WEB control interface for a camera. This template can be modified by user with additional parameters and functions. |
| VPipelineOnboard (Linux and Windows) | Template and usage example for the VPipeline library. It can be used as a base for your application. It runs a video processing pipeline with functions: video capture from a file, image flip, digital zoom, color processing, video stabilization, dehaze/defog, motion magnification, video tracking, motion detection and video change detection. Also, the application provides RTSP video streaming, UDP telemetry and remote control, and camera, lens and pan-tilt control through the VISCA protocol. |
| VPipelineControl (Linux and Windows) | Control application for the VPipelineOnboard application. It can be used as a base for your application. It provides a simple user interface and joystick control. |
| MultiRtspServer (Linux only) | The MultiRtspServer application is an example of streaming several video streams using multiple MServer objects. All the servers share one RTSP listener and are told apart by the stream suffix. |
| VOverlayTemplate (Linux and Windows) | The template for video overlay based on the SimpleUI library. |
| StreamerTemplate (Linux only) | The StreamerTemplate application captures video from a USB camera with the VSourceV4L2 library, encodes it with the hardware H264 encoder with the VCodecV4L2 library and sends it to the network as an RTP stream with the RtpPusher library. |
| StreamReceiver (Linux and Windows) | The StreamReceiver application receives a video stream with the VSourceStream library (RTSP, RTSPS, RTP and SRTP push streams, MPEG-TS) and shows the frames in a window with statistics with the SimpleUI library. |
Tutorials
| Tutorial | Description |
|---|---|
| YoloDnnPreparationExample (Linux only) | Provides a set of software tools for training, evaluating, and exporting neural network models, with a focus on object detection tasks. |
| SnapServiceUpdateTemplate (Linux only) | A step-by-step guide on how to create a snap package for a CMake project using Snapcraft, as well as instructions for setting up a systemd service for the snap application. Snap packages allow you to update an executable on a target machine by bundling all required packages into a single file. |
| YoloHailoTemplate (Linux only) | A general-purpose template for building an object detection pipeline using the YOLOv8 model and deploying it on the Hailo 8 NPU platform. It covers all essential steps: dataset annotation, data preparation, model training, ONNX export, conversion to Hailo HEF format, and deployment for real-time inference. |
| ABUpdateTemplate (Linux only) | A guide on how to build an A/B update system with RAUC and GRUB2 on x86_64 UEFI platforms. |