RapidPixel SDK
v1.4.0
RapidPixel SDK is a set of cross-platforms C++ libraries and sample applications that allows you to quickly create real-time video processing applications. The SDK includes all the necessary libraries as well as ready-to-use video processing pipelines for embedding into your applications (video capture, video codecs, video streaming, video tracking and object detection, video stabilization, communication, logging, video filtering, lenses and cameras controllers etc.). In addition, data exchange libraries are included, as well as pre-built applications that can be used as the foundation of your application. All algorithms are real-time and can be used on any CPU (Intel, AMD, Nvidia Jetson/Xavier, Raspberry PI 4/5, IMX8/9, Broadcom, Amlogic etc.). The motivation of RapidPixel SDK is to provide building blocks for video processing software with simple interface and easy usage principles. Compared to other popular SDKs and APIs (OpenCV, Intel IPP, Nvidia SDKs, Qualcomm Computer Vision SDK etc.) RapidPixel SDK offers building blocks of much larger scale that combine multiple algorithms to solve a specific problem (object detection, video tracking etc.). The SDK is cross-platforms (Linux and Windows) and provides various usage scenarios: using source code, using source code of particular libraries, compiling and installation into OS, compiling particular libraries. The SDK is CMake project and uses C++17 standard. Particular libraries may have dependency from external libraries (for example IntelOpenVINO or OpenCV, check description of particular libraries). Each library in SDK is self-sufficient, has detailed documentation, examples, test applications and can be used independently from others.
Functions
RapidPixel SDK includes libraries for:
- Video tracking.
- Video stabilization.
- Motion detection.
- Small changes detection on video (not moving objects).
- Detection with neural networks.
- Video filtering: motion magnification, image flip, digital zoom, defog / dehaze, denoise.
- Ready video processing pipelines.
- RTSP servers.
- Wide range of lenses and cameras software controllers.
- Laser range finder controllers.
- Video capture from different interfaces.
- Video codecs for various platforms.
- Pixel format conversion.
- Joysticks support.
- Licensing mechanism for your application, config files parsers, and lot of others service functions.
- Examples / test application for each library and ready application templates.
How to start
SDK structure
RapidPixel SDK is CMake project which includes set of libraries and examples. Each library is self-sufficient and can be used independently from others. Each library includes detailed documentation, examples and test applications. SDK structure by categories:
Video processing pipelines
Video trackers
Video stabilizers
Object detectors
Video streamers
Video filters
Video capture
Video codecs
Communication
Laser range finders controllers
Lens controllers
Camera controllers
Block camera controllers (lens and camera combined)
Service libraries
Examples and templates
Library / application | Description |
---|---|
VPipeline (Linux and Windows) | Provides full functional video processing pipeline with interfaces to connect different video filters, video trackers, object detectors and video stabilization modules. The motivation of VPipeline library is to provide simple interface to complex video processing pipeline as well as camera and lens control regardless of hardware. The library serves to two main goals: provides video processing pipeline and provides remote control interface for pipeline (encoding / decoding commands and telemetry). External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
VTracker (Linux and Windows) | Provides standard interface as well defines data structures and rules for different video trackers. VTracker interface class does nothing, just provides interface, defines data structures and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
CvTracker (Linux and Windows) | Video tracker implementation. The library provides fast calculation, compatibility with low-power processors, high accuracy and contains a lot of additional functions and modes, which allow using it in camera systems of any configuration. External dependencies: CUDA (optional for Nvidia Jetson platform). |
VStabiliser (Linux and Windows) | Provides standard interface as well defines data structures and rules for different video stabilization algorithms. VStabiliser interface class does nothing, just provides interface, defines data structures and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
VStabiliserOpenCv (Linux and Windows) | Video stabilizer implementation. Provides 2D and 3D digital video stabilization (horizontal offset, vertical offset and rotation). The library is used in camera systems for video vibration compensation. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
ObjectDetector (Linux and Windows) | Provides standard interface as well defines data structures and rules for different object detectors (motion detectors, changes detectors, neural networks etc.). ObjectDetector interface class does nothing, just provides interface, defines data structures and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
Gmd (Linux and Windows) | Motion object detector implementation. Provides automatic detection of moving objects on videos. This library is suitable for various types of cameras (daylight, SWIR, MWIR and LWIR), and it ensures accurate detection of small-size and low-contrast moving objects against complex backgrounds. No external dependencies. |
Ged (Linux and Windows) | Video changes detector implementation. Provides automatic detection of subtle changes on videos. This library is suitable for various types of cameras (daylight, SWIR, MWIR and LWIR), and it ensures accurate detection of small-size and low-contrast objects against complex backgrounds. No external dependencies. |
DnnOpenVinoDetector (Linux and Windows) | DNN object detector implementation. Provides an automatic detection of objects on videos through the utilization of neural networks based on OpenVINO™ runtime. It optimized and tested for Ultralytics YOLOv8 neural network model and provides simple interface. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license) and OpenVINO™ (Apache 2.0 license). |
DnnOpenCvDetector (Linux and Windows) | DNN object detector implementation. Provides automatic detection of objects on videos through the utilization of neural networks based on OpenCV. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
VStreamer (Linux and Windows) | Provides standard interface as well defines data structures and rules for different video streamers. VStreamer interface class does nothing, just provides interface, defines data structures and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
RtspServer (Linux and Windows) | RTSP server implementation. Provides RTSP server. The library supports H264 and HEVC video stream via TCP / UDP / UDP multicast. User can set general RTSP server parameters: RTSP server port, IP, stream name, user name, password etc. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license, for pixel format conversion and scaling functions). |
RtspServerLive555 (Linux only) | RTSP server implementation. Provides RTSP server based on popular open source library Live555 (source code not included, just linked, LGPL license). The library supports H264, HEVC or JPEG video stream via TCP / UDP / UDP multicast. User can set general RTSP server parameters: RTSP server port, IP, stream name, user name, password etc. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license, for pixel format conversion) and Live555 (LGPL license, this allows you to use these libraries (via linking) inside closed-source products). |
RtpPusher (Linux and Windows) | Simple RTP H264 video streamer which is compatible with all RTP clients (VLC, ffmpeg, gstreamer etc.). No external dependencies. |
VStreamerMediaMtx (Linux only) | RTSP, WebRTC, HLS, RTMP and SRC video server based on MediaMTX application. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license, for pixel format conversion, video scaling and running GStreamer pipeline) and GStreamer (external process, not included in code). |
VSource (Linux and Windows) | Provides standard interface as well defines data structures and rules for different video source classes (video capture classes). VSource interface class does nothing, just provides interface and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
VSourceV4L2 (Linux only) | Video source implementation. Provides video capture and video source (V4L2 compatible video sources) control function based on V4L2 API. External dependencies: V4L2 API (default subsystem in Linux). |
VSourceLibCamera (Linux only) | Video source implementation. Provides video capture and video source control function based on Libcamera API. External dependencies: Libcamera API (MIT license). |
VSourceOpenCv (Linux and Windows) | Video source implementation. Provides video capture and video source control function based on OpenCV library (version >=4.5). External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
VSourceFile (Linux and Windows) | Video source implementation. Provides video capture function for a encoded H264 and HEVC (*.h264 and *.hevc) video files. No external dependencies. |
VCodec (Linux and Windows) | Provides standard interface as well defines data structures and rules for different video codecs (video encoding and decoding). VCodec interface class doesn’t do anything, just provides interface. No external dependencies. |
VCodecV4L2 (Linux only) | Video codec implementation. Provides hardware accelerated video encoding / decoding (H264 and JPEG codecs) based on V4L2 API on Linux only. External dependencies: V4L2 API (default subsystem in Linux). |
VCodecImsdk (Windows only) | Video codec implementation. Provides hardware accelerated video encoding / decoding (H264, HEVC and JPEG codecs) for Intel HD Graphics based on Intel Media SDK. External dependencies: Intel Media SDK (source code included, MIT license). |
VCodecOneVpl (Linux and Windows) | Video codec implementation. Provides hardware accelerated video encoding / decoding (H264, HEVC and JPEG codecs) for Intel GPU based on Intel oneVPL API. External dependencies: oneVPL (linked, MIT license). |
VCodecJetPack (NVIDIA Linux only) | Video codec implementation. Provides hardware accelerated video encoding (H264 and HEVC codecs) for Nvidia Jetson platforms based on Jetson Multimedia API. External dependencies: Jetson Multimedia API (linked, proprietary license). |
VCodecLibav (Linux and Windows) | Video codec implementation. Provides hardware and software accelerated video encoding / decoding functions (H264, HEVC(H265) and JPEG codecs) for Linux and Windows OS based on FFmpeg. External dependencies: FFmpeg (linked, GPL or LGPL license depends on FFmpeg library build linked). |
VCodecOpenH264 (Linux only) | Video codec implementation. Provides video encoding / decoding functions (H264 codec only) for Linux based on Openh264 library. External dependencies: Openh264 (linked, BSD-2-Clause license). |
VFilter (Linux and Windows) | Provides interface as well defines data structures and rules for various video filters implementations. The VFilter interface represents video filter module which has video frames as input and output. VFilter interface class does nothing, just provides interface and methods to encode / decode commands and encode / decode params. No external dependencies. |
MotionMagnificator (Linux and Windows) | Video filter implementation. Provides subtle motion and temporal variations amplification in digital video content. It allows camera system operators to detect moving objects that are not visible to the naked eye. No external dependencies. |
Dehazer (Linux and Windows) | Video filter implementation. Provides video dehazing algorithm based on histogram manipulations. The library utilizes different histogram manipulations to improve the contrast of the video frame and to reduce the effect of haze. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
DigitalZoom (Linux and Windows) | Video filter implementation. Provides digital zoom functions based on OpenCV. The library supports different interpolation algorithms: bicubic interpolation, resampling using pixel area relation, Lanczos 8x8, bilinear interpolation, bit exact bilinear interpolation, nearest neighbor interpolation and bit exact nearest neighbor interpolation. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
ImageFlip (Linux and Windows) | Video filter implementation. Provides simple image flip function based on OpenCV. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
Denoiser (Linux and Windows) | Prelimary. Video filter implementation. Provides noise cancellation based on OpenCV. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
Palette (Linux and Windows) | Thermal camera color palette filter. No external dependencies. |
UdpDataChannel (Linux and Windows) | Provides reliable point-to-point communication between two applications based on UDP. The library includes two primary classes: UdpDataClient (connects to server) and UdpDataServer (wait connection from client). Both classes provide two-way communication. No external dependencies. |
NtpUdpChannel (Linux and Windows) | Provides mechanism to simplify sending and receiving data over the UDP transport layer via low bandwidth networks. The library streams data taking in account channel bandwidth and controls data loss. No external dependencies. |
SerialPort (Linux and Windows) | Provides simple interface to work with serial ports. No external dependencies. |
TcpClient (Linux and Windows) | Provides methods to work with TCP socket as client (connect, close, send data and receive data). No external dependencies. |
TcpServer (Linux and Windows) | Provides methods to work with TCP socket as server (wait connection, close, send data and receive data). No external dependencies. |
UdpSocket (Linux and Windows) | Provides methods to work with UDP ports (open, close, send data and receive data). No external dependencies. |
SbusParser (Linux and Windows) | Provides methods for encoding and decoding SBUS protocol data. No external dependencies. |
CrsfParser (Linux and Windows) | Provides mechanism for decoding and encoding packets of FPV drone control protocol. The library used for CRSF packets translation and extracts only CRSF_FRAMETYPE_RC_CHANNELS_PACKED packets. For other packets types the library just detects them in data buffer. No external dependencies. |
FileFinderByName (Linux and Windows) | Provides methods to find files by name part(s). It used in different projects to find data files of device files. No external dependencies. |
FormatConverterOpenCv (Linux and Windows) | Provides methods to convert pixels formats of images between each others. FormatConverterOpenCv supports all uncompressed pixel formats (conversion between each other) listed in Fourcc enum (RGB24, BGR24, YUYV, UYVY, GRAY, YUV24, NV12, NV21, YU12, YV12) of Frame library. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
Logger (Linux and Windows) | Provides logging functions: printing in terminal and(or) printing in file. No external dependencies. |
PlotOpenCv (Linux and Windows) | Provides methods for visualization of 2-dimensional line charts. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
ChildProcess (Linux only) | Provides methods to run / stop external process from your application in Linux. No external dependencies. |
VOutputV4L2 (Linux only) | Provide interface to write video frame to V4L2 video devices (for example V4L2 loopback devices) in Linux. External dependencies: V4L2 API (default subsystem in Linux). |
VOutputFb (Linux only) | Provide interface to write video frame to Linux framebuffer. No external dependencies. |
Joystick (Linux and Windows) | Provides simple interface to work with various types of joysticks. No external dependencies. |
SimpleFileDialog (Linux and Windows) | Provides simple dialog to chose file in Windows and Linux. No external dependencies. |
VideoRecorderOpenCv (Linux and Windows) | Provides methods for recording video to files with controlling file’s and folder’s size based on OpenCV. External dependencies: OpenCV (version >= 4.5, Apache 2.0 license). |
Frame (Linux and Windows) | Provides video frame data structures and pixel formats descriptions. No external dependencies. |
ConfigReader (Linux and Windows) | Provides methods to read / write config files or strings in JSON format. The library has simple interface to read application configuration parameters from file / string or write configuration parameters to file / string. External dependencies: JSON for modern C++ (source code included, MIT license). |
License (Linux and Windows) | Provides a licensing feature for C++ applications. It includes functions for generating a token (base on username and MAC address of the computer), creating a license based on it, and validating the license with control of its validity time. AES256 encryption method is implemented. No external dependencies. |
VOverlay (Linux and Windows) | Provides standard interface for overlaying information on video for different implementations. Standard for overlaying information in video streamers interfaces. No external dependencies. |
Camera (Linux and Windows) | Provides standard interface as well defines data structures and rules for different camera controllers. Camera interface class does nothing, just provides interface and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
FlirTau2Camera (Linux and Windows) | Software controller for Flir Tau 2 thermal cameras. No external dependencies. |
FlirTau2Parser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for Flir Tau 2 thermal cameras. No external dependencies. |
OwlCamera (Linux and Windows) | Software controller for Owl 640 SWIR cameras. No external dependencies. |
OwlParser (Linux and Windows) | Provides methods for encoding control commands and checking responses for Owl 640 SWIR cameras. No external dependencies. |
KowaCamera (Linux and Windows) | Software controller for KOWA cameras. No external dependencies. |
KowaProtocolParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for low-light KOWA camera. No external dependencies. |
HitachiCamera (Linux and Windows) | Software controller for Hitachi HD-SDI Series cameras. No external dependencies. |
IRayFtCamera (Linux and Windows) | Software controller for IRray FT thermal cameras. No external dependencies. |
FlirBosonCamera (Linux and Windows) | Software controller for Flir boson cameras. External dependencies: Flir Boson SDK (Proprietary license, user must control license permissions before use). |
Lens (Linux and Windows) | Provides standard interface as well defines data structures and rules for different lens controllers. Lens interface class does nothing, just provides interface and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
Slc (Linux and Windows) | Software controller for Stingray SWIR lenses. Includes reach auto-focus functions based on AFEngine library. No external dependencies. |
StingrayParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for Stingray SWIR lenses. No external dependencies. |
Olc (Linux and Windows) | Software controller for Ophir infrared lenses. Includes reach auto-focus functions based on AFEngine library. No external dependencies. |
Flc (Linux and Windows) | Software controller for Fujinon CCTV lenses. No external dependencies. |
FujiProtocolParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for Fujinon CCTV lenses. No external dependencies. |
Ylc (Linux and Windows) | Software controller for Yamano CCTV lenses. No external dependencies. |
SunnyLensParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses from Sunny continuous zoom lenses. No external dependencies. |
SunnyLens (Linux and Windows) | Software controller for Sunny continuous zoom lenses. Includes reach auto-focus functions based on AFEngine library. No external dependencies. |
RpLensParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses from RP thermal / SWIR lenses. No external dependencies. |
Rpl (Linux and Windows) | Sortware controller for RP thermal / SWIR lenses. No external dependencies. |
AFEngine (Linux and Windows) | Provides auto focus functions for different lens controllers. Supports ROI, auto ROI detection and other different parameters. Provides hardware agnostic interface to be implemented in different lens controllers. No external dependencies. |
KowaLensParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for KOWA CCTV lenses. No external dependencies. |
Klc (Linux and Windows) | Software controller for KOWA CCTV lenses. No external dependencies. |
FujiSxCamera (Linux and Windows) | Software controller for Fujinon Sx series CCTV cameras (block cameras which combine lens and camera (image sensor), and provides one control interface). No external dependencies. |
FujiSxParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for Fujinon Sx series CCTV cameras (block cameras which combine lens and camera (image sensor), and provides one control interface). No external dependencies. |
ViscaCamera (Linux and Windows) | Software controller for VISCA Protocol compatible cameras and includes camera, lens and pan-tilt control interfaces. No external dependencies. |
ViscaParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for VISCA Protocol compatible cameras. No external dependencies. |
PanTilt (Linux and Windows) | Provides standard interface as well defines data structures and rules for different pan-tilt controllers. PanTilt interface class does nothing, just provides interface and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
G5Camera (Linux and Windows) | Software controller for G5 thermal cameras (block cameras which combine lens and camera (image sensor), and provides one control interface). No external dependencies. |
G5ProtocolParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for G5 thermal cameras (block cameras which combine lens and camera (image sensor), and provides one control interface). No external dependencies. |
IRayBlockCamera (Linux and Windows) | Software controller for InfiRay FT block cameras (block cameras which combine lens and camera (image sensor), and provides one control interface). No external dependencies. |
IRay300Parser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for InfiRay FT block cameras (block cameras which combine lens and camera (image sensor), and provides one control interface).. No external dependencies. |
VentusCamera (Linux and Windows) | Software controller for WIND firmware compatible cameras, which mostly includes Ventus cameras produced by SIERRA-OLYMPIA TECHNOLOGIES INC. company. No external dependencies. |
Lrf (Linux and Windows) | Provides standard interface as well defines data structures and rules for different laser range finders. Lrf interface class does nothing, just provides interface and provides methods to encode / decode commands and encode / decode params. No external dependencies. |
LrfUltisenseParser (Linux and Windows) | Provides methods for encoding control commands and decoding responses for Safran Laser Range Finders. No external dependencies. |
LrfUltisense (Linux and Windows) | Software controller for Safran Laser Range Finders. No external dependencies. |
VPipelineDemo (Linux and Windows) | Demo application, template and usage example for VPipeline library. It can be uses as base for your application. It runs video processing pipeline with functions: video capture, image flip, digital zoom, video stabilization, dehaze / defog, motion magnification, video tracking, motion detection, video changes detection and objects detection with neural networks. Also the application provides simple user interface to display video and control video processing pipeline modules. |
RpiStreamer (Linux only) | Application template which users can use as basement of your applications. This application implements basic video processing pipeline: video capture > video stabilization > RTSP server for Raspberry PI4 and PI5. The application combines several libraries. |
RpiTracker (Linux only) | Application template which implements basic video processing pipeline: video capture -> video stabilization -> video tracker -> video encoding -> RTSP server. for Raspberry PI 4(5). The application shows how to combine several libraries. |
RpiOnboardControl (Linux and Windows) | Application template which shows how to control RpiTracker application remote. The application shows how to combine several libraries. |
VPipelineOnboard (Linux and Windows) | Template and usage example for VPipeline library. It can be uses as base for your application. It runs video processing pipeline with functions: video capture, image flip, digital zoom, video stabilization, dehaze / defog, motion magnification, video tracking, motion detection, video changes detection and objects detection with neural networks. Also the application provides video + telemetry streaming and remote control. |
VPipelineControl (Linux and Windows) | Control application for VPipelineOnboard application. It can be uses as base for your application. It provides simple user interface and Joystic control. |
MultiRtspServer (Linux only) | The MultiRtspServer application is example of streaming video with using of multiple RtspServer objects. Application supports Windows and Linux with Intel GPU. |
RpiTrackerOnboard (Linux only) | Application template which implements basic video processing pipeline: video capture -> video tracker -> video encoding -> custom video + telemetry streaming for Raspberry PI 4(5). The application shows how to combine several libraries. |
RpiTrackerControl (Linux only) | Application template which shows how to control RpiTrackerOnboard application remote. The application shows how to combine several libraries. |
YoloDnnPreparationExample (Linux only) | Provides comprehensive set of software tools designed for training, evaluating, and exporting neural network models, with a focus on object detection tasks. |