VPipelineOnboard application
An example of VPipeline C++ library usage on the video processing device
v2.0.0
Table of contents
Overview
The VPipelineOnboard is an example of VPipeline library usage on the device which captures, processes and streams video. The application (repository) is a CMake project and can be used as a template for your project. The repository contains the source code of the application only: the libraries the application uses are separate repositories which have to be placed next to it (see Application files). The application uses C++17 standard.
VPipelineOnboard is general-purpose software: it is not tied to any sector or application, and the developer of a program decides what the program does with it.
Upon starting, the application reads its parameters from VPipelineOnboard.json file (creates the file with default parameters if it doesn’t exist, see Launch) and initializes VPipeline with the following modules:
- VSourceFile - video source, reads video files. The file can be chosen in a dialog of SimpleFileDialog library.
- ViscaCamera - camera, lens and pan-tilt controller.
- VStabiliserCv - video stabiliser.
- CvTracker - video tracker.
- Gmd - motion detector (object detector 1).
- Ged - video changes detector (object detector 2).
- Video filters: ImageFlip (filter 0), DigitalZoom (filter 1), Dehazer (filter 3), ColorProcessor (filter 4) and MotionMagnificator (filter 6).
The other slots of the pipeline (object detectors 0 and 3 - 6, video filters 2 and 5) get dummy modules.
The processed video is streamed through RTSP by MServer library: the frames are encoded to JPEG by VEncoderJPEG library and VOverlayOnboard class of the application draws a title on the video. By default the stream is available as rtsp://127.0.0.1:8554/live. Control and telemetry go through the UDP port of UdpDataChannel server: the application receives commands for the video processing pipeline (first byte of the command is 0x00) and for the pan-tilt unit (first byte is 0x01, the command is executed only if the video tracker doesn’t track an object) and sends telemetry: the parameters of the pipeline modules (including object detection results) and of the pan-tilt unit.
Main application loop waits a video frame from VPipeline (YUV24 format), sends it to the RTSP server, gets the current parameters of the pipeline, moves the pan-tilt unit to keep the tracked object in the center of the frame (if the video tracker tracks an object) and sends the telemetry through the UDP port. The commands are received in a separate thread.
Application versions
Table 1 - Application versions.
| Version | Release date | What’s new |
|---|---|---|
| 1.0.0 | 06.06.2024 | First version. |
| 1.0.1 | 06.08.2024 | - Submodules updated. |
| 1.0.2 | 18.09.2024 | - Submodules updated. |
| 1.0.3 | 05.10.2024 | - Update submodules. |
| 1.0.4 | 04.12.2024 | - Update submodules. |
| 1.0.5 | 24.02.2025 | - Update submodules. |
| 1.0.6 | 18.03.2025 | - Update submodules. |
| 1.0.7 | 03.04.2025 | - Multiple submodules update. |
| 1.0.8 | 27.04.2025 | - Multiple submodules update. |
| 1.0.9 | 15.05.2025 | - CvTracker submodule update. |
| 1.0.10 | 26.05.2025 | - DnnOpenVinoDetector submodule update. |
| 1.0.11 | 22.06.2025 | - UdpDataChannel submodule update. |
| 1.0.12 | 19.07.2025 | - CvTracker submodule update. - Gmd submodule update. - ViscaCamera submodule update. |
| 1.0.13 | 10.08.2025 | - CvTracker submodule update. - Gmd submodule update. |
| 2.0.0 | 26.10.2026 | - Rewrite the application. |
Application files
The VPipelineOnboard is provided as source code in the form of a CMake project (repository). The repository contains the source code of the application only. The libraries used by the application are separate repositories: they have to be placed next to the VPipelineOnboard repository, in the same folder (the main CMakeLists.txt adds them as **../
CMakeLists.txt --------------------- Main CMake file of the application.
src -------------------------------- Folder with source code of the application.
CMakeLists.txt ----------------- CMake file of the application.
main.cpp ----------------------- Source code file of the application.
VOverlayOnboard.cpp ------------ Source code file of the video overlay.
VOverlayOnboard.h -------------- Header file of the video overlay.
VPipelineOnboardVersion.h ------ Header file with the application version.
VPipelineOnboardVersion.h.in --- File for CMake to generate the version header.
static ----------------------------- Folder with images for the documentation.
Build application
The VPipelineOnboard is a CMake project. The application has external dependencies:
- Libraries. They are separate repositories and have to be placed next to the VPipelineOnboard repository (see Application files):
- Camera
- ColorProcessor
- ConfigReader
- CryptoProvider
- CvTracker
- Dehazer
- DigitalZoom
- FormatConverter
- Frame
- Ged
- Gmd
- ImageFlip
- ImageResizer
- Lens
- Logger
- MotionMagnificator
- MServer
- ObjectDetector
- PanTilt
- SerialPort
- SimpleFileDialog
- SimpleUI
- UdpDataChannel
- UdpSocket
- VCodec
- VDecoderH264
- VDecoderH265
- VDecoderJPEG
- VEncoderJPEG
- VFilter
- ViscaCamera
- ViscaParser
- VOverlay
- VPipeline
- VSource
- VSourceFile
- VStabiliser
- VStabiliserCv
- VStreamer
- VTracker
- Software to be installed in OS:
- CMake 3.13 or newer and a C++17 compiler with OpenMP support (GCC provides it).
- OpenSSL 3.0 or newer with development files (Linux only,
libssl-devpackage): MServer library can’t be built without it. On Windows OpenSSL is not needed: MServer uses Windows CNG. - libsystemd development files (Linux, optional,
libsystemd-devpackage): SimpleFileDialog library uses XDG Desktop Portal with it, otherwise zenity or kdialog program.
Typical commands to clone the application and the libraries into the same folder (use the URLs of the repositories provided to you):
cd <work folder>
git clone <VPipelineOnboard repository URL>
for LIB in Camera ColorProcessor ConfigReader CryptoProvider CvTracker Dehazer \
DigitalZoom FormatConverter Frame Ged Gmd ImageFlip ImageResizer Lens \
Logger MotionMagnificator MServer ObjectDetector PanTilt SerialPort \
SimpleFileDialog SimpleUI UdpDataChannel UdpSocket VCodec VDecoderH264 \
VDecoderH265 VDecoderJPEG VEncoderJPEG VFilter ViscaCamera ViscaParser \
VOverlay VPipeline VSource VSourceFile VStabiliser VStabiliserCv \
VStreamer VTracker
do
git clone <repositories base URL>/${LIB}.git
done
If a library folder does not exist CMake stops with an error.
In the RapidPixel SDK the application is built from the root of the SDK (the libraries are taken from the SDK, the folders next to the application are not needed). The software of the OS listed above is needed too. The executable file is placed in the build/bin folder (build\bin\Release on Windows):
cd /path/to/RapidPixelSdk
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --target VPipelineOnboard
On Linux
Below are the steps to configure on Linux operating system:
-
To install the compiler, CMake, OpenSSL, libsystemd and zenity use following command:
sudo apt-get install build-essential cmake libssl-dev libsystemd-dev zenity -
Clone the VPipelineOnboard repository and the libraries into the same folder (see above).
-
Commands to build VPipelineOnboard application (Release mode):
cd VPipelineOnboard mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Release .. make
On Windows
Below are the steps to configure on Windows operating system:
-
Install Visual Studio with the Desktop development with C++ workload: Open the link, download installer and follow instructions: Visual Studio.
-
Install CMake: Open the link, download installer and follow instructions: CMake download. Version 3.13 or newer is supported, Visual Studio 2022 requires CMake 3.21 or newer.
-
Clone the VPipelineOnboard repository and the libraries into the same folder (see above).
-
Commands to build VPipelineOnboard application (Release mode):
cd VPipelineOnboard mkdir build cd build cmake .. cmake --build . --config Release
Launch
The result of compiling will be VPipelineOnboard.exe executable file on Windows or VPipelineOnboard executable file on Linux. To launch application on Linux run command:
./VPipelineOnboard
After start of the application, it will create configuration file (if it doesn’t exist already) VPipelineOnboard.json with default VPipeline parameters. Configuration file allows to modify all of the parameters from VPipeline, you can also copy configuration file (VPipelineOnboard.json). Default contents of the configuration file:
{
"Params": {
"general": {
"udpPort": 57001
},
"pipeline": {
"videoSource": {
"logLevel": 0,
"source": "file dialog",
"fourcc": "YUYV",
"width": 1920,
"height": 1080,
"gainMode": 1,
"exposureMode": 1,
"focusMode": 1,
"fps": 0.0,
"roiX": 0,
"roiY": 0,
"roiWidth": 0,
"roiHeight": 0,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0
},
"videoStabiliser": {
"scaleFactor": 0,
"xOffsetLimit": 0,
"yOffsetLimit": 0,
"aOffsetLimit": 0.0,
"xFilterCoeff": 0.80000001192092896,
"yFilterCoeff": 0.80000001192092896,
"aFilterCoeff": 0.80000001192092896,
"enable": true,
"transparentBorder": true,
"constXOffset": 0,
"constYOffset": 0,
"constAOffset": 0.0,
"type": 0,
"cutFrequencyHz": 0.0,
"fps": 0.0,
"logMod": 0,
"backend": 0,
"custom1": 1.0,
"custom2": 1.0,
"custom3": 0.0
},
"videoTracker": {
"rectWidth": 72,
"rectHeight": 72,
"searchWindowWidth": 256,
"searchWindowHeight": 256,
"lostModeOption": 0,
"frameBufferSize": 1,
"maxFramesInLostMode": 256,
"rectAutoSize": false,
"rectAutoPosition": false,
"multipleThreads": false,
"numChannels": 2,
"type": 0,
"custom1": 0.10000000149011612,
"custom2": 0.40000000596046448,
"custom3": 0.0
},
"objectDetector0": {
"initString": "",
"logMode": 0,
"frameBufferSize": 1,
"minObjectWidth": 4,
"maxObjectWidth": 128,
"minObjectHeight": 4,
"maxObjectHeight": 128,
"minXSpeed": 0.0,
"maxXSpeed": 30.0,
"minYSpeed": 0.0,
"maxYSpeed": 30.0,
"minDetectionProbability": 0.5,
"xDetectionCriteria": 1,
"yDetectionCriteria": 1,
"resetCriteria": 1,
"sensitivity": 0.039999999105930328,
"scaleFactor": 1,
"numThreads": 1,
"type": 0,
"enable": true,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0,
"classNames": [
""
]
},
"objectDetector1": {
"initString": "",
"logMode": 0,
"frameBufferSize": 1,
"minObjectWidth": 4,
"maxObjectWidth": 128,
"minObjectHeight": 4,
"maxObjectHeight": 128,
"minXSpeed": 0.10000000149011612,
"maxXSpeed": 15.0,
"minYSpeed": 0.10000000149011612,
"maxYSpeed": 15.0,
"minDetectionProbability": -1.0,
"xDetectionCriteria": 15,
"yDetectionCriteria": 15,
"resetCriteria": 5,
"sensitivity": 10.0,
"scaleFactor": 1,
"numThreads": 1,
"type": -1,
"enable": true,
"custom1": -1.0,
"custom2": -1.0,
"custom3": -1.0,
"classNames": []
},
"objectDetector2": {
"initString": "",
"logMode": 0,
"frameBufferSize": 30,
"minObjectWidth": 2,
"maxObjectWidth": 128,
"minObjectHeight": 2,
"maxObjectHeight": 128,
"minXSpeed": -1.0,
"maxXSpeed": -1.0,
"minYSpeed": -1.0,
"maxYSpeed": -1.0,
"minDetectionProbability": -1.0,
"xDetectionCriteria": 1,
"yDetectionCriteria": 1,
"resetCriteria": 6,
"sensitivity": 10.0,
"scaleFactor": 1,
"numThreads": 1,
"type": -1,
"enable": true,
"custom1": -1.0,
"custom2": -1.0,
"custom3": -1.0,
"classNames": []
},
"objectDetector3": {
"initString": "",
"logMode": 0,
"frameBufferSize": 1,
"minObjectWidth": 4,
"maxObjectWidth": 128,
"minObjectHeight": 4,
"maxObjectHeight": 128,
"minXSpeed": 0.0,
"maxXSpeed": 30.0,
"minYSpeed": 0.0,
"maxYSpeed": 30.0,
"minDetectionProbability": 0.5,
"xDetectionCriteria": 1,
"yDetectionCriteria": 1,
"resetCriteria": 1,
"sensitivity": 0.039999999105930328,
"scaleFactor": 1,
"numThreads": 1,
"type": 0,
"enable": true,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0,
"classNames": [
""
]
},
"objectDetector4": {
"initString": "",
"logMode": 0,
"frameBufferSize": 1,
"minObjectWidth": 4,
"maxObjectWidth": 128,
"minObjectHeight": 4,
"maxObjectHeight": 128,
"minXSpeed": 0.0,
"maxXSpeed": 30.0,
"minYSpeed": 0.0,
"maxYSpeed": 30.0,
"minDetectionProbability": 0.5,
"xDetectionCriteria": 1,
"yDetectionCriteria": 1,
"resetCriteria": 1,
"sensitivity": 0.039999999105930328,
"scaleFactor": 1,
"numThreads": 1,
"type": 0,
"enable": true,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0,
"classNames": [
""
]
},
"objectDetector5": {
"initString": "",
"logMode": 0,
"frameBufferSize": 1,
"minObjectWidth": 4,
"maxObjectWidth": 128,
"minObjectHeight": 4,
"maxObjectHeight": 128,
"minXSpeed": 0.0,
"maxXSpeed": 30.0,
"minYSpeed": 0.0,
"maxYSpeed": 30.0,
"minDetectionProbability": 0.5,
"xDetectionCriteria": 1,
"yDetectionCriteria": 1,
"resetCriteria": 1,
"sensitivity": 0.039999999105930328,
"scaleFactor": 1,
"numThreads": 1,
"type": 0,
"enable": true,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0,
"classNames": [
""
]
},
"objectDetector6": {
"initString": "",
"logMode": 0,
"frameBufferSize": 1,
"minObjectWidth": 4,
"maxObjectWidth": 128,
"minObjectHeight": 4,
"maxObjectHeight": 128,
"minXSpeed": 0.0,
"maxXSpeed": 30.0,
"minYSpeed": 0.0,
"maxYSpeed": 30.0,
"minDetectionProbability": 0.5,
"xDetectionCriteria": 1,
"yDetectionCriteria": 1,
"resetCriteria": 1,
"sensitivity": 0.039999999105930328,
"scaleFactor": 1,
"numThreads": 1,
"type": 0,
"enable": true,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0,
"classNames": [
""
]
},
"lens": {
"initString": "/dev/ttyUSB0;9600;20",
"focusMode": 0,
"filterMode": 0,
"afRoiX0": 0,
"afRoiY0": 0,
"afRoiX1": 0,
"afRoiY1": 0,
"zoomHwMaxSpeed": 50,
"focusHwMaxSpeed": 50,
"irisHwMaxSpeed": 50,
"zoomHwTeleLimit": 65535,
"zoomHwWideLimit": 0,
"focusHwFarLimit": 65535,
"focusHwNearLimit": 0,
"irisHwOpenLimit": 65535,
"irisHwCloseLimit": 0,
"afHwSpeed": 50,
"focusFactorThreshold": 0.0,
"refocusTimeoutSec": 0,
"irisMode": 0,
"autoAfRoiWidth": 150,
"autoAfRoiHeight": 150,
"autoAfRoiBorder": 100,
"afRoiMode": 0,
"extenderMode": 0,
"stabiliserMode": 0,
"afRange": 0,
"logMode": 0,
"type": 0,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0,
"fovPoints": []
},
"camera": {
"initString": "/dev/ttyUSB0;9600;20",
"width": 0,
"height": 0,
"displayMode": 0,
"videoOutput": 0,
"logMode": 0,
"exposureMode": 1,
"exposureTime": 0,
"whiteBalanceMode": 1,
"whiteBalanceArea": 0,
"wideDynamicRangeMode": 0,
"stabilizationMode": 0,
"isoSensitivity": 0,
"sceneMode": 0,
"fps": 0.0,
"brightnessMode": 1,
"brightness": 0,
"contrast": 0,
"gainMode": 1,
"gain": 0,
"sharpeningMode": 0,
"sharpening": 0,
"palette": 0,
"agcMode": 1,
"shutterMode": 1,
"shutterPos": 0,
"shutterSpeed": 0,
"digitalZoomMode": 0,
"digitalZoom": 1.0,
"exposureCompensationMode": 0,
"exposureCompensationPosition": 0,
"defogMode": 0,
"dehazeMode": 0,
"noiseReductionMode": 0,
"blackAndWhiteFilterMode": 0,
"filterMode": 0,
"nucMode": 0,
"autoNucIntervalMsec": 0,
"imageFlip": 0,
"ddeMode": 0,
"ddeLevel": 0.0,
"roiX0": 0,
"roiY0": 0,
"roiX1": 0,
"roiY1": 0,
"alcGate": 0,
"sensitivity": 0.0,
"changingMode": 0,
"changingLevel": 0.0,
"chromaLevel": 0,
"detail": 0,
"profile": 0,
"type": 0,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0
},
"videoFilter0": {
"mode": 0,
"level": -1.0,
"type": -1,
"custom1": -1.0,
"custom2": -1.0,
"custom3": -1.0
},
"videoFilter1": {
"mode": 0,
"level": 1.0,
"type": 0,
"custom1": 1.0,
"custom2": -1.0,
"custom3": -1.0
},
"videoFilter2": {
"mode": 1,
"level": 0.0,
"type": 0,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0
},
"videoFilter3": {
"mode": 0,
"level": 0.0,
"type": 0,
"custom1": -1.0,
"custom2": 1.0,
"custom3": -1.0
},
"videoFilter4": {
"mode": 0,
"level": 50.0,
"type": 0,
"custom1": 50.0,
"custom2": 50.0,
"custom3": 50.0
},
"videoFilter5": {
"mode": 1,
"level": 0.0,
"type": 0,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0
},
"videoFilter6": {
"mode": 1,
"level": 2.0,
"type": 0,
"custom1": 1.0,
"custom2": 0.0,
"custom3": 0.0
},
"general": {
"logMode": 0,
"enable": true,
"trackerCenteringEnable": false,
"trackerCenteringMaxOffsetX": 0,
"trackerCenteringMaxOffsetY": 0
}
},
"panTilt": {
"panMotorSpeed": 0.0,
"tiltMotorSpeed": 0.0,
"initString": "",
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0
},
"streamer": {
"enable": true,
"width": 1280,
"height": 720,
"directStreamIp": "127.0.0.1",
"rtspPort": 8554,
"rtspsPort": 8555,
"directStreamPort": 5004,
"webRtcPort": 7000,
"hlsPort": 8080,
"srtPort": 6000,
"rtmpPort": 1935,
"rtmpsPort": 1936,
"metadataPort": 9000,
"rtspEnable": true,
"directStreamEnable": false,
"webRtcEnable": false,
"hlsEnable": false,
"srtEnable": false,
"rtmpEnable": false,
"metadataEnable": false,
"rtspMulticastIp": "224.1.0.1/16",
"rtspMulticastPort": 18000,
"user": "no",
"password": "no",
"suffix": "live",
"metadataSuffix": "metadata",
"minBitrateKbps": 1000,
"maxBitrateKbps": 5000,
"bitrateKbps": 3000,
"bitrateMode": 0,
"fps": 30.0,
"gop": 30,
"h264Profile": 0,
"jpegQuality": 90,
"codec": "JPEG",
"fitMode": 0,
"cycleTimeUs": 0,
"overlayEnable": true,
"type": 0,
"custom1": 0.0,
"custom2": 0.0,
"custom3": 0.0,
"rtspKey": "no",
"rtspCert": "no",
"webRtcKey": "no",
"webRtcCert": "no",
"hlsKey": "no",
"hlsCert": "no",
"rtmpKey": "no",
"rtmpCert": "no",
"rtspEncryption": "no",
"webRtcEncryption": "no",
"rtmpEncryption": "no",
"hlsEncryption": "no",
"logLevel": 0,
"directStreamType": "rtp",
"directStreamBitrateKbps": 5000,
"directStreamMaxPayloadSize": 1472,
"directStreamPacingMode": 0,
"serverStreamType": "rtp",
"multicastTtl": 0,
"securityProfile": "no",
"publicAddress": "no",
"bindAddress": "0.0.0.0",
"rtpPortMin": 0,
"rtpPortMax": 0,
"webRtcMediaPort": 0,
"corsAllowedOrigin": "no",
"serverStreamMaxPayloadSize": 1472
},
"overlay": {
"title": "VPipelineOnboard"
}
}
}
The application parameters are stored in the Params object of the configuration file: general application parameters, VPipeline parameters (pipeline), pan-tilt parameters, RTSP server parameters and video overlay parameters. Table 2 shows short description of parameters.
Table 2 - Application parameters.
| Parameter | Description |
|---|---|
| general | General application parameters: udpPort - UDP port to receive control commands and to send telemetry. |
| pipeline.videoSource | Video source parameters. VSourceFile object will be initialized: it reads video files (H.264, H.265 and Motion JPEG in MP4, MOV, M4V and AVI files or raw streams). By default the “source” field has value “file dialog”: the application opens a file dialog to choose the video file. You can set the path to a video file instead. |
| pipeline.videoStabiliser | Video stabiliser parameters. VStabiliserCv object will be initialized. |
| pipeline.videoTracker | Video tracker parameters. CvTracker object will be initialized. |
| pipeline.objectDetector0 | Not used in the application: dummy object detector will be initialized. |
| pipeline.objectDetector1 | First object detector parameters. Gmd object will be initialized. Motion detector parameters. |
| pipeline.objectDetector2 | Second object detector parameters. Ged object will be initialized. Video changes detector parameters. |
| pipeline.objectDetector3 | Not used in the application: dummy object detector will be initialized. |
| pipeline.objectDetector4 | Not used in the application: dummy object detector will be initialized. |
| pipeline.objectDetector5 | Not used in the application: dummy object detector will be initialized. |
| pipeline.objectDetector6 | Not used in the application: dummy object detector will be initialized. |
| pipeline.lens | Lens controller parameters. ViscaCamera object will be initialized. |
| pipeline.camera | Camera controller parameters. ViscaCamera object will be initialized. |
| pipeline.videoFilter0 | Image flip parameters. ImageFlip object will be initialized. |
| pipeline.videoFilter1 | Digital zoom parameters. DigitalZoom object will be initialized. |
| pipeline.videoFilter2 | Not used in the application: dummy video filter will be initialized. |
| pipeline.videoFilter3 | Defog / dehaze parameters. Dehazer object will be initialized. |
| pipeline.videoFilter4 | Color processing parameters (palettes, brightness, contrast, saturation and white balance). ColorProcessor object will be initialized. |
| pipeline.videoFilter5 | Not used in the application: dummy video filter will be initialized. |
| pipeline.videoFilter6 | Motion magnification parameters. MotionMagnificator object will be initialized. |
| pipeline.general | General video processing pipeline parameters. |
| panTilt | Pan-tilt parameters of ViscaCamera (pan-tilt interface). The application sends them in telemetry, the values in the configuration file are not used. |
| streamer | RTSP server parameters. MServer object will be initialized, the frames are encoded by VEncoderJPEG (only “JPEG” codec is supported). By default the video is available as rtsp://<IP>:8554/live and the other streaming protocols are disabled. |
| overlay | Video overlay parameters: title - text in the top right corner of the video. The overlay is drawn if streamer.overlayEnable is true. |
After starting the application (running the executable file) the user should select the video file in the dialog box (if parameter “videoSource” in config file is set to “file dialog”). After that the UDP channel will start working, sending data from video source and current parameters and also reading control instructions from VPipelineControl or any UDP port connected to the VPipelineOnboard.