CvTracker C++ library

v11.0.2

Table of contents

• Overview
• Versions
• Library files
• Key features and capabilities
• Operating principles
  • Tracking process and operating modes
  • Scene tracking mode (TYPE = 2)
  • Object loss detection and re-acquisition
  • Stop-frame function
  • Communication channel time delay compensation
• Supported pixel formats
• CvTracker class description
  • CvTracker class declaration
  • How to use the library
  • getVersion method
  • initVTracker method
  • setParam method
  • getParam method
  • getParams method
  • executeCommand method
  • processFrame method
  • getImage method
  • decodeAndExecuteCommand method
  • encodeSetParamCommand method of VTracker class
  • encodeCommand method of VTracker class
  • decodeCommand method of VTracker class
• Data structures
  • VTrackerCommand enum
  • VTrackerParam enum
• VTrackerParams class description
  • VTrackerParams class declaration
  • Serialize video tracker params
  • Deserialize video tracker params
  • Read params from JSON file and write to JSON file
• Build and connect to your project
  • Build stand alone
  • CMake options
  • Connect to your project
• Simple example
• Demo application
  • Demo application overview
  • Launch and user interface
  • Control
• Limits and guarantees
• Portability

Overview

C++ library CvTracker is intended for automatic object video tracking. The library is written in C++ (C++17 standard) and is built and tested for x86-64 (Windows with MSVC, Linux with GCC and Clang) and ARM64 / aarch64 (Linux), see the section Portability. The library is compatible with low-power processors and contains additional functions and modes. The tracking algorithm is developed by ConstantRobotics Sp. z o.o. The library provides tracking of low-contrast and small-sized objects against complex backgrounds. The library contains a description of the C++ class CvTracker. A single instance of the CvTracker class provides tracking of a single object in video. To track several objects simultaneously, several instances of the CvTracker class must be created. The CvTracker library depends on the SDK libraries: VTracker interface library (defines programming interface), which depends on the Frame (video frame structure) and ConfigReader (JSON read/write) libraries, and on OpenMP (part of GCC and MSVC; with Clang the OpenMP runtime libomp must be installed, for example libomp-dev). The library does not depend on OpenCV. Additionally, the CvTracker demo application depends on the SimpleUI (window and drawing), VSourceFile (video file source), VideoRecorderMp4 (video recording) and SimpleFileDialog (file dialog) libraries. The libraries are separate repositories, see the section Build and connect to your project.

CvTracker is general-purpose software: it is not tied to any sector or application, and the developer of a program decides what the program does with it.

Versions

Table 1 - Library versions.

Version	Release date	What’s new
7.2.0	12.12.2022	Version 7.2.0.
8.0.0	25.12.2022	- The calculation speed has been increased. - Tracking stability has been improved. - Added support for YUV, YUYV, UYVY, NV12, RGB, BGR video formats for better tracking stability. - The number of configurable parameters has been reduced. - New control commands added.
8.1.0	10.06.2023	- Added new demo application for Windows and Linux. - Code optimized.
9.0.0	19.12.2023	- Classes names changed. - Added new pixel format support. - Code optimized. - Tracking stability improved. - Programming interface changed. - Set/get params methods thread-safe.
9.0.1	15.01.2024	- Submodules updated.
9.0.2	11.03.2024	- Typos in demo application fixed.
9.0.3	19.03.2024	- VTracker interface updated. - Demo mode fixed. - Frame ID according to source frame ID implemented. - Documentation updated.
9.0.4	21.05.2024	- Submodules updated. - Documentation updated.
9.0.5	07.07.2024	- Submodules updated. - CMake updated. - Files structure updated. - Added examples how to connect compiled version to the CMake project.
9.1.0	03.09.2024	- CUDA support is added. - Fixed move rectangle functions in FREE mode. - OpenCV dependency is excluded (except for demo application and example to provide user interface). - Performance (calculation speed) is improved.
9.2.0	02.12.2024	- Code optimized, calculation speed increased >30% in comparison with v9.1.0. - Add frame buffer size == 1 option to exclude copy frame data into internal buffer. - VSourceOpenCv library excluded from demo application.
9.2.1	09.12.2024	- Missing pixel formats are implemented.
9.3.0	20.02.2025	- Add parameters for object loss detection. - Add *.so file generation for Linux.
9.3.1	05.03.2025	- Fix mutex unlock issue.
9.4.0	14.05.2025	- Performance has been improved. - Code has been optimized for less CPU load (using maximum 2 CPU cores instead of 3 in previous version). - BGR24 and RGB24 pixel formats have been excluded due to less performance for tracking.
9.4.1	15.05.2025	- Optimization of the internal transforms.
9.5.0	17.07.2025	- Object loss detection mechanism was redefined.
9.5.1	09.08.2025	- Fix mistakes in matrices ROI operations.
9.5.2	12.08.2025	- Fix connecting CUDA to benchmark.
9.5.3	17.08.2025	- Add configurable max number of frames in demo mode.
9.5.4	06.01.2026	- Fixed object detection probability output value range.
10.0.0	06.01.2026	- Video tracking algorithm optimized separately for daylight and thermal cameras for better tracking stability. - CUDA for the internal transforms added by default on supported platforms. - Improved processing speed.
10.0.1	03.02.2026	- Fixed initialization of CUSTOM1 and CUSTOM2 parameters <= 0.
10.0.2	16.02.2026	- Updated demo application. - Implemented different processing for thermal and daylight cameras. - Processing speed improved (15% faster version 10.0.1).
10.0.3	26.03.2026	- Reduced calculation time in rectangle auto size mode. - Code optimized. - Bugs fixed.
10.0.4	25.04.2026	- Optimized color channels processing.
10.0.5	31.05.2026	- Fixed slow tracking rectangle drift off a contrasty object under very slow (sub-pixel) motion. - Fixed MOVE_SEARCH_WINDOW / SET_SEARCH_WINDOW_POSITION. - Fixed corrupted tracking when changing search window size during tracking. - Fixed YUYV/UYVY ROI copy one-byte row overrun and a GRAY frame data race.
10.0.6	18.06.2026	- Increased max search window size to 512x512.
10.0.7	16.07.2026	- Performance optimization. - Fixed mistakes in the model update in case of the 512x512 search window.
10.0.8	02.08.2026	- Performance optimization. - Added multithreading for thermal camera video processing (1 color channel used).
10.0.9	12.08.2026	- Added the offline quality harness (annotation and tracking quality evaluation tools). Library code is not affected.
10.0.10	16.09.2026	- Improved tracking of slow sub-pixel motion. Measured on the annotated-video quality harness: RMST +2.3% over 698 paired probes (p = 0.047).
10.1.0	20.09.2026	- Integrated specialized header-only internal transforms with zero heap allocations and direct SSE2/AVX2/NEON SIMD vectorization. - Improved processing of the color channels (U and V) for the daylight camera type. - Strictly enforced square search window dimensions (128x128, 256x256, 512x512) with automatic dimension synchronization. - Improved object loss detection and re-acquisition. - High-throughput execution across full tracking pipeline on x86 Windows, x86 Linux, and ARM Linux with 100% bit-identical numerical precision.
10.2.0	20.09.2026	- Added specialized Scene Tracking mode (TYPE = 2) for full-frame tracking and optical scene / gimbal stabilization. - Full-frame luminance (Y) is automatically downsampled to the configured search window size with zero heap allocations. - Full-frame tracking uses one channel without object size estimation and automatic rectangle sizing, captures the full frame center and reports whole-frame displacement and frame-to-frame velocity. - Added --scene automated verification mode to CvTrackerBenchmark.
11.0.0	10.10.2026	- Scene tracking mode (TYPE = 2) updated, STATIC and INERTIAL modes fixed. - Fixed systematic rightward tracking drift under camera vibration. - Internal transforms improved. - Demo application reworked: the OpenCV dependency is removed. - Security hardening and code review.
11.0.1	15.10.2026	- The repository does not contain sub-repositories: the libraries which CvTracker depends on are separate repositories next to it, the main CMakeLists.txt adds them when CvTracker is the top-level project (see Build stand alone and Connect to your project). - The compiled version of the library (folder output of the build folder) is created for any configuration of Visual Studio (the Debug build failed), the shared library for Linux is linked with the OpenMP runtime, the object files of the libraries are extracted into a separate folder, and the combined libraries are taken from the targets instead of fixed paths. - The test noise of the benchmark application is generated by a deterministic generator. - Documentation and comments updated, the file tree corrected. The tracking results are bit-identical to version 11.0.0.
11.0.2	25.10.2026	- The CMake option CVTRACKER_NATIVE_ARCH (-march=native) is replaced by CVTRACKER_AVX2 (ON by default): the x86-64 Release build uses -march=x86-64-v3 (MSVC: /arch:AVX2), so the library runs on every x86-64 CPU with AVX2 and FMA. The speed is within 1 % of the native build, the results are the same. CVTRACKER_NATIVE_ARCH is still accepted (deprecated).

Library files

The library is supplied as source code or as a compiled library. The user is provided with a set of files in the form of a CMake project (repository). The repository structure is shown below (source code of the library):

CMakeLists.txt ----------------- Main CMake file of the library.
README.md ---------------------- This document.
DemoApplicationManual.md ------- Manual of the demo application.
benchmark ---------------------- Folder with benchmark application.
    CMakeLists.txt ------------- CMake file for the benchmark application.
    main.cpp ------------------- Source code file of the benchmark application.
demo --------------------------- Folder with demo application.
    CMakeLists.txt ------------- CMake file for the demo application.
    main.cpp ------------------- Source code file of the demo application.
example ------------------------ Folder with CvTracker example application.
    CMakeLists.txt ------------- CMake file for the example application.
    main.cpp ------------------- Source code file of the example application.
static ------------------------- Folder with images used by the documentation.
src ---------------------------- Folder with source code of the library.
    CMakeLists.txt ------------- CMake file of the library.
    CvTracker.cpp -------------- C++ implementation file.
    CvTracker.h ---------------- Header file which includes CvTracker class declaration.
    CvTrackerVersion.h --------- Header file which includes version of the library.
    CvTrackerVersion.h.in ------ CMake service file to generate version file.
    Impl ----------------------- Folder with video tracker implementation files.
        CsrmTracker.h ---------- Header file with main video tracker implementation.
        CsrmTracker.cpp -------- Source code file with main video tracker implementation.
        Correlator.h ----------- Header file with video tracker implementation.
        Correlator.cpp --------- Source code file with video tracker implementation.
        CustomFft.h ------------ Header-only library of the internal transforms.

The repository does not contain the libraries which CvTracker depends on, they are separate repositories (see Build stand alone).

Files of the compiled version of the CvTracker library:

include -------------------- Folder with header files.
    CvTracker.h ------------ Main header file of the library.
    CvTrackerVersion.h ----- Header file with library version.
    ConfigReader.h --------- Header file of ConfigReader library.
    ConfigReaderVersion.h -- Header file with ConfigReader library version.
    CustomJson.hpp --------- Compatibility header of ConfigReader library.
    Frame.h ---------------- Header file of Frame library.
    FrameVersion.h --------- Header file with Frame library version.
    VTracker.h ------------- Header file of VTracker interface library.
    VTrackerVersion.h ------ Header file with VTracker interface library version.
lib ------------------------ Folder with libraries.
    CvTracker.lib ---------- CvTracker static library file file for Windows.
    or
    libCvTracker.a --------- CvTracker static library file for Linux.
    libCvTracker.so -------- CvTracker shared library file for Linux.

Key features and capabilities

Table 2 - Key features and capabilities.

Parameter	Value and description
Programming language	C++ (standard C++17). For multithreading processing the library uses OpenMP (part of GCC and MSVC; with Clang the libomp runtime must be installed) to process two color channels in parallel (two threads).
Compatibility with different operating systems	Built and tested on Windows (MSVC) and Linux (GCC, Clang) for x86-64 and on Linux for ARM64 (aarch64), see section Portability.
Maximum size of the tracking rectangle	Half of the search window size, but not more than 256x256 pixels: 64x64 pixels for the 128x128 search window, 128x128 for the 256x256 search window and 256x256 for the 512x512 search window. It is possible to track part of an object if it does not fit into the tracking rectangle. The shape of the tracking rectangle can be any within the minimum and maximum allowable limits.
Minimum size of the tracking rectangle	16x16 pixels. The object can be 2x2 pixels in size for normal tracking.
Minimum object size	2x2 pixels.
Minimum object contrast	5%. Contrast refers to the ratio of the difference between the average brightness of pixels belonging to the object and the average brightness of pixels belonging to the background. The demo application is designed to evaluate this parameter.
Maximum object offset per frame	The library provides tracking of objects as they change their position (change the position of the object center) per video frame. The maximum allowable object displacement per video frame is determined by the search window size, which is set by the user. For the default 256x256 search window this is approximately 110 pixels in any direction; a larger search window (e.g. 512x512) allows a proportionally larger displacement.
Discreteness of object coordinates	1 pixel when estimating the center of an object (center of a tracking rectangle).
Object size estimation	The library estimates the size and position of an object within the tracking rectangle to enable automatic adjustment of its position and size at the user’s command.
Auto adjustment of the tracking rectangle position	The library can automatically adjust the position of the tracking rectangle while tracking an object. This reduces the probability of tracking failure when the object changes its speed or direction quickly. The user can enable/disable this function. The auto adjustment function adds some processing time.
Auto adjustment of the tracking rectangle size	The library can automatically adjust the size of the tracking rectangle while tracking an object. This reduces the probability of tracking failure when the object changes its speed or direction quickly. The user can enable/disable this function.
Object speed estimation	The library calculates the horizontal and vertical components of object speed in video frames (pixels per frame).
Parameter modification	The library allows the user to change parameters of the tracking algorithm even while tracking, except for input pixel format and number of color channels for processing.
Supported pixel formats	Supported pixel formats of input video frames: GRAY, YUV24, YUYV, UYVY, NV12, NV21, YU12 and YV12. For each pixel format the algorithm can use one or more color channels for processing. See section “Supported pixel formats”.
Maximum and minimum video frame sizes to be processed	The maximum size of the video frames for processing is 8192x8192 pixels, the minimum is 240x240 pixels. A frame with a side outside the range 1…8192 pixels is rejected (processFrame(…) returns FALSE), see section Limits and guarantees.
Calculation time	The library performs calculations for each video frame. Calculation speed does not depend on video frame sizes, but depends on library parameters. The main parameters which determine calculation speed are: 1. search window size, 2. input pixel format, 3. number of color channels for processing (1 or 2). The library does not perform any background tasks. The library uses no more than two threads: one thread if multithreading is disabled or one color channel is processed and two threads (OpenMP, one thread per color channel) if multithreading is enabled and two color channels are processed.
Object loss detection	The library automatically detects when an object is lost and switches the algorithm into LOST mode - trajectory prediction mode. When the object detection criteria are met, the library automatically recaptures the object (TRACKING mode). There are some limitations. It is recommended to test this function with the demo application.
Adaptation to object shape and size changes	While tracking an object, the library adapts to object shape, size and brightness changes.
Obstacles processing	If an object is partially (up to 50%) blocked by a barrier, there is no tracking loss. The performance of the library in specific situations can be evaluated with the demo application.
Object search window	The size of the object’s search area is set by the user in the library’s parameters. The library’s tracking algorithm searches for an object in a search area whose center coincides with the center of the tracking rectangle in the previous video frame. The search area must be square: 128x128, 256x256, or 512x512 pixels. Setting either search window width or height automatically sets the companion dimension to the same value to keep the window square.
Tracking algorithm	The tracking algorithm is developed by ConstantRobotics Sp. z o.o.
STOP-FRAME function and compensation for communication delays	The library allows the user to compensate for time delays that occur in communication channels when transmitting object capture commands. The library also allows you to implement a STOP-FRAME mode to assist the user in capturing dynamic objects. The STOP-FRAME function is not only for dynamic objects but also for static objects in vibration scenarios. The duration of the STOP-FRAME mode can be set in the library parameters.

Note: The values in the table are applied to the concept of video frame(s) and the concept of pixel.

Operating principles

Tracking process and operating modes

The video tracker provides the following principle of operation: each video frame without dropping must be send to the tracker for processing regardless of the current tracker operation mode. If the tracker is in FREE mode, the tracker does not perform frame processing (the video frame is only added to the frame buffer), but the processing function must be called by user. In all other modes (TRACKING, LOST, INERTIAL and STATIC) the tracker processes each video frame given to it. Figure 1 shows basic principles of object search on video frames.

Figure 1 - Basic principles of object search. (1 - object image on the current frame, 2 - tracking rectangle calculated after processing of the current frame, 3 - position of the tracking rectangle on the previous frame, 4 - object search window on the current frame relative to the position of the tracking rectangle on the previous frame, 5 - current video frame)

At the moment of object capturing, the rectangular area of the video frame (capture rectangle) specified in the capture parameters (position and size) is taken as the reference image of the object, from which the object model is created. The library then searches for the object in each frame of the video in the search window. The search window is an area of the size set in the parameters with the center coinciding with the calculated center of the tracking rectangle on the previous video frame (or the center of the capture rectangle if the first frame after capture is being processed, the position of the search window can be changed by the user for any video frame). For each processed video frame, the library finds the most probable position of the object in the search window, which is taken as the coordinates of the object (calculated center of the tracking rectangle), and calculates the position and size of the object rectangle (the rectangle describes the size of the object image) in the tracking rectangle, the speed components of the tracking object on the video frames (pixels per frame) and the probability of the object detection. The tracking rectangle also follows objects which move slower than one pixel per frame and does not stay behind them (the coordinates are given with the discreteness of 1 pixel). Figure 1 shows a schematic representation of a video frame (5) that contains an image of a object (1). Assume that on the previous video frame the object was in the area corresponding to area (3), which is the area of the tracking rectangle (the most probable position of the object) in the previous video frame. The library performs object search in the area (4) whose center coincides with the position of the center of the tracking rectangle (3) in the previous video frame.

After capturing an object, the library estimates the position and the size of the object image within the tracking rectangle while the next video frames are processed (object rectangle: objectX, objectY, objectWidth and objectHeight fields of the VTrackerParams class; object mask image: getImage(…) method). This estimate is used to adjust the size and the position of the tracking rectangle automatically, once at the user’s command (ADJUST_RECT_SIZE, ADJUST_RECT_POSITION) or continuously (RECT_AUTO_SIZE, RECT_AUTO_POSITION). Tracker support follow modes:

Table 3 - Tracking algorithm operating modes.

Mode	Description
FREE - free mode.	In this mode, video tracker does not perform any calculations. Video tracker only adds video frames to the frame buffer (if frame buffer size > 1). Conditions for entering FREE mode: 1. Once the video tracker has been initialized. This mode is the default mode. 2. Automatically when the automatic tracking reset criteria are met. 3. After command RESET.
TRACKING - tracking mode.	In this mode the video tracker performs video tracking and updates all calculated (estimated) object parameters. Criteria for entering TRACKING mode: 1. After the CAPTURE command. 2. Automatically from LOST mode when object detection criteria are met.
LOST - object loss mode.	In this mode, the video tracker searches for an object for automatic re-capture (switching to TRACKING mode) and updates it’s coordinates in one of the ways specified in the parameters. LOST mode contains the following additional modes: 0. Tracking rectangle coordinates are not updated (remain the same as before entering LOST mode). 1. The tracking rectangle coordinates are updated based on the components of the object’s speed calculated before going into LOST mode. When the tracking rectangle reaches any edge of the frame, the coordinate update in the corresponding direction stops. 2. The tracking rectangle coordinates are updated based on the components of the speed of objects in the video frames calculated before going into LOST mode. When the tracking reset criteria is met, the tracker switches to FREE mode. Criteria for entering LOST mode: 1. Automatically when object loss is detected. 2. On command from TRACKING mode. 3. On command from INERTIAL mode. 4. On command from STATIC mode.
INERTIAL - inertial tracking mode.	In this mode the video tracker does not search for an object to recapture automatically, but only updates the coordinates of the tracking rectangle based on the velocity components of the objects calculated before. Criteria for entering INERTIAL mode: 1. On command from TRACKING mode. 2. On command from LOST mode. 3. On command from STATIC mode.
STATIC - static mode.	In this mode the video tracker processes each video frame but does not use the result: the tracking rectangle coordinates and the object model remain the same as before going into this mode. This mode is necessary to “freeze” the tracking algorithm for a certain number of frames. For example, if the tracking system is exposed to strong vibrations, it is possible to “freeze” the tracking algorithm until the vibration ends.

Figure 2 shows the operating mode graph and the possible transitions between them. The words “Auto” in figure 2 indicate the ability to change the mode automatically if the relevant criteria are met. The words “Command” indicate the ability to change mode by user’s command.

Figure 2 - Operation modes of the tracking algorithm. (Auto – automatic mode change capability)

Figure 2 shows the graph of operation modes. There are the following conditions for automatic mode changes (word “Auto” in figure 2): 1. Automatic switching from TRACKING to FREE mode is possible only if the tracking rectangle center has touched (coincided in coordinates) any of the video frame edges. This criterion does not depend on the LOST mode option. 2. The automatic switching from TRACKING to LOST mode is possible when an object loss is detected – when the calculated object detection probability falls below the threshold. 3. Automatic switching from LOST to TRACKING mode is possible when an object is detected again after a loss - when the calculated object detection probability exceeds the threshold. 4. Automatic reset of tracking in the LOST mode (switch to FREE mode) is possible when the center of the tracking rectangle touches the edge of the video frame (if the LOST mode option set to 2), as well as when the number of frames specified in the parameters has expired, at which the algorithm is continuously in LOST mode. 5. Automatic reset of tracking in INERTIAL mode (switch to FREE mode) is possible when the center of the tracking rectangle reaches the edge of the frame.

Scene tracking mode (TYPE = 2)

The CvTracker library includes a dedicated Scene Tracking mode (TYPE = 2) which tracks the motion of the whole scene.

Unlike object tracking (TYPE = 0 daylight or TYPE = 1 thermal) where the user selects an isolated object, Scene Tracking mode continuously tracks the full environmental scene as follows:

• Central Inscribed Square & Luminance: From the incoming video frame, a square region is selected inscribed in the center of the frame, with its side equal to 100% of the smaller frame dimension ($\min(W_{\text{frame}}, H_{\text{frame}})$). Only the luminance ($Y$) component of this central square is used. The square is processed at an internal resolution which is set by the search window size (128, 256 or 512): a larger search window gives a higher internal resolution and a longer processing time.
• Fixed Center Capture: In FREE mode, the capture rectangle is permanently locked at the exact center of the video frame and cannot be moved by mouse or manual positioning commands. Upon capture, tracking automatically initiates at the center of the inscribed square.
• Proportional Geometry (Percentages of Inscribed Frame Size):
  • Tracking Rectangle: Occupies 40% of the inscribed square dimension.
  • Search Window: Occupies 80% of the inscribed square dimension.
  • Motion Margin: The tracking rectangle can displace up to 30% of the inscribed square dimension from center before reaching any border of the inscribed square.
• Edge Clamping: As the camera or scene moves, the tracking rectangle and search window displace across the inscribed square. If the tracking rectangle reaches any boundary of the inscribed square, displacement along that respective axis halts (clamped to the boundary).
• Persistent Tracking (No Loss Transition): In Scene Tracking mode, transitions to LOST mode are disabled (mode remains in TRACKING = 1). The tracker continuously follows scene motion without entering target loss timeouts or occlusion state machines.
• Output Coordinate Scaling: All output parameters reported in VTrackerParams (rectX, rectY, rectWidth, rectHeight, searchWindowX, searchWindowY, searchWindowWidth, searchWindowHeight, velX, velY) are mapped back to source video frame coordinates. The output tracking rectangle size represents exactly 40% of the smaller frame dimension, the search window represents 80% of the smaller frame dimension, and velocities (velX, velY) report instantaneous frame-to-frame scene displacement in sensor pixels per frame.

Object loss detection and re-acquisition

The CvTracker library automatically detects object loss, moves the tracking rectangle while the object is lost (according to the LOST mode option) and automatically re-acquires the object once it is found again:

1. Detection Probability: When processing each video frame, the library searches for the object in the search window (128x128, 256x256, or 512x512 pixels, automatically kept square) and calculates the detection probability of the object from 0.0 to 1.0 (detectionProbability parameter in VTrackerParams class).

2. Loss Detection (TRACKING $\to$ LOST):
   • The library detects target loss and transitions from TRACKING mode (mode = 1) to LOST mode (mode = 2) when the detection probability falls below the user-defined loss threshold (custom1 field in VTrackerParams, default 0.1).
   • For the daylight camera type (TYPE = 0), fast changes of the object shape (such as changes of the aspect ratio, or rotations) are recognized so that they do not cause a premature switch to LOST mode.
3. Behavior in LOST Mode:
   • While in LOST mode, the tracking rectangle and the search window stay in place or move with the speed of the object calculated before the loss, according to the LOST mode option (lostModeOption field in VTrackerParams).
   • The lostModeFrameCounter field counts the frames in LOST mode. If the object is not found within the maximum allowed number of frames (maxFramesInLostMode field in VTrackerParams, default 256 frames, valid range 1 to 2048), the tracker terminates the search and switches to FREE mode (mode = 0).
   • The object model is not updated in LOST mode.
4. Object Re-acquisition (LOST $\to$ TRACKING):
   • On every frame in LOST mode, the library searches for the object in the search window.
   • Once the detection probability exceeds the re-acquisition threshold (custom2 field in VTrackerParams, default 0.4), the tracker:
     • Resets the lost-frame counter to zero.
     • Moves the tracking rectangle to the found position of the object.
     • Transitions back to TRACKING mode (mode = 1) and resumes the model update.

Stop-frame function

If a moving object needs to be captured, this can often be difficult for the user because they need to align the tracking rectangle with an object that keeps changing its position on the video frames. In addition, it is difficult to capture a stationary object in case of camera vibration. To help the user capture an object in challenging dynamic environments, the library has a STOP-FRAME function. This function allows the user to stop the video playback and accurately capture an object on a stopped video frame.

The function works as follows: video frames are put to the library for processing frame-by-frame (frame buffer size must be > 1). The library places frames in a ring buffer of the size specified by the user in library parameters. The tracking data contains the identifier (frameId field in VTrackerParams class) of the frame added to the ring buffer (user must assign unique frame ID to each frame) and is transmitted to the control system via a communication channel. The user of the control system sees the video from the cameras on the monitor. Each video frame has its own identifier (frameId field of the Frame object) assigned by the user - user must provide incremental frameId for each new video frame. The user can stop video playback, move the capture rectangle to an object on a stopped video frame and capture. When a capture command is formed, it includes a video frame identifier corresponding to the displayed video frame. When a capture command is received by the library, the object is captured on the frame in the frame buffer according to the identifier specified in the capture command. The video frame on which the object is captured will be some time behind the current video frame from the camera (the number of frames corresponding to the time the user of the control system “stops” the video, with the addition of the delay time in the communication channels). After a capture, the algorithm switches to tracking mode. When processing subsequent frames, the library sequentially processes the frames in the frame buffer, starting with the frame where the object was captured. When a processing method of the library method is called, multiple frames are processed to “catch up” the current video frame. In this way the library “catches up” the current video frame from the camera in a short time and enters normal tracking mode. WARNING: until the library has “caught up” the current video frame, it is not recommended to turn the pan-tilt systems. This may result in incorrect operation of the tracking systems.

Communication channel time delay compensation

When controlling the tracking system remotely (via communication channels), communication delays negatively affect the quality of the object being captured by the user. Video captured by the tracker device is compressed and transmitted to the control system with some delay. When the user captures an object, the generated capture command also arrives at the tracker device with some delay. The capture command contains the coordinates of the capture rectangle center. When capturing a dynamic object, due to time delays in the communication channels, the captured area will not match the object. Figure 2 shows the displacement between capture rectangle and real object position.

Figure 3 - Position error of the capture rectangle.

Figure 3 shows a video frame coming into the tracker device from the camera (left) and a video frame displayed to a user of the control system (right) at the same point of time. Assume that the video frame captured from the camera is simultaneously put to the tracking algorithm and sent to the control system. At time T1, the object was at one frame position. This frame after some time (encoding time + video frame transmission time + decoding time + display time) is displayed to the user, who performs object capture. For the user, the object position corresponds to the time T1. Once an object has been captured, the capture command is sent to the tracking algorithm with some delay (command generation time + command transmission time + command decoding time) (time moment T2). The algorithm captures an object in the current video frame. If the object is moving, the error (horizontally and vertically) in the position of the capture rectangle will be as follows:

\[\Delta X = (T_2 - T_1) \times V_x\] \[\Delta Y = (T_2 - T_1) \times V_y\]

where: ∆X – horizontal position error in pixels; ∆Y – vertical position error in pixels; T1 – the point in time corresponding to the frame displayed to the user; T2 – the point in time corresponding to the current video frame; Vx – the horizontal component of the speed of an object in video frames (pixels per frame); Vy – the vertical component of the speed of the object in the video frames (pixels per frame).

To compensate for errors that occur, a video frame identifier must be included in the capture command. The principle for compensating of time delays in communication channels is identical to that of the STOP-FRAME function.

Supported pixel formats

The library uses input video frames in form of Frame class object which declared in Frame.h (included in VTracker interface class). The library supports only 8 bit color depth RAW pixel formats: GRAY, YUV24, YUYV, UYVY, NV12, NV21, YU12 and YV12. Table 4 shows bytes layout for different pixels formats.

Table 4 - Bytes layout of supported pixel formats. Example of 4x4 pixels image.

YUV24	GRAY
YUYV	UYVY
NV12	NV21
YU12	YV12

CvTracker class description

CvTracker class declaration

CvTracker interface class declared in CvTracker.h file. Class declaration:

class CvTracker : public VTracker
{
public:

    /// Get string of the library version.
    static std::string getVersion();

    /// Class constructor.
    CvTracker();

    /// Class destructor.
    ~CvTracker() override;

    /// The tracker owns its working buffers: copying is not allowed.
    CvTracker(const CvTracker&) = delete;
    CvTracker& operator=(const CvTracker&) = delete;

    /// Init video tracker. All parameters will be set.
    bool initVTracker(VTrackerParams& params) override;

    /// Set video tracker param.
    bool setParam(VTrackerParam id, float value) override;

    /// Get video tracker parameter value.
    float getParam(VTrackerParam id) override;

    /// Get video tracker params (results).
    void getParams(VTrackerParams& params) override;

    /// Execute command.
    bool executeCommand(VTrackerCommand id,
                        float arg1 = 0,
                        float arg2 = 0,
                        float arg3 = 0) override;

    /// Process frame. Must be used for each input video frame.
    bool processFrame(rapidpixel::Frame& frame) override;

    /// Get image of internal surfaces.
    void getImage(int type, rapidpixel::Frame& image) override;

    /// Decode and execute command.
    bool decodeAndExecuteCommand(uint8_t* data, int size) override;
};

How to use the library

The library consists of a few source code files. To use the library, the developer must include the library in the C++ project. The CvTracker.h file contains the declaration of the CvTracker С++ class, which implements the tracking algorithm. The way to use the library:

1. Connect the library files to your project.
2. Create an instance of the CvTracker C++ class. The constructor allocates all working buffers of the tracker (see Limits and guarantees) and throws std::bad_alloc if the memory is not available.
3. Set parameters of the library through calling the setParam(…) method, if necessary. It is thread-safe. Check the returned value.
4. Call the processFrame(…) method to process each video frame. Check the returned value: FALSE means that the video frame was rejected.
5. To perform commands use executeCommand(…) method. It is thread-safe. Check the returned value.
6. The getParams(…) method is used to get the tracking results. It is thread-safe.

The processFrame(…) and getImage(…) methods must be called from one thread at a time and never concurrently with each other, the other public methods can be called from any thread, see Limits and guarantees.

getVersion method

The getVersion() method returns string of current version of CvTracker class. Method declaration:

static std::string getVersion();

Method can be used without CvTracker class instance:

cout << "CvTracker class version: " << CvTracker::getVersion() << endl;

Console output:

CvTracker class version: 11.0.2

initVTracker method

The initVTracker(…) method initializes the video tracker with a set of parameters. Method declaration:

bool initVTracker(VTrackerParams& params) override;

Parameter	Value
params	VTrackerParams class object which contains video tracker parameters. The library initializes all supported parameters listed in the VTrackerParams class. The VTrackerParams class is provided by the VTracker interface class.

Returns: TRUE if the video tracker initialized or FALSE if not. The method resets the tracker (it goes to FREE mode). A parameter value which is out of the allowed range is replaced by the nearest allowed or by the default value, and the corrected value is written back to params (see Table 6 and Table 7). initVTracker(…) must not be called concurrently with processFrame(…), getImage(…) or another initVTracker(…) call.

setParam method

The setParam(…) method sets a new video tracker parameter value. It is a thread-safe method. This means that the setParam(…) method can be safely called from any thread. NOTE: the new parameter value will be applied only when processing the next video frame. Method declaration:

bool setParam(VTrackerParam id, float value) override;

Parameter	Description
id	Parameter ID according to VTrackerParam enum. VTrackerParam enum provided by VTracker interface class. Not all parameters supported by CvTracker.
value	Parameter value. Value depends on parameter ID.

Returns: TRUE if the parameter was accepted or FALSE if not. FALSE is returned if the parameter ID is not supported by the library, if the value is NaN, infinite or its absolute value is greater than 2^30, or if the queue of the parameters which are not applied yet is full (256 entries; the queue is emptied by each processFrame(…) call). The method does not check the range of the other values: an out-of-range value is returned as TRUE and is then clamped or discarded when the parameter is applied before processing the next video frame.

getParam method

The getParam(…) method returns a video tracker parameter value. It is a thread-safe method. This means that the getParam(…) method can be safely called from any thread. Method declaration:

float getParam(VTrackerParam id) override;

Parameter	Description
id	Parameter ID according to VTrackerParam enum. VTrackerParam enum provided by VTracker interface class. Not all parameters supported by CvTracker.

Returns: parameter value or -1 if the parameters not supported by CvTracker.

getParams method

The getParams(…) method returns the video tracker params class and tracking results. It is a thread-safe method. This means that the getParams(…) method can be safely called from any thread. Method declaration:

void getParams(VTrackerParams& params) override;

Parameter	Description
params	Video tracker parameters class object (VTrackerParams class). VTrackerParams class provided by VTracker interface class.

executeCommand method

The executeCommand(…) method executes video tracker command. It is thread-safe executeCommand(…) method. This means that the executeCommand(…) method can be safely called from any thread. Method declaration:

bool executeCommand(VTrackerCommand id, float arg1 = 0, float arg2 = 0, float arg3 = 0) override;

Parameter	Description
id	Command ID according to VTrackerCommand enum. VTrackerCommand enum provided by VTracker interface class.
arg1	First argument. Value depends on command ID (see VTrackerCommand enum description).
arg2	Second argument. Value depends on command ID (see VTrackerCommand enum description).
arg3	Third argument. Value depends on command ID (see VTrackerCommand enum description).

Returns: TRUE if the command was accepted (it is executed when the next video frame is processed) or FALSE if not. FALSE is returned if the command ID is not supported, if an argument which is used by the command is NaN, infinite or has an absolute value greater than 16777216 (2147483520 for the frame ID of the CAPTURE and CAPTURE_PERCENTS commands), or if the queue of the commands which are not executed yet is full (256 entries; the queue is emptied by each processFrame(…) call).

processFrame method

The processFrame(…) method processes video frame. To get actual tracking results use getParams(…) method. Regardless of the operating mode, the tracker compiles frame data into internal buffers. Method declaration:

bool processFrame(rapidpixel::Frame& frame) override;

Parameter

Description

frame

Video frame for processing. Video tracker processes only RAW pixel formats (GRAY, YUV24, YUYV, UYVY, NV12, NV21, YV12, YU12, see Frame class description). The library will associate frameId from Frame object with internal ID which will be used for STOP-FRAME function or time delay compensation (see Stop-frame function). User must provide incremental frameId in Frame object - each new frame must have frameId + 1 from previous. Without it STOP-FRAME function or time delay compensation will not work properly.

The video frame is checked before it is used. The frame must have data (data is not null), a width and a height from 1 to 8192 pixels and size which is not less than the size of the pixel format (width x height for GRAY, x 3 for YUV24, x 2 for YUYV and UYVY, x 1.5 for NV12, NV21, YU12 and YV12). The width of YUYV and UYVY frames and the width and the height of NV12, NV21, YU12 and YV12 frames must be even. If the frame buffer size is greater than 1 (see frameBufferSize), size must be equal to the size of the pixel format.

Returns: TRUE if the video frame was processed or FALSE if not. If video tracker not in TRACKING mode the method will return TRUE. FALSE is returned if the video frame is rejected (the checks above failed or the pixel format is not supported) or the memory for the frame buffer is not available. The tracker is reset (FREE mode) when a frame is rejected, because the tracking can not continue without the frame. The method does not throw exceptions. Thread safety: the method must not be called concurrently with getImage(…), initVTracker(…) or another processFrame(…) call.

getImage method

The getImage(…) method designed to obtain images of internal matrices. Method declaration:

void getImage(int type, rapidpixel::Frame& image) override;

Parameter	Description
type	The type of image to be obtained. The following are available to the user: 0 - reference image of an object. 1 - object mask image. 2 - score map image (distribution of the probability of the object location in the search window).
image	Output frame (see Frame class description). The method will automatically initialize the frame with GRAY pixel format and a size equal to the configured search window (128x128, 256x256 or 512x512; default 256x256) if not properly initialized.

Thread safety: the method reads the internal surfaces which are updated by processFrame(…), so it must not be called concurrently with processFrame(…), initVTracker(…) or another getImage(…) call. For any other value of type the method gives a black image.

decodeAndExecuteCommand method

The decodeAndExecuteCommand(…) method decodes and executes a command on the video tracker side. It is a thread-safe method. This means that the decodeAndExecuteCommand(…) method can be safely called from any thread. The method only decodes commands encoded by the encodeSetParamCommand(…) or encodeCommand(…) methods of the VTracker interface class. Method declaration:

bool decodeAndExecuteCommand(uint8_t* data, int size) override;

Parameter	Description
data	Pointer to input command.
size	Size of command. Must be 11 bytes for SET_PARAM (change parameter command) or 19 bytes for COMMAND (action command).

Returns: TRUE if command decoded (SET_PARAM or COMMAND) and accepted by the setParam(…) or executeCommand(…) method (the command is executed when the next video frame is processed). FALSE is returned if data is null, the size is less than 11 bytes, the data is not a valid command (protocol version, command type, size or command ID mismatch) or the command was not accepted (see setParam(…) and executeCommand(…)).

encodeSetParamCommand method of VTracker class

The encodeSetParamCommand(…) static method designed to encode command to change any parameter in remote video tracker. To control video tracker remotely, the developer has to design his own protocol and according to it encode the command and deliver it over the communication channel. To simplify this, the VTracker class contains static methods for encoding the control command. The VTracker class provides two types of commands: a parameter change command (SET_PARAM) and an action command (COMMAND). encodeSetParamCommand(…) designed to encode SET_PARAM command. Method declaration:

static void encodeSetParamCommand(uint8_t* data, int& size, VTrackerParam id, float value);

Parameter	Description
data	Pointer to data buffer for encoded command. Must not be nullptr, must have size >= 11.
size	Size of encoded data. Will be 11 bytes (0 if data is nullptr).
id	Parameter ID according to VTrackerParam enum.
value	Parameter value. Value depends on parameter ID (see VTrackerParam enum description).

encodeSetParamCommand(…) is static and used without VTracker class instance. This method used on client side (control system). Command encoding example:

uint8_t data[1024];
int size = 0;
float outValue = static_cast<float>(rand() % 20);
VTracker::encodeSetParamCommand(
        data, size, VTrackerParam::MULTIPLE_THREADS, outValue);

encodeCommand method of VTracker class

The encodeCommand(…) static method designed to encode command for remote video tracker. To control video tracker remotely, the developer has to design his own protocol and according to it encode the command and deliver it over the communication channel. To simplify this, the VTracker class contains static methods for encoding the control command. The VTracker class provides two types of commands: a parameter change command (SET_PARAM) and an action command (COMMAND). encodeCommand(…) designed to encode COMMAND (action command). Method declaration:

static void encodeCommand(uint8_t* data, int& size, VTrackerCommand id, float arg1 = 0, float arg2 = 0, float arg3 = 0);

Parameter	Description
data	Pointer to data buffer for encoded command. Must not be nullptr, must have size >= 19.
size	Size of encoded data. Will be 19 bytes (0 if data is nullptr).
id	Command ID according to VTrackerCommand enum.
arg1	First argument. Value depends on command ID (see VTrackerCommand enum description).
arg2	Second argument. Value depends on command ID (see VTrackerCommand enum description).
arg3	Third argument. Value depends on command ID (see VTrackerCommand enum description).

encodeCommand(…) is static and used without VTracker class instance. This method used on client side (control system). Command encoding example:

uint8_t data[1024];
int size = 0;
float arg1 = static_cast<float>(rand() % 20);
float arg2 = static_cast<float>(rand() % 20);
float arg3 = static_cast<float>(rand() % 20);
VTracker::encodeCommand(data, size, VTrackerCommand::CAPTURE, arg1, arg2, arg3);

decodeCommand method of VTracker class

The decodeCommand(…) static method designed to decode command on video tracker side (edge device) which encoded by encodeSetParamCommand(…) or encodeCommand(…) methods. Method declaration:

static int decodeCommand(uint8_t* data, int size, VTrackerParam& paramId, VTrackerCommand& commandId, float& value1, float& value2, float& value3);

Parameter	Description
data	Pointer to input command. Must not be nullptr.
size	Size of command. Must be exactly 11 bytes for SET_PARAM or exactly 19 bytes for COMMAND.
paramId	Parameter ID according to VTrackerParam enum. After decoding SET_PARAM command the method will return parameter ID.
commandId	Command ID according to VTrackerCommand enum. After decoding COMMAND the method will return command ID.
value1	Parameter value (SET_PARAM) or first command argument (COMMAND).
value2	Second command argument. After decoding COMMAND.
value3	Third command argument. After decoding COMMAND.

Returns: 0 - in case decoding COMMAND, 1 - in case decoding SET_PARAM command or -1 in case errors: null pointer, wrong size, wrong version (the first two numbers of the library version must be equal to the version of the library which encoded the command), unknown data type or unknown command / parameter ID (IDs must be inside the range of VTrackerCommand enum / VTrackerParam enum). In case of an error paramId and commandId are not changed. The values of the arguments (value1, value2, value3) are not checked: the video tracker must validate them before use.

decodeCommand(…) is static and used without VTracker class instance. Command decoding example:

// Encode command.
uint8_t data[1024];
int size = 0;
float arg1 = static_cast<float>(rand() % 20);
float arg2 = static_cast<float>(rand() % 20);
float arg3 = static_cast<float>(rand() % 20);
VTracker::encodeCommand(data, size, VTrackerCommand::CAPTURE, arg1, arg2, arg3);

// Decode command.
VTrackerCommand commandId;
VTrackerParam paramId;
float inArg1 = static_cast<float>(rand() % 20);
float inArg2 = static_cast<float>(rand() % 20);
float inArg3 = static_cast<float>(rand() % 20);
if (VTracker::decodeCommand(data, size, paramId, commandId, inArg1, inArg2, inArg3) != 0)
{
    std::cout << "Command not decoded" << std::endl;
    return false;
}

Data structures

VTrackerCommand enum

VTrackerCommand enum describes commands supported by VTracker interface class. VTrackerCommand enum declared in VTracker.h file. Enum declaration:

enum class VTrackerCommand
{
    /// Object capture. Arguments:
    /// arg1 - Capture rectangle center X coordinate. -1 - default coordinate.
    /// arg2 - Capture rectangle center Y coordinate. -1 - default coordinate.
    /// arg3 - frame ID. -1 - Capture on last frame.
    CAPTURE = 1,
    /// Object capture command. Arguments:
    /// arg1 - Capture rectangle center X coordinate, percent of frame width.
    /// arg2 - Capture rectangle center Y coordinate, percent of frame height.
    CAPTURE_PERCENTS,
    /// Reset command. No arguments.
    RESET,
    /// Set INERTIAL mode. No arguments.
    SET_INERTIAL_MODE,
    /// Set LOST mode. No arguments.
    SET_LOST_MODE,
    /// Set STATIC mode. No arguments.
    SET_STATIC_MODE,
    /// Adjust tracking rectangle size automatically once. No arguments.
    ADJUST_RECT_SIZE,
    /// Adjust tracking rectangle position automatically once. No arguments.
    ADJUST_RECT_POSITION,
    /// Move tracking rectangle (change position). Arguments:
    /// arg1 - add to X coordinate, pixels. 0 - no change.
    /// arg2 - add to Y coordinate, pixels. 0 - no change.
    MOVE_RECT,
    /// Set tracking rectangle position in FREE mode. Arguments:
    /// arg1 - Rectangle center X coordinate.
    /// arg2 - Rectangle center Y coordinate.
    SET_RECT_POSITION,
    /// Set tracking rectangle position in FREE mode in percents of frame size.
    /// Arguments:
    /// arg1 - Rectangle center X coordinate, percent of frame width.
    /// arg2 - Rectangle center Y coordinate, percent of frame height.
    SET_RECT_POSITION_PERCENTS,
    /// Move search window (change position). Arguments:
    /// arg1 - add to X coordinate, pixels. 0 - no change.
    /// arg2 - add to Y coordinate, pixels. 0 - no change.
    MOVE_SEARCH_WINDOW,
    /// Set search window position. Arguments:
    /// arg1 - Search window center X coordinate.
    /// arg2 - Search window center Y coordinate.
    SET_SEARCH_WINDOW_POSITION,
    /// Set search window position in percent of frame size. Arguments:
    /// arg1 - Search window center X coordinate, percent of frame width.
    /// arg2 - Search window center Y coordinate, percent of frame height.
    SET_SEARCH_WINDOW_POSITION_PERCENTS,
    /// Change tracking rectangle size. Arguments:
    /// arg1 - horizontal size add, pixels.
    /// arg2 - vertical size add, pixels.
    CHANGE_RECT_SIZE
};

Table 5 - Video tracker commands description and description of necessary arguments for executeCommand(…) and encodeCommand(…) methods described.

Command	Description
CAPTURE	Object capture. Arguments: arg1 - Capture rectangle center X coordinate. -1 - default coordinate, arg2 - Capture rectangle center Y coordinate. -1 - default coordinate, arg3 - frame ID. -1 - Capture on last frame.
CAPTURE_PERCENTS	Object capture command. Arguments: arg1 - Capture rectangle center X coordinate, percent of frame width, arg2 - Capture rectangle center Y coordinate, percent of frame height, arg3 - frame ID. -1 - Capture on last frame.
RESET	Reset command. No arguments.
SET_INERTIAL_MODE	Set INERTIAL mode. No arguments.
SET_LOST_MODE	Set LOST mode. No arguments.
SET_STATIC_MODE	Set STATIC mode. No arguments.
ADJUST_RECT_SIZE	Adjust tracking rectangle size automatically once. No arguments.
ADJUST_RECT_POSITION	Adjust tracking rectangle position automatically once. No arguments.
MOVE_RECT	Move tracking rectangle (change position). Arguments: arg1 - add to X coordinate, pixels. 0 - no change, arg2 - add to Y coordinate, pixels. 0 - no change.
SET_RECT_POSITION	Set tracking rectangle position in FREE mode. Arguments: arg1 - Rectangle center X coordinate, arg2 - Rectangle center Y coordinate.
SET_RECT_POSITION_PERCENTS	Set tracking rectangle position in FREE mode in percent of frame size. Arguments: arg1 - Rectangle center X coordinate, percent of frame width, arg2 - Rectangle center Y coordinate, percent of frame height.
MOVE_SEARCH_WINDOW	Move search window (change position). Arguments: arg1 - add to X coordinate, pixels. 0 - no change, arg2 - add to Y coordinate, pixels. 0 - no change.
SET_SEARCH_WINDOW_POSITION	Set search window position. Arguments: arg1 - Search window center X coordinate, arg2 - Search window center Y coordinate.
SET_SEARCH_WINDOW_POSITION_PERCENTS	Set search window position in percent of frame size. Arguments: arg1 - Search window center X coordinate, percent of frame width, arg2 - Search window center Y coordinate, percent of frame height.
CHANGE_RECT_SIZE	Change tracking rectangle size. Arguments: arg1 - horizontal size add, pixels, arg2 - vertical size add, pixels.

VTrackerParam enum

VTrackerParam enum describes parameters supported by VTracker interface. VTrackerParam enum declared in VTracker.h file. Enum declaration:

enum class VTrackerParam
{
    /// Width of search window, pixels. Set by user.
    SEARCH_WINDOW_WIDTH = 1,
    /// Height of search window, pixels. Set by user.
    SEARCH_WINDOW_HEIGHT,
    /// Tracking rectangle width, pixels. Set by user or can be changed by
    /// tracking algorithm if rectAutoSize == true.
    RECT_WIDTH,
    /// Tracking rectangle height, pixels. Set by user or can be changed by
    /// tracking algorithm if rectAutoSize == true.
    RECT_HEIGHT,
    /// Option for LOST mode. Parameter that defines the behavior of the
    /// tracking  algorithm in LOST mode. Default is 0. Possible values:
    /// 0. In LOST mode, the coordinates of the center of the  tracking
    ///    rectangle are not updated and remain the  same as before entering
    ///    LOST mode.
    /// 1. The coordinates of the center of the tracking  rectangle are updated
    ///    based on the components  of the object’s speed calculated before
    ///    going into  LOST mode. If the tracking rectangle “touches” the  edge
    ///    of the video frame, the coordinate updating  for this component
    ///    (horizontal or vertical) will stop.
    /// 2. The coordinates of the center of the tracking  rectangle are
    ///    updated based on the components  of the object’s speed calculated
    ///    before going into  LOST mode. The tracking is reset if the center of 
    ///    the tracking rectangle touches any of the edges of  the video frame.
    LOST_MODE_OPTION,
    /// Size of frame buffer (number of frames to store). Set by user.
    FRAME_BUFFER_SIZE,
    /// Maximum number of frames in LOST mode to auto reset of algorithm. Set
    /// by user.
    MAX_FRAMES_IN_LOST_MODE,
    /// Use tracking rectangle auto size flag: 0 - no use, 1 - use. Set by user.
    RECT_AUTO_SIZE,
    /// Use tracking rectangle auto position: 0 - no use, 1 - use. Set by user.
    RECT_AUTO_POSITION,
    /// Use multiple threads for calculations. 0 - one thread, 1 - multiple. Set
    /// by user.
    MULTIPLE_THREADS,
    /// Number of channels for processing. E.g., number of color channels. Set
    /// by user.
    NUM_CHANNELS,
    /// Camera type: 0 - daylight, 1 - thermal. Set by user.
    TYPE,
    /// Custom parameter. Depends on implementation.
    CUSTOM_1,
    /// Custom parameter. Depends on implementation.
    CUSTOM_2,
    /// Custom parameter. Depends on implementation.
    CUSTOM_3
};

Table 6 - Video tracker params description. Some params may be unsupported by particular video tracker.

Parameter	Access	Description
SEARCH_WINDOW_WIDTH	read / write	Width of search window, pixels. Set by user. Must have values 128, 256 or 512 (default 256), any other value is ignored. Setting this parameter automatically sets SEARCH_WINDOW_HEIGHT to the same value to keep the window square. The size can be changed only in FREE mode (a change in another mode is ignored).
SEARCH_WINDOW_HEIGHT	read / write	Height of search window, pixels. Set by user. Must have values 128, 256 or 512 (default 256), any other value is ignored. Setting this parameter automatically sets SEARCH_WINDOW_WIDTH to the same value to keep the window square. The size can be changed only in FREE mode (a change in another mode is ignored).
RECT_WIDTH	read / write	Tracking rectangle width, pixels. Set by user or can be changed by tracking algorithm if RECT_AUTO_SIZE == 1. Valid range 16 to 256 pixels (default 72), but not more than half of SEARCH_WINDOW_WIDTH (so 64 pixels maximum for a 128 pixel wide search window, 128 for 256 and 256 for 512). Values outside the range are clamped.
RECT_HEIGHT	read / write	Tracking rectangle height, pixels. Set by user or can be changed by tracking algorithm if RECT_AUTO_SIZE == 1. Valid range 16 to 256 pixels (default 72), but not more than half of SEARCH_WINDOW_HEIGHT (so 64 pixels maximum for a 128 pixel high search window, 128 for 256 and 256 for 512). Values outside the range are clamped.
LOST_MODE_OPTION	read / write	Option for LOST mode. Parameter that defines the behavior of the tracking algorithm in LOST mode. Default is 0. Possible values: 0. In LOST mode, the coordinates of the center of the tracking rectangle are not updated and remain the same as before entering LOST mode. 1. The coordinates of the center of the tracking rectangle are updated based on the components of the object’s speed calculated before going into LOST mode. If the tracking rectangle “touches” the edge of the video frame, the coordinate updating for this component (horizontal or vertical) will stop. 2. The coordinates of the center of the tracking rectangle are updated based on the components of the object’s speed calculated before going into LOST mode. The tracking is reset if the center of the tracking rectangle touches any of the edges of the video frame.
FRAME_BUFFER_SIZE	read / write	Size of frame buffer (number of frames to store). Set by user. By default 1. Minimum value is 1 and maximum is 512. When frame buffer size is 1 the library doesn’t copy data to frame buffer. It reduces CPU load on low-power platforms.
MAX_FRAMES_IN_LOST_MODE	read / write	Maximum number of frames in LOST mode to auto reset of algorithm. Set by user. Valid range 1 to 2048 (default 256), a value outside the range is ignored.
RECT_AUTO_SIZE	read / write	Use tracking rectangle auto size flag: 0 - no use, 1 - use. Set by user.
RECT_AUTO_POSITION	read / write	Use tracking rectangle auto position: 0 - no use, 1 - use. Set by user.
MULTIPLE_THREADS	read / write	Use multiple threads for calculations. 0 - one thread, 1 - multiple. Set by user. When set to 1 and two color channels are processed, each color channel is processed in its own thread (two threads in total, OpenMP). If one color channel is processed, one thread is used. For the thermal camera type, the scene tracking type and GRAY frames the library resets this parameter to 0. When set to 0 all processing runs in the calling thread.
NUM_CHANNELS	read / write	Number of channels for processing. E.g., number of color channels. Set by user. The tracker can process 1 or 2 channels. First channel is GRAY (Y channel) channel. Second channel is color channel (combination of U and V channels). One channel is used usually for video from thermal cameras.
TYPE	read / write	Mode / camera type: 0 - daylight camera, 1 - thermal camera, 2 - scene tracking. The video tracking algorithm has separate optimizations for daylight and thermal cameras, and a specialized scene tracking mode for optical scene stabilization. Type can be changed at any time. - 0 (Daylight): Processes the Y channel and, if NUM_CHANNELS == 2, the color channel (combination of U and V), the two channels can be processed in two threads. - 1 (Thermal): Processes one channel (Y). Automatically sets NUM_CHANNELS == 1 and reports MULTIPLE_THREADS == 0. - 2 (Scene Tracking): Scene tracking and optical stabilization. Tracks the square inscribed in the center of the frame (side equal to 100% of the smaller frame dimension). The tracking rectangle occupies 40% of this inscribed dimension and the search area occupies 80%. When the tracking rectangle touches any boundary of the inscribed square, displacement along that axis is clamped. Transition to LOST mode is disabled. Output parameters are scaled to source frame coordinates. Automatically forces NUM_CHANNELS == 1, MULTIPLE_THREADS == 0, RECT_AUTO_SIZE == 0, and RECT_AUTO_POSITION == 0.
CUSTOM_1	read / write	Object loss threshold from 0 to 1. Default value 0.1. If detection probability (detectionProbability field in VTrackerParams class) for current frame less than threshold the library will switch to LOST mode. See Object loss detection and re-acquisition section.
CUSTOM_2	read / write	Object detection threshold from 0 to 1. Default value 0.4. If detection probability (detectionProbability field in VTrackerParams class) for current frame more than threshold the library will switch to TRACKING mode from LOST mode. See Object loss detection and re-acquisition section.
CUSTOM_3	read / write	Not used in the CvTracker library.

VTrackerParams class description

VTrackerParams class declaration

VTrackerParams class used for video tracker initialization (initVTracker(…) method) or to get all actual params and tracking results (getParams() method). Also VTrackerParams provide structure to write/read params from JSON files (JSON_READABLE macro, see ConfigReader class description) and provides methods to encode and decode params. Class declaration:

class VTrackerParams
{
public:
    /// Tracker mode index: 0 - FREE, 1 - TRACKING, 2 - LOST, 3 - INERTIAL, 4 - STATIC.
    /// Set by video tracker according to processing results or after command
    /// execution.
    int mode{0};
    /// Tracking rectangle horizontal center position. Calculated by tracking
    /// algorithm.
    int rectX{0};
    /// Tracking rectangle vertical center position. Calculated by tracking
    /// algorithm.
    int rectY{0};
    /// Tracking rectangle width, pixels. Set by user or can be changed by
    /// tracking algorithm if rectAutoSize == true.
    int rectWidth{72};
    /// Tracking rectangle height, pixels. Set by user or can be changed by
    /// tracking algorithm if rectAutoSize == true.
    int rectHeight{72};
    /// Estimated horizontal position of object center, pixels. Calculated by
    /// video tracker.
    int objectX{0};
    /// Estimated vertical position of object center, pixels.Calculated by
    /// video tracker.
    int objectY{0};
    /// Estimated object width, pixels. Calculated by video tracker.
    int objectWidth{72};
    /// Estimated object height, pixels. Calculated by video tracker.
    int objectHeight{72};
    /// Frame counter in LOST mode. After switching in LOST mode the video
    /// tracker start counting from 0. After switching to another mode from
    /// LOST mode the video tracker will reset this counter.
    int lostModeFrameCounter{0};
    /// Counter for processed frames after capture object. After reset tracking
    /// the video tracker will reset counter.
    int frameCounter{0};
    /// Width of processed video frame. Set by video tracker after processing
    /// first video frame.
    int frameWidth{0};
    /// Height of processed video frame. Set by video tracker after processing
    /// first video frame.
    int frameHeight{0};
    /// Width of search window, pixels. Set by user.
    int searchWindowWidth{256};
    /// Height of search window, pixels. Set by user.
    int searchWindowHeight{256};
    /// Horizontal position of search window center. This position will be used
    /// for next video frame. Usually it coincides with data tracking rectangle
    /// center but can be set by user to move search window for new video frame.
    int searchWindowX{0};
    /// Vertical position of search window center. This position will be used
    /// for next video frame. Usually it coincides with data tracking rectangle
    /// center but can be set by user to move search window for new video frame.
    int searchWindowY{0};
    /// Option for LOST mode. Parameter that defines the behavior of the
    /// tracking algorithm in LOST mode. Default is 0. Possible values:
    /// 0. In LOST mode, the coordinates of the center of the tracking
    ///    rectangle are not updated and remain the same as before entering
    ///    LOST mode.
    /// 1. The coordinates of the center of the tracking rectangle are updated
    ///    based on the components of the object's speed calculated before
    ///    going into LOST mode. If the tracking rectangle "touches" the edge
    ///    of the video frame, the coordinate updating for this component
    ///    (horizontal or vertical) will stop.
    /// 2. The coordinates of the center of the tracking rectangle are
    ///    updated based on the components of the object's speed calculated
    ///    before going into LOST mode. The tracking is reset if the center of
    ///    the tracking rectangle touches any of the edges of the video frame.
    int lostModeOption{0};
    /// Size of frame buffer (number of frames to store). Set by user.
    int frameBufferSize{128};
    /// Maximum number of frames in LOST mode to auto reset of algorithm. Set
    /// by user.
    int maxFramesInLostMode{128};
    /// ID of last processed frame in frame buffer. Set by video tracker.
    int processedFrameId{0};
    /// ID of last added frame to frame buffer. Set by video tracker.
    int frameId{0};
    /// Horizontal velocity of object on video frames (pixel/frame). Calculated
    /// by video tracker.
    float velX{0.0f};
    /// Vertical velocity of object on video frames (pixel/frame). Calculated
    /// by video tracker.
    float velY{0.0f};
    /// Estimated probability of object detection. Calculated by video tracker.
    float detectionProbability{0.0f};
    /// Use tracking rectangle auto size flag: false - no use, true - use. Set
    /// by user.
    bool rectAutoSize{false};
    /// Use tracking rectangle auto position: false - no use, true - use. Set
    /// by user.
    bool rectAutoPosition{false};
    /// Use multiple threads for calculations. Set by user.
    bool multipleThreads{false};
    /// Number of channels for processing. E.g., number of color channels. Set
    /// by user.
    int numChannels{3};
    /// Camera type: 0 - daylight, 1 - thermal, 2 - scene tracking. Set by user.
    int type{0};
    /// Processing time for last frame, mks. Calculated by video tracker.
    int processingTimeMks{0};
    /// Custom parameter. Depends on implementation.
    float custom1{0.0f};
    /// Custom parameter. Depends on implementation.
    float custom2{0.0f};
    /// Custom parameter. Depends on implementation.
    float custom3{0.0f};

    JSON_READABLE(VTrackerParams, rectWidth, rectHeight, searchWindowWidth,
                  searchWindowHeight, lostModeOption, frameBufferSize,
                  maxFramesInLostMode, rectAutoSize, rectAutoPosition,
                  multipleThreads, numChannels, type, custom1, custom2, custom3)

    /// Default constructor.
    VTrackerParams() = default;

    /// Copy constructor.
    VTrackerParams(const VTrackerParams& src) = default;

    /// operator =
    VTrackerParams& operator= (const VTrackerParams& src) = default;

    /// Encode params.
    bool encode(uint8_t* data, int bufferSize,
                int& size, VTrackerParamsMask* mask = nullptr);

    /// Decode params.
    bool decode(uint8_t* data, int dataSize);
};

Table 7 - VTrackerParams class fields description.

Field	Type	Description
mode	int	Tracker mode index: 0 - FREE. 1 - TRACKING. 2 - LOST. 3 - INERTIAL. 4 - STATIC. Set by video tracker according to processing results or after command execution.
rectX	int	Tracking rectangle horizontal center position, pixels. Calculated by tracking algorithm.
rectY	int	Tracking rectangle vertical center position, pixels. Calculated by tracking algorithm.
rectWidth	int	Tracking rectangle width, pixels. Set by user or can be changed by tracking algorithm if rectAutoSize == true.
rectHeight	int	Tracking rectangle height, pixels. Set by user or can be changed by tracking algorithm if rectAutoSize == true.
objectX	int	Estimated horizontal position of object center, pixels. Calculated by video tracker.
objectY	int	Estimated vertical position of object center, pixels. Calculated by video tracker.
objectWidth	int	Estimated object width, pixels. Calculated by video tracker.
objectHeight	int	Estimated object height, pixels. Calculated by video tracker.
lostModeFrameCounter	int	Frame counter in LOST mode. After switching in LOST mode the video tracker start counting from 0. After switching to another mode from LOST mode the video tracker will reset this counter.
frameCounter	int	Counter for processed frames after capture object. After reset tracking the video tracker will reset counter.
frameWidth	int	Width of processed video frame. Set by video tracker after processing first video frame.
frameHeight	int	Height of processed video frame. Set by video tracker after processing first video frame.
searchWindowWidth	int	Width of search window, pixels. Set by user. Must have values 128, 256 or 512 (default 256). Automatically synchronized with searchWindowHeight to ensure a square window.
searchWindowHeight	int	Height of search window, pixels. Set by user. Must have values 128, 256 or 512 (default 256). Automatically synchronized with searchWindowWidth to ensure a square window.
searchWindowX	int	Horizontal position of search window center. This position will be used for next video frame. Usually it coincides with data tracking rectangle center but can be set by user to move search window for new video frame.
searchWindowY	int	Vertical position of search window center. This position will be used for next video frame. Usually it coincides with data tracking rectangle center but can be set by user to move search window for new video frame.
lostModeOption	int	Option for LOST mode. Parameter that defines the behavior of the tracking algorithm in LOST mode. Default is 0. Possible values: 0. In LOST mode, the coordinates of the center of the tracking rectangle are not updated and remain the same as before entering LOST mode. 1. The coordinates of the center of the tracking rectangle are updated based on the components of the object’s speed calculated before going into LOST mode. If the tracking rectangle “touches” the edge of the video frame, the coordinate updating for this component (horizontal or vertical) will stop. 2. The coordinates of the center of the tracking rectangle are updated based on the components of the object’s speed calculated before going into LOST mode. The tracking is reset if the center of the tracking rectangle touches any of the edges of the video frame.
frameBufferSize	int	Size of frame buffer (number of frames to store). Set by user. By default 1. Minimum value is 1 and maximum is 512. When frame buffer size is 1 the library doesn’t copy data to frame buffer. It reduces CPU load on low-power platforms. The frame buffer memory is allocated when the first video frame (or a video frame of another size) is processed and is equal to frameBufferSize x the size of the video frame (see Limits and guarantees).
maxFramesInLostMode	int	Maximum number of frames in LOST mode to auto reset of algorithm. Set by user. Valid range 1 to 2048, default 256 (a value outside the range is replaced by the default in initVTracker(…) and ignored by setParam(…)).
processedFrameId	int	ID of last processed frame in frame buffer. Set by video tracker.
frameId	int	ID of last added frame to frame buffer. Set by video tracker and has the same value as input frameId from Frame object.
velX	float	Horizontal velocity of object on video frames (pixel/frame). Calculated by video tracker.
velY	float	Vertical velocity of object on video frames (pixel/frame). Calculated by video tracker.
detectionProbability	float	Estimated probability of object detection. Calculated by video tracker.
rectAutoSize	bool	Use tracking rectangle auto size flag: false - no use, true - use.
rectAutoPosition	bool	Use tracking rectangle auto position: false - no use, true - use.
multipleThreads	bool	Use multiple threads for calculations. Set by user. When true and two color channels are processed, each color channel is processed in its own thread (two threads in total, OpenMP). If one color channel is processed, one thread is used. For the thermal camera type, the scene tracking type and GRAY frames the library resets this parameter to false. When false all processing runs in the calling thread.
numChannels	int	Number of channels for processing. E.g., number of color channels. Set by user. The tracker can process 1 or 2 channels. First channel is GRAY (Y channel) channel. Second channel is color channel (combination of U and V channels). One channel is used usually for video from thermal cameras.
type	int	Mode / camera type: 0 - daylight camera, 1 - thermal camera, 2 - scene tracking (optical scene stabilization). See detailed description in Table 6.
processingTimeMks	int	Dual-purpose field, microseconds. As result (output): processing time of the last processFrame(…) call. Calculated by video tracker. As setting (input, applied on initVTracker(…) only; the VTrackerParam enum has no ID for this field, so setParam(…) can’t change it): maximum calculation-time budget for a single processFrame(…) call. When the frame buffer holds several frames accumulated during the STOP-FRAME function (see frameBufferSize), the tracker processes this backlog frame by frame and stops as soon as the budget is exceeded, resuming on the next processFrame(…) call. This bounds the delay of one processFrame(…) call so it does not block while catching up on the whole backlog. Set by user. By default 0, which keeps the internal default limit of 30 ms; any positive value overrides it (a value below 1000 mks is clamped to a 1 ms minimum).
custom1	float	Object loss threshold from 0 to 1. Default value 0.1. If detection probability for current frame less than threshold the library will switch to LOST mode. See Object loss detection and re-acquisition section.
custom2	float	Object detection threshold from 0 to 1. Default value 0.4. If detection probability for current frame more than threshold the library will switch to TRACKING mode from LOST mode. See Object loss detection and re-acquisition section.
custom3	float	Not used in the CvTracker library.

Serialize video tracker params

VTrackerParams class provides method encode(…) to serialize video tracker params (fields of VTrackerParams class, see Table 7). Serialization of params necessary in case when you need to send params via communication channels. Method provides options to exclude particular parameters from serialization. To do this method inserts binary mask (5 bytes) where each bit represents particular parameter and decode(…) method recognizes it. The encoded data has the following layout: 1 byte data type (0x02), 2 bytes of the library version (major and minor numbers: data encoded by a library with other major or minor number is rejected by decode(…)), 5 bytes of the mask and the fields selected by the mask in the order of Table 7. Integer and float fields are stored as 4 bytes in the byte order of the host (little endian, see Portability), boolean fields as 1 byte (0 or 1). The first field (mode) is the most significant bit of the first mask byte. Method declaration:

bool encode(uint8_t* data, int bufferSize, int& size, VTrackerParamsMask* mask = nullptr);

Parameter	Value
data	Pointer to data buffer. Must not be nullptr.
bufferSize	Data buffer size. Must be at least 135 bytes (size of all fields), otherwise the method returns FALSE regardless of the mask.
size	Size of encoded data. Set to 0 if the method returns FALSE.
mask	Parameters mask - pointer to VTrackerParamsMask structure. VTrackerParamsMask (declared in VTracker.h file) determines flags for each field (parameter) declared in VTrackerParams class. If the pointer is nullptr all fields are encoded. If the user wants to exclude a particular parameter from serialization, he should clear (set to false) the corresponding flag in the VTrackerParamsMask structure (all flags are true by default).

Returns: TRUE if data encoded (serialized) or FALSE if not (data is nullptr or the buffer is too small).

VTrackerParamsMask structure declaration:

typedef struct VTrackerParamsMask
{
    bool mode{true};
    bool rectX{true};
    bool rectY{true};
    bool rectWidth{true};
    bool rectHeight{true};
    bool objectX{true};
    bool objectY{true};
    bool objectWidth{true};
    bool objectHeight{true};
    bool lostModeFrameCounter{true};
    bool frameCounter{true};
    bool frameWidth{true};
    bool frameHeight{true};
    bool searchWindowWidth{true};
    bool searchWindowHeight{true};
    bool searchWindowX{true};
    bool searchWindowY{true};
    bool lostModeOption{true};
    bool frameBufferSize{true};
    bool maxFramesInLostMode{true};
    bool processedFrameId{true};
    bool frameId{true};
    bool velX{true};
    bool velY{true};
    bool detectionProbability{true};
    bool rectAutoSize{true};
    bool rectAutoPosition{true};
    bool multipleThreads{true};
    bool numChannels{true};
    bool type{true};
    bool processingTimeMks{true};
    bool custom1{true};
    bool custom2{true};
    bool custom3{true};
} VTrackerParamsMask;

Example without parameters mask:

// Prepare random parameters.
VTrackerParams in;
in.mode = rand() % 255;
in.rectX = rand() % 255;
in.rectY = rand() % 255;

// Encode data.
uint8_t data[1024];
int size = 0;
if (!in.encode(data, 1024, size))
{
    std::cout << "Can't encode data" << std::endl;
    return false;
}
std::cout << "Encoded data size: " << size << " bytes" << std::endl;

Example with parameters mask:

// Prepare random parameters.
VTrackerParams in;
in.mode = rand() % 255;
in.rectX = rand() % 255;
in.rectY = rand() % 255;

// Prepare mask.
VTrackerParamsMask mask;
mask.mode = true;
mask.rectX = false;
mask.rectY = true;

// Encode data.
uint8_t data[1024];
int size = 0;
if (!in.encode(data, 1024, size, &mask))
{
    std::cout << "Can't encode data" << std::endl;
    return false;
}
std::cout << "Encoded data size: " << size << " bytes" << std::endl;

Deserialize video tracker params

VTrackerParams class provides method decode(…) to deserialize params (fields of VTrackerParams class, see Table 7). Deserialization of params necessary in case when you need to receive params via communication channels. Method automatically recognizes which parameters were serialized by encode(…) method. Method declaration:

bool decode(uint8_t* data, int dataSize);

Parameter	Value
data	Pointer to encoded data buffer. Must not be nullptr.
dataSize	Size of the encoded data, bytes. Bytes after the last decoded field are ignored.

Returns: TRUE if data decoded (deserialized) or FALSE if not (null pointer, data shorter than the header or than the fields selected by the mask, wrong data type or wrong version). If the method returns FALSE the object is not changed. Fields which are not selected by the mask are set to 0 (false for boolean fields). Any non zero byte of a boolean field is decoded as true. The method does not check the values of the fields (for example, frameBufferSize can be negative or very large): the video tracker must validate the values before use (see Limits and guarantees).

Example:

// Prepare random parameters.
VTrackerParams in;
in.mode = rand() % 255;
in.rectX = rand() % 255;
in.rectY = rand() % 255;

// Encode data.
uint8_t data[1024];
int size = 0;
if (!in.encode(data, 1024, size))
{
    std::cout << "Can't encode data" << std::endl;
    return false;
}
std::cout << "Encoded data size: " << size << " bytes" << std::endl;

// Decode data.
VTrackerParams out;
if (!out.decode(data, size))
{
    std::cout << "Can't decode data" << std::endl;
    return false;
}

Read params from JSON file and write to JSON file

VTracker library depends on ConfigReader library which provides methods to read parameters from a JSON file and to write parameters to a JSON file. Example of writing to and reading from a JSON file:

// Prepare random parameters.
VTrackerParams in;
in.mode = rand() % 255;
in.rectX = rand() % 255;
in.rectY = rand() % 255;

// Write parameters to the file.
rapidpixel::ConfigReader inConfig;
if (!inConfig.set(in, "VTrackerParams") ||
    !inConfig.writeToFile("VTrackerParams.json"))
{
    std::cout << "Can't write config file" << std::endl;
    return false;
}

// Read parameters from the file.
rapidpixel::ConfigReader outConfig;
if(!outConfig.readFromFile("VTrackerParams.json"))
{
    std::cout << "Can't open config file" << std::endl;
    return false;
}

// Obtain parameters from the config reader.
VTrackerParams out;
if(!outConfig.get(out, "VTrackerParams"))
{
    std::cout << "Can't read params from file" << std::endl;
    return false;
}

VTrackerParams.json will look like:

{
    "VTrackerParams": {
        "rectWidth": 2,
        "rectHeight": 18,
        "searchWindowWidth": 195,
        "searchWindowHeight": 19,
        "lostModeOption": 65,
        "frameBufferSize": 130,
        "maxFramesInLostMode": 111,
        "rectAutoSize": true,
        "rectAutoPosition": true,
        "multipleThreads": true,
        "numChannels": 47,
        "type": 213,
        "custom1": 155.0,
        "custom2": 239.0,
        "custom3": 79.0
    }
}

Only the fields listed in the JSON_READABLE macro of VTrackerParams class are stored in the file (the fields calculated by the video tracker are not stored). All these fields are required: if the file is not a valid JSON document, if the object is missing, if any field is missing or if a value can not be converted to the type of the field (a string, null, an array or an integer outside of the range of int), the get(…) method of ConfigReader returns FALSE and the params object is not changed. Numbers are converted to the type of the field (a fractional number is truncated for an integer field, any non zero number is true for a boolean field). The values are not checked against any range (for example, frameBufferSize can be negative or very large): the video tracker must validate the values before use (see Limits and guarantees).

Build and connect to your project

The library requires a C++17 compiler (MSVC 2019 or newer, GCC 9 or newer, Clang 10 or newer), CMake 3.13 or newer and OpenMP. The benchmark, the example and the demo application don’t depend on OpenCV. The window of the demo application uses WinAPI on Windows and the X11 libraries on Linux (optional, without them a headless window is built, see the SimpleUI library). Installation commands for Linux (Ubuntu):

sudo apt-get install build-essential cmake libx11-dev libxext-dev
# with Clang also:
sudo apt-get install libomp-dev

Build stand alone

The repository does not contain sub-repositories. The libraries which CvTracker depends on are separate repositories and must be cloned into the same folder as CvTracker: Frame, ConfigReader and VTracker are required by the library and the benchmark application, the example and the demo application additionally require SimpleUI, SimpleFileDialog, VSourceFile, VideoRecorderMp4 and the libraries which they depend on (VSource, VCodec, FormatConverter, VDecoderH264, VDecoderH265, VDecoderJPEG and VEncoderJPEG). The folder structure must be:

<work folder>
    CvTracker
    Frame
    ConfigReader
    VTracker
    SimpleUI
    SimpleFileDialog
    VSourceFile
    VideoRecorderMp4
    VSource
    VCodec
    FormatConverter
    VDecoderH264
    VDecoderH265
    VDecoderJPEG
    VEncoderJPEG

Typical commands to build the library, the benchmark, the example and the demo application on Linux (use the URLs of the repositories provided to you):

git clone <CvTracker repository URL>
git clone <Frame repository URL>
git clone <ConfigReader repository URL>
git clone <VTracker repository URL>
git clone <SimpleUI repository URL>
git clone <SimpleFileDialog repository URL>
git clone <VSourceFile repository URL>
git clone <VideoRecorderMp4 repository URL>
git clone <VSource repository URL>
git clone <VCodec repository URL>
git clone <FormatConverter repository URL>
git clone <VDecoderH264 repository URL>
git clone <VDecoderH265 repository URL>
git clone <VDecoderJPEG repository URL>
git clone <VEncoderJPEG repository URL>
cd CvTracker
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make

Typical commands to build on Windows (Visual Studio 2019 or newer):

cd CvTracker
cmake -B build -S .
cmake --build build --config Release

If CvTracker is the top-level CMake project, the main CMakeLists.txt file adds the folders ../Frame, ../ConfigReader and ../VTracker (in this order) before the library itself, and the folders of the libraries of the example and the demo application before these applications. If a folder does not exist CMake stops with an error. If the example and the demo application are not needed, only the repositories Frame, ConfigReader and VTracker are required, switch the applications off: cmake .. -D_SUBMODULE_CACHE_OVERWRITE=OFF -D_CVTRACKER_DEMO=OFF -D_CVTRACKER_EXAMPLE=OFF (see CMake options).

The default build type is Release. The applications are placed in the bin folder of the build folder.

The build creates the compiled version of the library in the folder output of the build folder: the header files in output/include and the library in output/lib (CvTracker.lib on Windows, libCvTracker.a and libCvTracker.so on Linux), see the section Library files. The library file contains the CvTracker, VTracker, Frame and ConfigReader libraries. An application which uses the compiled library must be built with the compiler which built the library (the OpenMP runtime of another compiler does not match), with the same runtime library (/MD or /MDd with Visual Studio) and with OpenMP (/openmp or -fopenmp); the shared library for Linux is linked with the OpenMP runtime.

CMake options

Table 8 - CMake options. In the options below ${PARENT} is the prefix defined by the top-level project (empty if CvTracker is the top-level project).

Option	Default	Description
${PARENT}_CVTRACKER	ON	Build the library.
${PARENT}_CVTRACKER_BENCHMARK	ON / OFF	Build the benchmark application. ON if CvTracker is the top-level project, OFF if it is a sub-repository.
${PARENT}_CVTRACKER_DEMO	ON / OFF	Build the demo application (same defaults).
${PARENT}_CVTRACKER_EXAMPLE	ON / OFF	Build the example application (same defaults).
${PARENT}_CVTRACKER_DEMO_MODE	OFF	Build the demo version of the library: after ${PARENT}_CVTRACKER_DEMO_FRAME_LIMIT processed frames the tracking is deliberately degraded (the search window position is shifted). A demo build is not suitable for production use.
${PARENT}_CVTRACKER_DEMO_FRAME_LIMIT	256	Number of processed frames after which the demo version degrades the tracking. Used only if the demo mode is ON.
${PARENT}_SUBMODULE_CACHE_OVERWRITE	ON	Forced rewriting of the options above in the CMake cache. The top-level project sets it to OFF when it includes CvTracker as a sub-repository.
CVTRACKER_AVX2	ON	On x86-64 build the Release library for the instruction set level x86-64-v3 (-march=x86-64-v3 for GCC and Clang: AVX2, FMA, BMI2; /arch:AVX2 for MSVC). The binaries run on every CPU with AVX2 and FMA (Intel Haswell, AMD Excavator / Zen and newer) and do not depend on the build machine. When OFF the library is built for the baseline instruction set (SSE2), for CPUs without AVX2: that build is about 1.5 times slower and its results are not guaranteed to be bit-identical to the AVX2 build. No effect on ARM64. The former option CVTRACKER_NATIVE_ARCH is accepted and mapped to this option. See Portability.
CMAKE_BUILD_TYPE	Release (if not specified)	Build type.

The options of the top-level CMakeLists.txt (the rows with the prefix ${PARENT}_) are set with the FORCE keyword while ${PARENT}_SUBMODULE_CACHE_OVERWRITE is ON, so a value given in the command line is ignored. To change them from the command line pass also -D_SUBMODULE_CACHE_OVERWRITE=OFF (for a top-level project, ${PARENT} is empty), for example: cmake .. -D_SUBMODULE_CACHE_OVERWRITE=OFF -D_CVTRACKER_DEMO=OFF. CVTRACKER_NATIVE_ARCH is a normal CMake option and can be set directly.

Connect to your project

If you want connect CvTracker library to your CMake project as source code you can make follow. When CvTracker is included in another project, its main CMakeLists.txt file does not add the libraries which it depends on (Frame, ConfigReader and VTracker): your project must add them (in this order: Frame, ConfigReader, VTracker, then CvTracker). If a library is not added, the compilation stops with the error “VTracker.h: No such file or directory” (the name of the missing header file is in the message). For example, if your repository has structure:

CMakeLists.txt
src
    CMakeList.txt
    yourLib.h
    yourLib.cpp

Create folder 3rdparty in your repository and copy the repository folders Frame, ConfigReader, VTracker and CvTracker in 3rdparty folder. New structure of your repository:

CMakeLists.txt
src
    CMakeList.txt
    yourLib.h
    yourLib.cpp
3rdparty
    Frame
    ConfigReader
    VTracker
    CvTracker

Create CMakeLists.txt file in 3rdparty folder. CMakeLists.txt should contain:

cmake_minimum_required(VERSION 3.13)

################################################################################
## 3RD-PARTY
## dependencies for the project
################################################################################
project(3rdparty LANGUAGES CXX)

################################################################################
## SETTINGS
## basic 3rd-party settings before use
################################################################################
# To inherit the top-level architecture when the project is used as a submodule.
SET(PARENT ${PARENT}_YOUR_PROJECT_3RDPARTY)
# Disable self-overwriting of parameters inside included subdirectories.
SET(${PARENT}_SUBMODULE_CACHE_OVERWRITE OFF CACHE BOOL "" FORCE)

################################################################################
## CONFIGURATION
## 3rd-party submodules configuration
################################################################################
SET(${PARENT}_SUBMODULE_FRAME                           ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_FRAME)
    SET(${PARENT}_FRAME                                 ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_FRAME_TEST                            OFF CACHE BOOL "" FORCE)
endif()

SET(${PARENT}_SUBMODULE_CONFIG_READER                   ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_CONFIG_READER)
    SET(${PARENT}_CONFIG_READER                         ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_CONFIG_READER_EXAMPLES                OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_CONFIG_READER_TESTS                   OFF CACHE BOOL "" FORCE)
endif()

SET(${PARENT}_SUBMODULE_VTRACKER                        ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_VTRACKER)
    SET(${PARENT}_VTRACKER                              ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_VTRACKER_TEST                         OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_VTRACKER_EXAMPLE                      OFF CACHE BOOL "" FORCE)
endif()

SET(${PARENT}_SUBMODULE_CVTRACKER                       ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_CVTRACKER)
    SET(${PARENT}_CVTRACKER                             ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_CVTRACKER_BENCHMARK                   OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_CVTRACKER_DEMO                        OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_CVTRACKER_EXAMPLE                     OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_CVTRACKER_DEMO_MODE                   OFF CACHE BOOL "" FORCE)
endif()

################################################################################
## INCLUDING SUBDIRECTORIES
## Adding subdirectories according to the 3rd-party configuration
################################################################################
if (${PARENT}_SUBMODULE_FRAME)
    add_subdirectory(Frame)
endif()

if (${PARENT}_SUBMODULE_CONFIG_READER)
    add_subdirectory(ConfigReader)
endif()

if (${PARENT}_SUBMODULE_VTRACKER)
    add_subdirectory(VTracker)
endif()

if (${PARENT}_SUBMODULE_CVTRACKER)
    add_subdirectory(CvTracker)
endif()

File 3rdparty/CMakeLists.txt adds the folders Frame, ConfigReader, VTracker and CvTracker to your project and excludes the test applications, the benchmark, the demo application and the example from compiling (by default they are excluded from compiling if CvTracker is included as sub-repository). Your repository new structure will be:

CMakeLists.txt
src
    CMakeList.txt
    yourLib.h
    yourLib.cpp
3rdparty
    CMakeLists.txt
    Frame
    ConfigReader
    VTracker
    CvTracker

Next you need include folder 3rdparty in main CMakeLists.txt file of your repository. Add string at the end of your main CMakeLists.txt:

add_subdirectory(3rdparty)

Next you have to include CvTracker library in your src/CMakeLists.txt file:

target_link_libraries(${PROJECT_NAME} CvTracker)

If you want connect compiled library to your CMake project you have to add following to your CMake file (Example for Linux; on Windows use the file CvTracker.lib). The compiled static library requires OpenMP.

find_package(OpenMP REQUIRED)
target_include_directories(${PROJECT_NAME} PRIVATE <path to the folder with the header files>)
add_library(CvTracker STATIC IMPORTED)
set_target_properties(CvTracker PROPERTIES IMPORTED_LOCATION <path to the folder with the library file>/libCvTracker.a)
target_link_libraries(${PROJECT_NAME} CvTracker OpenMP::OpenMP_CXX)

Simple example

The simple example shows how to build a basic application with a user interface. The application opens a video file, shows the video in a window and provides basic functions to control the tracker (capture, reset, changing the size and the position of the tracking rectangle). The example uses the SimpleUI library (window and drawing), the VSourceFile library (video file source) and the CvTracker library, it doesn’t depend on OpenCV. The example checks the returned values of the library methods and stops if the video source does not provide frames or the tracker does not accept them. The tracker object is created inside main (under try), so a failed memory allocation is reported by the application. Source code of the example (example/main.cpp):

#include <iostream>
#include <memory>
#include <string>
#include "SimpleUi.h"
#include "VSourceFile.h"
#include "CvTracker.h"

// Link namespaces.
using namespace std;
using namespace rapidpixel;

namespace
{

/// Number of consecutive failures of getting or processing of a video frame
/// after which the application stops.
constexpr int MAX_FRAME_FAILURES = 30;

/// Get the video tracker object. It is created at the first call (inside main),
/// so a failed memory allocation is reported by the application instead of
/// terminating the process before main starts.
/// @return Reference to the video tracker object.
CvTracker& getTracker()
{
    static CvTracker tracker;
    return tracker;
}
/// Tracker data.
VTrackerParams g_trackerData;

/// Mouse callback function: the tracking rectangle follows the mouse in FREE
/// mode, the left button captures an object, the right button resets the tracker.
/// @param event Mouse event.
void mouseCallBackFunction(const MouseEvent& event)
{
    // Update mouse position.
    if (g_trackerData.type != 2)
        getTracker().executeCommand(VTrackerCommand::SET_RECT_POSITION, static_cast<float>(event.x), static_cast<float>(event.y));

    switch (event.type)
    {
    case MouseEventType::LEFT_DOWN: // Capture/Reset function
        getTracker().executeCommand(VTrackerCommand::CAPTURE, static_cast<float>(event.x), static_cast<float>(event.y));
        break;
    case MouseEventType::RIGHT_DOWN:
        getTracker().executeCommand(VTrackerCommand::RESET);
        break;
    default:
        break;
    }
}



/// Run the example.
/// @return Exit code of the application.
int run()
{
    cout << "Simple Demo Application v" << CvTracker::getVersion() << endl;

    // Set video source. The file name can contain spaces.
    string fileName;
    cout << "Enter video file name: ";
    if (!getline(cin, fileName) || fileName.empty())
    {
        cout << "ERROR: Video file name is not set" << endl;
        return -1;
    }

    // Open video source.
    VSourceFile sourceFile;
    if (!sourceFile.openVSource(fileName))
    {
        cout << "ERROR: Video source not open" << endl;
        return -1;
    }

    // Create window. Mouse and slider callbacks are called from getKey().
    unique_ptr<SimpleUiWindow> window;
    try
    {
        window = make_unique<SimpleUiWindow>("VIDEO TRACKER v" + CvTracker::getVersion());
    }
    catch (const exception& e)
    {
        cout << "ERROR: Can't create window: " << e.what() << endl;
        return -1;
    }
    window->setMouseCallback(mouseCallBackFunction);
    window->show();

    // Set initial tracking parameters. The values are applied when the first
    // video frame is processed.
    const struct
    {
        VTrackerParam id;
        float value;
    } initialParams[] =
    {
        {VTrackerParam::RECT_WIDTH, 72},
        {VTrackerParam::RECT_HEIGHT, 72},
        {VTrackerParam::LOST_MODE_OPTION, 0},
        {VTrackerParam::NUM_CHANNELS, 2},
        {VTrackerParam::MULTIPLE_THREADS, 1},
        {VTrackerParam::TYPE, 0},
        {VTrackerParam::CUSTOM_1, 0},
        {VTrackerParam::RECT_AUTO_SIZE, 1},
        {VTrackerParam::RECT_AUTO_POSITION, 1}
    };
    for (const auto& param : initialParams)
    {
        if (!getTracker().setParam(param.id, param.value))
        {
            cout << "ERROR: Can't set tracker parameter" << endl;
            return -1;
        }
    }

    // Main loop.
    Frame yuvFrame;
    yuvFrame.fourcc = Fourcc::YUV24;
    int failures = 0;
    while (true)
    {
        // Capture next frame. Stop if the source does not give frames.
        if (!sourceFile.getFrame(yuvFrame, 1000))
        {
            if (++failures >= MAX_FRAME_FAILURES)
            {
                cout << "ERROR: Video source does not provide frames" << endl;
                return -1;
            }
            continue;
        }

        // Process video frame.
        if (!getTracker().processFrame(yuvFrame))
        {
            if (++failures >= MAX_FRAME_FAILURES)
            {
                cout << "ERROR: Video frames are not accepted by the tracker" << endl;
                return -1;
            }
            continue;
        }
        failures = 0;

        // Get current tracker data.
        getTracker().getParams(g_trackerData);

        // Choose tracking rectangle color according to tracker mode.
        Color rectColor{255, 255, 255};
        switch (g_trackerData.mode)
        {
        case 1: // TRACKING.
            rectColor = {255, 0, 0};
            break;
        case 2: // LOST.
            rectColor = {0, 0, 255};
            break;
        case 3: // INERTIAL.
            rectColor = {0, 255, 0};
            break;
        case 4: // STATIC.
            rectColor = {0, 255, 255};
            break;
        default: // FREE.
            break;
        }

        // Calculate tracking rectangle corners.
        int x1 = g_trackerData.rectX - g_trackerData.rectWidth / 2;
        int y1 = g_trackerData.rectY - g_trackerData.rectHeight / 2;
        int x2 = x1 + g_trackerData.rectWidth - 1;
        int y2 = y1 + g_trackerData.rectHeight - 1;

        // Draw tracking rectangle.
        SimpleUiDraw::drawRectangle(yuvFrame, x1 - 2, y1 - 2, x2 + 2, y2 + 2, rectColor, 2 * 2);

        // Show image.
        window->render(yuvFrame);

        // Wait keyboard events.
        switch (window->getKey(1))
        {
        // ESC - Exit.
        case 27:
            window->close();
            return 0;
        // W - Increase tracking rectangle height.
        case 119:
            getTracker().executeCommand(VTrackerCommand::CHANGE_RECT_SIZE, 0, 8);
            break;
        // S - Decrease tracking rectangle height.
        case 115:
            getTracker().executeCommand(VTrackerCommand::CHANGE_RECT_SIZE, 0, -8);
            break;
        // D - Increase tracking rectangle width.
        case 100:
            getTracker().executeCommand(VTrackerCommand::CHANGE_RECT_SIZE, 8, 0);
            break;
        // A - Decrease tracking rectangle width.
        case 97:
            getTracker().executeCommand(VTrackerCommand::CHANGE_RECT_SIZE, -8, 0);
            break;
        // T - Move strobe UP (change position in TRACKING mode).
        case 116:
            getTracker().executeCommand(VTrackerCommand::MOVE_RECT, 0, -4);
            break;
        // G - Move strobe DOWN (change position in TRACKING mode).
        case 103:
            getTracker().executeCommand(VTrackerCommand::MOVE_RECT, 0, 4);
            break;
        // H - Move strobe RIGHT (change position in TRACKING mode).
        case 104:
            getTracker().executeCommand(VTrackerCommand::MOVE_RECT, 4, 0);
            break;
        // F - Move strobe LEFT (change position in TRACKING mode).
        case 102:
            getTracker().executeCommand(VTrackerCommand::MOVE_RECT, -4, 0);
            break;
        default:
            break;
        }
    }
}

}



// Entry point.
int main()
{
    try
    {
        return run();
    }
    catch (const exception& e)
    {
        cout << "ERROR: " << e.what() << endl;
        return -1;
    }
}

Demo application

Demo application overview

The demo application is intended to evaluate the performance of the CvTracker C++ library. The application allows you to evaluate the operation of the algorithm with different parameters. It is an application with a window which serves as an example of using the CvTracker library. The application plays a video file (VSourceFile library), shows the video in a window (SimpleUI library), records the video with the tracking results into an MP4 file (VideoRecorderMp4 library) and forms a simple user interface. The video file is selected with the file dialog (SimpleFileDialog library). The application doesn’t depend on OpenCV. The libraries are separate repositories, see the section Build and connect to your project.

Launch and user interface

The demo application does not require installation. It is built from the repository (see Build and connect to your project) for Windows x64 and for Linux. Table 9 shows the list of files of the demo application.

Table 9 - List of files of demo application (example for Windows OS).

File	Description
CvTrackerDemo.exe	Demo application executable file for Windows OS.
CvTrackerDemo.json	Demo application config file in the current folder. The application creates it if it does not exist.
record_DD_MM_YYYY_HH_MM_SS.mp4	Video file created in the current folder each time the video recording is started (key R). The name contains the local date and time of the start (for example record_09_10_2026_14_35_07.mp4), so the previous recordings are not overwritten.

To launch demo application run CvTrackerDemo.exe executable file on Windows x64 OS or run commands on Linux:

chmod +x CvTrackerDemo
./CvTrackerDemo

If a message about missing system libraries appears (on Windows OS) when launching the application, you must install the Microsoft Visual C++ Redistributable for Visual Studio (x64), which will install the system libraries required for operation. Config file CvTrackerDemo.json includes video capture params (videoSource section) and video tracker params (videoTracker section). If the demo application does not find the file in the current folder after startup, it creates it with the default parameters of the library. The same is done if the file is damaged (it can’t be read or does not contain valid parameters): the application prints an error message to the console, uses the default parameters of the library and overwrites the file with them. Values of custom1 and custom2 outside of the range from 0 to 1 are limited to this range. Config file content:

{
    "Params": {
        "videoSource": {
            "source": "file dialog"
        },
        "videoTracker": {
            "custom1": 0.1,
            "custom2": 0.4,
            "custom3": 0.0,
            "frameBufferSize": 1,
            "lostModeOption": 0,
            "maxFramesInLostMode": 256,
            "multipleThreads": false,
            "numChannels": 2,
            "rectAutoPosition": false,
            "rectAutoSize": false,
            "rectHeight": 72,
            "rectWidth": 72,
            "searchWindowHeight": 256,
            "searchWindowWidth": 256,
            "type": 0
        }
    }
}

Table 10 - Parameters description.

Parameter	Description
source	Video source initialization string. If the parameter is set to “file dialog”, then after start the program will offer to select video file via file selection dialog. The parameter can contain a video file name (e.g. “test.mp4”), the file formats are listed in the description of the VSourceFile library (.mp4, .mov, .m4v, .avi, .h264, .264, .h265, .hevc, .mjpeg, .mjpg, .jpeg, .jpg). The software plays the video with repetition i.e. when the end of the video file is reached, playback starts again. If the file was selected in the file dialog, the parameter stays “file dialog” in the config file.
frameBufferSize	Size of frame buffer (number of frames to store). Determines possible duration of STOP-FRAME function. The default value is 1: the library does not store frames, and an object is captured on the last frame, so the capture on the stopped frame (SPACE key) is not compensated. To use the STOP-FRAME function set a value greater than 1 (for example 128).
lostModeOption	Option for LOST mode. Parameter that defines the behavior of the tracking algorithm in LOST mode. Default is 0. Possible values: 0. In LOST mode, the coordinates of the center of the tracking rectangle are not updated and remain the same as before entering LOST mode. 1. The coordinates of the center of the tracking rectangle are updated based on the components of the object’s speed calculated before going into LOST mode. If the tracking rectangle “touches” the edge of the video frame, the coordinate updating for this component (horizontal or vertical) will stop. 2. The coordinates of the center of the tracking rectangle are updated based on the components of the object’s speed calculated before going into LOST mode. The tracking is reset if the center of the tracking rectangle touches any of the edges of the video frame.
maxFramesInLostMode	Maximum number of frames in LOST mode to auto reset of algorithm. Valid range 1 to 2048, default 256.
multipleThreads	Use multiple threads for calculations: false - no use, true - use. If true and two color channels are processed (numChannels = 2), the algorithm uses a separate thread for each color channel (two threads in total), otherwise one thread is used. Can be changed after start (key 3).
rectHeight, rectWidth	Size of the tracking rectangle, pixels. The demo application ignores these values and starts with the 64x64 tracking rectangle, the size can be changed with the keys W, S, D and A.
numChannels	Number of color channels for processing. Demo application uses YUV frame formats for processing. More number of channel - more processing time. It is recommended to use 1 channel for thermal video or 2 for daylight video. Can be changed after start (key 2).
rectAutoPosition	Use tracking rectangle auto position: false - no use, true - use. Tracker will try adjust tracking rectangle position after object capture. Can be changed after start (key E).
rectAutoSize	Use tracking rectangle auto size flag: false - no use, true - use. Tracker will try adjust tracking rectangle size after object capture. Can be changed after start (key Q).
searchWindowHeight	Height of search window, pixels. Recommended value is 256. Valid values 128, 256 or 512. Automatically synchronized with searchWindowWidth to ensure a square window.
searchWindowWidth	Width of search window, pixels. Recommended value is 256. Valid values 128, 256 or 512. Automatically synchronized with searchWindowHeight to ensure a square window.
custom1	Object loss threshold from 0 to 1. Default value 0.1. If detection probability for current frame less than threshold the library will switch to LOST mode. Can be changed after start with the slider “Object loss threshold, %” (custom1 x 100). See Object loss detection and re-acquisition section.
custom2	Object detection threshold from 0 to 1. Default value 0.4. If detection probability for current frame more than threshold the library will switch to TRACKING mode from LOST mode. Can be changed after start with the slider “Object detection threshold, %” (custom2 x 100). See Object loss detection and re-acquisition section.
custom3	Not used in the CvTracker library.
type	Camera type: 0 - daylight, 1 - thermal, 2 - scene tracking. Can be changed after start (key 1).

When the video tracker parameters are changed in the window (the sliders and the keys 1, 2, 3, Q and E), the application writes the current parameters to the config file, so they are used at the next start.

After starting the application (running the executable file) the user should select the video file in the dialog box (if parameter “source” in config file set to “file dialog”). After that the user will see the user interface as shown below.

Table 11 - User interface elements description.

Element	Description
Video (left part of the window)	Source video in original resolution, tracking rectangle position, actual video tracker parameters and information about control buttons (text in the upper left corner).
Sliders (top of the window)	Object loss threshold, % (custom1 x 100 parameter) and Object detection threshold, % (custom2 x 100 parameter). A change is applied immediately and is saved to the config file.
PATTERN	Reference image of the object. The image is taken at the object capture and, for the daylight and thermal camera types, is updated while the object is tracked.
MASK	Object mask. After capture video tracker tries estimate real position of object inside tracking rectangle. The result of this estimation is object mask.
CORRELATION	Score map which is distribution of probability of object location on search window (the size of the map is 128, 256 or 512 pixels, depending on the search window). Bright pixels - high probability, dark pixels - low probability.
PROBABILITY	Detection probability chart (from 0 to 1, the last 120 frames) and position of object detection threshold (yellow line) and object loss threshold (red line). A line is not drawn if the threshold is 0.
Status (right of the PATTERN image)	Time - processing time of the last video frame by the video tracker, ms. Video recording state: NO REC (green) or RECORDING (red). Video tracker mode: FREE, TRACKING, LOST, INERTIAL or STATIC.
Rectangles on the video	In FREE mode: the capture rectangle (white), which follows the mouse cursor (in scene tracking mode it is drawn at the center of the frame). In other modes: the tracking rectangle (TRACKING - red, LOST - blue, INERTIAL - green, STATIC - cyan) and the estimated object position inside the tracking rectangle (object rectangle, white). In TRACKING mode the center of the tracking rectangle is marked with a cross.

Control

To control the application, it is necessary that the main video display window was active (in focus), and also it is necessary that the English keyboard layout was activated without CapsLock mode. The program is controlled by the keyboard and mouse.

Table 12 - Control buttons.

Button	Description
ESC	Exit the application (the application also exits when the window is closed). If video recording is active, it will be stopped and the video file will be completed.
SPACE	Turn on/off STOP-FRAME mode: the displayed video frame is frozen while the video tracker continues to process the video. Capture the object on the frozen frame with the left mouse button (the frameBufferSize parameter must be greater than 1, see Table 10). The mode is also turned off by the left and the right mouse buttons.
ENTER	Pause/resume the video playback to capture an object on a stopped video. The pause is also turned off by the left and the right mouse buttons.
Left mouse button	Capture an object at the cursor position (in FREE mode the capture rectangle follows the cursor). In scene tracking mode the center of the frame is captured. A click outside the video (on the side panels) is ignored.
Right mouse button	Reset the tracker (switch to FREE mode).
R	Start/stop video recording. Each start creates a new file record_DD_MM_YYYY_HH_MM_SS.mp4 (30 FPS) in the current folder, where the name contains the local date and time of the start, for example record_09_10_2026_14_35_07.mp4. The previous recordings are not overwritten (if a file with the same name exists, _1, _2 and so on is added to the name). Recording is performed of the source video in original resolution with the tracking rectangle and the object rectangle drawn on it (without the side panels and the parameters text). To stop the recording, press the R key again. During video recording the status block shows RECORDING (red), otherwise NO REC (green). If the recording can’t be started, the application prints an error message to the console.
W	Increase tracking rectangle height by 8 pixels. Can be changed in tracking mode.
S	Decrease tracking rectangle height by 8 pixels. Can be changed in tracking mode.
D	Increase tracking rectangle width by 8 pixels. Can be changed in tracking mode.
A	Decrease tracking rectangle width by 8 pixels. Can be changed in tracking mode.
T	Move the tracking rectangle UP by 4 pixels.
G	Move the tracking rectangle DOWN by 4 pixels.
F	Move the tracking rectangle LEFT by 4 pixels.
H	Move the tracking rectangle RIGHT by 4 pixels.
Q	Enable/disable tracking rectangle auto size.
E	Enable/disable tracking rectangle auto position.
Z	Adjust tracking rectangle size and position once.
1	Change camera type: daylight, thermal, scene tracking, then daylight again. Note: the library sets one color channel and multithreading off for the thermal camera and for scene tracking. When the type is changed from scene tracking to daylight, the application sets two color channels and multithreading on.
2	Change number of color channels 1 or 2. Note: one color channel is used for the thermal camera and for scene tracking.
3	Enable / disable multithreading. Note: multithreading is used only when two color channels are processed.

Limits and guarantees

Table 13 - Limits of the library.

Item	Limit	Behavior when the limit is exceeded
Size of a video frame	Width and height from 1 to 8192 pixels.	processFrame(…) returns FALSE and resets the tracker.
Video frame data	data is not null and size is not less than the size of the pixel format. If frameBufferSize > 1 size is equal to the size of the pixel format.	processFrame(…) returns FALSE and resets the tracker.
Pixel format	GRAY, YUV24, YUYV, UYVY, NV12, NV21, YU12, YV12.	processFrame(…) returns FALSE and resets the tracker.
Even sizes	Width of YUYV and UYVY frames, width and height of NV12, NV21, YU12 and YV12 frames.	processFrame(…) returns FALSE and resets the tracker.
Search window	Square, 128, 256 or 512 pixels.	Replaced by the default (256) in initVTracker(…), ignored by setParam(…).
Tracking rectangle	From 16 pixels to half of the search window, but not more than 256 pixels.	Clamped.
Frame buffer	From 1 to 512 frames.	Clamped by initVTracker(…), ignored by setParam(…).
Maximum number of frames in LOST mode	From 1 to 2048.	Replaced by the default (256) in initVTracker(…), ignored by setParam(…).
Parameters waiting for the next frame	256.	setParam(…) returns FALSE.
Commands waiting for the next frame	256.	executeCommand(…) returns FALSE.
Value of a parameter	A finite number, absolute value not more than 2^30.	setParam(…) returns FALSE.
Argument of a command	A finite number, absolute value not more than 16777216 (2147483520 for the frame ID).	executeCommand(…) returns FALSE.
Binary command	11 bytes (SET_PARAM) or 19 bytes (COMMAND).	decodeAndExecuteCommand(…) returns FALSE.

Memory. The constructor of the CvTracker class allocates the working buffers for the largest search window (about 44 MB) and the queues of parameters and commands. The buffers are zero filled; the operating system commits the memory when it is used, so the resident memory depends on the search window. Measured on Linux x86-64 (GCC 13, 1280x720 YUV24 frames, frameBufferSize = 1): after the first 5 video frames the resident memory of the process grows by about 13 MB for the 128x128, 18 MB for the 256x256 and 36 MB for the 512x512 search window. When frameBufferSize is greater than 1, the frame buffer (frameBufferSize x the size of the video frame) is allocated when the first video frame, or a video frame of another size, is processed. The library does not allocate memory while it processes video frames of the same size, and setParam(…), executeCommand(…), getParam(…) and getParams(…) do not allocate memory. If a memory allocation fails, the method returns FALSE (the constructor throws std::bad_alloc).

Thread safety. setParam(…), getParam(…), getParams(…), executeCommand(…) and decodeAndExecuteCommand(…) can be called from any thread, also while processFrame(…) runs. initVTracker(…), processFrame(…) and getImage(…) must be called from one thread at a time and never concurrently with each other. New parameters and commands are applied by the next processFrame(…) call. A CvTracker object can not be copied; different objects are independent and can be used in different threads.

Ownership. The library does not keep a reference to the video frame after processFrame(…) returns (when frameBufferSize > 1 it copies the frame data into its own buffer). The caller owns the frame and the image given to getImage(…).

Determinism. The results are calculated only from the sequence of video frames, parameters and commands. They do not depend on the contents of the memory, on the number of threads or on the other objects of the process. The processing time (processingTimeMks) and the number of buffered frames which are processed by one call in STOP-FRAME mode (limited by the time budget) depend on the real time.

Portability

The processing code of the library uses SIMD instructions which are selected at compile time: AVX2 if the compiler defines __AVX2__ (x86-64), SSE2 otherwise (x86-64), NEON on ARM64, and plain scalar code for the other targets. The library has no run-time selection of the instruction set.

By default the Release build on x86-64 uses the instruction set level x86-64-v3 (GCC and Clang: -Ofast -march=x86-64-v3 -funroll-loops, MSVC: /O2 /GL /fp:fast /arch:AVX2 /Qpar); on ARM64 it uses the baseline instruction set (-Ofast -funroll-loops). Consequences:

• The x86-64 binaries run on every CPU with AVX2 and FMA, whatever CPU built them; on a CPU without AVX2 the process stops with an illegal instruction exception. For such CPUs set CVTRACKER_AVX2 to OFF (see CMake options) to build for the baseline instruction set (SSE2): the tracker then needs about 1.5 times more time per frame. The ARM64 build uses NEON, which is part of the ARM64 architecture.
• The results are reproducible (bit-exact) for the same source code, compiler, flags, instruction set and run-time environment (CPU class and C library, see the next item). Builds with different instruction sets (for example AVX2 with FMA instructions and SSE2) or with different compilers calculate the same algorithm, but the floating point results differ in the last bits (fast floating point model, contraction of multiplications and additions), so they are not bit-identical.
• With GCC on Linux the fast floating point model together with OpenMP lets the compiler call the vector mathematical functions of the GNU C library (libmvec) for exp, cos, sinf, cosf and expf. The C library selects the implementation of these functions at run time by the features of the CPU (for example SSE4.1, AVX2 with FMA), so one binary can give results which differ in the last bits on CPUs of different classes and with different versions of the C library. The Clang builds do not use libmvec.

Built and tested configurations: Windows x64 (MSVC 2022), Linux x86-64 (GCC 12 and 13, Clang 15 and 18; x86-64-v3 and baseline instruction set) and Linux ARM64 (GCC 13, cross build, tested under QEMU user-mode emulation). 32-bit targets are not supported.

---

Table of contents