vstabilisercv_web_logo

VStabiliserCv C++ library

v6.3.3

Table of contents

Overview

VStabiliserCv is a C++ library for fast 2D (X and Y offsets) and 3D (X and Y offsets + rotation) digital video stabilization. It provides two algorithm types selected by the TYPE parameter: 2D translation-only stabilization (horizontal and vertical offset, TYPE 0) and 3D translation + rotation stabilization (TYPE 1). The library has no mandatory third-party dependencies: it is written in standard C++17 and needs only a C++17 compiler, the CMake build system and the Frame, ConfigReader and VStabiliser libraries. For every frame the library measures the motion of the image against the previous frame, filters the measured motion separately on every axis (steady camera movement such as a pan is followed, shaking is compensated) and shifts or rotates the frame by the resulting correction. The FFT used internally is part of the library (src/impl/Fft2d*); its SIMD level (AVX2, SSE2, NEON or portable C++) is selected at run time from the CPU features. Optionally, if EGL and OpenGL ES 3.0 are found at build time, the output warp (rotation together with the sub-pixel offset) of the packed three-byte formats (RGB24, BGR24 and YUV24) is executed on the GPU, falling back to the CPU warp automatically when the device has no working GL driver. It inherits the interface from the VStabiliser interface library (a separate repository, see Build and connect to your project), providing a standardized and interchangeable contract compatible with other VStabiliser-based implementations. The library supports the RAW pixel formats RGB24, BGR24, GRAY, YUV24, YUYV, UYVY, NV12, NV21, YV12 and YU12 of the Frame class (Frame library) and performs calculations in a single computational thread (optionally, the TYPE 1 output warp can be parallelised with OpenMP; see the CUSTOM_1 parameter).

VStabiliserCv is general-purpose software: it is not tied to any sector or application, and the developer of a program decides what the program does with it.

Versions

Table 1 - Library versions.

Version Release date What’s new
1.0.0 22.09.2020 First version.
1.1.0 22.09.2020 - Library interface updated.
- Performance improved.
1.2.0 06.11.2020 - Added parameters for limiting the possible offset and rotation of the video frame.
- Added the ability to scale video frames to increase the speed.
1.3.0 15.12.2020 - The speed of image rotation operation is increased by 40%.
- Bugs fixed.
1.4.0 19.01.2021 - Added “transparent borders” mode.
1.5.0 08.07.2021 - The naming of the methods changed.
- Code refactored.
- Added method to get string of current library version.
2.0.0 04.01.2022 - Calculation speed is increased by 30%.
- Added support of new pixels formats: YUV, YUY2(YUYV) (2D stabilization only), UYVY (2D stabilization only) and NV12 (2D stabilization only).
- Reduced number of adjustable parameters for ease of use.
- Redesigned programming interface.
- The mechanism of changing library parameters was redesigned.
- Added the ability to use third-party libraries to rotate the image.
- Fixed the bug with checking the offset and rotation limitations.
- Fixed the bug of picture jumping when it is impossible to calculate the parameters of displacements on a particular frame of video.
3.0.0 10.05.2022 - Simplified software interface: access to software library properties via text-value key replaced by simple enumeration.
- The error of transformation matrix values return has been fixed.
- Redesigned control protocol parser: simplified interface and eliminated additional third-party dependencies.
3.1.0 12.06.2023 - Added constant offsets (X, Y and Rotation).
4.0.0 04.08.2023 - Changed programming interface according to VStabiliser interface class.
- Added new calculation algorithm. Calculation speed increased 80% in comparison with the previous version.
- Added auto calculation of filter parameters.
- Added support for all RAW pixel format listed in Frame class.
4.0.1 10.08.2023 - Added support for python.
4.0.2 13.11.2023 - Frame class updated.
4.0.3 27.12.2023 - VStabiliser interface class updated.
- getParams(…) method usage fixed in test applications.
- Demo application updated (removed dependency from VSourceOpenCv library).
4.0.4 15.01.2024 - VStabiliser submodule updated.
4.0.5 27.03.2024 - VStabiliser submodule updated.
- Documentation updated.
4.0.6 21.05.2024 - Submodules updated.
- Documentation updated.
4.1.0 25.07.2024 - CMake structure updated.
- Moved implementation to separate folder.
4.2.0 29.09.2024 - Update VStabiliser interface.
- Add HOLD_MSEC action command.
- Change executeCommand(…) method signature.
5.0.0 02.02.2025 - Main class name changed from VStabiliserOpenCv to VStabiliserCv.
- Update documentation.
5.1.0 03.03.2025 - Camera trajectory filtering changed.
- Code optimization.
5.2.0 21.04.2025 - Added a GPU-accelerated (VPI) processing backend.
5.2.1 06.01.2026 - Fixed frame ID and source ID copying for output frame.
5.2.2 13.05.2026 - Fixed initVStabiliser(...) dropping the custom1, custom2, custom3, cutFrequencyHz and logMod values from the input parameters (previously these slots were hard-coded to the internal defaults).
- Added missing setParam(...) cases for CUT_FREQUENCY_HZ and LOG_MODE (previously readable but not writable).
- Fixed both processing paths reading m_params instead of the per-frame snapshot m_localParams for offset-limit clamps and scaleFactor (race-prone, mutex-protected).
- Fixed the VPI initialization block in stabilise(...) calling prepareGrayscaleResizedImage(...) before the per-frame parameter snapshot was taken (the first frame could use stale defaults).
- Documentation fixes.
6.0.0 14.07.2026 - Reimplemented the stabilization algorithms in pure C++17 with no external dependencies — the OpenCV and VPI requirements were removed; the library now needs only a C++17 compiler and CMake.
- Motion estimation reimplemented; the bundled third-party component PocketFFT (header-only, BSD-3-Clause, src/impl/fft.h) added.
- Rotation estimation (TYPE 1) reimplemented.
- Camera trajectory filtering reimplemented, separately for every axis.
- Optional OpenMP parallelisation of the output warp (CUSTOM_1 parameter).
6.1.0 09.09.2026 - Added an optional OpenGL ES 3.0 / EGL GPU pipeline for the output warp of the RGB24, BGR24 and YUV24 pixel formats (CMake option VSTAB_OPENGL, default ON; automatic fall back to the CPU warp when the GPU path is unavailable).
- Added the CUSTOM_2 parameter — a run-time switch for the GPU output warp: 1 — enabled (default), 0 — forces the CPU warp. Not used in builds without the GPU path.
- Fixed cross-compilation for ARM platforms: -march=native now defaults to OFF for cross-builds and is probed before use.
6.2.0 06.10.2026 - The third-party component PocketFFT (BSD-3-Clause, src/impl/fft.h) removed; the FFT is computed by the bundled header-only CustomFft engine (src/impl/CustomFft.h, SIMD: AVX2 / SSE2 / NEON): the FFT is ~1.35x faster on AVX2 builds, stabilization results are unchanged.
- Demo application reworked without OpenCV: video files are read by VSourceFile, the result and mix videos are recorded by VideoRecorderMp4 (MJPEG in MP4 at the source frame rate instead of AVI), the user interface (window, sliders, text) is built on SimpleUI. Cameras and network streams are no longer supported; the stabiliser is no longer reset automatically when the video file restarts (SPACE resets it).
- Demo application: the window can be resized and closing it exits the application, the letter keys work in any keyboard layout and regardless of CapsLock, the sliders apply the coefficients when moved and start from the configured values.
- Examples reworked without OpenCV: VSourceFile delivers the frames directly in the pixel format of the example and SimpleUI shows them (GRAY, NV12, NV21, UYVY, YUYV, YU12 and YV12 frames are converted to BGR24 for display by FormatConverter). The video file can be passed as the first command-line argument.
- New submodules: VSourceFile, VideoRecorderMp4 and SimpleUI for the demo, VSourceFile and SimpleUI for the examples. OpenCV is no longer required to build any part of the project.
- Documentation updated.
6.3.0 07.10.2026 - Implementation split into internal components (src/impl) and brought to the coding standard; the output for valid input is bit-identical to 6.2.0 for the same build configuration (verified with GCC 12 / 13 and Clang 15 / 18 on x86-64 and with GCC 12 / 13 on AArch64; on architectures without SSE2 or NEON only without FMA contraction, see Portability).
- FFT SIMD level selected at run time (AVX2 / SSE2 / NEON / portable): portable builds (VSTAB_NATIVE_ARCH=OFF) use AVX2 where the CPU has it and run on every CPU of the architecture; new CMake option VSTAB_ENABLE_SIMD. The FFT spectra are stored densely: about 0.25 MB less memory per instance.
- Fixed the CPU rotation warp for constant rotations beyond ±90 degrees (CONST_A_OFFSET): most of the frame was filled as border; exactly ±90 degrees caused a division by zero. Frames whose warp coordinates exceed the 16.16 fixed-point range of the CPU warp (about ±32000 pixels) are compensated by the integer shift without rotation (an integer overflow skipped the compensation).
- Input validation: stabilise(...) returns FALSE (instead of reading or writing outside the frame buffers) when the frame size does not match the pixel format, when a 4:2:2 / 4:2:0 frame has an odd width (or an odd height for 4:2:0), for an unknown pixel format, and when the source and the result frame share memory; decodeAndExecuteCommand(...) rejects a null buffer; HOLD_MSEC rejects NaN; a result frame with a matching description but no buffer is reallocated; stabilise(...) returns FALSE instead of throwing when the result frame cannot be allocated.
- Every first-party target is built with strict warnings (0 warnings) and exploit mitigations (stack protector, _FORTIFY_SOURCE, CET / BTI, RELRO, non-executable stack; MSVC /sdl /GS /guard:cf).
- Benchmark, demo and examples cleaned up (argument validation, checked return values); the unused OpenCV lookup of the benchmark removed.
- Documentation updated: security, limits, SIMD, third-party components and testing sections.
6.3.1 14.10.2026 - Documentation updated: the getOffsets method description now explains the meaning of the corrections and how to map a point of the result frame back to the source frame (formula, example code, accuracy).
- Description of getOffsets extended in VStabiliserCv.h.
6.3.2 20.10.2026 - The repository has no git submodules: the dependencies (Frame, ConfigReader, VStabiliser and, for the examples, the demo and the test application, VSource, VCodec, VDecoderH264, VDecoderH265, VDecoderJPEG, FormatConverter, VSourceFile, SimpleUI, VEncoderJPEG, VideoRecorderMp4, SimpleFileDialog) are separate repositories next to this one (stand alone build) or are added by the top-level project.
- Release builds without the instruction-set and floating-point flags for the whole library (MSVC /arch:AVX2, /fp:fast and /Ob3, GCC / Clang -march=native; the CMake option VSTAB_NATIVE_ARCH is removed): the library runs on every CPU of the architecture (only the FFT uses AVX2, selected at run time) and gives the same results with MSVC, GCC and Clang on x86-64. The results of Release builds differ from 6.3.1 within the floating-point rounding (offsets by less than 1e-5 pixel); Debug builds give identical results.
- Fixed: stabilise(...) crashed on a compressed frame (JPEG / H264 / HEVC) whose width × height × 4 does not fit an int (the result frame could not be allocated); such a frame is rejected (FALSE).
- New test application (test): checks the mapping between the source and the result frame (getOffsets) against the output of the library.
- Examples: the exceptions of SimpleUI (no display) are caught, the result of the RESET command is checked.
- Documentation updated.
6.3.3 25.10.2026 - TYPE 1: the rotation estimation uses a new approximation of the logarithm (absolute error about 5e-6 instead of 1e-4, monotone). The rotation and offset estimates differ from 6.3.2 by fractions of a pixel; the processing takes about 10 us more per frame.

Library files

The library is supplied only as source code. The repository structure is shown below:

CMakeLists.txt --------------------- Main CMake file of the library.
src -------------------------------- Folder with library source code.
    CMakeLists.txt ----------------- CMake file of the library.
    VStabiliserCv.h ---------------- Main library header file.
    VStabiliserCvVersion.h --------- Header file with library version.
    VStabiliserCvVersion.h.in ------ Service CMake file to generate version file.
    VStabiliserCv.cpp -------------- C++ class definitions file.
    impl --------------------------- Folder with algorithm implementation files.
        VStabiliserCvImpl.h -------- Implementation class: parameters, commands, frame pipeline.
        VStabiliserCvImpl.cpp ------ Implementation class source.
        EstimationGrid.h ----------- Geometry constants of the motion-estimation grid.
        FrameFormat.h -------------- Validation of the input frames.
        FrameFormat.cpp ------------ Validation of the input frames source.
        LumaSampler.h -------------- Decimation of the frame luma into the estimation grid.
        LumaSampler.cpp ------------ Luma sampler source.
        Fft2d.h -------------------- Real 2D FFT interface with run-time SIMD level selection.
        Fft2d.cpp ------------------ SIMD level selection (CPUID), built without ISA flags.
        Fft2dLevels.h -------------- Interface of the FFT level units.
        FftEngine.h ---------------- FFT engine templates shared by the level units.
        Fft2dPortable.cpp ---------- Portable (plain C++) FFT level.
        Fft2dSse2.cpp -------------- SSE2 FFT level (x86).
        Fft2dAvx2.cpp -------------- AVX2 FFT level (x86, the only unit built with -mavx2).
        Fft2dNeon.cpp -------------- NEON FFT level (ARM).
        SpectrumCorrelator.h ------- Shift estimation with sub-pixel precision.
        SpectrumCorrelator.cpp ----- Shift estimation source.
        LogPolarMapper.h ----------- Input of the rotation estimation (TYPE 1).
        LogPolarMapper.cpp --------- Rotation estimation input source.
        MotionEstimator.h ---------- 2D / 3D motion estimation pipeline.
        MotionEstimator.cpp -------- Motion estimation source.
        MotionFilter.h ------------- Motion filter of one stabilised axis.
        MotionFilter.cpp ----------- Motion filter source.
        PixelShift.h --------------- Integer pixel-shift compensation.
        PixelShift.cpp ------------- Pixel-shift source.
        FrameWarp.h ---------------- CPU rotation warp (fixed-point bilinear).
        FrameWarp.cpp -------------- CPU warp source.
        GlWarp.h ------------------- GPU rotation warp (EGL / OpenGL ES 3.0, optional).
        GlWarp.cpp ----------------- GPU warp source.
demo ------------------------------- Folder for demo application.
    CMakeLists.txt ----------------- CMake file for demo application.
    main.cpp ----------------------- Source C++ file of demo application.
benchmark -------------------------- Folder for benchmark application.
    CMakeLists.txt ----------------- CMake file for benchmark application.
    main.cpp ----------------------- Source C++ file of benchmark application.
static ----------------------------- Folder with the images of this document.
examples --------------------------- Folder with examples.
    CMakeLists.txt ----------------- CMake file to include examples.
    BgrExample --------------------- Example for BGR24 pixel format.
    BgrWithoutCopyingExample ------- Example for BGR24 pixel format.
    GrayExample -------------------- Example for GRAY pixel format.
    Nv12Example -------------------- Example for NV12 pixel format.
    Nv21Example -------------------- Example for NV21 pixel format.
    RgbExample --------------------- Example for RGB24 pixel format.
    UyvyExample -------------------- Example for UYVY pixel format.
    Yu12Example -------------------- Example for YU12 pixel format.
    YuvExample --------------------- Example for YUV24 pixel format.
    YuyvExample -------------------- Example for YUYV pixel format.
    Yv12Example -------------------- Example for YV12 pixel format.
test ------------------------------- Folder for test application.
    CMakeLists.txt ----------------- CMake file for test application.
    main.cpp ----------------------- Source C++ file of test application.
    StabCoordinates.h -------------- Mapping between source and result frame coordinates.

Key features and capabilities

Table 2 - Key features and capabilities.

Parameter and feature Description
Programming language C++ (standard C++17, no compiler extensions). No mandatory third-party dependencies: the library needs only the Frame, ConfigReader and VStabiliser libraries (OpenMP and EGL / OpenGL ES 3.0 are optional and affect only the output warp).
Supported OS Any operating system with a C++17 compiler and CMake 3.13 or higher.
Number of stabilization axes Horizontal and vertical offset in every algorithm type. Algorithm TYPE 1 (3D) additionally stabilises rotation (roll).
Supported pixel formats RGB24, BGR24, GRAY, YUV24, YUYV, UYVY, NV12, NV21, YV12, YU12. Both algorithm types (2D TYPE 0 and 3D TYPE 1) are supported for every format — see Table 4 for per-format details.
Frame size Both sides at least 256 pixels; the size field must match the pixel format; 4:2:2 formats need an even width, 4:2:0 formats an even width and height. Other frames are rejected (see stabilise method).
SIMD The FFT selects its instruction-set level at run time: AVX2, SSE2, NEON or portable C++; the AVX2 and SSE2 levels give identical results (see Portability).
Allocated memory About 0.9 MB of fixed internal buffers per instance, without input and output image data. It does not depend on the video frame size.
Maximum compensated frame shift The detectable inter-frame shift is up to ≈30 % of the frame dimension: for TYPE 0 — of the width horizontally and of the height vertically, for TYPE 1 — of min(width, height) on both axes.
Constant image offset The library provides methods to add constant offsets to the result image: signed horizontal and vertical offset (pixels, accepted range -8192..8192) and a constant rotational offset (radians, accepted range -π..π). They are applied to every processed frame.
GPU (OpenGL ES) output warp Optional. When the library is built with EGL and OpenGL ES 3.0 (CMake option VSTAB_OPENGL, default ON), the output warp — rotation together with the sub-pixel offset — of the RGB24, BGR24 and YUV24 pixel formats runs on the GPU in an offscreen EGL context. It is used only on frames that are actually rotated (the translation-only path stays on the lossless integer pixel shift) and falls back to the CPU warp automatically if the device hardware or driver does not support it. All other pixel formats always use the CPU warp. The GPU path can additionally be switched off at run time with the CUSTOM_2 parameter (see Table 5), without rebuilding the library. Windows carries no system EGL / OpenGL ES and needs ANGLE to be supplied separately.

Supported pixel formats

The Frame library contains the Fourcc enum which defines the pixel formats (Frame.h file). VStabiliserCv library supports RAW pixel formats. The input frame is not converted to another pixel format: motion estimation samples only the luma (brightness) information from the frame (for RGB24/BGR24 — the mean of the three channels, for the YUV family — the Y bytes) into a small internal working buffer, while the compensation (pixel shift or rotation warp) is applied to the original frame with all its colour planes/channels — the output frame has the same pixel format and full colour content as the input. Fourcc enum declaration:

enum class Fourcc
{
    /// RGB 24bit pixel format.
    RGB24 = MAKE_FOURCC_CODE('R', 'G', 'B', '3'),
    /// BGR 24bit pixel format.
    BGR24 = MAKE_FOURCC_CODE('B', 'G', 'R', '3'),
    /// YUYV 16bits per pixel format.
    YUYV  = MAKE_FOURCC_CODE('Y', 'U', 'Y', 'V'),
    /// UYVY 16bits per pixel format.
    UYVY  = MAKE_FOURCC_CODE('U', 'Y', 'V', 'Y'),
    /// Grayscale 8bit.
    GRAY  = MAKE_FOURCC_CODE('G', 'R', 'A', 'Y'),
    /// YUV 24bit per pixel format.
    YUV24  = MAKE_FOURCC_CODE('Y', 'U', 'V', '3'),
    /// NV12 pixel format.
    NV12  = MAKE_FOURCC_CODE('N', 'V', '1', '2'),
    /// NV21 pixel format.
    NV21  = MAKE_FOURCC_CODE('N', 'V', '2', '1'),
    /// YU12 (YUV420) - Planar pixel format.
    YU12 = MAKE_FOURCC_CODE('Y', 'U', '1', '2'),
    /// YV12 (YVU420) - Planar pixel format.
    YV12 = MAKE_FOURCC_CODE('Y', 'V', '1', '2'),
    /// JPEG compressed format.
    JPEG  = MAKE_FOURCC_CODE('J', 'P', 'E', 'G'),
    /// H264 compressed format.
    H264  = MAKE_FOURCC_CODE('H', '2', '6', '4'),
    /// HEVC compressed format.
    HEVC  = MAKE_FOURCC_CODE('H', 'E', 'V', 'C')
};

Table 3 - Bytes layout of supported RAW pixel formats. Example of 4x4 pixels image.

rgbRGB24 bgrBGR24
yuvYUV24 grayGRAY
yuyvYUYV uyvyUYVY
nv12NV12 nv21NV21
yu12YU12 yv12YV12

Both stabilization types are fully supported for every listed pixel format: motion estimation, the translation compensation (integer pixel shift) and the TYPE 1 rotation warp all handle all ten formats. The per-format differences are only in the luma source used for motion estimation and in the granularity of the translation-only (memcpy) compensation path, where the shift is rounded so the sub-sampled chroma planes stay aligned with the luma. The TYPE 1 rotation warp resamples every plane (including chroma) bilinearly with sub-pixel precision, so it has no such granularity restriction. Table 4 summarizes the per-format behaviour.

Table 4 - Stabilization support per pixel format.

Pixel format 2D (TYPE 0) 3D (TYPE 1, rotation + translation) Luma source for motion estimation Translation granularity on the pixel-shift path Frame dimension requirements
GRAY Yes Yes Pixel bytes as-is 1 px (both axes) ≥ 256×256
RGB24 / BGR24 Yes Yes Mean of R, G, B channels 1 px (both axes) ≥ 256×256
YUV24 Yes Yes Y channel 1 px (both axes) ≥ 256×256
YUYV / UYVY (4:2:2 packed) Yes Yes Y bytes 2 px horizontally, 1 px vertically (chroma is sub-sampled horizontally) ≥ 256×256, width must be even
NV12 / NV21 (4:2:0 semi-planar) Yes Yes Y plane 2 px on both axes (chroma is sub-sampled on both axes) ≥ 256×256, width and height must be even
YU12 / YV12 (4:2:0 planar) Yes Yes Y plane 2 px on both axes (chroma is sub-sampled on both axes) ≥ 256×256, width and height must be even

The frame dimension requirements of Table 4 are checked: a frame that violates them (or whose size field does not match the pixel format) is rejected — stabilise(...) returns FALSE and leaves the result frame unchanged.

Compressed formats (JPEG, H264, HEVC) are not supported: stabilise(...) copies such frames (with a size of 1 to width × height × 4 bytes, where width × height × 4 must not exceed 2147483647) to the output without processing; other compressed frames are rejected.

VStabiliserCv class description

VStabiliserCv class declaration

VStabiliserCv.h contains the VStabiliserCv class declaration. VStabiliserCv inherits the interface from VStabiliser. Class declaration:

namespace rapidpixel
{
class VStabiliserCv : public VStabiliser
{
public:

    /// Class constructor. Throws std::bad_alloc if the implementation
    /// cannot be allocated.
    VStabiliserCv();

    /// Class destructor.
    ~VStabiliserCv() override;

    /// Not copyable, not movable.
    VStabiliserCv(const VStabiliserCv&) = delete;
    VStabiliserCv(VStabiliserCv&&) = delete;
    VStabiliserCv& operator=(const VStabiliserCv&) = delete;
    VStabiliserCv& operator=(VStabiliserCv&&) = delete;

    /// Get string of current VStabiliserCv class version.
    static std::string getVersion();

    /// Init all video stabiliser parameters by params structure.
    bool initVStabiliser(const VStabiliserParams& params) override;

    /// Set value to parameter with given id.
    bool setParam(VStabiliserParam id, float value) override;

    /// Get parameter with given id.
    float getParam(VStabiliserParam id) override;

    /// Get params.
    void getParams(VStabiliserParams& params) override;

    /// Execute command.
    bool executeCommand(VStabiliserCommand id, float value = 0.0f) override;

    /// Stabilise video frame.
    bool stabilise(rapidpixel::Frame& src, rapidpixel::Frame& dst) override;

    /// Get offsets: horizontal, vertical and rotation.
    void getOffsets(float& dX, float& dY, float& dA) override;

    /// Decode and execute command.
    bool decodeAndExecuteCommand(uint8_t* data, int size) override;
};
}

getVersion method

The getVersion() method returns the current VStabiliserCv class version as a string. Method declaration:

static std::string getVersion();

Method can be used without a VStabiliserCv class instance. Example:

std::cout << "VStabiliserCv version: " << VStabiliserCv::getVersion();

Console output:

VStabiliserCv version: 6.3.3

initVStabiliser method

The initVStabiliser(…) method initializes the video stabilizer from a set of parameters. Method declaration:

bool initVStabiliser(const rapidpixel::VStabiliserParams& params) override;
Parameter Description
params VStabiliserParams class object.

Returns: always TRUE. The method applies all parameters supported by VStabiliserCv (enable, xFilterCoeff, yFilterCoeff, aFilterCoeff, transparentBorder, constXOffset, constYOffset, constAOffset, type, custom1, custom2); if a particular value is invalid, the documented default is applied instead of it. Note: custom1 values below 1 (including fractions) are mapped to 1 (serial warp), so the interface-struct default keeps the serial behaviour — the “auto / all cores” mode is reachable only via setParam(CUSTOM_1, 0); custom2 enables (≥ 1) or disables (< 1) the OpenGL warp in builds with the GPU path and is ignored otherwise.

setParam method

The setParam(…) method changes a single video stabilizer parameter. NOTE: the new value is applied only on the next processed video frame. Every numeric parameter is validated: NaN / Inf values and values outside the documented range are rejected (the method returns FALSE). Toggling MODE to a different value or switching TYPE resets the algorithm (the same behaviour as the ON / OFF commands). Thread-safe — backed by std::atomic<T> per field; safe to invoke from any thread, including concurrently with stabilise(...). Method declaration:

bool setParam(rapidpixel::VStabiliserParam id, float value) override;
Parameter Description
id Parameter ID according to VStabiliserParam enum.
value Parameter value. Depends on parameter ID.

Returns: TRUE if param was accepted or FALSE if not.

getParam method

The getParam(…) method returns the current value of a single video stabilizer parameter. Thread-safe — lock-free atomic load; safe to invoke from any thread, including concurrently with stabilise(...). Method declaration:

float getParam(rapidpixel::VStabiliserParam id) override;
Parameter Description
id Parameter ID according to VStabiliserParam enum.

Returns: Parameter value, or -1 if the parameter is not supported / not used by VStabiliserCv (as documented by the base VStabiliser interface) — see Table 5 below for the per-parameter behaviour.

getParams method

The getParams(…) method returns all video stabilizer parameters. Thread-safe — may be invoked from any thread. Method declaration:

void getParams(VStabiliserParams& params) override;
Parameter Description
params VStabiliserParams class object to be filled.

executeCommand method

The executeCommand(…) method executes a video stabilizer action command. Thread-safe — atomic field updates for RESET / ON / OFF; HOLD_MSEC publishes the deadline through a single atomic store. Safe to invoke from any thread, including concurrently with stabilise(...). Method declaration:

bool executeCommand(VStabiliserCommand id, float value = 0.0f) override;
Parameter Description
id Command ID according to VStabiliserCommand enum.
value Command argument. Depends on command ID.

Returns: TRUE if the command was executed or FALSE if not.

stabilise method

The stabilise(…) method performs video stabilization. The method is not re-entrant: the internal work buffers, the data of the previous frame and the motion filter state are not protected by any lock, so call it from a single processing thread per VStabiliserCv instance (setParam(...) / getParam(...) / getParams(...) / executeCommand(...) / getOffsets(...) / decodeAndExecuteCommand(...) may be invoked concurrently from other threads; every frame reads the parameters once, so a change takes effect from the next frame). The stabiliser resets itself automatically when the input frame resolution or pixel format changes mid-stream. Motion estimation costs the same for every frame size (the frame is decimated to a fixed-size internal grid), the compensation is proportional to the frame area, and nothing grows with the number of processed frames. The output does not depend on the warp thread count (CUSTOM_1). In builds with the GPU warp the GPU context belongs to the thread that created it; calls from another thread use the CPU warp. Method declaration:

bool stabilise(rapidpixel::Frame& src, rapidpixel::Frame& dst) override;
Parameter Description
src Source frame. Supported RAW formats: RGB24, BGR24, GRAY, YUV24, YUYV, UYVY, NV12, NV21, YV12, YU12 (compressed formats such as JPEG, H264, HEVC are copied unchanged). Both frame dimensions must be at least 256 pixels, size must match the pixel format, 4:2:2 formats need an even width and 4:2:0 formats an even width and height.
dst Result frame. Pixel format matches the source frame. It is (re)allocated when its size, pixel format or buffer does not match the source frame; a frame wrapping a user buffer of the right size and format is filled in place. It must not share memory with src (in-place processing is not supported). If stabilization is disabled (MODE = 0) or held (HOLD_MSEC), no correction is applied: the result is the source frame with only the constant offsets (CONST_X/Y/A_OFFSET) applied. Translation-only corrections are applied as a lossless integer pixel shift; when the total rotation (correction + CONST_A_OFFSET) moves a frame corner by at least one pixel, the correction is applied as a single rigid warp (rotation + translation) with the fixed-point 2x2 bilinear kernel (thread count set by CUSTOM_1). A frame wider or taller than 32768 pixels, or whose rotated source coordinates exceed ±32765 pixels at a corner (frames of about 16000 pixels per side and more with large offsets), is compensated by the integer shift without rotation.

Returns: TRUE if the video frame was processed, FALSE if the source frame was rejected (no data, a side below 256, size not matching the pixel format, odd sizes of a subsampled format, unknown pixel format, a compressed frame whose width × height × 4 does not fit an int) or shares memory with the result frame (the result frame is not changed then), or if memory cannot be allocated (the content of the result frame is then undefined).

getOffsets method

The getOffsets(…) method returns the filtered corrections of the last processed video frame. The constant offsets (CONST_X/Y/A_OFFSET) are not included; on the integer pixel-shift path the frame is shifted by the corrections plus the constant offsets rounded to whole pixels (and to even pixels where the chroma is subsampled). The rotation angle dA is non-zero only for algorithm TYPE 1; the 2D translation-only type (TYPE 0) always reports 0. The three values are independent atomic reads and may belong to two adjacent frames while stabilise(...) runs concurrently. Method declaration:

void getOffsets(float& dX, float& dY, float& dA) override;
Parameter Description
dX Horizontal offset in pixels applied to the last processed frame.
dY Vertical offset in pixels applied to the last processed frame.
dA Rotation correction in radians applied to the last processed frame. Non-zero only for TYPE 1; 0 for TYPE 0.

What the values mean. The corrections describe one rigid transform of the source frame into the result frame: first a rotation by dA about the frame centre, then a translation by (dX, dY). The rotation centre is fixed: c = ((width − 1) / 2, (height − 1) / 2) in pixel-index coordinates (the centre of the top-left pixel is (0, 0); in coordinates whose origin is the top-left corner of the frame the same point is (width / 2, height / 2)). It does not depend on the parameters or on any tracker. The axes are x to the right and y down, dX > 0 moves the image content right, dY > 0 moves it down and dA > 0 rotates it clockwise on the screen. The values are the correction applied to the image (its sign is opposite to the image jitter): the whole correction in force for this frame, not a frame-to-frame difference. A steady pan does not stay in them, and a sudden jolt is cancelled mostly in the same frame (the motion filter cancels about 90 % of a measured jolt at the default strength 0.8 after a quiet start) and then released gradually. The constant offsets are applied together with them: the total rotation is a = dA + constA, the total translation is (sx, sy) = (dX + constX, dY + constY). The dX, dY and dA values are also available as INSTANT_X_OFFSET, INSTANT_Y_OFFSET and INSTANT_A_OFFSET (getParam(…), float) and as the instantXOffset, instantYOffset and instantAOffset fields of VStabiliserParams (getParams(…); the first two are integers, see the note about them below).

Mapping a point of the result frame back to the source frame

To find where a point (xStab, yStab) of the stabilised (result) frame was located on the source (not stabilised) frame, call stabilise(…), take the corrections of this frame from getOffsets(…) (or getParam(INSTANT_X_OFFSET / INSTANT_Y_OFFSET / INSTANT_A_OFFSET)) and the constant offsets from getParam(CONST_X_OFFSET / CONST_Y_OFFSET / CONST_A_OFFSET) (or getParams(…)), and apply the inverse transform:

a        = dA + constA
(sx, sy) = (dX + constX, dY + constY)
c        = ((width - 1) / 2, (height - 1) / 2)

xSrc = cx + cos(a) * (xStab - cx - sx) + sin(a) * (yStab - cy - sy)
ySrc = cy - sin(a) * (xStab - cx - sx) + cos(a) * (yStab - cy - sy)

The forward transform (source to result) is xStab = cx + sx + cos(a) * (xSrc - cx) - sin(a) * (ySrc - cy), yStab = cy + sy + sin(a) * (xSrc - cx) + cos(a) * (ySrc - cy). The library applies the corrections in two different ways, and the mapping has to follow the way that was actually used for the frame:

  • Rotation warp. Used when ** a * halfDiagonal >= 1 pixel, where **halfDiagonal = 0.5 * sqrt(width² + height²) (the corner of the frame moves by at least one pixel). The formula above describes the warp; the translation is applied with sub-pixel precision.
  • Integer pixel shift. Used when the rotation is smaller: the frame is not rotated and is shifted by whole pixels. In this case use a = 0 and the rounded translation, xSrc = xStab − round(sx), ySrc = yStab − round(sy) (rounding half away from zero). Formats with subsampled chroma move by even amounts: an odd shift is rounded away from zero to the next even value on both axes for NV12, NV21, YU12 and YV12, and on the horizontal axis only for YUYV and UYVY. A shift of half the frame or more on one of the axes is not applied (the frame is copied).

The function below implements both cases. It is the function of test/StabCoordinates.h, which the test application checks against the output of the library; the same header contains the inverse function sourceToStabilised(…).

#include <cmath>
#include <cstdlib>

/**
 * @brief Map a point of the stabilised frame to the source (unstabilised)
 * frame, reproducing the transform VStabiliserCv applied to the last
 * processed frame:
 *   stabilised = c + t + R(a) * (source - c),
 *   c = ((W - 1) / 2, (H - 1) / 2),  a = dA + constA,
 *   t = (dX + constX, dY + constY),  R(a) = [cos a, -sin a; sin a, cos a]
 * (rotation about the frame centre, then shift), or, if the rotation moves a
 * frame corner by less than 1 px, a whole-pixel shift only.
 * Coordinates are pixel indices: (0, 0) is the centre of the top-left pixel,
 * x right, y down (a positive angle is clockwise on screen). For multi-plane
 * formats they are luma pixels.
 *
 * The CPU warp of the library resamples in 16.16 fixed point, so its output
 * follows this mapping to within (width + 1) / 131072 + 1 / 256 px
 * (0.014 px at 1280, 0.019 px at 1920). With the GPU warp (builds with
 * OpenGL ES, BGR24 / RGB24 / YUV24 frames) the precision is that of the GPU
 * texture filter. A frame wider or taller than 32768 pixels, or whose
 * rotated source coordinates exceed +-32765 pixels at a corner, gets the
 * whole-pixel shift without the rotation; this function does not model
 * that case.
 * The corrections dX, dY and dA are the values of getOffsets(dX, dY, dA) or
 * of getParam(INSTANT_X_OFFSET / INSTANT_Y_OFFSET / INSTANT_A_OFFSET), read
 * after stabilise() of this frame and before the next call (not
 * instantXOffset / instantYOffset of VStabiliserParams: they are int, the
 * fraction is lost). The constant offsets are the values of
 * getParam(CONST_X_OFFSET / CONST_Y_OFFSET / CONST_A_OFFSET) or constXOffset /
 * constYOffset / constAOffset of VStabiliserParams. evenX and evenY follow the
 * pixel format: NV12, NV21, YU12, YV12 - TRUE, TRUE; UYVY, YUYV - TRUE,
 * FALSE; GRAY, RGB24, BGR24, YUV24 - FALSE, FALSE.
 * @param xStab Horizontal coordinate in the stabilised frame, pixels.
 * @param yStab Vertical coordinate in the stabilised frame, pixels.
 * @param width Frame width, pixels.
 * @param height Frame height, pixels.
 * @param dX Horizontal correction of the frame, pixels.
 * @param dY Vertical correction of the frame, pixels.
 * @param dA Rotation correction of the frame, radians.
 * @param constX Constant horizontal offset, pixels.
 * @param constY Constant vertical offset, pixels.
 * @param constA Constant rotation offset, radians.
 * @param evenX TRUE if the format shifts by even pixels horizontally.
 * @param evenY TRUE if the format shifts by even pixels vertically.
 * @param xSrc Result: horizontal coordinate in the source frame, pixels.
 * @param ySrc Result: vertical coordinate in the source frame, pixels.
 * @return TRUE if the point has a source pixel, FALSE if the stabilised pixel
 *         is border fill.
 */
inline bool stabilisedToSource(double xStab, double yStab, int width,
                               int height, double dX, double dY, double dA,
                               double constX, double constY, double constA,
                               bool evenX, bool evenY,
                               double& xSrc, double& ySrc)
{
    const double a{dA + constA};
    const double tx{dX + constX};
    const double ty{dY + constY};
    const double halfDiagonal{
        0.5 * std::sqrt(static_cast<double>(width) * width +
                        static_cast<double>(height) * height)};
    if (std::fabs(a) * halfDiagonal >= 1.0)
    {
        // Rotation about the frame centre, then sub-pixel shift.
        const double cx{(width - 1) * 0.5};
        const double cy{(height - 1) * 0.5};
        const double cosA{std::cos(a)};
        const double sinA{std::sin(a)};
        const double x{xStab - cx - tx};
        const double y{yStab - cy - ty};
        xSrc = cx + cosA * x + sinA * y;
        ySrc = cy - sinA * x + cosA * y;
    }
    else
    {
        // No rotation: shift by whole pixels.
        long ix{std::lround(tx)};
        long iy{std::lround(ty)};
        if (evenX && ix % 2 != 0) { ix += (ix > 0) ? 1 : -1; }
        if (evenY && iy % 2 != 0) { iy += (iy > 0) ? 1 : -1; }
        // A shift of half the frame or more is not applied at all.
        if (std::labs(ix) >= width / 2 || std::labs(iy) >= height / 2)
        {
            ix = 0;
            iy = 0;
        }
        xSrc = xStab - static_cast<double>(ix);
        ySrc = yStab - static_cast<double>(iy);
    }
    return xSrc >= 0.0 && xSrc <= width - 1.0 &&
           ySrc >= 0.0 && ySrc <= height - 1.0;
}

Notes:

  • Accuracy. With the float values of getOffsets(…) or getParam(INSTANT_X_OFFSET / INSTANT_Y_OFFSET / INSTANT_A_OFFSET) the function describes the integer pixel shift exactly and the CPU rotation warp to within its 16.16 fixed-point resampling, (width + 1) / 131072 + 1 / 256 pixel (0.014 pixel at a width of 1280, 0.019 pixel at 1920). The GPU warp of BGR24, RGB24 and YUV24 frames (builds with OpenGL ES) has the precision of the GPU texture filter. The instantXOffset and instantYOffset fields of VStabiliserParams are integers truncated toward zero: a point mapped with them can be off by up to 1 pixel on each axis (2 pixels for the formats with subsampled chroma, because of the even rounding). The instantAOffset field is a float and has no such loss.
  • Verification. The test application (test folder) renders a scene with a known camera shift and roll, stabilises it and checks the function against every frame the library produced: on the integer pixel shift every pixel matches exactly; on the rotation warp the pixels match within the fixed-point rounding, the remaining misalignment of the frame is below 0.015 pixel and the marker positions mapped back to the source agree with the ground truth within 0.03 pixel (TYPE 0 and 1; GRAY, NV12 and UYVY; constant offsets up to 20 degrees; 1280 × 720 and 1920 × 1080). Alternative formulas (rotation about another centre, opposite sign of the rotation, translation before the rotation, the integer fields) are rejected by the same check.
  • No transformation. If stabilisation is disabled (MODE = 0) or held (HOLD_MSEC), or the algorithm has just been restarted, dX, dY and dA are 0, but the constant offsets are still applied. Compressed frames (JPEG, H264, HEVC) are copied without any transformation, so xSrc = xStab and ySrc = yStab. If stabilise(…) returned FALSE the values are not related to the result frame. A frame wider or taller than 32768 pixels, or whose rotated source coordinates exceed ±32765 pixels at a corner, is compensated by the integer shift without the rotation; the function does not model that case.
  • Outside of the frame. A point of the result frame can be mapped outside of the source frame (xSrc < 0, xSrc > width − 1, ySrc < 0 or ySrc > height − 1; the function returns FALSE): such a point lies in the border that the compensation has uncovered (filled with the neutral colour, or transparent, see TRANSPARENT_BORDER) and has no source pixel.
  • Threading. Read the corrections right after stabilise(…) in the same thread: if stabilise(…) runs concurrently, the three values are independent atomic reads and may belong to two adjacent frames.

encodeSetParamCommand method of VStabiliser interface class

The encodeSetParamCommand(…) static method encodes command to change any parameters of remote video stabiliser. To control a stabiliser remotely, the developer has to develop his own protocol and according to it encode the command and deliver it over the communication channel. To simplify this, the VStabiliser class contains static methods for encoding the control commands. The VStabiliser class provides two types of commands: a parameter change command (SET_PARAM) and an action command (COMMAND). encodeSetParamCommand(…) designed to encode SET_PARAM command. Method declaration:

static void encodeSetParamCommand(uint8_t* data, int& size, VStabiliserParam id, float value);
Parameter Description
data Pointer to data buffer for encoded command. Must have size >= 11.
size Size of encoded data. Will be 11 bytes.
id Parameter ID according to VStabiliserParam enum.
value Parameter value.

encodeSetParamCommand(…) is static and used without VStabiliser class instance. This method used on client side (control system). Example:

// Buffer for encoded data.
uint8_t data[11]{};
// Size of encoded data.
int size{0};
// Encode command.
VStabiliser::encodeSetParamCommand(data, size, VStabiliserParam::X_FILTER_COEFF, 0.8f);

encodeCommand method of VStabiliser interface class

The encodeCommand(…) static method encodes video stabiliser action command. To control a stabiliser remotely, the developer has to develop his own protocol and according to it encode the command and deliver it over the communication channel. To simplify this, the VStabiliser class contains static methods for encoding the control commands. The VStabiliser class provides two types of commands: a parameter change command (SET_PARAM) and an action command (COMMAND). encodeCommand(…) designed to encode COMMAND command (action command). Method declaration:

static void encodeCommand(uint8_t* data, int& size, VStabiliserCommand id, float value = 0.0f);
Parameter Description
data Pointer to data buffer for encoded command. Must have size >= 11.
size Size of encoded data. Will be 11 bytes.
id Command ID according to VStabiliserCommand enum.
value Command argument. Depends on command ID.

encodeCommand(…) is static and used without VStabiliser class instance. This method used on client side (control system). Encoding example:

// Buffer for encoded data.
uint8_t data[11]{};
// Size of encoded data.
int size{0};
// Encode command.
VStabiliser::encodeCommand(data, size, VStabiliserCommand::HOLD_MSEC, 1200);

decodeCommand method of VStabiliser interface class

The decodeCommand(…) static method designed to decode command on video stabiliser side. To control a stabiliser remotely, the developer has to develop his own protocol and according to it encode the command and deliver it over the communication channel. To simplify this, the VStabiliser interface class contains static method to decode input command (commands should be encoded by methods encodeSetParamCommand(…) or encodeCommand(…)). The VStabiliser class provides two types of commands: a parameter change command (SET_PARAM) and an action command (COMMAND). Method declaration:

static int decodeCommand(uint8_t* data, int size, VStabiliserParam& paramId, VStabiliserCommand& commandId, float& value);
Parameter Description
data Pointer to input command.
size Size of command. Must be 11 bytes (both COMMAND and SET_PARAM are encoded as 11 bytes).
paramId Video stabiliser parameter ID according to VStabiliserParam enum. After decoding SET_PARAM command the method will return parameter ID.
commandId Video stabiliser command ID according toVStabiliserCommand enum. After decoding COMMAND the method will return command ID.
value Video stabiliser parameter value after decoding SET_PARAM command or command argument after decoding COMMAND.

Returns: 0 - in case decoding COMMAND, 1 - in case decoding SET_PARAM command or -1 in case errors.

decodeCommand(…) is static and used without VStabiliser class instance. Command decoding example:

// Buffer for encoded data.
uint8_t data[11]{};
int size{0};
VStabiliser::encodeCommand(data, size, VStabiliserCommand::ON);

// Decode command.
VStabiliserCommand commandId{VStabiliserCommand::RESET};
VStabiliserParam paramId{VStabiliserParam::MODE};
float value{0.0f};
if (VStabiliser::decodeCommand(data, size, paramId, commandId, value) != 0)
{
    std::cout << "Command not decoded" << std::endl;
    return false;
}

decodeAndExecuteCommand method

The decodeAndExecuteCommand(…) method decodes and executes command on video stabiliser side. The decodeAndExecuteCommand(…) method can be safely called from any thread. Method declaration:

bool decodeAndExecuteCommand(uint8_t* data, int size) override;
Parameter Description
data Pointer to input command.
size Size of command. Must be 11 bytes (both SET_PARAM and COMMAND are encoded as 11 bytes).

Returns: TRUE if command decoded (SET_PARAM or COMMAND) and executed (action command or set param command). FALSE for a null data pointer, a non-positive size, a packet that does not decode or a command / value that is rejected.

Data structures

VStabiliserParam enum

VStabiliserParam enum lists video stabilizer parameters. Declared in VStabiliser.h of the VStabiliser library. Enum declaration:

namespace rapidpixel
{
enum class VStabiliserParam
{
    /// Not used by VStabiliserCv (kept for VStabiliser-interface
    /// compatibility — the internal decimation is fixed in code).
    SCALE_FACTOR = 1,
    /// Not used by VStabiliserCv. The per-frame shift limit is
    /// hard-wired in code (~30 % of the frame dimension). setParam(...)
    /// returns false, getParam(...) returns -1.
    X_OFFSET_LIMIT,
    /// Not used by VStabiliserCv. Same rules as X_OFFSET_LIMIT.
    Y_OFFSET_LIMIT,
    /// Not used by VStabiliserCv (the rotation correction is not
    /// limited by this parameter).
    A_OFFSET_LIMIT,
    /// Horizontal stabilization strength. Range [0, 1]: 0 - near pass-through
    /// (almost no stabilization), 1 - strongest stabilization.
    X_FILTER_COEFF,
    /// Vertical stabilization strength. Range [0, 1]: 0 - near pass-through
    /// (almost no stabilization), 1 - strongest stabilization.
    Y_FILTER_COEFF,
    /// Rotation stabilization strength (TYPE 1). Range [0, 1], same
    /// strength-to-time-constant mapping as X/Y_FILTER_COEFF: 0 - rotation
    /// axis is not stabilised at all, 1 - strongest stabilization.
    A_FILTER_COEFF,
    /// Stabilisation mode: 0 - off (no correction, only the constant
    /// offsets are applied), 1 - stabilisation on. Toggling the value resets
    /// the algorithm.
    MODE,
    /// Transparent border mode: 0 - uncovered borders are filled with the
    /// neutral colour, 1 - they keep the previous content of the result
    /// frame (pixel-shift path) or the source pixel (rotation warp).
    TRANSPARENT_BORDER,
    /// Constant horizontal image offset in pixels added to each processed
    /// frame. Signed, accepted range -8192..8192.
    CONST_X_OFFSET,
    /// Constant vertical image offset in pixels added to each processed
    /// frame. Signed, accepted range -8192..8192.
    CONST_Y_OFFSET,
    /// Constant rotational angle in radians added to each processed frame.
    /// Accepted range -pi..pi.
    CONST_A_OFFSET,
    /// Read only. Horizontal offset (pixels) applied to the last frame.
    INSTANT_X_OFFSET,
    /// Read only. Vertical offset (pixels) applied to the last frame.
    INSTANT_Y_OFFSET,
    /// Read only. Rotation correction (radians) applied to the last frame.
    /// Non-zero only for TYPE 1; the 2D translation-only type reports 0.
    INSTANT_A_OFFSET,
    /// Algorithm type: 0 - 2D stabilisation (translation only), 1 - 3D
    /// stabilisation (translation + rotation). Switching the type resets the
    /// algorithm.
    TYPE,
    /// Not used by VStabiliserCv.
    CUT_FREQUENCY_HZ,
    /// Not used by VStabiliserCv. The motion filter runs on a fixed
    /// 30 FPS nominal time step regardless of the actual input frame rate;
    /// tune X/Y_FILTER_COEFF empirically for the host pipeline instead.
    FPS,
    /// Read only. Processing time of the last video frame, in microseconds.
    PROCESSING_TIME_MKS,
    /// Not used by VStabiliserCv.
    LOG_MODE,
    /// Not used by VStabiliserCv (single plain-C++ backend).
    BACKEND,
    /// TYPE 1 output warp thread count via OpenMP: 1 - serial (default),
    /// N > 1 - N threads, 0 - auto (all cores). Has effect only when the
    /// library is built with OpenMP.
    CUSTOM_1,
    /// GPU (OpenGL ES) output warp: 1 - enabled (default), 0 - disabled.
    /// Not used in builds without the GPU path.
    CUSTOM_2,
    /// Not used by VStabiliserCv.
    CUSTOM_3
};
}

Table 5 - Video stabilizer parameters supported by VStabiliserCv.

Unused parameters (rows marked not used below) are kept in the enum only for VStabiliser interface compatibility. For every such parameter setParam(...) returns false and getParam(...) returns -1 — the library does not store the value.

Parameter Description
SCALE_FACTOR Not used — the internal decimation is fixed in code (see TYPE for the sampled region).
X_OFFSET_LIMIT Not used. The per-frame shift limit is hard-wired in code (≈30 % of the frame dimension — see Table 2). setParam(...) returns false, getParam(...) returns -1.
Y_OFFSET_LIMIT Not used. Same rules as X_OFFSET_LIMIT.
A_OFFSET_LIMIT Not used — the rotation correction is not limited by this parameter.
X_FILTER_COEFF Horizontal stabilization strength. Range [0, 1]: 0 — near pass-through (almost no stabilization), 1 — strongest stabilization. Default: 0.8.
Y_FILTER_COEFF Vertical stabilization strength. Range [0, 1]: 0 — near pass-through (almost no stabilization), 1 — strongest stabilization. Default: 0.8.
A_FILTER_COEFF Rotation stabilization strength for TYPE 1. Range [0, 1], same strength-to-time-constant mapping as X/Y_FILTER_COEFF: 0 — the rotation axis is not stabilised at all, 1 — strongest stabilization. Default: 0.8.
MODE Stabilization mode: 0 — off (no correction; the result is the source frame with only the constant offsets applied), 1 — on. Toggling the value (by setParam(...) or the ON / OFF commands) resets the algorithm. Default: 1.
TRANSPARENT_BORDER 0 — the borders uncovered by the compensation are filled with the neutral colour (black; luma 0 and chroma 128 for the YUV family), 1 — transparent borders: on the integer pixel-shift path they keep the previous content of the result frame (the previous output when the same result frame is reused), on the rotation warp they keep the source pixel at the same position. Default: 1.
CONST_X_OFFSET Constant horizontal offset in pixels added to every processed frame. Signed, accepted range: -8192..8192; setParam(...) returns false for values outside this range or non-finite. Default: 0.
CONST_Y_OFFSET Constant vertical offset in pixels added to every processed frame. Signed, accepted range: -8192..8192; setParam(...) returns false for values outside this range or non-finite. Default: 0.
CONST_A_OFFSET Constant rotational angle in radians added to every processed frame. Accepted range: -π..π; setParam(...) returns false for values outside this range or non-finite. Default: 0.
INSTANT_X_OFFSET Read only. Horizontal offset (pixels) applied to the last processed frame.
INSTANT_Y_OFFSET Read only. Vertical offset (pixels) applied to the last processed frame.
INSTANT_A_OFFSET Read only. Rotation correction (radians) applied to the last processed frame. Non-zero only for TYPE 1; the 2D translation-only type reports 0.
TYPE Algorithm type: 0 — 2D translation-only stabilisation (the motion is measured on the central part of the frame, with separate scales for the width and the height), 1 — 3D translation + rotation stabilisation (the motion is measured on the central part of the central min(width, height) square, with one scale for both axes). Only 0 and 1 are accepted — any other value is rejected (setParam(...) returns false). Switching the type resets the algorithm. Default: 0.
CUT_FREQUENCY_HZ Not used.
FPS Not used. The internal motion filter runs on a fixed 30 FPS nominal time step regardless of the actual input frame rate. Estimating the real FPS reliably is a host-side concern; the recommended practice is to tune X_FILTER_COEFF / Y_FILTER_COEFF empirically for the deployed pipeline (typically a fixed frame rate per system).
PROCESSING_TIME_MKS Read only. Processing time of the last frame in microseconds.
LOG_MODE Not used — the library produces no log output.
BACKEND Not used — single plain-C++ backend.
CUSTOM_1 TYPE 1 output warp thread count via OpenMP: 1 — serial (default), N > 1 — N threads, 0 — auto (all cores). Accepted range: 0..1024. Has effect only when the library is built with OpenMP.
CUSTOM_2 Enable the OpenGL ES GPU output warp: 1 — enabled (default), 0 — disabled (the CPU warp is used). Accepted range: 0..1; any other value is rejected (setParam(...) returns false). Affects only the RGB24, BGR24 and YUV24 pixel formats and only the frames that are actually rotated (TYPE 1) — every other format and the translation-only path always use the CPU warp regardless of this parameter. In a build without the GPU path (VSTAB_OPENGL=OFF, or EGL / OpenGL ES not found at configure time) the parameter is not used: setParam(...) returns false and getParam(...) returns -1.
CUSTOM_3 Not used.

VStabiliserCommand enum

VStabiliserCommand enum lists video stabilizer commands. Declared in VStabiliser.h. Enum declaration:

namespace rapidpixel
{
enum class VStabiliserCommand
{
    /// Reset stabilisation algorithm. No params.
    RESET = 1,
    /// Enable stabilisation. After execution parameter MODE must be set to 1. No params.
    ON,
    /// Disable stabilisation. After execution parameter MODE must be set to 0. No params.
    OFF,
    /// Hold processing for some time. Parameter - time in milliseconds.
    HOLD_MSEC
};
}

Table 6 - Video stabilizer commands.

Command Description
RESET Reset the stabilization algorithm state (clears frame buffers, zeroes offsets).
ON Enable stabilization. After execution parameter MODE is set to 1.
OFF Disable stabilization. After execution parameter MODE is set to 0.
HOLD_MSEC Suspend stabilization for the given number of milliseconds and reset the algorithm. Accepted range: 0..100000 ms (values outside the range and NaN return FALSE). During the timeout no correction is applied: the result is the source frame with only the constant offsets applied. Used to pause stabilization while a pan-tilt unit is moving.

VStabiliserParams class description

VStabiliserParams class is used for video stabilizer initialization (initVStabiliser(…)) or to retrieve all current parameters (getParams(…)). It also provides methods to serialize/deserialize parameters and to read/write JSON configuration files. VStabiliserParams is declared in VStabiliser.h of the VStabiliser library.

VStabiliserParams class declaration 

namespace rapidpixel
{
class VStabiliserParams
{
public:
    /// Not used by VStabiliserCv (initVStabiliser does not forward it).
    int   scaleFactor{1};
    /// Not used by VStabiliserCv: initVStabiliser(...) ignores this
    /// field and getParams(...) reports 0. The per-frame shift limit is
    /// hard-wired in code (~30 % of the frame dimension).
    int   xOffsetLimit{150};
    /// Not used by VStabiliserCv. Same rules as xOffsetLimit.
    int   yOffsetLimit{150};
    /// Not used by VStabiliserCv.
    float aOffsetLimit{10.0f};
    /// Horizontal stabilization strength. Range [0, 1]: 0 - near pass-through,
    /// 1 - strongest stabilization.
    float xFilterCoeff{0.9f};
    /// Vertical stabilization strength. Range [0, 1]: 0 - near pass-through,
    /// 1 - strongest stabilization.
    float yFilterCoeff{0.9f};
    /// Rotation stabilization strength for TYPE 1. Range [0, 1]:
    /// 0 - the rotation axis is not stabilised, 1 - strongest stabilization.
    float aFilterCoeff{0.9f};
    /// Enable/disable stabilisation.
    bool  enable{true};
    /// Enable/disable transparent borders.
    bool  transparentBorder{true};
    /// Constant horizontal image offset in pixels.
    /// Signed, accepted range -8192..8192.
    int   constXOffset{0};
    /// Constant vertical image offset in pixels.
    /// Signed, accepted range -8192..8192.
    int   constYOffset{0};
    /// Constant rotational angle in radians.
    /// Accepted range -pi..pi.
    float constAOffset{0.0f};
    /// Read only. Horizontal offset (pixels) applied to the last frame.
    int   instantXOffset{0};
    /// Read only. Vertical offset (pixels) applied to the last frame.
    int   instantYOffset{0};
    /// Read only. Rotation correction (rad) applied to the last frame.
    /// Non-zero only for TYPE 1.
    float instantAOffset{0.0f};
    /// Algorithm type: 0 - 2D stabilisation (translation only), 1 - 3D
    /// stabilisation (translation + rotation). Switching the type resets the
    /// algorithm. VStabiliserCv default: 0.
    int   type{2};
    /// Not used by VStabiliserCv.
    float cutFrequencyHz{2.0f};
    /// Not used by VStabiliserCv — the internal motion filter runs on a
    /// fixed 30 FPS nominal time step regardless of the actual input frame
    /// rate. Tune xFilterCoeff/yFilterCoeff empirically for the deployed
    /// pipeline instead.
    float fps{30.0f};
    /// Read only. Processing time of the last video frame, in microseconds.
    int   processingTimeMks{0};
    /// Not used by VStabiliserCv.
    int   logMod{0};
    /// Not used by VStabiliserCv (single plain-C++ backend).
    int   backend{0};
    /// TYPE 1 output warp thread count via OpenMP. initVStabiliser(...)
    /// maps values below 1 to 1 (serial, default); N > 1 - N threads; auto
    /// mode (all cores) is reachable only via setParam(CUSTOM_1, 0).
    float custom1{0};
    /// GPU (OpenGL ES) output warp: >= 1 - enabled, < 1 - disabled.
    float custom2{0};
    /// Not used by VStabiliserCv.
    float custom3{0};

    JSON_READABLE(VStabiliserParams, scaleFactor, xOffsetLimit, yOffsetLimit,
                  aOffsetLimit, xFilterCoeff, yFilterCoeff, aFilterCoeff,
                  enable, transparentBorder, constXOffset, constYOffset,
                  constAOffset, type, cutFrequencyHz, fps, logMod, backend,
                  custom1, custom2, custom3);

    bool encode(uint8_t* data, int bufferSize, int& size,
                VStabiliserParamsMask* mask = nullptr);
    bool decode(uint8_t* data, int dataSize);
};
}

Table 7 - VStabiliserParams class field descriptions.

Field Type Description
scaleFactor int Not used — the internal decimation is fixed in code; initVStabiliser does not forward this field.
xOffsetLimit int Not used. setParam(X_OFFSET_LIMIT, ...) returns false, getParam returns -1, getParams reports 0; the per-frame shift limit is hard-wired (≈30 % of the frame dimension).
yOffsetLimit int Not used. Same rules as xOffsetLimit.
aOffsetLimit float Not used — the rotation correction is not limited by this field.
xFilterCoeff float Horizontal stabilization strength. Range [0, 1]: 0 — near pass-through, 1 — strongest stabilization. Default 0.8.
yFilterCoeff float Vertical stabilization strength. Range [0, 1]: 0 — near pass-through, 1 — strongest stabilization. Default 0.8.
aFilterCoeff float Rotation stabilization strength for TYPE 1. Range [0, 1]: 0 — the rotation axis is not stabilised, 1 — strongest stabilization. Default 0.8.
enable bool Enable/disable stabilization. Default true.
transparentBorder bool Enable/disable transparent borders. Default true.
constXOffset int Constant horizontal offset in pixels. Signed, accepted range -8192..8192. Default 0.
constYOffset int Constant vertical offset in pixels. Signed, accepted range -8192..8192. Default 0.
constAOffset float Constant rotational angle in radians added to every processed frame. Accepted range -π..π. Default 0.
instantXOffset int Read only. Horizontal correction (pixels) applied to the last frame, truncated toward zero. Use getOffsets for the exact float value and for mapping points back to the source frame.
instantYOffset int Read only. Vertical correction (pixels) applied to the last frame, truncated toward zero. Use getOffsets for the exact float value.
instantAOffset float Read only. Rotation correction (radians) applied to the last frame about the frame centre. Non-zero only for TYPE 1.
type int Algorithm type: 0 — 2D translation-only stabilisation, 1 — 3D translation + rotation stabilisation. Only 0 and 1 are valid. Switching the type resets the algorithm. Library default: 0.
cutFrequencyHz float Not used.
fps float Not used. The internal motion filter runs on a fixed 30 FPS nominal time step. Tune xFilterCoeff / yFilterCoeff empirically for the deployed pipeline instead.
processingTimeMks (read only) int Processing time of the last frame in microseconds.
logMod int Not used.
backend int Not used — single plain-C++ backend.
custom1 float TYPE 1 output warp thread count via OpenMP: 1 — serial (default), N > 1 — N threads, 0 — auto (all cores, via setParam only). initVStabiliser(...) maps values below 1 to 1, so the struct default keeps the serial behaviour.
custom2 float GPU (OpenGL ES) output warp switch: initVStabiliser(...) enables it for values ≥ 1 and disables it otherwise; getParams(...) reports 1 / 0. In builds without the GPU path it is not used and getParams(...) reports 0.
custom3 float Not used. Same rules as custom2.

Serialize video stabiliser params

The encode(…) method serializes the video stabilizer parameters for transmission over a communication channel. Method declaration:

bool encode(uint8_t* data, int bufferSize, int& size,
            VStabiliserParamsMask* mask = nullptr);
Parameter Description
data Pointer to the output data buffer.
bufferSize Capacity of the output buffer in bytes.
size Output: size in bytes of the encoded payload written to data.
mask Optional pointer to VStabiliserParamsMask to select which fields to include. Pass nullptr to serialize all fields.

Returns: TRUE if the parameters were encoded successfully or FALSE if the buffer was too small.

Example:

VStabiliserParams in{};
uint8_t data[1024]{};
int size{0};
in.encode(data, 1024, size);
std::cout << "Encoded data size: " << size << " bytes" << std::endl;

Deserialize video stabiliser params

The decode(…) method deserializes the video stabilizer parameters. Method declaration:

bool decode(uint8_t* data, int dataSize);
Parameter Description
data Pointer to the input data buffer (produced by encode(...)).
dataSize Size of the input data buffer in bytes.

Returns: TRUE if decoded successfully or FALSE otherwise.

Example:

VStabiliserParams in{};
uint8_t data[1024]{};
int size{0};
in.encode(data, 1024, size);

VStabiliserParams out{};
if (!out.decode(data, size))
{
    std::cout << "Can't decode data" << std::endl;
}

Read params from JSON file and write to JSON file

The ConfigReader library provides methods to read and write parameters from/to a JSON file:

VStabiliserParams in{};
in.scaleFactor = 2;
in.fps = 25.0f;
// Note: xOffsetLimit / yOffsetLimit are not used by VStabiliserCv and
// ignored by initVStabiliser(...) — assigning them has no effect.

// Write params to file.
rapidpixel::ConfigReader inConfig{};
inConfig.set(in, "VStabiliserParams");
inConfig.writeToFile("VStabiliserCvParams.json");

// Read params from file.
rapidpixel::ConfigReader outConfig{};
outConfig.readFromFile("VStabiliserCvParams.json");
VStabiliserParams out{};
outConfig.get(out, "VStabiliserParams");

VStabiliserCvParams.json will look like:

{
    "VStabiliserParams": {
        "scaleFactor": 2,
        "xOffsetLimit": 150,
        "yOffsetLimit": 150,
        "aOffsetLimit": 10.0,
        "xFilterCoeff": 0.89999997615814209,
        "yFilterCoeff": 0.89999997615814209,
        "aFilterCoeff": 0.89999997615814209,
        "enable": true,
        "transparentBorder": true,
        "constXOffset": 0,
        "constYOffset": 0,
        "constAOffset": 0.0,
        "type": 2,
        "cutFrequencyHz": 2.0,
        "fps": 25.0,
        "logMod": 0,
        "backend": 0,
        "custom1": 0.0,
        "custom2": 0.0,
        "custom3": 0.0
    }
}

Note: Several fields above (scaleFactor, xOffsetLimit, yOffsetLimit, aOffsetLimit, cutFrequencyHz, fps, logMod, backend, custom3) are part of the shared VStabiliserParams serialization format defined by the VStabiliser interface but are not used by VStabiliserCv — setParam(...) for any of them returns false, getParam(...) returns -1, and initVStabiliser(...) does not copy them from the input struct. The numbers in the JSON snippet above are the parent-struct defaults; getParams(...) on a freshly-constructed VStabiliserCv instance reports the actual runtime defaults instead — most notably xFilterCoeff / yFilterCoeff / aFilterCoeff = 0.8, type = 0, custom1 = 1, custom2 = 1 in builds with the GPU warp (0 otherwise), and all “not used” fields = 0. The float fields are written with the precision of their float value (0.9f is 0.89999997615814209).

Build and connect to your project

The library and the benchmark need a C++17 compiler, CMake 3.13 or higher and the repositories of the dependencies of the library: Frame, ConfigReader and VStabiliser (the optional GPU warp also uses the system EGL and OpenGL ES libraries, see below). The examples also use VSourceFile (video file reading; it needs VSource, VCodec, VDecoderH264, VDecoderH265, VDecoderJPEG and FormatConverter) and SimpleUI (display), the demo in addition VideoRecorderMp4 (with VEncoderJPEG) and SimpleFileDialog, the test application SimpleUI. On Linux the windows need the X11 development libraries (sudo apt install libx11-dev libxext-dev); without them SimpleUI is built with a headless stub and no window is shown.

In the Release configuration the library is compiled with the following optimisation flags (src/CMakeLists.txt). None of them selects an instruction set for the whole library or relaxes the floating-point model (see Portability):

  • GCC / Clang: -O3 -fno-math-errno -fno-trapping-math -funroll-loops. Deliberately not -Ofast: fast-math lets the compiler reorder floating-point operations (the results would depend on the compiler) and assume that NaN and infinity never occur.
  • MSVC: /O2 /GL /Qpar /Gy (with /LTCG propagated to consumers).
  • OpenMP is optional: if CMake finds it, it is linked and used to parallelise the output warp row loop (run-time thread count set by the CUSTOM_1 parameter); without OpenMP the warp always runs serially.
  • OpenGL ES is optional and controlled by the VSTAB_OPENGL CMake option (default ON): if the EGL and GLESv2 libraries and the EGL/egl.h and GLES3/gl3.h headers are found (in the toolchain sysroot for a cross-build), they are linked and the output warp of the RGB24, BGR24 and YUV24 formats runs on the GPU; otherwise the CPU warp is used. Disable it at build time with -DVSTAB_OPENGL=OFF, or leave the path compiled in and switch it off at run time with the CUSTOM_2 parameter (see Table 5). Windows needs an EGL / ES implementation installed separately — see Windows and the GPU warp.
  • Every first-party target (library, examples, benchmark, demo, test application) is compiled with strict warnings and exploit-mitigation flags (function vstab_cv_harden() of src/CMakeLists.txt); the headers of the dependencies are included as system headers.

Build stand alone

The repository does not contain sub-repositories. The libraries VStabiliserCv depends on must be cloned into the same folder as VStabiliserCv (the ones marked in brackets only for the parts that use them). The folder structure must be:

<work folder>
    VStabiliserCv
    Frame
    ConfigReader
    VStabiliser
    VSource               (examples, demo)
    VCodec                (examples, demo)
    VDecoderH264          (examples, demo)
    VDecoderH265          (examples, demo)
    VDecoderJPEG          (examples, demo)
    FormatConverter       (examples, demo)
    VSourceFile           (examples, demo)
    SimpleUI              (examples, demo, test application)
    VEncoderJPEG          (demo)
    VideoRecorderMp4      (demo)
    SimpleFileDialog      (demo)

Typical commands to build the library, the examples, the benchmark, the demo and the test application (use the URLs of the repositories provided to you):

git clone <VStabiliserCv repository URL>
git clone <Frame repository URL>
git clone <ConfigReader repository URL>
git clone <VStabiliser repository URL>
git clone <VSource repository URL>
git clone <VCodec repository URL>
git clone <VDecoderH264 repository URL>
git clone <VDecoderH265 repository URL>
git clone <VDecoderJPEG repository URL>
git clone <FormatConverter repository URL>
git clone <VSourceFile repository URL>
git clone <SimpleUI repository URL>
git clone <VEncoderJPEG repository URL>
git clone <VideoRecorderMp4 repository URL>
git clone <SimpleFileDialog repository URL>
cd VStabiliserCv
mkdir build
cd build
cmake ..
make

On Windows use cmake --build . --config Release instead of make. If VStabiliserCv is the top-level CMake project, the main CMakeLists.txt file adds the folders ../Frame, ../ConfigReader and ../VStabiliser (in this order) before the library itself, and the folders needed by the examples, the demo and the test application before each of them. If a folder does not exist CMake stops with an error. Only Frame, ConfigReader and VStabiliser are needed when the other parts are switched off (see CMake options).

The examples read the video file given as the first command-line argument (default static/test.mp4 relative to the working directory; the repository does not contain a video, so pass one or place it there). The demo asks for the file at startup. The benchmark needs no input files:

# From the project root:
./build/bin/VStabiliserCvRgbExample /path/to/video.mp4
./build/bin/VStabiliserCvDemo
./build/bin/VStabiliserCvBenchmark all
./build/bin/VStabiliserCvMappingTest --batch

The test application checks the mapping between the source and the result frame coordinates (see getOffsets method) on synthetic video and prints one result line per scenario; the exit code is 0 when every scenario passes. Without --batch it shows the source and the result frames in a window (click the result frame to map a point). Options: --frames N, --size WxH, --scenario K, --save DIR (BMP snapshots).

CMake options

Table 8 - CMake options of the repository. In the options below ${PARENT} is the prefix defined by the top-level project (empty if VStabiliserCv is the top-level project).

Option Default Description
${PARENT}_VSTABILISER_CV ON Build the library.
${PARENT}_VSTABILISER_CV_EXAMPLES ON (stand alone), OFF (included in other project) Build the examples.
${PARENT}VSTABILISER_CV_EXAMPLE ON Build one example: <FORMAT> is BGR, RGB, BGR_COPYLESS, GRAY, YUV, UYVY, YUYV, NV12, NV21, YU12 or YV12.
${PARENT}_VSTABILISER_CV_BENCHMARK ON (stand alone), OFF (included in other project) Build the benchmark.
${PARENT}_VSTABILISER_CV_DEMO ON (stand alone), OFF (included in other project) Build the demo.
${PARENT}_VSTABILISER_CV_TEST ON (stand alone), OFF (included in other project) Build the test application.
${PARENT}_SUBMODULE_CACHE_OVERWRITE ON Forced rewriting of the options above in the CMake cache. The top-level project sets it to OFF when it includes VStabiliserCv as a sub-repository. A stand alone build rewrites the options above with FORCE unless it is OFF (see Windows and the GPU warp).
CMAKE_BUILD_TYPE Release (if not specified) Build type.
VSTAB_ENABLE_SIMD ON Build the SIMD levels of the FFT. OFF defines VSTAB_NO_SIMD and keeps the portable level only (slower FFT; same results only without FMA contraction, see Portability).
VSTAB_OPENGL ON Look for EGL and OpenGL ES 3.0 and build the GPU output warp (defines OPENGL_AVAILABLE).
VSTAB_NO_AVX2 (macro) not defined Defined automatically when the compiler cannot build the AVX2 unit (or for universal macOS binaries); can also be added to CMAKE_CXX_FLAGS to test the SSE2 level.

Connect to your project

To use VStabiliserCv in your CMake project, add the repositories of VStabiliserCv and of its dependencies (Frame, ConfigReader, VStabiliser) as sub-repositories, for example in the 3rdparty folder of your project:

<your project folder>
    3rdparty
        CMakeLists.txt
        Frame
        ConfigReader
        VStabiliser
        VStabiliserCv
    src
    CMakeLists.txt

Create the file 3rdparty/CMakeLists.txt. It adds the dependencies in dependency order (Frame, ConfigReader, VStabiliser) and then the library. When VStabiliserCv is included in another project it does not add its dependencies (they must be added by the project) and it does not build its examples, benchmark, demo and test application:

cmake_minimum_required(VERSION 3.13)

################################################################################
## 3RD-PARTY
## dependencies for the project
################################################################################
project(3rdparty LANGUAGES CXX)

################################################################################
## SETTINGS
## basic 3rd-party settings before use
################################################################################
# To inherit the top-level architecture when the project is used as a submodule.
SET(PARENT ${PARENT}_YOUR_PROJECT_3RDPARTY)
# Disable self-overwriting of parameters inside included subdirectories.
SET(${PARENT}_SUBMODULE_CACHE_OVERWRITE OFF CACHE BOOL "" FORCE)

################################################################################
## CONFIGURATION
## 3rd-party submodules configuration
################################################################################
SET(${PARENT}_SUBMODULE_FRAME                           ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_FRAME)
    SET(${PARENT}_FRAME                                 ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_FRAME_TEST                            OFF CACHE BOOL "" FORCE)
endif()

SET(${PARENT}_SUBMODULE_CONFIG_READER                   ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_CONFIG_READER)
    SET(${PARENT}_CONFIG_READER                         ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_CONFIG_READER_EXAMPLES                OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_CONFIG_READER_TESTS                   OFF CACHE BOOL "" FORCE)
endif()

SET(${PARENT}_SUBMODULE_VSTABILISER                     ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_VSTABILISER)
    SET(${PARENT}_VSTABILISER                           ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_VSTABILISER_TEST                      OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_VSTABILISER_EXAMPLE                   OFF CACHE BOOL "" FORCE)
endif()

SET(${PARENT}_SUBMODULE_VSTABILISER_CV                  ON  CACHE BOOL "" FORCE)
if (${PARENT}_SUBMODULE_VSTABILISER_CV)
    SET(${PARENT}_VSTABILISER_CV                        ON  CACHE BOOL "" FORCE)
    SET(${PARENT}_VSTABILISER_CV_EXAMPLES               OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_VSTABILISER_CV_BENCHMARK              OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_VSTABILISER_CV_DEMO                   OFF CACHE BOOL "" FORCE)
    SET(${PARENT}_VSTABILISER_CV_TEST                   OFF CACHE BOOL "" FORCE)
endif()

################################################################################
## INCLUDING SUBDIRECTORIES
## Adding subdirectories according to the 3rd-party configuration
################################################################################
if (${PARENT}_SUBMODULE_FRAME)
    add_subdirectory(Frame)
endif()

if (${PARENT}_SUBMODULE_CONFIG_READER)
    add_subdirectory(ConfigReader)
endif()

if (${PARENT}_SUBMODULE_VSTABILISER)
    add_subdirectory(VStabiliser)
endif()

if (${PARENT}_SUBMODULE_VSTABILISER_CV)
    add_subdirectory(VStabiliserCv)
endif()

Add the 3rdparty folder in the main CMakeLists.txt of your project (add_subdirectory(3rdparty)) and link the library in the CMakeLists.txt of your target:

target_link_libraries(${PROJECT_NAME} VStabiliserCv)

If the project does not add the dependencies, the build stops with a compile error such as Frame.h: No such file or directory.

Windows and the GPU warp

Windows ships no EGL and no OpenGL ES — only desktop OpenGL — so VSTAB_OPENGL=ON finds nothing and the warp is compiled for the CPU only. An EGL / ES 3.0 implementation has to be supplied, and ANGLE, which translates ES to Direct3D 11, is the usual one; vcpkg builds it from source. Run the steps below from an x64 Native Tools command prompt.

1. Install ANGLE for the same triplet the project is built with. It is compiled from source, so the command takes a while:

vcpkg install angle:x64-windows

2. Configure the build:

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release ^
      -DCMAKE_TOOLCHAIN_FILE=<vcpkg>/scripts/buildsystems/vcpkg.cmake

A stand alone build always enables the examples, the benchmark, the demo and the test application: the top-level CMakeLists.txt writes those cache variables with FORCE, so -D…=OFF on its own cannot turn them off. To build the library alone, turn the forced cache rewriting off as well — the leading underscore is the empty ${PARENT} prefix:

cmake .. -DCMAKE_BUILD_TYPE=Release ^
      -DCMAKE_TOOLCHAIN_FILE=<vcpkg>/scripts/buildsystems/vcpkg.cmake ^
      -D_SUBMODULE_CACHE_OVERWRITE=OFF -D_VSTABILISER_CV_EXAMPLES=OFF ^
      -D_VSTABILISER_CV_DEMO=OFF -D_VSTABILISER_CV_BENCHMARK=OFF ^
      -D_VSTABILISER_CV_TEST=OFF

3. Check the configure log. VStabiliserCv: OpenGL ES ENABLED (…/libEGL.lib, …/libGLESv2.lib) means the GPU warp is compiled in; EGL/GLESv2 (libs or headers) NOT found means CMake did not find ANGLE — check the toolchain file and the triplet.

4. Build with cmake --build . --config Release.

Deployment. The vcpkg toolchain file copies libEGL.dll and libGLESv2.dll next to every executable in the build, and the d3dcompiler_47.dll that ANGLE loads at run time is part of Windows. Configuring with -DCMAKE_PREFIX_PATH=<vcpkg>/installed/x64-windows instead of the toolchain file also builds, but then nothing copies the DLLs and they have to be shipped by hand.

Portability

Only the FFT uses intrinsics, and its instruction-set level is selected at run time:

  • src/impl/Fft2d.cpp is compiled without any instruction-set flag. On the first use it runs CPUID / XGETBV (AVX2 = CPUID.7.0:EBX bit 5, with OSXSAVE, AVX and the XMM / YMM state enabled in XCR0) and selects the best level compiled into the library: AVX2 (x86), else SSE2 (x86-64 baseline) or NEON (AArch64 baseline, or 32-bit ARM built with NEON), else the portable C++ level.
  • src/impl/Fft2dAvx2.cpp is the only unit compiled with the AVX2 flag (-mavx2, MSVC /arch:AVX2); the CMake configuration probes that the compiler accepts it and defines VSTAB_NO_AVX2 otherwise. The AVX2 unit does not enable FMA, so every level computes the same operations in the same order.
  • Every level unit keeps its code in an unnamed namespace, so the linker cannot merge code compiled for one instruction set into another level.
  • No other flag selects an instruction set: the library runs on every CPU of the target architecture and uses AVX2 where it is available. The CMake option VSTAB_ENABLE_SIMD and the macros VSTAB_NO_SIMD / VSTAB_NO_AVX2 switch the SIMD levels off (see CMake options).
  • The SSE2 and AVX2 levels give bit-identical results. All levels, the portable one included, are bit-identical when the compiler does not contract multiply-adds into FMA instructions (x86-64 builds of the repository). Where it may (AArch64, where FMA is part of the baseline, or an instruction-set flag added by the user, e.g. -march=native on a CPU with FMA), it contracts the scalar code of the portable level differently from the vector code, so a build with VSTAB_ENABLE_SIMD=OFF then differs from the default build in the last bits of the results.
  • On x86-64 the results are the same with MSVC 2022, GCC 12 / 13 and Clang 15 / 18 (Release and Debug). On AArch64 the compilers contract multiply-adds into FMA instructions (GCC in optimised builds, Clang in every build), so the results differ from x86-64, and between GCC and Clang, in the last bits.
  • Built and run: Windows x86-64 (MSVC 2022), Linux x86-64 (GCC 12 / 13, Clang 15 / 18), Linux AArch64 (GCC 13 and Clang 18 cross builds, run under QEMU).

Example

Below is a minimal example (identical to examples/BgrExample). It uses VSourceFile to read BGR24 frames from the video file given as the first command-line argument (default static/test.mp4, not part of the repository), stabilises each frame and displays the source and stabilised video in two SimpleUI windows. Press ESC to exit or SPACE to reset the stabiliser. The other examples do the same for the other pixel formats: VSourceFile delivers the frames in the format of the example, and the formats the window can’t show (GRAY, NV12, NV21, UYVY, YUYV, YU12, YV12) are converted to BGR24 for display by FormatConverter. For a test without video files and windows, see the benchmark application — a pure C++ synthetic accuracy-and-speed test: it renders a textured canvas through a known jitter trajectory (shift + rotation), runs five configurations (TYPE 0 2D translation-only; TYPE 1 3D rotation + translation with a 1-, 2- and 4-thread output warp; and a TYPE 1 translation-only rotation-leakage check) and reports per-frame timing, the inter-frame MAD (mean absolute difference) reduction and the correction tracking errors against the known trajectory, with PASS/FAIL verdicts per configuration. Usage: VStabiliserCvBenchmark [width height [frames]] or VStabiliserCvBenchmark all [frames] (FullHD and HD); sides 256..8192, default 240 frames; without arguments an interactive menu asks for the resolution.

#include <exception>
#include <iostream>
#include <string>
#include <SimpleUi.h>
#include <VSourceFile.h>
#include <VStabiliserCv.h>



namespace
{
// Stabilise and show the video file (argument 1, default
// static/test.mp4) until a window is closed or ESC is pressed.
// Returns the exit code.
int run(int argc, char** argv)
{
    // Init video source. VSourceFile decodes the file, converts the frames to
    // the requested pixel format and restarts the file at the end.
    std::string source{argc > 1 ? argv[1] : "static/test.mp4"};
    rapidpixel::VSourceFile videoSource{};
    if (!videoSource.openVSource(source))
    {
        std::cout << "Can't open " << source << std::endl;
        return -1;
    }

    // Init frames.
    rapidpixel::Frame srcBgrFrame{};
    rapidpixel::Frame dstBgrFrame{};

    // Create windows.
    rapidpixel::SimpleUiWindow srcWindow{"SOURCE VIDEO"};
    rapidpixel::SimpleUiWindow dstWindow{"RESULT VIDEO"};
    srcWindow.show();
    dstWindow.show();

    // Create video stabilizer object.
    rapidpixel::VStabiliserCv videoStabilizer{};

    // Main loop, until a window is closed.
    while (srcWindow.isOpen() && dstWindow.isOpen())
    {
        // Capture next video frame in BGR24 pixel format.
        srcBgrFrame.fourcc = rapidpixel::Fourcc::BGR24;
        if (!videoSource.getFrame(srcBgrFrame, 3000) ||
            srcBgrFrame.fourcc != rapidpixel::Fourcc::BGR24)
        {
            std::cout << "No BGR24 input frame" << std::endl;
            return -1;
        }

        // Stabilise frame.
        if (!videoStabilizer.stabilise(srcBgrFrame, dstBgrFrame))
        {
            std::cout << "Stabilisation not calculated" << std::endl;
            return -1;
        }

        // Show video (the windows show BGR24 frames as they are).
        srcWindow.render(srcBgrFrame);
        dstWindow.render(dstBgrFrame);

        // Process keyboard events of both windows.
        int key{srcWindow.getKey(1)};
        if (key < 0)
        {
            key = dstWindow.getKey(1);
        }
        if (key == 27)
        {
            // ESC - exit.
            return 0;
        }
        if (key == ' ')
        {
            // SPACE - reset video stabilizer.
            if (!videoStabilizer.executeCommand(
                    rapidpixel::VStabiliserCommand::RESET))
            {
                std::cout << "Can't reset video stabilizer" << std::endl;
            }
        }
    }

    return 0;
}
}



// Entry point. Optional argument: path to the video file
// (default static/test.mp4).
int main(int argc, char** argv)
{
    // SimpleUiWindow throws std::runtime_error when it cannot create a
    // window (e.g. no display) or render a frame.
    try
    {
        return run(argc, argv);
    }
    catch (const std::exception& e)
    {
        std::cout << "ERROR: " << e.what() << std::endl;
        return -1;
    }
}

Demo application

Demo application overview

The demo application is intended to evaluate the performance of the VStabiliserCv C++ library. The application allows you to evaluate the operation of the algorithm. It is a small application and serves as an example of using the VStabiliserCv library. The application uses the VSourceFile library for reading and decoding video files, the VideoRecorderMp4 library for recording video, the SimpleUI library for displaying video and forming a simple user interface (window, sliders, text), and the SimpleFileDialog library to open a video file through a native file dialog.

Launch and user interface

The demo application does not require installation. It is compiled for Windows x64 (Windows 10 and newer) as well as for Linux (several distros; to get the demo application for a specific Linux distro, send us a request). Table 9 shows the files that make up the demo application.

Table 9 - List of files of demo application (example for Windows OS).

File Description
VStabiliserCvDemo.exe Demo application executable file for windows OS.
VC_redist.x64 Installer of necessary system libraries for Windows x64.
src Folder with application source code.
static/test.mp4 Test video file of the demo package (not part of the repository; loaded when the user selects it at startup).

To launch demo application run VStabiliserCvDemo.exe executable file on Windows x64 OS or run commands on Linux:

sudo chmod +x VStabiliserCvDemo
./VStabiliserCvDemo

If a message about missing system libraries appears (on Windows OS) when launching the application, you must install the VC_redist.x64.exe program, which will install the system libraries required for operation.

The demo application sets the video stabiliser parameters in code (enable = true, xFilterCoeff = 0.97, yFilterCoeff = 0.97, aFilterCoeff = 0.97, transparentBorder = true, type = 1 — 3D rotation + translation, custom1 = 4 warp threads), so no configuration file is required. The per-frame shift limit is hard-wired (≈30 % of the frame dimension — see Table 2) and not user-tunable. Command-line arguments are not parsed. At startup the application prompts on stdin:

Open file dialog? (y/n)
  • Answering y/Y/yes/Yes opens a native file-open dialog via SimpleFileDialog. Cancelling the dialog produces an empty string and the demo exits with ERROR: No video source selected.
  • Any other answer reads the video file path from stdin. Only video files are supported (cameras and network streams are not).

The software plays the video file with repetition at the file frame rate: when the end of the file is reached, VSourceFile starts again from the first frame. The stabiliser is not reset on the wrap — press SPACE to reset it.

After the source is opened the user will see the user interface. The window shows the original video (top) and the video processed by the stabilisation algorithm (bottom). The upper part of the window shows the processing time of the current video frame, the current algorithm configuration (algorithm type and, for TYPE 1, the warp thread count) and the keyboard hints. The window can be resized: the video is scaled to fit it keeping the aspect ratio. Closing the window exits the application.

vstabilisercv_demo_app

Control

To control the application, it is necessary that the main video display window was active (in focus). The letter keys work in any keyboard layout and regardless of CapsLock. The program is controlled by the keyboard and by three on-screen sliders.

Table 10 - Control buttons.

Button Description
ESC Exit the application. If video recording is active, it will be stopped.
SPACE Reset stabilization algorithm.
H Hold (pause) stabilisation for 1 second. Invokes the HOLD_MSEC command with a 1000 ms argument: input frames are passed through unchanged for one second, after which stabilisation resumes automatically.
T Toggle the algorithm type (TYPE) between 0 (2D, translation only) and 1 (3D, rotation + translation). Switching the type resets the stabiliser.
1/2/4/8 Set the output-warp thread count (CUSTOM_1) to 1, 2, 4 or 8 threads (OpenMP builds only).
R Start/stop video recording. When video recording is enabled, a files dst_[date and time].mp4 (result video) and mix_[date and time].mp4 (combined source and result video, MJPEG in MP4 container at the source frame rate) are created in the current working directory of the application. Recording is performed of what is displayed to the user. To stop the recording, press the R key again. During video recording, the application shows a warning message.

In addition to the keyboard, the application provides three on-screen sliders (Table 11) on the panel above the video.

Table 11 - On-screen sliders.

Control Description
X coeff % Slider for the horizontal smoothing coefficient (X_FILTER_COEFF). Integer range 0–100 is scaled to [0..1] and applied when the slider is moved. 0 disables horizontal compensation, 1 is maximum horizontal stabilization.
Y coeff % Slider for the vertical smoothing coefficient (Y_FILTER_COEFF). Integer range 0–100 is scaled to [0..1] and applied when the slider is moved. 0 disables vertical compensation, 1 is maximum vertical stabilization.
A coeff % Slider for the rotation (roll) smoothing coefficient (A_FILTER_COEFF). Integer range 0–100 is scaled to [0..1] and applied when the slider is moved. 0 disables rotation compensation, 1 is maximum rotation stabilization. Has effect only in algorithm TYPE 1 (3D, rotation + translation), which the demo runs by default.

Table of contents