MvExtractorH264 C++ library

v1.1.0

Table of contents

• Overview
• Versions
• Library files
• MvExtractorH264 class description
  • MvExtractorH264 class declaration
  • getVersion method
  • getMotionVector method
  • Motion vector format
  • End of the stream
• Supported streams and damaged data
• Security and hardening
• Limits and guarantees
• Build and connect to your project
• Simple example
• Example application

Overview

The MvExtractorH264 C++ library extracts the motion vectors of H.264 (AVC) elementary video streams (Annex B) without decoding the video. The library parses the bitstream (parameter sets, slice headers, CAVLC and CABAC macroblock data), runs the motion vector prediction of the standard (median prediction, P_Skip, spatial and temporal direct prediction of B slices) and returns one motion vector [x, y, dx, dy] for every macroblock. Key features:

• Motion vectors only: no sample of any picture is reconstructed. The library has no inverse quantization and transforms, no intra and inter sample prediction (interpolation, weighted prediction), no deblocking filter and no picture buffers with samples. It parses the residual data only to follow the bitstream.
• Identical vectors to the VDecoderH264 decoder: the parsing and the motion vector derivation are the code of the verified VDecoderH264 decoder without the sample reconstruction. The vectors, the order of the pictures and the macroblock order are bit-for-bit the same as the vectors returned by VDecoderH264::getMotionVector(), also for damaged streams.
• Fast: 2.4 to 2.5 times faster than VDecoderH264 with the reconstruction of the samples and 7 to 10 % faster than its parse-only mode (see Portability and performance).
• Pure portable C++17: no external libraries, no SIMD code and no instruction-set flags, no threads, no dynamic loading, no network, no files, no logging, no environment variables. It runs on every CPU of the target architecture (tested on x86-64 and AArch64).
• Safe against damaged and hostile input: every size and index read from the stream is checked, the memory use is bounded (see Limits and guarantees), the code is built with the strictest warning level, stack protection and control flow protection, and was tested with sanitizers and fuzzers.
• Supported streams: progressive 8-bit 4:2:0 Baseline, Main and High profile streams: CAVLC and CABAC, I, P and B slices, multiple slices, multiple reference frames, weighted prediction, 4x4 / 8x8 transforms, scaling matrices, constrained intra prediction. See Supported streams and damaged data for the streams that give no pictures.
• Simple interface: one class with the PIMPL pattern, one method that takes a frame with H.264 data and returns the vectors. The class depends only on the Frame class that describes the data of a video frame.

The example application MvExtractorH264Example reads an H.264 file and shows the vectors on a black background in a window or prints them to the terminal.

Versions

Table 1 - Library versions.

Version	Release date	What’s new
1.0.0	23.09.2026	- First version.

Library files

The repository structure is organized as follows:

CMakeLists.txt ---------------- Main CMake build configuration.
README.md --------------------- Documentation and usage guide.
3rdparty ---------------------- External dependencies.
    CMakeLists.txt ------------ CMake inclusion script for third-party libraries.
    Frame --------------------- Frame class (video frame description).
src --------------------------- Library source code.
    CMakeLists.txt ------------ CMake script for the library (warnings, exploit mitigation).
    MvExtractorH264.h --------- Public library interface (PIMPL).
    MvExtractorH264.cpp ------- Public library implementation wrapper.
    MvExtractorH264Version.h -- Version definition header.
    MvExtractorH264Version.h.in Template for CMake version generation.
    impl ---------------------- Internal implementation files.
        MvExtractorH264Impl.h -- Internal class declaration.
        MvExtractorH264Impl.cpp Splitting of the data into NAL units, queue of the pictures.
        H264Engine.h ---------- Engine interface: pictures and motion vector extraction.
        H264Engine.cpp -------- Engine core: bitstream, parameter sets, picture management.
        H264EngineInternal.h -- Internal data structures (parameter sets, pictures, macroblock data).
        H264Compat.h ---------- Bit operations and unaligned access (portable C++).
        H264Slice.inl --------- Slice data: macroblock layer, CABAC / CAVLC parsing, motion vector prediction.
example ----------------------- Example application.
    SimpleUI ------------------ Lightweight window and drawing library.
    H264Parser.h -------------- H.264 elementary stream file reader header.
    H264Parser.cpp ------------ H.264 elementary stream file reader implementation.
    CMakeLists.txt ------------ Example build configuration.
    main.cpp ------------------ Example: motion vectors in a window or in the terminal.
static ------------------------ Images of the documentation.

MvExtractorH264 class description

MvExtractorH264 class declaration

The MvExtractorH264 class is declared in MvExtractorH264.h:

namespace cr
{
namespace video
{
class MvExtractorH264
{
public:

    /// Class constructor.
    MvExtractorH264();

    /// Class destructor.
    ~MvExtractorH264();

    /// Copying is deleted: an object owns the state of one stream.
    MvExtractorH264(const MvExtractorH264&) = delete;
    MvExtractorH264& operator=(const MvExtractorH264&) = delete;

    /// Get library version.
    static std::string getVersion();

    /// Get the motion vectors of the next picture in output order.
    const std::vector<std::vector<float>>& getMotionVector(Frame& frame);
};
}
}

One object processes one stream. The methods of one object must not be called from several threads at the same time (use one object per thread or lock the calls). Different objects are independent of each other and can work in different threads. The constructor throws std::bad_alloc if the memory for the object cannot be allocated.

getVersion method

The getVersion() method returns a string with the current version of the library:

static std::string getVersion();

Example usage:

std::cout << "MvExtractorH264 version: " << cr::video::MvExtractorH264::getVersion() << std::endl;

Terminal output:

MvExtractorH264 version: 1.1.0

getMotionVector method

The getMotionVector(…) method parses H.264 data and returns the motion vectors of the next picture in output (display) order.

const std::vector<std::vector<float>>& getMotionVector(Frame& frame);

Parameter	Type	Description
frame	Frame&	Source frame. frame.fourcc must be Fourcc::H264. frame.data points to frame.size bytes with one or several NAL units in the Annex B format (start codes 00 00 01 or 00 00 00 01): an access unit, a part of the stream or the whole stream. If the data has no start code, it is one NAL unit. The data is not changed and is not used after the call (the library copies what it needs).

Returns: a reference to the motion vectors of one picture (see Motion vector format), or an empty vector if no picture is ready yet or the frame is not an H264 frame (fourcc is not Fourcc::H264, size is negative, or size is positive and data is null). The reference stays valid until the next call of the method or until the object is destroyed.

Behavior:

• A picture is returned when the library knows that it is complete and next in the output order. Pictures with B-frame reordering are returned a few calls after their data was passed, so a call with the data of a picture can return the vectors of an earlier picture or an empty vector.
• Pictures are returned one per call. If the data of one call completes several pictures (for example a whole file in one call), the first picture is returned and the others wait in an internal queue for the next calls: call the method again (with the next data, or with an H264 frame of size 0 at the end of the stream) to get them. A waiting picture needs 8 bytes per macroblock and the memory of all waiting pictures is limited to 256 MiB (about 4000 pictures of 1920x1080): if a single call produces more, the oldest waiting pictures are dropped.
• The method does not throw exceptions: if the memory runs out, it returns an empty vector.
• The class does not write logs, does not use the file system, the network or environment variables.

Motion vector format

Every element of the returned vector belongs to one macroblock (16x16 pixels) and contains exactly four float values:

\[\mathbf{v} = [x, y, dx, dy]\]

• x: horizontal pixel coordinate of the visible top-left corner of the macroblock after the cropping of the stream (0 <= x < width).
• y: vertical pixel coordinate of the visible top-left corner of the macroblock after the cropping of the stream (0 <= y < height).
• dx: horizontal displacement in pixels: the average of the 16 motion vectors of the 4x4 blocks of the macroblock (positive - right). Motion vectors of the stream have a quarter pixel precision, so dx is a multiple of 1/64 pixel.
• dy: vertical displacement in pixels (positive - down), calculated like dx.

The vector contains one element for every macroblock that overlaps the visible picture, in raster order (for example 8160 elements for 1920x1080). If the stream crops the picture inside a macroblock, the position of that macroblock is clipped to the visible picture and its displacement still averages the whole macroblock. The displacement is taken from the prediction list 0, or from the list 1 for macroblocks predicted from the list 1 only. For intra macroblocks (all macroblocks of I and IDR pictures) and for concealed macroblocks dx = dy = 0.

End of the stream

At the end of the stream the pictures that wait for the output order are still inside the library. To read them, call the method with an H264 frame of size 0 (no data pointer is needed) until it returns an empty vector:

cr::video::Frame endOfStream;
endOfStream.fourcc = cr::video::Fourcc::H264;
endOfStream.size = 0;
while (!extractor.getMotionVector(endOfStream).empty())
{
    // use the vectors of the delayed picture
}

To process another stream, create a new object.

Supported streams and damaged data

The library supports the same streams as the VDecoderH264 decoder:

Feature	Supported
Profiles	Baseline, Main, High (also Constrained, with and without scaling matrices)
Entropy coding	CAVLC and CABAC
Slice types	I, P and B (direct prediction spatial and temporal), several slices per picture
Pictures	progressive frames, 8 bits, 4:2:0
Reference pictures	up to 16 reference frames, reordering of the lists, sliding window and adaptive marking, long-term references, gaps in frame_num

Streams and NAL units of the following kinds are not supported and give no pictures (the NAL units are ignored): interlaced coding (field pictures, MBAFF), 4:2:2, 4:4:4 and monochrome video, bit depth above 8, lossless coding (transform bypass), slice groups (FMO), arbitrary slice order (ASO), redundant pictures, SP and SI slices, data partitioning, pictures larger than level 6.2 allows (139264 macroblocks, 1055 macroblocks per side), SVC and MVC extension NAL units.

Damaged data never makes the library fail:

• A NAL unit with invalid syntax is ignored (a damaged parameter set is not stored, a damaged slice header drops the slice).
• If the data of a slice becomes invalid in the middle, the decoded macroblocks are kept and the rest of the slice is dropped. Macroblocks that no slice covers are concealed: they have zero motion (dx = dy = 0) and refer to the most recent reference picture.
• Reference pictures that are missing from the stream are replaced by an internal picture without motion.
• The object stays usable after any data: the pictures that follow valid data are returned as usual.

The vectors of damaged streams are the same as the vectors of VDecoderH264 for the same data.

The memory of the pictures that wait for retrieval is limited in both libraries. If one call completes very many pictures (for example a file of 9000 pictures of 1280x720 passed in one call), VDecoderH264 drops the oldest waiting pictures earlier than this library, which stores a waiting picture in 8 bytes per macroblock. This is the only difference: every picture that both libraries return has the same vectors, and the pictures are in the same order.

Security and hardening

The library parses data from outside of the process, so it follows the code level requirements of the EU Cyber Resilience Act (Annex I) for an input parser:

• Untrusted input. The only input is the data of the frames passed to getMotionVector(...). Every size, count, index, offset and enumeration read from the stream is checked against the real size of the data and against a hard limit before it is used for an allocation, a loop bound or an index: the reader of the bits returns zeros past the end of the data and reports the error, parameter sets with sizes above the limits are rejected, the numbers of reference pictures, slices and macroblocks are limited, the picture buffers are allocated for the size of the sequence parameter set and limited by level 6.2. A truncated or inconsistent NAL unit is rejected, the object stays in a defined state.
• Memory and CPU use are bounded: see Limits and guarantees. Every loop is bounded by the size of the data or by a constant of the standard.
• No undefined behavior. No out-of-bounds access, no unaligned loads through casted pointers (memcpy is used), no signed overflow, no shift by an invalid count. Checked with AddressSanitizer and UndefinedBehaviorSanitizer on a large set of valid, damaged and hostile streams, with libFuzzer, and with static analysis (MSVC /analyze, clang-tidy, cppcheck).
• What the library does not do: it does not execute programs, load plugins or libraries, open files or network connections, create threads, read environment variables, write logs, use random numbers or contain cryptography. No secrets are processed.
• Exception safety. Allocation failures are handled at the interface: getMotionVector(...) returns an empty vector, no exception leaves the method, the destructor does not throw.
• Build hardening (CMake, every target of the repository): MSVC - /W4 /permissive- /sdl /GS /guard:cf, linker /DYNAMICBASE /NXCOMPAT /guard:cf; GCC and Clang - -Wall -Wextra -Wpedantic -Wshadow -Wconversion -Wsign-conversion -Wold-style-cast -Wcast-qual -Wformat=2 -Wnon-virtual-dtor -Woverloaded-virtual -Wdouble-promotion -Wimplicit-fallthrough -Wmissing-declarations -Wundef -Wextra-semi -Wzero-as-null-pointer-constant, -fstack-protector-strong, -D_FORTIFY_SOURCE=2 (not in Debug), -fstack-clash-protection, -fcf-protection=full (x86), -mbranch-protection=standard (AArch64) where the compiler supports them, position independent code, full RELRO and non-executable stack for shared builds. No -ffast-math, no -march=native, no instruction-set flags.
• Verification is described in Testing and verification.

Limits and guarantees

What	Limit / guarantee
Picture size	up to 139264 macroblocks and 1055 macroblocks per side (level 6.2, 16880 pixels). Larger sequence parameter sets are ignored.
Parameter sets	32 sequence and 256 picture parameter set identifiers.
Reference pictures	up to 16 reference frames, up to 32 entries per reference picture list.
Slices	up to 65534 slices per picture (further slices of the picture are ignored).
Pictures in memory	at most 32 pictures at a time (decoded picture buffer, output queue and the picture being parsed). The memory of a picture is 268 bytes per macroblock (about 2.1 MiB for 1920x1080), so the library needs at most about 66 MiB for 1920x1080 and about 1.1 GiB for the largest picture of level 6.2, only if the stream really uses all pictures.
Waiting pictures	a picture that waits for retrieval needs 8 bytes per macroblock and 128 bytes of bookkeeping; the pictures that wait are limited to 256 MiB in total (about 4000 pictures of 1920x1080, the oldest are dropped).
Input data	one call takes up to 2 GiB - 1 bytes (Frame::size is an int). The data of a NAL unit is copied once into a buffer that is reused.
Time	the time of a call is proportional to the number of macroblocks of the pictures it completes and to the size of the data. It is not bounded by the size of the data alone: a slice of a few dozen bytes (all macroblocks skipped) completes a picture of any size, and a picture of 139264 macroblocks costs about 8 ms (measured: 400 such pictures in a 29 KB stream took 3.2 s on a desktop CPU). There is no waiting and no blocking.
Thread safety	one object - one thread at a time; different objects are independent; no global mutable state (the constant tables are built at the first use in a thread-safe way).
Ownership	the returned vector belongs to the object and stays valid until the next call. The input data is not kept.
Errors	no exceptions leave the methods (only the constructor can throw std::bad_alloc); invalid data gives fewer pictures or concealed macroblocks, never a crash.
Same input, same output	the result depends only on the data and on the split of the data into calls, not on the platform, the compiler or the CPU.

Build and connect to your project

Prerequisites

• CMake version 3.13 or higher.
• C++17 compliant compiler (tested with MSVC 2022, GCC 12 and 13, Clang 15 and 18).
• Submodules of the repository (3rdparty/Frame, example/SimpleUI): clone with --recurse-submodules or run git submodule update --init --recursive.
• On Linux the window of the example application needs the packages libx11-dev and libxext-dev; without them the example is built with a stub window instead (use the --terminal mode then).

Build

Typical commands to build the library and the example application:

cd MvExtractorH264
cmake -B build -S .
cmake --build build --config Release

On Windows with Visual Studio: cmake -B build -S . -G "Visual Studio 17 2022" -A x64. The example application is built to build/bin (build/bin/Release for multi-configuration generators).

Cross-compilation for AArch64 (for example from x86-64 Ubuntu):

sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
cmake -B build_arm -S . \
    -DCMAKE_SYSTEM_NAME=Linux \
    -DCMAKE_SYSTEM_PROCESSOR=aarch64 \
    -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \
    -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build_arm -j

CMake options and macros: the library has no options and no macros that change the code. The repository builds the library and the example application when it is the top level project; when it is used as a subproject (add_subdirectory) only the library is built. The cache variables ${PARENT}_MVEXTRACTORH264 (build the library, default ON) and ${PARENT}_MVEXTRACTORH264_EXAMPLE (build the example application, ON for the top level project and OFF for a subproject) switch the parts; ${PARENT} is empty for the top level project (see Connect to CMake project).

Connect to CMake project

If you want to connect the MvExtractorH264 library to your CMake project as source code, you can do the following. For example, if your repository has the structure:

CMakeLists.txt
src
    CMakeList.txt
    yourLib.h
    yourLib.cpp

Create folder 3rdparty in your repository and copy MvExtractorH264 folder there. The new structure of your repository:

CMakeLists.txt
src
    CMakeList.txt
    yourLib.h
    yourLib.cpp
3rdparty
    MvExtractorH264

Create a CMakeLists.txt file in the 3rdparty folder. The CMakeLists.txt file should contain:

cmake_minimum_required(VERSION 3.13)

################################################################################
## 3RD-PARTY
## dependencies for the project
################################################################################
project(3rdparty LANGUAGES CXX)

################################################################################
## SETTINGS
## basic 3rd-party settings before use
################################################################################
# To inherit the top-level architecture when the project is used as a submodule.
SET(PARENT ${PARENT}_YOUR_PROJECT_3RDPARTY)
# Disable self-overwriting of parameters inside included subdirectories.
SET(${PARENT}_SUBMODULE_CACHE_OVERWRITE OFF CACHE BOOL "" FORCE)

################################################################################
## INCLUDING SUBDIRECTORIES
## Adding subdirectories according to the 3rd-party configuration
################################################################################
add_subdirectory(MvExtractorH264)

Next, you need to include the 3rdparty folder in the main CMakeLists.txt file of your repository. Add the following line at the end of your main CMakeLists.txt:

add_subdirectory(3rdparty)

Next, you need to include the MvExtractorH264 library in your src/CMakeLists.txt file:

target_link_libraries(${PROJECT_NAME} MvExtractorH264)

Include the public header in your code:

#include "MvExtractorH264.h"

The library links the Frame library (3rdparty/Frame) and exports its include directory, so #include "Frame.h" works in your code too.

Simple example

The example reads an H.264 Annex B file, passes it to the library one NAL unit at a time, prints the number of the macroblocks with motion for every picture and reads the delayed pictures at the end:

#include <cmath>
#include <cstddef>
#include <cstdint>
#include <fstream>
#include <iostream>
#include <iterator>
#include <vector>
#include "Frame.h"
#include "MvExtractorH264.h"

// Print the number of macroblocks with motion.
static void printPicture(int index, const std::vector<std::vector<float>>& vectors)
{
    int moving{0};
    for (const std::vector<float>& mv : vectors)
    {
        // mv = [x, y, dx, dy]
        if (std::abs(mv[2]) > 0.5f || std::abs(mv[3]) > 0.5f)
        {
            ++moving;
        }
    }
    std::cout << "Picture " << index << ": " << vectors.size() << " macroblocks, " << moving << " with motion" << std::endl;
}

int main(int argc, char* argv[])
{
    if (argc < 2)
    {
        std::cerr << "Usage: " << argv[0] << " file.h264" << std::endl;
        return 1;
    }

    // Read the file (a file of any size should be read in parts, see example/H264Parser.cpp).
    std::ifstream file(argv[1], std::ios::binary);
    std::vector<uint8_t> stream((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
    if (stream.empty())
    {
        std::cerr << "Can not read " << argv[1] << std::endl;
        return 1;
    }

    // Start of every NAL unit: the position of its 00 00 01 start code.
    std::vector<std::size_t> starts;
    for (std::size_t i{0}; i + 3 <= stream.size(); ++i)
    {
        if (stream[i] == 0 && stream[i + 1] == 0 && stream[i + 2] == 1)
        {
            starts.push_back(i);
            i += 2;
        }
    }

    cr::video::MvExtractorH264 extractor;
    cr::video::Frame frame;
    frame.fourcc = cr::video::Fourcc::H264;
    int index{0};
    for (std::size_t n{0}; n < starts.size(); ++n)
    {
        // The frame points to the data of the NAL unit (the frame does not own it).
        const std::size_t end{n + 1 < starts.size() ? starts[n + 1] : stream.size()};
        frame.data = stream.data() + starts[n];
        frame.size = static_cast<int>(end - starts[n]);
        const std::vector<std::vector<float>>& vectors = extractor.getMotionVector(frame);
        if (!vectors.empty())
        {
            printPicture(index++, vectors);
        }
    }

    // End of the stream: an H264 frame of size 0 returns the delayed pictures.
    frame.data = nullptr;
    frame.size = 0;
    for (;;)
    {
        const std::vector<std::vector<float>>& vectors = extractor.getMotionVector(frame);
        if (vectors.empty())
        {
            break;
        }
        printPicture(index++, vectors);
    }
    std::cout << index << " pictures" << std::endl;
    return 0;
}

The data of one call may contain any number of NAL units: one NAL unit as in the example, an access unit (the data of one picture) or more. A default constructed Frame owns no memory, so the example can point frame.data to its own buffer; do not assign data of a frame that allocated its memory (call release() first, see the Frame class).

Example application

The repository includes the example application MvExtractorH264Example (example/main.cpp). It reads an H.264 elementary stream file (*.h264, Annex B) with the H264Parser class, passes the access units to MvExtractorH264 and shows the motion vectors, not the video: the vectors are drawn as green lines on a black background, or printed to the terminal.

MvExtractorH264Example [file.h264] [--window | --terminal] [--summary]

Option	Description
file.h264	The file to process. If it is not given, the application asks for it.
--window	Show the vectors in a window. A line goes from the center of every macroblock with motion (vectors with both components not above 0.5 pixel are not drawn) to the displaced position. The window is refreshed with the frame rate of the stream (33 ms if the stream does not tell it). ESC or the close button stops the application.
--terminal	Print the vectors to the terminal: a line with the number of macroblocks and of macroblocks with motion for every picture and the time of the extraction, then the vector of every macroblock with motion (x, y - position of the macroblock, dx, dy - displacement in pixels; vectors with both components not above 0.5 pixel are not printed).
--summary	In the terminal mode print only the line of every picture.

If neither --window nor --terminal is given, the application asks whether to show the vectors in a window (the answer n or an empty answer prints them to the terminal). At the end the application prints the number of pictures and the total and mean time of the extraction. The exit code is 0 on success, 1 for an invalid command line, a missing file or a window that can not be created, and 2 if the file has no pictures.

Examples:

./build/bin/MvExtractorH264Example video.h264 --window
./build/bin/MvExtractorH264Example video.h264 --terminal --summary

You can make a test file with ffmpeg:

ffmpeg -f lavfi -i testsrc2=s=1280x720:r=30 -t 5 -pix_fmt yuv420p -c:v libx264 -profile:v high -bf 2 -f h264 video.h264

---

Table of contents