RpiTrackerControl application
v1.1.4
Table of contents
- Overview
- Versions
- Source code files
- Config file
- Run application
- How to send tracker commands
- Source code
Overview
RpiTrackerControl is template control application that works with RpiTrackerOnboard. It provides remote control for RpiTrackerOnboard as well as showing real time stream that comes from RpiTrackerOnboard. The application combines libraries: VCodecLibav (for decoding H264 frame with software codec), VTracker (for preparing tracker commands and decoding tracker results), UdpDataChannel (for communication with RpiTrackerOnboard), FormatConverterOpenCv (for converting raw pixel format) and Logger (for printing logs). The application shows how to control tracker and receive real time stream from remote RpiTrackerOnboard. Structure of video processing pipeline:
After start, application reads JSON config file which includes communication port and ip. If there is no config file the application will create new one with default parameters. The application utilizes UdpDataClient to read the input buffer, from which it extracts and parses both tracker results and the encoded frame. Following this, it decodes the frame and tracker results. Finally, it displays the real-time stream along with the tracker results.
Versions
Table 1 - Application versions.
| Version | Release date | What’s new |
|---|---|---|
| 1.0.0 | 05.03.2024 | - First version. |
| 1.1.0 | 13.03.2024 | - Codec implementation is moved to software codec. |
| 1.1.1 | 31.05.2024 | - Documentation updated. |
| 1.1.2 | 06.08.2024 | - Submodules updated. |
| 1.1.3 | 18.09.2024 | - VCodecLibav submodule updated. |
| 1.1.4 | 04.12.2024 | - VCodecLibav submodule updated. |
Source code files
The application is supplied by source code only. The user is given a set of files in the form of a CMake project (repository). The repository structure is shown below:
CMakeLists.txt --------------------- Main CMake file.
3rdparty --------------------------- Folder with third-party libraries.
CMakeLists.txt ----------------- CMake file to include third-party libraries.
UdpDataChannel ----------------- Folder with UdpDataChannel library source code.
VCodecLibav -------------------- Folder with VCodecLibav library source code.
VTracker ----------------------- Folder with VTracker library source code.
FormatConverterOpenCv ---------- Folder with FormatConverterOpenCv library source code.
src -------------------------------- Folder with application source code.
CMakeLists.txt ----------------- CMake file.
RpiTrackerControlVersion.h ----- Header file with application version.
RpiTrackerControlVersion.h.in -- File for CMake to generate version header.
main.cpp ----------------------- Application source code file.
Config file
RpiTrackerControl application reads config file RpiTrackerControl.json in the same folder with application executable file. Config file content:
{
"Params":
{
"communication":
{
"dataUdpPort": 50021,
"serverIp": "192.168.0.2"
}
}
}
Table 2 - Config file parameters description.
| Parameter | type | Description |
|---|---|---|
| Communication: | ||
| dataUdpPort | int | UDP port number to send commands for tracker and get video stream. |
| serverIp | string | Ip of server (system that RpiTrackerOnboard runs) |
Run application
-
Copy application (RpiTrackerControl executable and RpiTrackerControl.json) to any folder.
-
Copy all files under /3rdparty/dllForExecutable into same folder that you have RpiTrackerControl executable.
-
Run :
./RpiTrackerControl -vv
-vv or -v arguments enables logger to print logs to console and file.
When the application is started for the first time, it creates a configuration file named RpiTrackerControl.json if file does not exist (refer to the Config file section). If the application is run as a superuser using sudo, the file will be owned by the root user. Therefore, to modify the configuration file, superuser privileges will be necessary.
How to send tracker commands
The RpiTrackerControl application transmits various tracker commands to the RpiTrackerOnboard application, some of which are specifically related to keyboard and mouse functions. Below is a list of these tracker commands that pertain to keyboard and mouse operations.
-
Main tracker commands.
-
Start tracking : Click mouse left button.
-
Reset tracker : Click mouse right button.
-
-
Tracking rectangle dimension commands.
-
Increase tracking rectangle width : Press D on keyboard.
-
Decrease tracking rectangle width : Press A on keyboard.
-
Increase tracking rectangle height : Press W on keyboard.
-
Decrease tracking rectangle height : Press S on keyboard.
-
-
Move tracking rectangle by keyboard.
-
Move right : Press H on keyboard.
-
Mode left : Press F on keyboard.
-
Move up : Press T on keyboard.
-
Move down : Press G on keyboard.
-
Source code
Bellow source code of main.cpp file:
#include <iostream>
#include "VTracker.h"
#include "FormatConverterOpenCv.h"
#include "VCodecLibav.h"
#include "UdpDataClient.h"
#include "RpiTackerControlVersion.h"
#include "ConfigReader.h"
#include "Logger.h"
/// Log folder.
#define LOG_FOLDER "Log"
/// Config file name.
#define CONFIG_FILE_NAME "RpiTackerControl.json"
/// Application parameters.
class Params
{
public:
/// Communication.
class Communication
{
public:
/// Output UDP port.
int dataUdpPort{50021};
/// Server ip.
std::string serverIp{"192.168.0.2"};
JSON_READABLE(Communication, dataUdpPort, serverIp)
};
/// Control channel.
Communication communication;
JSON_READABLE(Params, communication)
};
/// Application params.
Params g_params;
/// Log flag.
cr::utils::PrintFlag g_logFlag{cr::utils::PrintFlag::DISABLE};
/// Logger.
cr::utils::Logger g_log;
/// Upd channel.
cr::clib::UdpDataClient g_client;
/// Tracker params.
cr::vtracker::VTrackerParams g_sharedTrackerParams;
/// Init shared frame with initial default size.
int g_width = 1280, g_height = 720;
cr::video::Frame g_sharedFrame(g_width, g_height, cr::video::Fourcc::BGR24);
/// Shared data (shared frame and shared tracker params) mutex.
std::mutex g_sharedDataMtx;
/// Flag to stop communication thread.
std::atomic<bool> g_stopCommunicationThread{false};
/**
* @brief Load configuration params from JSON file.
* @return TRUE if parameters loaded or FALSE if not.
*/
bool loadConfig();
/**
* @brief Mouse callback function.
* @param event Event ID.
* @param x Mouse cursor horizontal position.
* @param y Mouse cursor vertical position.
* @param flags Flags.
* @param userdata User data.
*/
void mouseCallBackFunction(int event, int x, int y, int flags, void* userdata);
/**
* @brief Process keyboard events.
* @param key Key code.
*/
bool processKeyboardEvents(int key);
/**
* @brief Communication thread function.
*/
void communicationThreadFunction();
int main(int argc, char **argv)
{
// Check arguments and set console output if necessary.
if (argc > 1)
{
std::string str = std::string(argv[1]);
if (str == "-v" || str == "-vv")
{
g_logFlag = cr::utils::PrintFlag::CONSOLE_AND_FILE;
}
}
// Configure logger.
cr::utils::Logger::setSaveLogParams(LOG_FOLDER, "log", 20, 1);
// Welcome message.
g_log.print(cr::utils::PrintColor::YELLOW, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] " <<
"RpiTackerControl application v" << RPI_TRACKER_CONTROL_VERSION << std::endl;
// Load config file.
if (!loadConfig())
{
g_log.print(cr::utils::PrintColor::RED, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] ERROR: " <<
"Can't load config file." << std::endl;
return -1;
}
g_log.print(cr::utils::PrintColor::CYAN, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] " <<
"Config file loaded." << std::endl;
// Init UDP channel.
if (!g_client.init(g_params.communication.serverIp, g_params.communication.dataUdpPort))
{
g_log.print(cr::utils::PrintColor::RED, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] ERROR: " <<
"Can't initialize UDP client." << std::endl;
return -1;
}
// Init buffer for input data. Enough big for any frame.
const int bufferSize = 1920 * 1080 * 5;
uint8_t* buffer = new uint8_t[bufferSize];
// Create window and set mouse callback function.
cv::namedWindow("RpiTackerControl application", cv::WINDOW_AUTOSIZE);
cv::setMouseCallback("RpiTackerControl application", mouseCallBackFunction, nullptr);
// Start communication thread.
std::thread communicationThread(communicationThreadFunction);
// Frame for shwoing.
cv::Mat showFrame(g_height, g_width, CV_8UC3);
// Tracker params.
cr::vtracker::VTrackerParams trackerParams;
while (true)
{
// Get shared data.
g_sharedDataMtx.lock();
if (showFrame.cols != g_sharedFrame.width || showFrame.rows != g_sharedFrame.height)
{
showFrame.release();
showFrame = cv::Mat(g_sharedFrame.height, g_sharedFrame.width, CV_8UC3);
}
memcpy(showFrame.data, g_sharedFrame.data, g_sharedFrame.size);
trackerParams = g_sharedTrackerParams;
g_sharedDataMtx.unlock();
std::string info = "Processing time : " + std::to_string(trackerParams.processingTimeMks / 1000) + " ms";
// Draw text.
cv::putText(showFrame, info, cv::Point(40, 30), cv::FONT_HERSHEY_SIMPLEX, 1, cv::Scalar(0, 255, 0), 2);
// Draw rectangle.
cv::Scalar rectangleColor(255, 128, 128);
if (trackerParams.mode == 0)
{
rectangleColor = cv::Scalar(255, 128, 128); // Light pink in free mode.
}
else if (trackerParams.mode == 1)
{
rectangleColor = cv::Scalar(76, 84, 255); // Medium blue in tracking mode.
}
else
{
rectangleColor = cv::Scalar(29, 255, 107); // Bright green in other modes like inertial and static.
}
cv::Mat image(showFrame.rows, showFrame.cols, CV_8UC3, showFrame.data);
int x0 = trackerParams.rectX - trackerParams.rectWidth / 2;
int y0 = trackerParams.rectY - trackerParams.rectHeight / 2;
cv::rectangle(image, cv::Rect(x0, y0, trackerParams.rectWidth , trackerParams.rectHeight), rectangleColor, 2);
// Show frame.
cv::imshow("RpiTackerControl application", showFrame);
int key = cv::waitKey(1);
if (!processKeyboardEvents(key))
{
g_stopCommunicationThread = true;
break;
}
}
communicationThread.join();
return 1;
}
bool loadConfig()
{
// Init variables.
cr::utils::ConfigReader config;
// Open config json file (if not exist - create new and exit).
if(config.readFromFile(CONFIG_FILE_NAME))
{
// Read values and set to params.
if(!config.get(g_params, "Params"))
{
g_log.print(cr::utils::PrintColor::RED, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] ERROR: " <<
"Params were not read" << std::endl;
return false;
}
}
else
{
// Set default params.
config.set(g_params, "Params");
// Save config file.
if (!config.writeToFile(CONFIG_FILE_NAME))
{
g_log.print(cr::utils::PrintColor::CYAN, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] INFO: " <<
"Config file created." << std::endl;
return false;
}
}
return true;
}
void mouseCallBackFunction(int event, int x, int y, int flags, void* userdata)
{
// Prepare command.
uint8_t command[32];
int size = 0;
switch (event)
{
/// Capture object on current coordinates.
case cv::EVENT_LBUTTONDOWN:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::CAPTURE, -1, -1, -1);
break;
}
/// Reset tracker.
case cv::EVENT_RBUTTONDOWN:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::RESET);
break;
}
case cv::EVENT_MOUSEMOVE:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::SET_RECT_POSITION,
static_cast<float>(x), static_cast<float>(y));
break;
}
}
g_client.send(command, size);
}
bool processKeyboardEvents(int key)
{
// Command buffer.
uint8_t command[32];
int size = 0;
// Process keyboard events.
switch (key)
{
// ESC - Exit.
case 27:
{
cv::destroyAllWindows();
return false;
}
// W - Increase tracking rectangle height.
case 119:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::CHANGE_RECT_SIZE, 0, 4);
break;
}
// S - Decrease tracking rectangle height.
case 115:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::CHANGE_RECT_SIZE, 0, -4);
break;
}
// D - Increase tracking rectangle width.
case 100:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::CHANGE_RECT_SIZE, 4, 0);
break;
}
// A - Decrease tracking rectangle width.
case 97:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::CHANGE_RECT_SIZE, -4, 0);
break;
}
// T - Move strobe UP (change position in TRACKING mode).
case 116:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::MOVE_RECT, 0, -4);
break;
}
// G - Move strobe DOWN (change position in TRACKING mode).
case 103:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::MOVE_RECT, 0, 4);
break;
}
// H - Move strobe RIGHT (change position in TRACKING mode).
case 104:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::MOVE_RECT, 4, 0);
break;
}
// F - Move strobe LEFT (change position in TRACKING mode).
case 102:
{
cr::vtracker::VTracker::encodeCommand(
command, size, cr::vtracker::VTrackerCommand::MOVE_RECT, -4, 0);
break;
}
}
// Send command.
g_client.send(command, size);
return true;
}
void communicationThreadFunction()
{
/// Init buffer for input data.
const int bufferSize = 1920 * 1080 * 4;
uint8_t* buffer = new uint8_t[bufferSize];
/// Frames.
cr::video::Frame encodedFrame;
encodedFrame.fourcc = cr::video::Fourcc::H264;
cr::video::Frame decodedFrame;
decodedFrame.fourcc = cr::video::Fourcc::NV12;
cr::video::Frame bgrFrame;
bgrFrame.fourcc = cr::video::Fourcc::BGR24;
cr::video::Frame emptyFrame (g_width, g_height, cr::video::Fourcc::BGR24);
// Error message when there is no data.
cv::Mat errorMessage(emptyFrame.height, emptyFrame.width, CV_8UC3, emptyFrame.data);
std::string text = "NO INPUT DATA";
int fontFace = cv::FONT_HERSHEY_SIMPLEX;
double fontScale = 2;
int thickness = 2;
// Calculate text size
int baseline = 0;
cv::Size textSize = cv::getTextSize(text, fontFace, fontScale, thickness, &baseline);
baseline += thickness;
// Center the text
cv::Point textOrg((errorMessage.cols - textSize.width) / 2, (errorMessage.rows + textSize.height) / 2);
// Render the text on the frame
cv::putText(errorMessage, text, textOrg, fontFace, fontScale, cv::Scalar(0, 0, 255), thickness, 8);
/// Tracker params.
cr::vtracker::VTrackerParams emptyParams;
cr::vtracker::VTrackerParams trackerParams;
/// Codec.
cr::video::VCodecLibav codec;
codec.setParam(cr::video::VCodecParam::TYPE, 1); // set codec type to software codec.
/// Convertor.
cr::video::FormatConverterOpenCv convertor;
while (true)
{
if (g_stopCommunicationThread)
{
break;
}
// Wait new data.
int size = 0;
if (!g_client.get(buffer, bufferSize, size, 1000))
{
g_sharedDataMtx.lock();
if (emptyFrame.width != g_sharedFrame.width || emptyFrame.height != g_sharedFrame.height)
{
emptyFrame.release();
emptyFrame = cr::video::Frame(g_sharedFrame.width, g_sharedFrame.height, cr::video::Fourcc::BGR24);
errorMessage.release();
errorMessage = cv::Mat(emptyFrame.height, emptyFrame.width, CV_8UC3, emptyFrame.data);
// Center the text
cv::Point textOrg((errorMessage.cols - textSize.width) / 2, (errorMessage.rows + textSize.height) / 2);
cv::putText(errorMessage, text, textOrg, fontFace, fontScale, cv::Scalar(0, 0, 255), thickness, 8);
}
g_sharedFrame = emptyFrame;
g_sharedTrackerParams = emptyParams;
g_sharedDataMtx.unlock();
continue;
}
// Get tracker data size.
int trackerParamsSize = 0;
memcpy(&trackerParamsSize, &buffer[0], sizeof(int));
if (!trackerParams.decode(&buffer[sizeof(int)], trackerParamsSize))
{
g_log.print(cr::utils::PrintColor::RED, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] ERROR: " <<
"Can't decode input data." << std::endl;
continue;
}
// Deserialize frame data.
int frameSize = 0;
memcpy(&frameSize, &buffer[sizeof(int) + trackerParamsSize], sizeof(int));
if (!encodedFrame.deserialize(&buffer[sizeof(int) + trackerParamsSize + sizeof(int)], frameSize))
{
g_log.print(cr::utils::PrintColor::RED, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] ERROR: " <<
"Can't deserialize frame data." << std::endl;
continue;
}
// Decode frame.
if (!codec.transcode(encodedFrame, decodedFrame))
{
g_log.print(cr::utils::PrintColor::RED, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] ERROR: " <<
"Can't decode frame." << std::endl;
continue;
}
// Convert frame to BGR.
if (!convertor.convert(decodedFrame, bgrFrame))
{
g_log.print(cr::utils::PrintColor::RED, g_logFlag) <<
"[" << __LOGFILENAME__ << "][" << __LINE__ << "] ERROR: " <<
"Can't convert frame to BGR." << std::endl;
continue;
}
g_sharedDataMtx.lock();
if (g_sharedFrame.width != bgrFrame.width || g_sharedFrame.height != bgrFrame.height)
{
g_sharedFrame.release();
g_sharedFrame = cr::video::Frame(bgrFrame.width, bgrFrame.height, cr::video::Fourcc::BGR24);
}
g_sharedFrame = bgrFrame;
g_sharedTrackerParams = trackerParams;
g_sharedDataMtx.unlock();
}
// Give main thread time to join.
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}