[mrpt-img]

Overview

Basic computer vision data structures and tools: bitmap images, canvas, color maps, pinhole camera models, and image undistortion.

Library mrpt-img

This C++ library is part of MRPT and can be installed in Debian-based systems with:

sudo apt install libmrpt-img-dev

Image handling

The class mrpt::img::CImage is a portable image container supporting grayscale, RGB, and RGBA images in 8-bit depth. It provides loading/saving (via STB), pixel access, drawing primitives, resizing, and color conversion.

In MRPT 3.0, CImage no longer depends on OpenCV. All image I/O and processing is handled natively (STB for codecs, custom kernels for filtering and resampling).

Camera models and distortion

mrpt::img::TCamera stores pinhole intrinsics and lens distortion coefficients for three models (see mrpt::img::DistortionModel):

DistortionModel::none — ideal pinhole
DistortionModel::plumb_bob — radial + tangential (Brown-Conrady, 5 params)
DistortionModel::kannala_brandt — fisheye / equidistant (4 params)

Distortion/undistortion math is in mrpt::img::camera_geometry (header: <mrpt/img/camera_geometry.h>).

Image undistortion

mrpt::img::CUndistortMap — precomputes a remap table for a single camera, then applies it efficiently to any number of frames. Much faster than calling CImage::undistort() repeatedly with the same camera params.
mrpt::img::CStereoRectifyMap — Bouguet-style stereo rectification for a calibrated stereo pair. Produces remap tables for both cameras so that epipolar lines become horizontal.
CImage::undistort() — convenience one-shot undistortion (internally creates a CUndistortMap).

All three support every DistortionModel defined in TCamera.

Image pyramids

mrpt::img::CImagePyramid builds a multi-scale (octave) image pyramid by repeated half-size decimation, with optional Gaussian smoothing. buildPyramidFast() moves the source image into the pyramid to avoid a copy.

Eigen interop

<mrpt/img/CImage_Eigen.h> provides zero-copy Eigen::Map views over grayscale CImage data:

#include <mrpt/img/CImage_Eigen.h>
mrpt::img::CImage img(640, 480, mrpt::img::CH_GRAY);
auto map = mrpt::img::asEigenMap(img);  // Eigen::Map with stride
map.setZero();  // works directly on image memory

SIMD (SSE/AVX) optimizations

MRPT supports optional SIMD-optimized code paths for image operations on Intel/AMD CPUs. The system uses compile-time feature detection via #if MRPT_ARCH_INTEL_COMPATIBLE and separate translation units per instruction set:

** CImage.SSE2.cpp **: SSE2 optimizations (scale-half for 1-channel 8-bit, smooth scale-half).
** CImage.SSSE3.cpp **: SSSE3 optimizations (scale-half for 3-channel RGB, RGB/BGR to grayscale).
** CImage.SSEx.h **: Shared declarations for all SIMD-optimized functions (private header in src/).

Current SIMD-optimized functions:

image_SSE2_scale_half_1c8u() — Grayscale 1:2 decimation
image_SSE2_scale_half_smooth_1c8u() — Grayscale smooth (2x2 average) decimation
image_SSSE3_scale_half_3c8u() — RGB 1:2 decimation
image_SSSE3_rgb_to_gray_8u() — RGB to grayscale conversion
image_SSSE3_bgr_to_gray_8u() — BGR to grayscale conversion

Pattern for adding new SIMD kernels:

Create a new .cpp file named CImage.<ISA>.cpp (e.g., CImage.AVX2.cpp).
Guard the entire file with #if MRPT_ARCH_INTEL_COMPATIBLE and the appropriate #include <immintrin.h>.
Add the function declaration to CImage.SSEx.h.
Call the SIMD function from the main CImage.cpp with a runtime fallback to scalar code.
Functions return bool indicating whether the SIMD path was taken (used by CImagePyramid::buildPyramid()).

The bilinear remap in remap_bilinear.h (used by CUndistortMap and CStereoRectifyMap) is a candidate for future SSE2/AVX2 optimization.

Library contents

Color map functions (in #include

// typedefs

typedef TPixelCoordBase<int32_t> mrpt::img::TPixelCoord;
typedef TPixelCoord mrpt::img::TImageSize;
typedef TPixelCoordBase<float> mrpt::img::TPixelCoordf;

// enums

enum mrpt::img::DistortionModel;
enum mrpt::img::TInterpolationMethod;
enum mrpt::img::VideoCodec;

// structs

struct mrpt::img::TColor;
struct mrpt::img::TColorf;

template <typename T>
struct mrpt::img::TPixelCoordBase;

// classes

class mrpt::img::CCanvas;
class mrpt::img::CImage;
class mrpt::img::CImagePyramid;
class mrpt::img::CMappedImage;
class mrpt::img::CStereoRectifyMap;
class mrpt::img::CUndistortMap;
class mrpt::img::CVideoFileWriter;
class mrpt::img::TCamera;

// global functions

void mrpt::img::registerAllClasses_mrpt_img();

void mrpt::img::camera_geometry::projectPoints(
    const std::vector<mrpt::math::TPoint3D>& points3D,
    const mrpt::math::TPose3D& cameraPose,
    const mrpt::math::CMatrixDouble33& intrinsicParams,
    std::vector<mrpt::img::TPixelCoordf>& projectedPoints,
    bool acceptPointsBehind = false
    );

void mrpt::img::camera_geometry::projectPoints_with_distortion(
    const std::vector<mrpt::math::TPoint3D>& points3D,
    const mrpt::math::TPose3D& cameraPose,
    const mrpt::img::TCamera& cameraParams,
    std::vector<mrpt::img::TPixelCoordf>& projectedPoints,
    bool acceptPointsBehind = false
    );

void mrpt::img::camera_geometry::projectPoints_with_distortion(
    const std::vector<mrpt::math::TPoint3D>& points3D,
    const mrpt::img::TCamera& cameraParams,
    const mrpt::math::TPose3D& cameraPose,
    std::vector<mrpt::img::TPixelCoordf>& projectedPoints,
    bool acceptPointsBehind = false
    );

void mrpt::img::camera_geometry::projectPoint_with_distortion(
    const mrpt::math::TPoint3D& pointInCamFrame,
    const mrpt::img::TCamera& cameraParams,
    mrpt::img::TPixelCoordf& pixel,
    bool acceptPointsBehind = false
    );

template <bool INVERSE_CAM_POSE>
mrpt::img::TPixelCoordf mrpt::img::camera_geometry::projectPoint(
    const mrpt::img::TCamera& cameraParams,
    const mrpt::math::TPose3D& cameraPose,
    const mrpt::math::TPoint3D& point3D
    );

template <typename POINT>
mrpt::img::TPixelCoordf mrpt::img::camera_geometry::projectPoint(const POINT& pointInCamFrame, const mrpt::img::TCamera& cameraParams);

void mrpt::img::camera_geometry::undistort_points(
    const std::vector<mrpt::img::TPixelCoordf>& distortedPixels,
    std::vector<mrpt::img::TPixelCoordf>& undistortedPixels,
    const mrpt::img::TCamera& cameraParams
    );

void mrpt::img::camera_geometry::undistort_points(
    const std::vector<mrpt::img::TPixelCoordf>& distortedPixels,
    std::vector<mrpt::img::TPixelCoordf>& undistortedPixels,
    const mrpt::math::CMatrixDouble33& intrinsicParams,
    const std::vector<double>& distortionParams
    );

void mrpt::img::camera_geometry::undistort_point(const mrpt::img::TPixelCoordf& distortedPt, mrpt::img::TPixelCoordf& undistortedPt, const mrpt::img::TCamera& cameraParams);

void mrpt::img::camera_geometry::undistort_points_to_unit_plane(
    const std::vector<mrpt::img::TPixelCoordf>& distortedPixels,
    std::vector<mrpt::math::TPoint2D>& normalizedCoords,
    const mrpt::img::TCamera& cameraParams
    );

Typedefs

typedef TPixelCoordBase<int32_t> mrpt::img::TPixelCoord

A pair (x,y) of pixel coordinates (integer resolution).

<>

typedef TPixelCoord mrpt::img::TImageSize

A type for image sizes.

<>

typedef TPixelCoordBase<float> mrpt::img::TPixelCoordf

A pair (x,y) of pixel coordinates (subpixel resolution).

<>

Global Functions

void mrpt::img::registerAllClasses_mrpt_img()

Forces manual RTTI registration of all serializable classes in this namespace.

Should never be required to be explicitly called by users, except if building MRPT as a static library.

void mrpt::img::camera_geometry::projectPoints(
    const std::vector<mrpt::math::TPoint3D>& points3D,
    const mrpt::math::TPose3D& cameraPose,
    const mrpt::math::CMatrixDouble33& intrinsicParams,
    std::vector<mrpt::img::TPixelCoordf>& projectedPoints,
    bool acceptPointsBehind = false
    )

Project 3D points to image plane without distortion (pinhole model only).

This applies only the intrinsic camera matrix transformation:

\[u = f_x \frac{X}{Z} + c_x, \quad v = f_y \frac{Y}{Z} + c_y\]

Points behind the camera (Z≤0 in camera frame) are marked with pixel coordinates (-1,-1) unless acceptPointsBehind=true.

Parameters:

points3D	[IN] 3D points in world coordinates (meters)
cameraPose	[IN] Camera pose in world coordinates
intrinsicParams	[IN] 3x3 camera calibration matrix K
projectedPoints	[OUT] Projected pixel coordinates (resized automatically)
acceptPointsBehind	[IN] If false, points with Z≤0 are marked as (-1,-1)

See also:

projectPoints_with_distortion

void mrpt::img::camera_geometry::projectPoints_with_distortion(
    const std::vector<mrpt::math::TPoint3D>& points3D,
    const mrpt::math::TPose3D& cameraPose,
    const mrpt::img::TCamera& cameraParams,
    std::vector<mrpt::img::TPixelCoordf>& projectedPoints,
    bool acceptPointsBehind = false
    )

Project 3D points to image plane with lens distortion.

Applies the full camera model:

Transform point to camera frame: P_cam = cameraPose^{-1} * P_world
Normalize: (x,y) = (X/Z, Y/Z)
Apply distortion: (x_d, y_d) = distort(x, y)
Project to pixels: (u,v) = (f_x*x_d + c_x, f_y*y_d + c_y)

Parameters:

points3D	[IN] 3D points in world coordinates (meters)
cameraPose	[IN] Camera pose in world coordinates
cameraParams	[IN] Complete camera model (intrinsics + distortion)
projectedPoints	[OUT] Projected pixel coordinates (resized automatically)
acceptPointsBehind	[IN] If false, points with Z≤0 are marked as (-1,-1)

void mrpt::img::camera_geometry::projectPoint_with_distortion(
    const mrpt::math::TPoint3D& pointInCamFrame,
    const mrpt::img::TCamera& cameraParams,
    mrpt::img::TPixelCoordf& pixel,
    bool acceptPointsBehind = false
    )

Project a single 3D point (in camera frame) to image plane with distortion.

Parameters:

pointInCamFrame	[IN] 3D point in camera coordinate frame (X right, Y down, Z forward)
cameraParams	[IN] Complete camera model (intrinsics + distortion)
pixel	[OUT] Projected pixel coordinates
acceptPointsBehind	[IN] If false, points with Z≤0 are marked as (-1,-1)

See also:

projectPoints_with_distortion

template <bool INVERSE_CAM_POSE>
mrpt::img::TPixelCoordf mrpt::img::camera_geometry::projectPoint(
    const mrpt::img::TCamera& cameraParams,
    const mrpt::math::TPose3D& cameraPose,
    const mrpt::math::TPoint3D& point3D
    )

Project a single 3D point without distortion (template version).

Parameters:

INVERSE_CAM_POSE	How camera pose F is interpreted: false: Point in camera = P ⊖ F (inverse composition, typical use) true: Point in camera = F ⊕ P (direct composition)
cameraParams	[IN] Camera intrinsic parameters
cameraPose	[IN] Camera pose
point3D	[IN] 3D point in world coordinates

Returns:

Projected pixel coordinates

template <typename POINT>
mrpt::img::TPixelCoordf mrpt::img::camera_geometry::projectPoint(
    const POINT& pointInCamFrame,
    const mrpt::img::TCamera& cameraParams
    )

Project a single 3D point (already in camera frame) without distortion.

Parameters:

POINT	Any type with .x, .y, .z members
pointInCamFrame	[IN] 3D point in camera coordinate frame
cameraParams	[IN] Camera intrinsic parameters

Returns:

Projected pixel coordinates

void mrpt::img::camera_geometry::undistort_points(
    const std::vector<mrpt::img::TPixelCoordf>& distortedPixels,
    std::vector<mrpt::img::TPixelCoordf>& undistortedPixels,
    const mrpt::img::TCamera& cameraParams
    )

Remove lens distortion from pixel coordinates (batch version).

Converts distorted pixel coordinates to undistorted pixel coordinates using iterative refinement. The output pixels can be used with the undistorted camera model (distortion=none).

Parameters:

distortedPixels	[IN] Distorted pixel coordinates as captured by camera
undistortedPixels	[OUT] Undistorted pixel coordinates (resized automatically)
cameraParams	[IN] Complete camera model including distortion parameters

void mrpt::img::camera_geometry::undistort_point(
    const mrpt::img::TPixelCoordf& distortedPt,
    mrpt::img::TPixelCoordf& undistortedPt,
    const mrpt::img::TCamera& cameraParams
    )

Remove lens distortion from a single pixel coordinate.

Parameters:

distortedPt	[IN] Distorted pixel coordinates
undistortedPt	[OUT] Undistorted pixel coordinates
cameraParams	[IN] Complete camera model

See also:

undistort_points

void mrpt::img::camera_geometry::undistort_points_to_unit_plane(
    const std::vector<mrpt::img::TPixelCoordf>& distortedPixels,
    std::vector<mrpt::math::TPoint2D>& normalizedCoords,
    const mrpt::img::TCamera& cameraParams
    )

Convert distorted pixels to normalized image plane coordinates.

This is the key function for 3D reconstruction and ray casting. It:

Removes lens distortion
Converts to normalized coordinates on the z=1 plane
Returns coordinates that satisfy: pixel ray direction = (x, y, 1)

The output normalized coordinates (x,y) represent the intersection of the pixel’s ray with the z=1 plane in the camera coordinate system. For a 3D point P=(X,Y,Z) in camera frame, it projects to normalized coords (X/Z, Y/Z).

Usage example for 3D ray computation:

TPoint2D norm = normalizedCoords[i];
TPoint3D ray_direction(norm.x, norm.y, 1.0);
// Ray from camera origin: P(t) = camera_origin + t * ray_direction

Output are NOT pixel coordinates but normalized 3D ray parameters

Parameters:

distortedPixels	[IN] Distorted pixel coordinates from camera
normalizedCoords	[OUT] Normalized coordinates on z=1 plane (resized automatically)
cameraParams	[IN] Complete camera model

See also:

undistort_points