Vision Systems & Engineering

Robot Operating System (ROS)

The open-source robotics middleware ROS 2 as of its Jazzy distribution, covering nodes, topics, services and actions, DDS and the RMW layer, quality of service, discovery, launch and composition, tf2 and the REP 105 frames, the image and camera-info messages, message_filters, bags and distributions, with an example that ports the Python time synchronizers.

intermediate

The Robot Operating System (ROS) is not an operating system. It is open-source middleware and a set of libraries and tools for building robot software out of many cooperating programs. Each program, a node, publishes and subscribes to named, typed streams of messages. Cameras, depth sensors, detectors, trackers and planners all exchange data this way. This article covers ROS 2, the line that the ROS 2 documentation, the REP 2000 release schedule and the packages read here describe. The original ROS, now called ROS 1, appears only where those sources compare the two. The code is developed in public GitHub repositories, package by package, each under its own licence: the message_filters licence names Willow Garage and the Open Source Robotics Foundation as copyright holders.

For a vision engineer, ROS is the layer that carries frames and their metadata between processes and machines. Images travel as sensor_msgs/Image with a capture timestamp and a frame name; calibration travels beside them as CameraInfo; tf2 relates the camera’s frame to the robot and the world; and message_filters groups frames from several sensors that describe the same moment. Robot perception covers what robots do with such data. Multi-camera systems covers rig-level synchronization and calibration, and rotation representations covers REP 103’s axes, the _optical camera frames and the quaternion order of ROS messages. Facts below come from the Jazzy branches of the ROS 2 documentation (commit 929e37c), message_filters (05a6b5f), common_interfaces (a941f14) and rosbag2 (12b5e8b), and from the REP repository at commit 11ca24a, unless another source is named.

Purpose

A robot is a distributed system even when it fits in one chassis. A driver reads the camera, another the IMU, a third the wheels; perception, mapping, planning and control run at their own rates, often in different processes and sometimes on different computers. ROS gives these programs:

  • a common way to find each other and exchange data (topics, services, actions), with automatic discovery;
  • typed message definitions with agreed semantics, such as SI units and the image and camera-calibration messages;
  • conventions for coordinate frames and their relations over time (REP 103, REP 105, tf2);
  • tools to record, replay and inspect the traffic (ros2 bag, and the command-line tools);
  • a release process, with distributions supported for fixed periods on named platforms (REP 2000).

The Topics page describes the result as “a strongly-typed, anonymous publish/subscribe system”: a subscriber need not know which publisher produced a message, so components can be swapped without touching the rest of the system.

Core Capabilities

Nodes, topics, services and actions

A node is “a participant in the ROS 2 graph” that uses a client library (rclcpp for C++, rclpy for Python) to communicate. Nodes can talk within one process, across processes or across machines, and the Nodes page recommends that each node “do one logical thing”. In C++, a node can create sub-nodes that share its context and graph identity under a sub-namespace, avoiding the discovery overhead of separate nodes.

ROS applications communicate through three kinds of interface, each described in a small interface definition language (.msg, .srv and .action files):

  • Topics are for continuous data streams such as sensor data and robot state. Any number of publishers and subscribers may share a topic, and every subscriber receives what any publisher sends. Message fields are typed, and the core types carry fixed semantics; the Topics page’s example is the IMU message’s angular velocity, specified in radians per second.
  • Services are remote procedure calls that should return quickly, because the client is usually waiting; a .srv file holds a request and a response separated by ---. There should be one server per service name; which server answers is undefined if there are several.
  • Actions are long-running calls with feedback that can be cancelled or pre-empted, such as driving to a waypoint. An .action file holds a request, a result and a feedback part, each separated by ---.

Nodes also expose parameters that change their behaviour at run time.

DDS and the RMW layer

ROS 2 is built on DDS and its wire protocol RTPS, which provide discovery, serialization and transport. DDS is an industry standard with several implementations, and ROS 2 reaches them through an abstract ROS middleware interface, rmw, implemented once per vendor. The middleware page of the Jazzy documentation lists:

Product Licence RMW implementation Status
eProsima Fast DDS Apache 2 rmw_fastrtps_cpp Full support, default RMW, packaged with binary releases
Eclipse Cyclone DDS Eclipse Public License v2.0 rmw_cyclonedds_cpp Full support, packaged with binary releases
RTI Connext DDS commercial, research rmw_connextdds Full support; Connext installed separately
GurumNetworks GurumDDS commercial rmw_gurumdds_cpp Community support; GurumDDS installed separately

Fast DDS is the default when installed; otherwise the implementation whose package name comes first alphabetically is used. Nodes on different RMW implementations can often communicate, but this is not guaranteed, and the page lists combinations that fail for WString; it suggests that every part of a distributed system use the same ROS version and the same RMW implementation.

Quality of service

Each publisher and subscription carries a QoS profile made of policies:

  • history: keep last NN samples (with the depth as queue size) or keep all;
  • reliability: best effort, which may lose samples, or reliable, which may retry;
  • durability: volatile, or transient local, in which the publisher keeps samples for late-joining subscriptions;
  • deadline, lifespan, liveliness and lease duration.

The defaults for publishers and subscriptions are keep last with depth 10, reliable, volatile and system-default liveliness, chosen to resemble ROS 1. The sensor data profile uses best effort and a smaller queue, because timely readings matter more for sensor data than receiving every one. The page notes that even reliable delivery in ROS 2 runs over UDP.

Compatibility follows a request-versus-offer model: a connection is made only if no policy the subscription requests is more stringent than what the publisher offers. A best-effort publisher and a reliable subscription are incompatible; a reliable publisher serves a best-effort subscription. A volatile publisher and a transient-local subscription do not communicate. When a pair is incompatible, no messages pass, and both sides can register callbacks for “incompatible QoS” events. In our reading, this is the first thing to check when a camera topic is visible but a subscriber receives nothing: a publisher using the best-effort sensor data profile and a subscription left at the reliable default never connect.

Discovery

Discovery is automatic, through the middleware. A starting node advertises itself to other nodes with the same ROS domain, set by the ROS_DOMAIN_ID environment variable; nodes answer, advertise periodically so that late arrivals are found, and announce when they go offline. Connections are made only between compatible QoS settings. In our reading, different domain IDs therefore separate robots, or experiments, that share a network.

Launch and composition

A system of many nodes is started from a launch file, written in XML, YAML or Python and run with ros2 launch. It describes which programs to run, where, with which arguments and configuration, and the launch system monitors the processes it started.

Composition decides at deployment time whether nodes run in separate processes, with fault isolation and easier debugging, or together in one process, with lower overhead and optionally more efficient intra-process communication. The recommended way to write C++ code is as a component, a node built into a shared library, similar to a ROS 1 nodelet, which a component container loads at run time. The Jazzy documentation lists three generic containers: component_container (one single-threaded executor), component_container_mt (one multi-threaded executor) and component_container_isolated (an executor per component). For image pipelines, in our reading, this is how large frames avoid serialization between a driver and the nodes that consume them; the Composition page itself states only the lower overhead and the optional intra-process path.

tf2 and coordinate frames

tf2 keeps track of coordinate frames over time. It stores the relations between frames as a tree buffered in time and transforms points, vectors and poses between any two frames at a requested time, answering questions such as where the head frame was relative to the world five seconds ago. Transforms are broadcast by many nodes, each for part of the robot, and gathered by listeners; static transforms, such as a camera’s mounting on the chassis, are published once and kept without history. The Tf2 page cites a TePRA 2013 paper on the library, “tf: The transform library” (not read). The page warns about direction: a transform published in geometry_msgs/msg/Transform describes the frame, which is the inverse of the transform that maps data from one frame into the other, and the library inverts it as it walks the tree.

Two REPs fix the conventions:

  • REP 103 fixes SI units and right-handed systems, with xx forward, yy left and zz up for a body, and the _optical frames of cameras, with zz forward, xx right and yy down. The Image message’s header comment repeats the optical convention: origin at the optical centre, +x+x to the right of the image, +y+y down, +z+z into the image.
  • REP 105 names the frames of mobile platforms. base_link is attached rigidly to the robot base. odom is world-fixed; a pose in it is continuous but drifts without bound, and it is typically computed from wheel, visual or inertial odometry. map is world-fixed with zz up; a pose in it should not drift significantly but may jump when localization corrects it. earth is the ECEF origin, for several robots in different maps. Because each frame has exactly one parent, the tree is earth → map → odom → base_link, not map and odom both attached to base_link.

A camera’s optical frame hangs below base_link through the robot’s mounting transforms. REP 105 assigns the odom → base_link transform to an odometry source, visual odometry among them, and the corrective map → odom transform to a localization component, whose jumps make map a poor frame for local sensing; in our reading, SLAM is one such component.

Images and camera calibration

sensor_msgs/Image carries an uncompressed image:

  • a header whose stamp “should be acquisition time of image” and whose frame_id should be the camera’s optical frame;
  • height (rows) and width (columns), with (0,0)(0, 0) at the top-left corner;
  • encoding, a string from image_encodings.hpp: for example rgb8, bgr8, mono8, mono16, OpenCV-style types such as 16UC1 and 32FC1, Bayer patterns such as bayer_rggb8, and YUV formats such as uyvy, yuyv, nv21 and nv24;
  • is_bigendian and step, the full row length in bytes, with data of size step × rows.

In our reading of “full row length”, step may exceed width × bytes per pixel, so a reader must take each row from its own offset; the example below shows what happens otherwise.

Depth images follow REP 118. The canonical form is a 32-bit float per pixel giving depth along the camera’s zz axis in metres; drivers may instead publish 16-bit unsigned integers in millimetres on the image_raw topic, with 0 denoting an invalid depth. REP 118 keeps depth in the ordinary Image message, rather than a new type, so that image tools, viewers and image_transport work unchanged. Depth map covers the conversion of such images to points.

sensor_msgs/CameraInfo travels on a camera_info topic beside up to five image topics (image_raw, image, image_color, image_rect and image_rect_color). It holds:

  • the calibrated image size and a distortion model with its parameters DD; for plumb_bob, the five parameters (k1,k2,t1,t2,k3)(k_1, k_2, t_1, t_2, k_3);
  • KK, the 3 × 3 intrinsic matrix of the raw image, and RR, the rectification rotation for stereo, both row-major arrays of nine values;
  • PP, the 3 × 4 projection matrix of the rectified image, row-major. For a monocular camera Tx=Ty=0T_x = T_y = 0; for the right camera of a horizontal stereo pair Tx=−fx′BT_x = -f'_x B, with BB the baseline, as in stereo camera;
  • binning_x, binning_y and a region of interest, which may change without recalibration. Binning values of 0 mean 1, and the ROI is given in full-resolution, unbinned sensor coordinates (REP 104).

The message also says that an uncalibrated camera leaves DD, KK, RR and PP zeroed, and that clients may take K[0]=0K[0] = 0 to mean uncalibrated. Camera parameters shows how KK changes under binning and cropping.

Compression. CompressedImage carries a format string and a compressed buffer, and the values of format depend on the image transport used, that is, on the image_transport plugin that produced it. For the compressed_image_transport plugin the codec is one of JPEG, PNG or TIFF; for compressed_depth_image_transport the source is usually 16UC1 or 32FC1 and the codec PNG or RVL. In our reading, JPEG suits colour frames for viewing or logging, while depth needs a lossless codec; image file formats and image compression cover the codecs.

message_filters and time synchronization

Many vision nodes need frames from several sensors at once: a stereo pair, colour with depth, a camera with a LiDAR scan. message_filters provides filters that sit between subscriptions and a callback: a Subscriber source, a Cache that keeps the last NN messages for lookup by time, and synchronizers that call back with one message from each input. The index page of the documentation lists three policies for the C++ Synchronizer, each reading the timestamp from the message header:

  • ExactTime calls back only when every channel has a message with exactly the same stamp;
  • ApproximateEpsilonTime accepts stamps equal within an epsilon;
  • ApproximateTime “uses an adaptive algorithm” to match messages.

The C++ synchronizers take up to nine channels.

The Python module implements two classes, TimeSynchronizer and ApproximateTimeSynchronizer. Their names differ from the C++ policies, and so does the approximate algorithm.

The Python classes (__init__.py, commit 05a6b5f). Both keep one dictionary per input, keyed by stamp in nanoseconds; a second message with the same stamp replaces the first. On every incoming message they apply these rules:

  1. Store the message in its input’s queue. While that queue holds more than queue_size messages, delete the one with the smallest stamp, which is not necessarily the earliest to arrive.
  2. TimeSynchronizer finds the stamps present in every queue, emits the set for the oldest of them, and then deletes from every queue all messages older than the emitted stamp.
  3. ApproximateTimeSynchronizer takes slop, documented as a delay in seconds. For each other input it collects the queued stamps whose distance to the new message’s stamp is at most slop, sorted nearest first; if any input has none, it returns. It then walks the combinations of those candidates with itertools.product, in that nearest-first order, and emits the first combination whose spread, the newest stamp minus the oldest, is strictly less than slop. It deletes only the emitted messages; unmatched older ones stay until the trim of rule 1 removes them.
  4. Each class emits at most one set per incoming message.

The approximate search thus considers only messages that have already arrived, and pairs the newest message with them at once. It never waits for a later message that would match better. The number of combinations is the product of the candidate counts of the other inputs, each at most queue_size, so it grows exponentially with the number of inputs; in our reading, this keeps the Python class to a few inputs. The Python class can also accept headerless messages with allow_headerless=True, stamping them with the current ROS time; its docstring advises against this, “since the delays are unpredictable”.

The C++ ApproximateTime policy (sync_policies/approximate_time.h, same commit) is a different algorithm, and nothing in the example below tests it. It keeps a deque per channel and maintains a candidate set and a pivot channel. It compares a new candidate interval with the current one, weighting how much later its end is with age_penalty_, which defaults to 0.1. Intervals longer than max_interval_duration_ are rejected; that bound defaults to the largest int32_t number of seconds plus 999,999,999 ns and is changed with setMaxIntervalDuration. Optional inter-message lower bounds per channel let it prove that a candidate is optimal and publish it without waiting for more messages. Its constructor takes a queue size but no slop. It asserts a queue size above zero, with the comment that “at least 2 is recommended”. When a channel’s queue overflows, it drops the oldest message of that channel and marks the channel as having dropped messages.

The documentation of message_filters warns that each subscriber must use the same QoS profile as the corresponding publisher, otherwise the callback is never called and no warning is given. The QoS page’s rule is compatibility rather than equality; in our reading the advice is a safe way to meet it.

TUM RGB-D associates colour and depth stamps offline with its associate.py script: it forms every pair within a maximum difference and accepts them closest first. The ROS synchronizers instead decide online, in arrival order, from what is queued when each message arrives. Real-time vision pipelines covers the levels of synchronization and the clock domains these stamps come from.

Bags

ros2 bag record creates an ordinary subscriber to the requested topics and writes what it receives, without disturbing the other subscribers. rosbag2 stores bags through storage plugins. The repository provides mcap and sqlite3, and the README names mcap as the default. Recording can split files by size or duration and compress with zstd, by file or by message; the README notes that MCAP’s own compression keeps files indexable, whereas rosbag2’s compression does not. A snapshot mode keeps messages in an in-memory circular buffer and writes it to disk only when asked to, through a service on the recorder. ros2 bag play detects the storage format, and ros2 bag convert rewrites bags. A bag’s recording times are not the sensors’ stamps: EuRoC MAV’s tools re-stamp each message of its bags with its header timestamp, because the two differ.

Minimal Example

ROS 2 cannot be installed on the machine used here. The example therefore ports the two Python synchronizers from message_filters jazzy at commit 05a6b5f (src/message_filters/__init__.py) and runs them on synthetic streams whose capture instants are known. rclpy’s Time and Duration become integer nanoseconds, the lock and logging are removed, and a counter is added for messages deleted by the queue_size trim; the code carries the file’s BSD licence text. A second part packs Image-style buffers. Every outcome below tests our port, not ROS. The C++ policy is not tested.

Set-up. The streams last 60 s. Frame kk of a 30 Hz stream is captured at k⋅109/30k \cdot 10^9/30 ns, rounded. The values below are our choices, not values from the sources: the drop rate, offsets, jitter, latencies, the correctness tolerance and the seed.

  • A, a hardware-triggered pair: two 30 Hz cameras whose stamps equal their capture instants, identical in both streams. Each message is dropped independently with probability 0.05.
  • A0, the base check: A without drops.
  • B, a free-running pair: camera 1 at 30 Hz from time 0, camera 2 at 29.97 Hz from 7 ms. Each stamp is the capture instant plus independent integer jitter, uniform in ±1 ms.
  • C, a camera with depth: a 30 Hz camera and a 15 Hz depth stream captured at the instants of the even camera frames, both with ±1 ms stamp jitter.

Stream 1 arrives 5 ms after capture. Stream 2 arrives either 5 ms after capture (no lag) or 105 ms after (lag), and messages are fed in order of arrival, stream 1 first on ties. Each policy is run with queue_size 2 and 10: ApproximateTime with slop 5, 17 and 40 ms, given directly in nanoseconds because rclpy’s conversion from seconds was not read, and ExactTime. A set is correct if its two capture instants differ by less than half the 30 Hz period, 16,666,667 ns. Each message carries its capture instant, so the script also counts the possible correct sets in each case. The data seed is 0.

Packing. A 37 × 53 colour image (R = (5c + 40) mod 256, G = (7r + 90) mod 256, B = (3r + 11c + 200) mod 256) and a 16-bit image (D = (4660 + 1777r + 613c) mod 65536) are packed as rgb8, bgr8 and 16UC1 in both byte orders. step is rounded up to a multiple of 64 bytes, and the padding is filled with 0xAB. Our decoder reads rows of step bytes and applies the byte order and channel order. P1 checks the round trips, P2 the data length and P3 the bytes of pixel (0, 0) against values written independently of the packer. P4 tries three wrong decodes: the bgr8 buffer read as rgb8, the big-endian buffer read as little-endian, and the rgb8 buffer read as if rows had no padding.

Hypotheses, written before any script ran, with the derivations that motivated them:

  • C0: in A0, every policy at both queue sizes emits all 1,800 possible sets, all correct.
  • H1a: ExactTime emits nothing in B and C. Jitter is drawn per message, so stamps never coincide.
  • H1b: in A, ExactTime emits exactly the possible sets, all correct, at queue_size 10 with and without lag, and at 2 without lag. Its stamps are equal across streams, and partners arrive at most one frame apart.
  • H2a: every ApproximateTime set has a stamp spread below the slop. This follows from the code’s own test, so it checks the port only.
  • H2b: in B without lag, slop 40 ms gives wrong sets at both queue sizes. With equal latencies, a new message can pair only with already-arrived messages of the other stream. The 30/29.97 Hz drift sweeps the phase through more than one period, so the already-arrived neighbour is sometimes more than half a period away yet within 40 ms, and it is taken before the right partner arrives.
  • H2c: in B, slop 5 ms gives no wrong set under any condition. A stamp spread below 5 ms plus at most 1 ms of jitter per stamp bounds the capture spread below 7 ms.
  • H2d: in A without lag, slop 40 ms gives wrong sets. When camera 1’s frame kk is dropped, camera 2’s frame kk waits in its queue, and camera 1’s frame k+1k+1 finds it 33.3 ms away, within 40 ms.
  • H2e: in A without lag, slop 5 and 17 ms give no wrong set, since stamps are equal or at least 33.3 ms apart.
  • H3a: with the lag, queue_size 2 emits fewer sets than 10, for every case and slop and for ExactTime in A. A 100 ms lag is three frame periods, and a queue of two holds only the two newest stamps of the other stream when the late message arrives.
  • H3b: with the lag, the trim discards more messages at queue_size 2 than at 10, in every case and policy.
  • H4: every round trip is exact and every wrong decode differs from the original.

Planning and runs. Before the pilot, a planning script computed the ground truth of the same streams without running any synchronizer: the message counts, the possible sets (1,800, 1,611, 1,798 and 900) and the packing reference bytes. The expected counts in C0 and H1b were therefore written as numbers after that script printed them; the hypotheses themselves were written before. The batch plan’s outline of this example was extended before the pilot, and the editor approved these departures as piloted:

  • the arrival-lag condition;
  • the parameters listed above as our choices;
  • ExactTime run without a slop;
  • the printed counts of correct, wrong and possible sets and the data checks;
  • the split hypotheses and those on case A;
  • the byte-layout and wrong-decode checks;
  • the slop given in nanoseconds.

The pilot and the run shown are the same script; the pilot and two runs printed identical output. It runs in under a second with NumPy 2.3.5.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                       # one BLAS thread, set before NumPy is imported
import itertools
import numpy as np

# TimeSync and ApproxSync below are ported from message_filters (ros2/message_filters, branch jazzy,
# commit 05a6b5f), src/message_filters/__init__.py: TimeSynchronizer.add and
# ApproximateTimeSynchronizer.add. Modified: rclpy Time and Duration are replaced by integer
# nanoseconds, the lock and logging are removed, messages are tuples, and a counter of messages
# deleted by the queue_size trim is added. The file carries this licence:
#
# Copyright (c) 2009, Willow Garage, Inc.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
#     * Redistributions of source code must retain the above copyright
#       notice, this list of conditions and the following disclaimer.
#     * Redistributions in binary form must reproduce the above copyright
#       notice, this list of conditions and the following disclaimer in the
#       documentation and/or other materials provided with the distribution.
#     * Neither the name of the Willow Garage, Inc. nor the names of its
#       contributors may be used to endorse or promote products derived from
#       this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

class TimeSync:                                   # port of TimeSynchronizer (exact stamps)
    def __init__(self, n, queue_size):
        self.queues = [{} for _ in range(n)]      # one {stamp_ns: msg} per input
        self.queue_size = queue_size
        self.out, self.discarded = [], 0

    def trim(self, my_queue):
        while len(my_queue) > self.queue_size:
            del my_queue[min(my_queue)]           # the smallest stamp goes first
            self.discarded += 1

    def add(self, msg, i):
        my_queue = self.queues[i]
        my_queue[msg[2]] = msg
        self.trim(my_queue)
        common = set.intersection(*[set(q) for q in self.queues])
        signaled = None
        for t in sorted(common):
            self.out.append(tuple(q[t] for q in self.queues))
            for q in self.queues:
                del q[t]
            signaled = t
            break
        if signaled is not None:                  # drop everything older than the emitted stamp
            for q in self.queues:
                for s in list(q):
                    if s < signaled:
                        del q[s]

class ApproxSync(TimeSync):                       # port of ApproximateTimeSynchronizer
    def __init__(self, n, queue_size, slop_ns):
        TimeSync.__init__(self, n, queue_size)
        self.slop = slop_ns

    def add(self, msg, i):
        stamp = msg[2]
        my_queue = self.queues[i]
        my_queue[stamp] = msg
        self.trim(my_queue)
        search_queues = self.queues[:i] + self.queues[i + 1:]
        stamps = []
        for queue in search_queues:               # stamps within slop of the new one, nearest first
            topic_stamps = []
            for s in queue:
                delta = abs(s - stamp)
                if delta > self.slop:
                    continue
                topic_stamps.append((s, delta))
            if not topic_stamps:
                return
            topic_stamps = sorted(topic_stamps, key=lambda x: x[1])
            stamps.append(topic_stamps)
        for vv in itertools.product(*[list(zip(*s))[0] for s in stamps]):
            vv = list(vv)
            vv.insert(i, stamp)
            qt = list(zip(self.queues, vv))
            if (max(vv) - min(vv)) < self.slop and all(t in q for q, t in qt):
                self.out.append(tuple(q[t] for q, t in qt))
                for q, t in qt:
                    del q[t]
                break                             # one set per incoming message

# ----- synthetic streams with exact ground truth (ours) -----
T_NS = 60 * 10**9                                 # 60 s of data
TOL = 16_666_667                                  # correct: capture instants < half a 30 Hz period apart
CASES = {"A0": 0, "A": 1, "B": 2, "C": 3}

def frames(rate, start_ns, stop_ns):
    k, out = 0, []
    while True:
        t = int(round(start_ns + k * 1e9 / rate))
        if t >= stop_ns:
            return np.array(out, dtype=np.int64)
        out.append(t); k += 1

def make_case(name):
    """Returns two streams, each (capture_ns, stamp_ns)."""
    rng = np.random.default_rng([0, CASES[name]])
    if name in ("A0", "A"):                       # hardware trigger: equal stamps
        c1 = frames(30, 0, T_NS); c2 = c1.copy()
        s1, s2 = c1.copy(), c2.copy()
        if name == "A":                           # independent drops, p = 0.05 per stream
            k1 = rng.random(c1.size) >= 0.05; k2 = rng.random(c2.size) >= 0.05
            c1, s1, c2, s2 = c1[k1], s1[k1], c2[k2], s2[k2]
    elif name == "B":                             # free-running 30 Hz and 29.97 Hz, jittered stamps
        c1 = frames(30, 0, T_NS); c2 = frames(29.97, 7_000_000, T_NS)
        s1 = c1 + rng.integers(-1_000_000, 1_000_001, c1.size)
        s2 = c2 + rng.integers(-1_000_000, 1_000_001, c2.size)
    else:                                         # camera 30 Hz, depth 15 Hz on even camera frames
        c1 = frames(30, 0, T_NS); c2 = c1[::2].copy()
        s1 = c1 + rng.integers(-1_000_000, 1_000_001, c1.size)
        s2 = c2 + rng.integers(-1_000_000, 1_000_001, c2.size)
    return (c1, s1), (c2, s2)

LAT1, LAT2 = 5_000_000, {"no lag": 5_000_000, "lag": 105_000_000}

def feed(streams, lat2):
    """Messages (stream, capture, stamp, arrival) in arrival order; ties: stream 0 first."""
    msgs = []
    for i, ((c, s), lat) in enumerate(zip(streams, (LAT1, lat2))):
        msgs += [(i, int(ci), int(si), int(ci) + lat) for ci, si in zip(c, s)]
    return sorted(msgs, key=lambda m: (m[3], m[0]))

def run(sync, msgs):
    for m in msgs:
        sync.add(m, m[0])
    sets = len(sync.out)
    caps = [abs(a[1] - b[1]) for a, b in sync.out]
    spreads = [abs(a[2] - b[2]) for a, b in sync.out]
    correct = sum(d < TOL for d in caps)
    return sets, correct, sets - correct, (max(spreads) if spreads else -1), sync.discarded

POLICIES = [("approx", 5_000_000), ("approx", 17_000_000), ("approx", 40_000_000), ("exact", None)]

def make_sync(pol, slop, q):
    return ApproxSync(2, q, slop) if pol == "approx" else TimeSync(2, q)

# ----- data checks -----
print("NumPy", np.__version__)
data, possible = {}, {}
for name in CASES:
    (c1, s1), (c2, s2) = data[name] = make_case(name)
    d = np.abs(c1[:, None] - c2[None, :]); close = d < TOL
    possible[name] = int(close.sum())
    print("D1 %-2s messages %d + %d | max |stamp - capture| %d ns" % (
        name, c1.size, c2.size, max(int(np.abs(s1 - c1).max()), int(np.abs(s2 - c2).max()))), end="")
    if name == "A":
        print(" | drops %d, %d" % (1800 - c1.size, 1800 - c2.size), end="")
    if name == "B":
        per = 1e9 / 30
        print(" | phase drift %.3f periods" % ((c2[-1] - 7_000_000 - (c2.size - 1) * per) / per), end="")
    if name == "C":
        print(" | depth captures on camera captures %d" % int(np.isin(c2, c1).sum()), end="")
    print()
    print("D2 %-2s possible correct sets %d | max partners per message %d" % (
        name, possible[name], max(int(close.sum(1).max()), int(close.sum(0).max()))))
print("D3 arrival latency: stream 1 %d ms; stream 2 %d ms (no lag), %d ms (lag)" % (
    LAT1 // 10**6, LAT2["no lag"] // 10**6, LAT2["lag"] // 10**6))

# ----- synchronizer grid -----
hdr = "%-4s %-6s %-12s %3s %6s %7s %6s %7s %15s %9s" % (
    "case", "lag", "policy", "q", "sets", "correct", "wrong", "sets/s", "max spread ns", "discarded")
print(hdr)
rows = {}
for name in ["A0", "A", "B", "C"]:
    for lag in (["no lag"] if name == "A0" else ["no lag", "lag"]):
        msgs = feed(data[name], LAT2[lag])
        for pol, slop in POLICIES:
            for q in (2, 10):
                r = rows[(name, lag, pol, slop, q)] = run(make_sync(pol, slop, q), msgs)
                label = "exact" if pol == "exact" else "approx %2d ms" % (slop // 10**6)
                print("%-4s %-6s %-12s %3d %6d %7d %6d %7.2f %15d %9d" % (
                    name, lag, label, q, r[0], r[1], r[2], r[0] / 60, r[3], r[4]))

# ----- checks and hypotheses -----
A = [("approx", 5_000_000), ("approx", 17_000_000), ("approx", 40_000_000)]
c0 = all(rows[("A0", "no lag", p, s, q)][:3] == (possible["A0"], possible["A0"], 0)
         for p, s in POLICIES for q in (2, 10))
print("C0 A0 every policy and q: sets == correct == %d, wrong 0:" % possible["A0"], c0)
h1a = [rows[(n, l, "exact", None, q)][0] for n in "BC" for l in ("no lag", "lag") for q in (2, 10)]
print("H1a exact sets in B, C (8 conditions):", h1a, "->", all(x == 0 for x in h1a))
h1b = [(l, q, rows[("A", l, "exact", None, q)][:3]) for l, q in (("no lag", 10), ("lag", 10), ("no lag", 2))]
print("H1b exact in A (sets, correct, wrong) vs possible %d:" % possible["A"], h1b, "->",
      all(r == (possible["A"], possible["A"], 0) for _, _, r in h1b))
h2a = [(k, v[3]) for k, v in rows.items() if k[2] == "approx" and v[0] > 0 and not v[3] < k[3]]
print("H2a approx rows with spread >= slop:", len(h2a), "->", len(h2a) == 0)
for tag, case, slops, want in (("H2b", "B", [40_000_000], "wrong>0"), ("H2d", "A", [40_000_000], "wrong>0"),
                               ("H2e", "A", [5_000_000, 17_000_000], "wrong=0")):
    w = [rows[(case, "no lag", "approx", s, q)][2] for s in slops for q in (2, 10)]
    ok = all(x > 0 for x in w) if want == "wrong>0" else all(x == 0 for x in w)
    print("%s %s no lag, slop %s ms, q 2/10: wrong %s -> %s" % (
        tag, case, "/".join(str(s // 10**6) for s in slops), w, ok))
w = [rows[("B", l, "approx", 5_000_000, q)][2] for l in ("no lag", "lag") for q in (2, 10)]
print("H2c B slop 5 ms, wrong over (no lag, lag) x (q 2, 10):", w, "->", all(x == 0 for x in w))
h3a, h3b = [], []
for n in "ABC":
    for p, s in (A + ([("exact", None)] if n == "A" else [])):
        a2, a10 = rows[(n, "lag", p, s, 2)], rows[(n, "lag", p, s, 10)]
        h3a.append((n, p, s, a2[0], a10[0], a2[0] < a10[0]))
    for p, s in POLICIES:
        a2, a10 = rows[(n, "lag", p, s, 2)], rows[(n, "lag", p, s, 10)]
        h3b.append((n, p, s, a2[4], a10[4], a2[4] > a10[4]))
print("H3a lag, sets q=2 < q=10:", [(n, s and s // 10**6, a, b) for n, p, s, a, b, _ in h3a], "->",
      all(x[-1] for x in h3a))
print("H3b lag, discarded q=2 > q=10:", [(n, s and s // 10**6, a, b) for n, p, s, a, b, _ in h3b], "->",
      all(x[-1] for x in h3b))

# ----- packing check: sensor_msgs/Image-style buffers (ours) -----
r, c = np.mgrid[0:37, 0:53]
rgb = np.stack([(5 * c + 40) % 256, (7 * r + 90) % 256, (3 * r + 11 * c + 200) % 256], -1).astype(np.uint8)
depth = ((4660 + 1777 * r + 613 * c) % 65536).astype(np.uint16)
BPP = {"rgb8": 3, "bgr8": 3, "16UC1": 2}

def pack(img, encoding, is_bigendian=0, align=64):
    h, w = img.shape[:2]
    row = w * BPP[encoding]
    step = -(-row // align) * align
    if encoding == "bgr8":
        img = img[..., ::-1]
    if encoding == "16UC1":
        img = img.astype(">u2" if is_bigendian else "<u2")
    buf = np.full((h, step), 0xAB, np.uint8)
    buf[:, :row] = np.frombuffer(np.ascontiguousarray(img).tobytes(), np.uint8).reshape(h, row)
    return dict(height=h, width=w, encoding=encoding, is_bigendian=is_bigendian, step=step, data=buf.tobytes())

def unpack(msg, ignore_step=False):
    h, w, enc = msg["height"], msg["width"], msg["encoding"]
    row = w * BPP[enc]
    a = np.frombuffer(msg["data"], np.uint8)
    a = a[:h * row].reshape(h, row) if ignore_step else a.reshape(h, msg["step"])[:, :row]
    if enc == "16UC1":
        return a.copy().view(">u2" if msg["is_bigendian"] else "<u2").reshape(h, w).astype(np.uint16)
    a = a.reshape(h, w, 3)
    return a[..., ::-1] if enc == "bgr8" else a

def maxdiff(x, y):
    return int(np.abs(x.astype(np.int64) - y.astype(np.int64)).max())

msgs = {"rgb8": pack(rgb, "rgb8"), "bgr8": pack(rgb, "bgr8"),
        "16UC1 BE": pack(depth, "16UC1", 1), "16UC1 LE": pack(depth, "16UC1", 0)}
want = {"rgb8": (40, 90, 200), "bgr8": (200, 90, 40), "16UC1 BE": (0x12, 0x34), "16UC1 LE": (0x34, 0x12)}
p1 = []
for k, m in msgs.items():
    ref = depth if k.startswith("16") else rgb
    p1.append(maxdiff(unpack(m), ref))
    first = tuple(m["data"][:len(want[k])])
    print("P %-8s step %3d | len(data) %4d = step x height %4d: %s | pixel (0,0) bytes %s, want %s: %s"
          " | round trip max diff %d" % (k, m["step"], len(m["data"]), m["step"] * m["height"],
          len(m["data"]) == m["step"] * m["height"], first, want[k], first == want[k], p1[-1]))
wrong = {"bgr8 read as rgb8": maxdiff(unpack(dict(msgs["bgr8"], encoding="rgb8")), rgb),
         "BE read as LE": maxdiff(unpack(dict(msgs["16UC1 BE"], is_bigendian=0)), depth),
         "rgb8 ignoring step": maxdiff(unpack(msgs["rgb8"], ignore_step=True), rgb)}
print("P4 wrong decodes, max diff:", wrong)
print("H4 round trips exact:", all(x == 0 for x in p1), "| every wrong decode differs:",
      all(v > 0 for v in wrong.values()))
NumPy 2.3.5
D1 A0 messages 1800 + 1800 | max |stamp - capture| 0 ns
D2 A0 possible correct sets 1800 | max partners per message 1
D1 A  messages 1695 + 1711 | max |stamp - capture| 0 ns | drops 105, 89
D2 A  possible correct sets 1611 | max partners per message 1
D1 B  messages 1800 + 1798 | max |stamp - capture| 999873 ns | phase drift 1.799 periods
D2 B  possible correct sets 1798 | max partners per message 1
D1 C  messages 1800 + 900 | max |stamp - capture| 999676 ns | depth captures on camera captures 900
D2 C  possible correct sets 900 | max partners per message 1
D3 arrival latency: stream 1 5 ms; stream 2 5 ms (no lag), 105 ms (lag)
case lag    policy         q   sets correct  wrong  sets/s   max spread ns discarded
A0   no lag approx  5 ms   2   1800    1800      0   30.00               0         0
A0   no lag approx  5 ms  10   1800    1800      0   30.00               0         0
A0   no lag approx 17 ms   2   1800    1800      0   30.00               0         0
A0   no lag approx 17 ms  10   1800    1800      0   30.00               0         0
A0   no lag approx 40 ms   2   1800    1800      0   30.00               0         0
A0   no lag approx 40 ms  10   1800    1800      0   30.00               0         0
A0   no lag exact          2   1800    1800      0   30.00               0         0
A0   no lag exact         10   1800    1800      0   30.00               0         0
A    no lag approx  5 ms   2   1611    1611      0   26.85               0       182
A    no lag approx  5 ms  10   1611    1611      0   26.85               0       166
A    no lag approx 17 ms   2   1611    1611      0   26.85               0       182
A    no lag approx 17 ms  10   1611    1611      0   26.85               0       166
A    no lag approx 40 ms   2   1652     812    840   27.53        33333334        99
A    no lag approx 40 ms  10   1652     812    840   27.53        33333334        83
A    no lag exact          2   1611    1611      0   26.85               0        10
A    no lag exact         10   1611    1611      0   26.85               0         0
A    lag    approx  5 ms   2     22      22      0    0.37               0      3361
A    lag    approx  5 ms  10   1611    1611      0   26.85               0       169
A    lag    approx 17 ms   2     22      22      0    0.37               0      3361
A    lag    approx 17 ms  10   1611    1611      0   26.85               0       169
A    lag    approx 40 ms   2    175       3    172    2.92        33333334      3054
A    lag    approx 40 ms  10   1652     812    840   27.53        33333334        86
A    lag    exact          2     22      22      0    0.37               0      3345
A    lag    exact         10   1611    1611      0   26.85               0         0
B    no lag approx  5 ms   2    461     461      0    7.68         4978765      2674
B    no lag approx  5 ms  10    461     461      0    7.68         4978765      2658
B    no lag approx 17 ms   2   1779    1754     25   29.65        16998142        38
B    no lag approx 17 ms  10   1779    1754     25   29.65        16998142        22
B    no lag approx 40 ms   2   1798     798   1000   29.97        34634043         1
B    no lag approx 40 ms  10   1798     798   1000   29.97        34634043         0
B    no lag exact          2      0       0      0    0.00              -1      3594
B    no lag exact         10      0       0      0    0.00              -1      3578
B    lag    approx  5 ms   2      2       2      0    0.03         1461114      3593
B    lag    approx  5 ms  10    461     461      0    7.68         4978765      2661
B    lag    approx 17 ms   2      2       2      0    0.03         1461114      3593
B    lag    approx 17 ms  10   1776    1749     27   29.60        16998142        31
B    lag    approx 40 ms   2    406       0    406    6.77        39960556      2784
B    lag    approx 40 ms  10   1798    1745     53   29.97        18345132         0
B    lag    exact          2      0       0      0    0.00              -1      3594
B    lag    exact         10      0       0      0    0.00              -1      3578
C    no lag approx  5 ms   2    900     900      0   15.00         1978528       898
C    no lag approx  5 ms  10    900     900      0   15.00         1978528       890
C    no lag approx 17 ms   2    900     900      0   15.00         1978528       898
C    no lag approx 17 ms  10    900     900      0   15.00         1978528       890
C    no lag approx 40 ms   2    900     900      0   15.00         1978528       898
C    no lag approx 40 ms  10    900     900      0   15.00         1978528       890
C    no lag exact          2      0       0      0    0.00              -1      2696
C    no lag exact         10      0       0      0    0.00              -1      2680
C    lag    approx  5 ms   2      1       1      0    0.02         1271782      2696
C    lag    approx  5 ms  10    900     900      0   15.00         1978528       892
C    lag    approx 17 ms   2      1       1      0    0.02         1271782      2696
C    lag    approx 17 ms  10    900     900      0   15.00         1978528       892
C    lag    approx 40 ms   2      1       1      0    0.02         1271782      2696
C    lag    approx 40 ms  10    900     900      0   15.00         1978528       892
C    lag    exact          2      0       0      0    0.00              -1      2696
C    lag    exact         10      0       0      0    0.00              -1      2680
C0 A0 every policy and q: sets == correct == 1800, wrong 0: True
H1a exact sets in B, C (8 conditions): [0, 0, 0, 0, 0, 0, 0, 0] -> True
H1b exact in A (sets, correct, wrong) vs possible 1611: [('no lag', 10, (1611, 1611, 0)), ('lag', 10, (1611, 1611, 0)), ('no lag', 2, (1611, 1611, 0))] -> True
H2a approx rows with spread >= slop: 0 -> True
H2b B no lag, slop 40 ms, q 2/10: wrong [1000, 1000] -> True
H2d A no lag, slop 40 ms, q 2/10: wrong [840, 840] -> True
H2e A no lag, slop 5/17 ms, q 2/10: wrong [0, 0, 0, 0] -> True
H2c B slop 5 ms, wrong over (no lag, lag) x (q 2, 10): [0, 0, 0, 0] -> True
H3a lag, sets q=2 < q=10: [('A', 5, 22, 1611), ('A', 17, 22, 1611), ('A', 40, 175, 1652), ('A', None, 22, 1611), ('B', 5, 2, 461), ('B', 17, 2, 1776), ('B', 40, 406, 1798), ('C', 5, 1, 900), ('C', 17, 1, 900), ('C', 40, 1, 900)] -> True
H3b lag, discarded q=2 > q=10: [('A', 5, 3361, 169), ('A', 17, 3361, 169), ('A', 40, 3054, 86), ('A', None, 3345, 0), ('B', 5, 3593, 2661), ('B', 17, 3593, 31), ('B', 40, 2784, 0), ('B', None, 3594, 3578), ('C', 5, 2696, 892), ('C', 17, 2696, 892), ('C', 40, 2696, 892), ('C', None, 2696, 2680)] -> True
P rgb8     step 192 | len(data) 7104 = step x height 7104: True | pixel (0,0) bytes (40, 90, 200), want (40, 90, 200): True | round trip max diff 0
P bgr8     step 192 | len(data) 7104 = step x height 7104: True | pixel (0,0) bytes (200, 90, 40), want (200, 90, 40): True | round trip max diff 0
P 16UC1 BE step 128 | len(data) 4736 = step x height 4736: True | pixel (0,0) bytes (18, 52), want (18, 52): True | round trip max diff 0
P 16UC1 LE step 128 | len(data) 4736 = step x height 4736: True | pixel (0,0) bytes (52, 18), want (52, 18): True | round trip max diff 0
P4 wrong decodes, max diff: {'bgr8 read as rgb8': 254, 'BE read as LE': 64260, 'rgb8 ignoring step': 252}
H4 round trips exact: True | every wrong decode differs: True

Data checks. D1–D3 held. In A, the drops removed 105 and 89 messages. In B, the stamps stay within 999,873 ns of their capture instants, and the phase drifts by 1.799 periods, against 60×0.03=1.860 \times 0.03 = 1.8 by our arithmetic. In C, all 900 depth captures fall on camera captures. No message has more than one possible partner.

C0 held. In A0, every policy at both queue sizes emits 1,800 sets, all correct. Without drops or jitter, the ported classes recover the ground truth.

H1 held. H1a held: ExactTime emits no set in any of the eight B and C conditions, as expected from free-running stamps that never coincide. H1b held: in A, ExactTime emits 1,611 sets, all correct, at queue_size 10 with and without lag and at 2 without lag, which equals the 1,611 possible sets. In our port, exact matching suits hardware-triggered cameras that share a stamp and pairs nothing when stamps carry independent jitter.

H2a held, as a check of the port. No ApproximateTime row has a stamp spread at or above its slop.

H2b held. In B without lag at slop 40 ms, 1,000 of the 1,798 sets are wrong at both queue sizes. H2c held: at slop 5 ms in B, no set is wrong in any of the four conditions. Without lag, the price is the rate: 461 sets, 7.68 per second, against 1,798 possible.

H2d held. In A without lag at slop 40 ms, 840 of 1,652 sets are wrong at both queue sizes. The run emits more sets than there are possible correct ones (1,611). H2e held: at slop 5 and 17 ms the same streams give 1,611 sets with none wrong.

H3 held. H3a held in all ten comparisons. With the lag, queue_size 2 emits 22 sets in A at slop 5 and 17 ms and with ExactTime, against 1,611 at 10. At slop 40 ms the counts are 175 against 1,652 in A and 406 against 1,798 in B. In B at slop 5 and 17 ms they are 2 against 461 and 2 against 1,776. In C, queue_size 2 emits 1 set and 10 emits 900, at every slop. H3b held in all twelve comparisons: each count of messages discarded at queue_size 2 exceeds its count at 10, for example 3,361 against 169 in A at slop 5 ms, 2,784 against 0 in B at slop 40 ms, and 2,696 against 2,680 for ExactTime in C.

Rows without a prediction, reported as printed:

  • B at slop 17 ms gives 25 wrong sets of 1,779 without lag (both queue sizes), and 27 of 1,776 with lag at queue_size 10.
  • B with lag at queue_size 10 and slop 40 ms gives 53 wrong sets of 1,798.
  • A at slop 40 ms gives 840 wrong sets of 1,652 with lag at queue_size 10, the same count as without lag.
  • A with lag at queue_size 2 and slop 40 ms gives 172 wrong sets of 175.
  • B with lag at queue_size 2 and slop 40 ms gives 406 sets, all wrong.
  • C without lag gives 900 sets, all correct, at every slop and both queue sizes.

H4 held, and the packing checks P1–P4 held. All four buffers round-trip with a maximum difference of 0; each holds step × height bytes (7,104 for the colour encodings, 4,736 for 16UC1); and the first pixel’s bytes are (40, 90, 200) for rgb8, (200, 90, 40) for bgr8, (0x12, 0x34) big-endian and (0x34, 0x12) little-endian. Each wrong decode differs from the original, with maximum differences of 254, 64,260 and 252. Reading bgr8 as rgb8 swaps red and blue. Reading big-endian data as little-endian swaps the bytes of every 16-bit value. Ignoring step shifts every row after the first by the accumulated padding. The 16-bit big-endian layout is the one PNG uses, as image file formats describes.

In our reading, the example shows that the approximate synchronizer’s slop is an acceptance window, not a guarantee of correct pairing. A window wider than half the frame period admits neighbouring frames whenever the true partner is missing or has not yet arrived. A queue shorter than the lag between streams removes true partners before they can be matched.

Ecosystem and Integrations

  • Vision packages. The CameraInfo message names image_pipeline, whose image_proc and stereo_image_proc packages produce the processed and rectified image topics from image_raw and camera_info, and image_geometry, which it recommends for projecting points with the calibration. CompressedImage names the compressed_image_transport and compressed_depth_image_transport plugins of image_transport.
  • OpenCV and other libraries. The encodings in image_encodings.hpp include OpenCV’s type names (8UC3, 16UC1, 32FC1 and others). REP 2000 lists each distribution’s dependency versions per platform at its first release, as a low watermark, among them OpenCV, PCL, Qt, Python and NumPy; for Jazzy on Ubuntu Noble these are OpenCV 4.6.0, PCL 1.14.0, NumPy 1.26.4 and Python 3.12.3.
  • Simulation. REP 2000 lists Gazebo Harmonic among Jazzy’s dependencies on Ubuntu, Debian and macOS. CARLA offers a native ROS 2 interface and a separate ROS bridge.
  • Datasets. Several published datasets were recorded through ROS or ship ROS bags. TUM RGB-D was recorded with ROS Diamondback and its openni_camera package, as described in the TUM RGB-D paper, and EuRoC MAV offered its sequences as bags. In drivable-area segmentation, the HybridNets repository includes a ROS package for path planning.
  • Tools. The Jazzy Concepts pages also include pages on the command-line tools and on RQt, which were not read for this article.

Alternatives

The sources do not rank middleware, and neither does this article.

  • ROS 1. The ROS 2 pages contrast it in three places. ROS 1 connected every publisher and subscriber of the same type on the same topic and primarily used TCP. It offered queue sizes and latching, where ROS 2 has QoS policies. It had separate node and nodelet APIs, where ROS 2 has one component API. Nothing about ROS 1’s current support is claimed here.
  • DDS directly. In our reading, since ROS 2 runs on DDS, a system that needs DDS features without the rest of the ROS ecosystem can use a DDS implementation’s own API, giving up ROS’s message packages, tools and conventions.
  • Media pipelines. For pure video work, a media framework such as GStreamer moves frames between elements in one process. Real-time vision pipelines describes GStreamer’s queues and latency model.
  • In-process code. A single program with threads and queues avoids serialization and discovery entirely. Real-time vision pipelines discusses its queue policies.

Versions and Compatibility

Distributions. As of November 2026, REP 2000 at commit 11ca24a (23 October 2025, the newest commit of the REP repository) gives this schedule:

  • a new ROS 2 release every 12 months;
  • in even years, an LTS release one month after the Ubuntu LTS, usually in May, supported for 4 years and 11 months;
  • in odd years, a non-LTS release on the previous LTS’s Ubuntu, supported for 1.5 years.

The entries current at that commit are:

  • Humble Hawksbill, May 2022 to May 2027;
  • Jazzy Jalisco, May 2024 to May 2029;
  • Kilted Kaiju, May 2025 to November 2026;
  • Rolling Ridley, from June 2020, ongoing. It is a rolling development distribution which, as of May 2025, targets the same platforms as Kilted.

The copy lists no distribution released after Kilted, and none is described here. The facts in this article come from Jazzy branches.

Jazzy’s platforms (REP 2000):

  • Tier 1: Ubuntu 24.04 (Noble) on amd64 and arm64, and Windows 10 (VS2019) on amd64.
  • Tier 2: RHEL 9 on amd64.
  • Tier 3, community-supported: Ubuntu 22.04, macOS, Debian Bookworm and OpenEmbedded/Yocto.

Tier 1 platforms get continuous integration, nightly and packaging jobs, and their errors are prioritized; Tier 3 platforms are those that community reports say work. Jazzy’s minimum language requirements are C++17 and Python 3.8. Its middleware support is:

  • Tier 1: Fast DDS (the default), Cyclone DDS, and Connext on all architectures except arm64;
  • Tier 2: rmw_fastrtps_dynamic_cpp;
  • Tier 3: GurumDDS.

A middleware gets no better support than the tier of the platform it runs on.

Compatibility. Mixing RMW implementations, or ROS versions, across one system is not guaranteed to work (middleware page). In our reading, a bag recorded under one distribution is safest to replay under the same one.

Licences. ROS 2 has no single licence; each package carries its own. Among those read here, message_filters is BSD-3-Clause, sensor_msgs and rosbag2 are Apache-2.0, the documentation is CC BY 4.0, and the REPs are in the public domain. The middleware licences in the table above range from Apache 2.0 to commercial.

Versions used here. The example ran with NumPy 2.3.5. It ports message_filters jazzy at 05a6b5f; no ROS distribution was installed.

Limitations

  • Pairing is only as good as the stamps. The synchronizers match header stamps, which the Image message requires to be acquisition times. A driver that stamps on arrival, or clocks that are not mapped into one domain, defeat every policy.
  • Approximate synchronization can pair the wrong frames, as the example shows for our port of the Python class; the C++ policy was not tested.
  • QoS mismatches fail silently. An incompatible pair does not connect and a filter chain never calls back; only monitored incompatibility events reveal it.
  • Large images cost bandwidth and copies. Image carries raw pixels; compression needs an image_transport plugin, and composition offers an intra-process path. In our reading, how the middleware handles large messages should be measured on the target system.
  • Frame conventions must be followed. Optical frames, REP 105’s frame tree and the direction of tf2 transforms are conventions, not checks: a camera published with body-frame axes, or a transform inverted, produces data that look valid and are wrong. Rotation representations and camera pose cover common mistakes.
  • A distribution is a moving target. Non-LTS distributions are supported for 1.5 years and LTS ones for about five, and binary support depends on the platform tier, so a deployed robot is tied to an Ubuntu release and a distribution’s end of life.

Related

  • Real-Time Vision Pipelines

    How a vision system turns a stream of camera frames into timely decisions, covering frame budgets, latency against throughput, pipelining and threads, queues and dropped frames, timestamps and clock synchronization, and how end-to-end latency is measured.

  • Robot Perception

    How robots use cameras, depth sensors, LiDAR, inertial and joint sensors and touch to estimate their own state and the world around them fast enough to act on it, covering the perception–action loop, state estimation, latency and robustness, and the main robot settings from mobile robots and arms to drones, legged robots and cars.

  • TUM RGB-D

    The TUM RGB-D SLAM dataset and benchmark, Kinect and Asus Xtion sequences at 30 Hz with motion-capture ground truth at 100 Hz; its sequence categories, file formats and calibration, timestamp association, and the absolute trajectory error and relative pose error as the official scripts compute them.

  • Camera Intrinsic and Extrinsic Parameters

    The parameters of the pinhole camera matrix P = K[R | t], split into intrinsics K, which map camera coordinates to pixels, and extrinsics R and t, which place the camera in the world.

  • Image File Formats

    What PNG, TIFF, JPEG files, OpenEXR and RAW containers store, including chunks, tags, sample types, bit depths and metadata; how far each one's dynamic range reaches; and what Pillow, imageio and tifffile return when they read them.

  • EuRoC MAV

    Eleven visual-inertial sequences flown by a hexacopter in an ETH Zurich machine hall and a motion-capture room, with stereo images, IMU data and laser-tracker or Vicon ground truth; the standard test set for visual-inertial odometry and SLAM, and how trajectory alignment changes the errors reported on it.

References

  1. ROS 2 documentation (Jazzy), Concepts: Nodes, Topics, Services, Actions, Interfaces, Discovery and Launch (source/Concepts/Basic), ros2/ros2_documentation at commit 929e37c (CC BY 4.0; read November 2026).
  2. ROS 2 documentation (Jazzy), Concepts: Quality of Service settings, Different ROS 2 middleware vendors, Composition and Tf2 (source/Concepts/Intermediate), ros2/ros2_documentation at commit 929e37c (CC BY 4.0; read November 2026).
  3. message_filters (jazzy): src/message_filters/__init__.py, include/message_filters/sync_policies/approximate_time.h, doc/index.rst and the Approximate Time Synchronizer (Python) tutorial, ros2/message_filters at commit 05a6b5f (BSD-3-Clause; read November 2026).
  4. sensor_msgs (jazzy): msg/Image.msg, msg/CameraInfo.msg, msg/CompressedImage.msg and include/sensor_msgs/image_encodings.hpp, ros2/common_interfaces at commit a941f14 (Apache-2.0 package licence; read November 2026).
  5. Foote, T. & Purvis, M. (2010). REP 103: Standard Units of Measure and Coordinate Conventions. ROS Enhancement Proposals, ros-infrastructure/rep at commit 11ca24a (read November 2026).
  6. Mihelich, P. (2010). REP 104: CameraInfo updates for Diamondback. ROS Enhancement Proposals, ros-infrastructure/rep at commit 11ca24a (read November 2026).
  7. Meeussen, W. (2010). REP 105: Coordinate Frames for Mobile Platforms. ROS Enhancement Proposals, ros-infrastructure/rep at commit 11ca24a (read November 2026).
  8. Mihelich, P. (2011). REP 118: Depth Images. ROS Enhancement Proposals, ros-infrastructure/rep at commit 11ca24a (read November 2026).
  9. Arguedas, M., Ragnarok, S., Thomas, D., Nash, A., Biggs, G. & Gutierrez, M. A. REP 2000: ROS 2 Releases and Target Platforms. ROS Enhancement Proposals, ros-infrastructure/rep at commit 11ca24a, 23 October 2025 (read November 2026).
  10. rosbag2 (jazzy): README.md, ros2/rosbag2 at commit 12b5e8b (Apache-2.0; read November 2026).
  11. tf: The transform library. TePRA 2013, as cited by the ROS 2 documentation's Tf2 page (not read).