Vision Systems & Engineering

CARLA

An open-source driving simulator built on Unreal Engine, with a server–client architecture, synchronous stepping, camera, depth, segmentation and LiDAR sensors, a Traffic Manager and a recorder, described from its 0.9.16 documentation and LibCarla source, with an example that reproduces its depth encoding and camera coordinates.

intermediate

CARLA (Car Learning to Act) is an open-source simulator for autonomous-driving research. Dosovitskiy et al. (2017) introduced it as an open layer over Unreal Engine 4. It comes with its own free urban assets: road layouts, buildings, vehicles and pedestrians. The code is developed in the carla-simulator/carla repository, whose MIT licence names the Computer Vision Center (CVC) at the Universitat Autonoma de Barcelona as copyright holder. A vision engineer uses CARLA as a source of rendered driving scenes with exact ground truth: depth, semantic and instance segmentation, optical flow and LiDAR points. It also serves as a closed-loop test bed where an agent’s actions change what its cameras see next.

Synthetic data covers what rendered data is good for and where it falls short. End-to-end autonomous driving covers the 2017 paper’s driving benchmark, CARLA100, NoCrash, the CARLA Leaderboard and its driving score. This article covers the software itself: how the server and its clients divide the work, how simulated time advances, what each sensor returns and how it is encoded, which coordinate conventions apply, and what the Traffic Manager and recorder do. It also covers the release lines and licences. Facts come from the 0.9.16 documentation and LibCarla source at commit 294096e, unless another version is named. Nothing in the example below runs CARLA, which is not installed on the machine used here.

Purpose

The 2017 paper argues that testing urban driving in the physical world costs too much for most research groups. Some scenarios are also too dangerous to stage, such as a child running onto the road. In the authors’ view, existing platforms fell short as well. Racing simulators such as TORCS have no pedestrians, intersections, cross traffic or traffic rules. Commercial games such as Grand Theft Auto V offer little control over the environment, scripting or sensors (Dosovitskiy et al., 2017, §1). CARLA was built to support training, prototyping and validation of both perception and control, with configurable sensor suites, weather and time of day.

For computer vision the simulator does two jobs:

  • A labelled-data generator. Every camera frame can come with pixel-aligned depth, semantic tags, instance identifiers and flow, all rendered from the same scene. Synthetic data discusses how such labels are used and where they mislead.
  • A closed-loop environment. The client reads the sensors, acts, and advances the simulation. Perception errors therefore feed into driving, which open-loop evaluation on logged data cannot show; the end-to-end driving article discusses this. The gap between simulated and real images belongs to sim-to-real transfer.

Core Capabilities

Server, clients, world and actors

CARLA has a server–client architecture. The server runs the simulation and renders the scene. Clients connect over the network, read the state of the world and request changes. The 2017 paper describes the client API as Python, talking to the server through sockets. Clients send commands (steering, accelerating and braking) and meta-commands (resetting, changing the environment, changing the sensor suite) (Dosovitskiy et al., 2017, §2).

In 0.9.16:

  • carla.Client(host, port) defaults to 127.0.0.1 and port 2000 (python_api.md).
  • The quick-start page lists two TCP ports, 2000 and 2001 by default.
  • Several clients may connect at once. The foundations page warns that “only one client should tick”, because the server treats every tick as if it came from the same client.

The world is the single object that represents the running simulation. It spawns actors, sets the weather and gives access to the map; loading another map destroys the world and creates a new one. Actors are vehicles, walkers, sensors, the spectator, traffic signs and traffic lights. They are spawned from blueprints, ready-made actor layouts whose attributes can partly be customized. The core-concepts page of 0.9.16 says that eight maps are available and that all of them describe their roads in the ASAM OpenDRIVE 1.4 standard. Weather is a set of parameters (sun orientation, cloudiness, wind, fog and others). The world page notes that weather changes are visual only and do not affect physics.

Simulated time and synchrony

The documentation separates two settings that together decide how the simulation advances (adv_synchrony_timestep.md).

  • Time step. With a variable time step, the default, each step lasts as long as the server took to compute it. With a fixed time step (settings.fixed_delta_seconds = 0.05), each step advances simulated time by the same amount; at 0.05 s, the server takes 20 steps per simulated second.
  • Synchrony. In asynchronous mode, the default, the server runs as fast as it can without waiting. In synchronous mode, it waits for a client’s world.tick() before computing the next step.

The page recommends synchronous mode with a fixed time step when “synchrony and precision is relevant”, and warns: “In synchronous mode, always use a fixed time-step”. If the server waits for a slow client under a variable time step, the steps grow large. The page says physics cannot run properly when the time step is bigger than 0.1 s.

Physics substepping is enabled by default, with at most 10 substeps of at most 0.01 s. With a fixed step, the settings must satisfy fixed_delta_seconds <= max_substep_delta_time * max_substeps. The page also notes that data from GPU-based sensors, mostly cameras, “is usually generated with a delay of a couple of frames”. Its example loop puts a camera’s images in a queue and ticks the server after taking each one out.

The page lists the conditions under which CARLA supports physics and collision determinism:

  • synchronous mode with a fixed delta;
  • synchronous mode set before the world is loaded;
  • a world reload for each repetition;
  • commands batched with apply_batch_sync, which is guaranteed to execute them or return a failure.

For vision work, these settings decide whether frames from several sensors describe the same instant. That is the simulated counterpart of the synchronization problems in real-time vision pipelines.

Sensors and their encodings

A sensor is an actor, usually attached to a vehicle. Its blueprint attributes set what it measures. Its listen() method registers a callback for the data, and every measurement carries a frame number, a simulation timestamp and the sensor’s world transform (core_sensors.md). sensor_tick sets the simulated seconds between captures; its default of 0.0 means “as fast as possible”, and the camera sections list one output per step. A rigid attachment is the one the docs recommend for data, while the spring-arm attachments smooth the motion for video.

The RGB, depth and segmentation cameras return a carla.Image. Its raw_data is, in the 0.9.16 sensor reference, an “Array of BGRA 32-bit pixels”, and LibCarla’s ImageView.h views sensor images as bgra8 pixels. The same byte layout carries different quantities. The optical-flow camera’s raw_data is instead an “Array of BGRA 64-bit pixels containing two float values” (a carla.OpticalFlowImage in the Python API), and the DVS camera returns events:

Sensor What the pixel holds (0.9.16 sensor reference)
RGB camera A rendered colour image; optional post-processing (vignette, grain, bloom, auto exposure, lens flares, depth of field)
Depth camera A 24-bit depth value spread over R, G and B, least significant byte in R
Semantic segmentation The semantic tag in the red channel; CityScapesPalette converts it to colours
Instance segmentation The tag in R and a unique object identifier in G and B
Optical flow Image-plane velocity per pixel, two values in the range [−2, 2] that scale with the image size to pixels
DVS (event camera) A carla.DVSEventArray of brightness-change events, not an image

The semantic tag table notes that tags changed from 0.9.13 to 0.9.14, so label maps from different versions do not necessarily agree.

The 2017 paper had RGB cameras and two pseudo-sensors, ground-truth depth and semantic segmentation with 12 classes (Dosovitskiy et al., 2017, §2). By 0.9.16 the sensor reference also lists:

  • LiDAR by ray casting, which returns points with x,y,zx, y, z and an intensity value;
  • semantic LiDAR, radar, GNSS, IMU, collision, lane-invasion and obstacle detectors;
  • RSS;
  • V2X.

The depth camera’s encoding matters most for vision, and the example reproduces it. The sensor reference gives the decoding as

normalized=R+256 G+2562B2563−1,metres=1000⋅normalized,\text{normalized} = \frac{R + 256\,G + 256^2 B}{256^3 - 1}, \qquad \text{metres} = 1000 \cdot \text{normalized},

so depth is stored over 0–1000 m in 2242^{24} steps. The page calls the value the distance of each pixel to the camera, “also known as depth buffer or z-buffer”. It does not say in prose whether this is depth along the optical axis or the Euclidean range along the ray; depth map explains why the two differ. Two converters turn the image into a viewable grey map, Depth and the logarithmic one. The sensor reference calls the logarithmic one “Logaritmic depth” [sic], and the Python API calls it carla.ColorConverter.LogarithmicDepth. The sensor reference says, as quoted: “The precision is milimetric [sic] in both, but the logarithmic approach provides better results for closer objects.” The Python API reference of the same version says, of carla.ColorConverter, that depth conversions “cause a loss of accuracy”. It explains that depth is detected as a float that is then converted to a grey value between 0 and 255. The two pages of the 0.9.16 documentation therefore disagree; the example measures what the conversion does.

In LibCarla 0.9.16, ColorConverter::Depth computes the normalized value as a 32-bit float. The LogarithmicDepth struct is empty. ImageView.h implements the logarithmic view by running Depth into a 32-bit float grey image and applying LogarithmicLinear to it. That function computes 1+ln⁡(normalized)/5.703781 + \ln(\text{normalized})/5.70378 and clamps the result to [0.005,1][0.005, 1].

Coordinates

CARLA uses Unreal Engine’s coordinate system. The Python API reference calls it “a Z-up left-handed system”. The sensor reference gives the axes as xx forward, yy right, zz up, and notes that many visualization tools invert the yy axis, so data shown directly “may result in mirrored outputs”. Locations are in metres. Some camera attributes use Unreal units instead; focal_distance is “Measured in cm (UE units)”.

The constructor is carla.Rotation(pitch, yaw, roll) in degrees. The Python API maps these to rotations about yy, zz and xx, and warns that the Unreal Editor declares them in the order (roll, pitch, yaw). LibCarla stores the three angles as 32-bit floats. Its Rotation::RotateVector applies, in the words of its comment, “Rz(yaw) * Ry(pitch) * Rx(roll) = first x, then y, then z”, with Unreal’s signs. Transform::GetMatrix and GetInverseMatrix return the 4 × 4 forms as 16 floats.

The bounding-box tutorial projects world points into a camera in four steps:

  1. Build KK from the horizontal field of view, f=w/(2tan⁡(fov⋅π/360))f = w / (2\tan(\text{fov}\cdot\pi/360)), with the principal point at (w/2,h/2)(w/2, h/2).
  2. Transform world points with get_inverse_matrix().
  3. Swap Unreal’s camera axes to the “standard” ones by (x,y,z)→(y,−z,x)(x, y, z) \to (y, -z, x).
  4. Apply KK and divide by depth.

That swap maps a left-handed frame to the right-handed xx right, yy down, zz forward frame that camera parameters and camera pose use. Rotation representations treats such conventions in general.

Traffic Manager

The Traffic Manager (TM) controls every vehicle set to autopilot, to fill the world with plausible urban traffic (adv_traffic_manager.md):

  • Where it runs. It runs on the client side as a pipeline of stages, each on its own thread: localization, collision, traffic light, motion planner and vehicle lights.
  • Ports. A client creates a TM on a port, 8000 by default. Further TMs on the same port become TM-clients of the first, the TM-server.
  • Synchronous mode. The docs say the TM “is designed to work in synchronous mode”. When the server is synchronous, the TM must be set to synchronous mode too (tm.set_synchronous_mode(True)); the synchrony page carries the same warning.
  • Deterministic mode. Available only in synchronous mode. set_random_device_seed makes repeated runs of a script give the same traffic, provided the seed is set again after each world reload.
  • Hybrid physics mode. Turns off most physics for autopilot vehicles outside a radius around a vehicle tagged hero, 50 m by default. Vehicles without physics move by teleportation.

Recorder and rendering modes

The recorder saves what is needed to replay a simulation into a binary file, and the server can then reproduce it (foundations.md):

  • What a file can hold: actor creation and destruction with bounding and trigger boxes, traffic-light states and time settings, vehicle and pedestrian positions, orientations and velocities, vehicle light state and physics control, and the light states of buildings, streets and vehicles.
  • Extra data: by default only what playback needs is stored; start_recorder(path, True) adds the rest, which the page lists as velocities, traffic-light time settings, execution time, trigger and bounding boxes, and vehicle physics controls.
  • Replay: replay_file takes a start time, a duration and a camera to follow.
  • Size: the page estimates one hour with 50 traffic lights and 100 vehicles at about 200 MB.

The synchrony page says that a recording made with a fixed time step is easy to reenact. With a variable step, the steps differ between runs, and floating-point truncation of real time accumulates and prevents a precise repetition.

The rendering page separates two modes:

  • No-rendering mode (settings.no_rendering_mode = True): nothing is drawn, and GPU-based sensors return empty data. It suits traffic simulation.
  • Off-screen mode (-RenderOffScreen, from 0.9.12): frames are rendered without a display, and sensors still return data. In our reading, it is the mode for headless data generation.

Minimal Example

CARLA needs Unreal Engine and a GPU and cannot run on the machine used for this article. The example therefore reproduces two documented mechanisms of version 0.9.16 in NumPy and SciPy:

  • the depth camera’s encoding and its two grey-level converters;
  • the tutorial’s projection of world points into the camera.

The ported functions follow LibCarla’s ColorConverter.h (Depth, LogarithmicLinear), Transform.h (GetInverseMatrix) and Math.h (ToRadians), and the bounding-box tutorial’s build_projection_matrix and get_image_point. They carry the repository’s MIT licence text from that commit. The encoder that writes the 24-bit code, the scene, and the independent projection are ours. Every outcome below tests our reproduction of these files, not CARLA itself.

Set-up, part A. The depth camera uses its default attributes, 800 × 600 pixels and a 90° horizontal field of view, so the tutorial’s formula gives f=400f = 400. We place it 2 m above a ground plane at rotation (0, 0, 0). Two boxes stand in front of it: A spans xx 8–12 m, yy −3 to −1 m and zz 0–1.5 m; B spans 30–34, 1–3.5 and 0–3 m. Rays that hit nothing, and hits beyond 1,000 m, get the encodable maximum of 1,000 m. Pixel centres sit at half-integer coordinates, which is our convention. Each pixel’s ray has forward component 1, so the ray parameter at the hit is the depth along xx.

Because the docs do not say whether the camera stores that planar depth or the Euclidean range, the round-trip check runs on both readings. For the other measures we added a second synthetic image: a 1000 × 1000 ramp of depths from 1 mm to 1,000 m in 1 mm steps. The scene has few pixels beyond 500 m, and its sky decodes exactly, so it cannot show far-range effects or locate grey-level steps.

Our encoder takes the nearest 24-bit code, splits it into R (least significant), G and B, and stores the bytes in BGRA order. The server’s shader was not read. The script decodes in three ways:

  • with the docs’ formula in 64-bit floats;
  • with the Depth port, whose normalized value is a 32-bit float, multiplied by 1,000 in 64-bit floats (variant i);
  • with the same port, multiplied by 1,000 in 32-bit floats (variant ii).

The docs’ in_meters = 1000 * normalized does not say in which precision the product is formed, so both variants are printed. For the 8-bit grey outputs we round 255x255x to the nearest integer. This is our assumption: the conversion is done by boost::gil, whose code was not read. The steps between grey levels do not depend on whether it rounds or truncates. Either way, consecutive level boundaries are 1/2551/255 apart in the converted value; only the position of the first boundary moves (our derivation).

Set-up, part B. A camera at (−2,0.5,2.5)(-2, 0.5, 2.5) m with carla.Rotation(pitch=-6, yaw=5, roll=8) views the 16 corners of the two boxes. The port projects them with GetInverseMatrix in 32-bit floats, as LibCarla computes it, followed by the tutorial’s axis swap and KK. The independent route never uses CARLA’s composite matrix:

  1. It flips yy to obtain a right-handed frame (xx forward, yy left, zz up).
  2. It builds the camera’s rotation with SciPy as roll about xx, then pitch about yy, then yaw about zz, the order of Rotation.h’s comment. The signs come from what each angle alone does to the axis vectors in Math.cpp: positive yaw turns forward towards +y+y, positive pitch raises forward towards +z+z, and positive roll lowers the right vector. In the flipped frame these become rotations by −yaw and −pitch and by +roll.
  3. It reads OpenCV’s axes (right, down, forward) off the camera’s left, up and forward axes.

The script also computes the determinant of the axis swap.

Before the run. Every hypothesis and bar below was written down by derivation before any script ran. A planning script then computed the values a correct method reaches on the same set-up: the rounding error of the exact quotient, the 32-bit rounding of the exact normalized value, the closed-form level boundaries, and a 64-bit version of the port. It changed no bar. Each bar is listed with its result. q=1000/(224−1)q = 1000/(2^{24}-1) m is the 24-bit quantum (our arithmetic).

Changes to the plan. The batch plan’s outline of this example was extended before the run, and the editor approved it as piloted:

  • the depth ramp, and the round-trip check on both readings;
  • the two 32-bit variants, with their bar derived rather than taken from the planning script;
  • the split of the grey-level hypothesis into a linear step, a logarithmic step and the clamp distance;
  • our definition of the documentation check: “holds if the smallest step per grey level of each 8-bit output is ≤ 1 mm”;
  • porting GetInverseMatrix, which the tutorial calls, instead of GetMatrix, which the batch plan named;
  • the sign derivation of the independent route;
  • three data checks;
  • the sizes and poses above, which are our choices.

Runs. A pilot of the same script printed the same output as the run shown. Nothing in the script is random, two runs printed identical output, and it runs in under a second.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                        # one BLAS thread, set before NumPy is imported
import itertools
import numpy as np
import scipy
from scipy.spatial.transform import Rotation as SciRot

# The functions marked "port" follow CARLA 0.9.16 (commit 294096e): LibCarla's image/ColorConverter.h
# (Depth, LogarithmicLinear), geom/Transform.h (GetInverseMatrix), geom/Math.h (ToRadians), and the
# bounding-box tutorial Docs/tuto_G_bounding_boxes.md (build_projection_matrix, get_image_point).
# They are ported from C++ and the tutorial into NumPy and carry the licence of the repository at that commit:
#
# MIT License
#
# Copyright (c) 2017 Computer Vision Center (CVC) at the Universitat Autonoma de
# Barcelona (UAB).
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

F32 = np.float32
CODE_MAX = 256 * 256 * 256 - 1
q = 1000.0 / CODE_MAX                          # 24-bit quantum in metres (our arithmetic)

def depth_converter(bgra):                     # port: ColorConverter::Depth
    b, g, r = (bgra[..., i].astype(np.int64) for i in range(3))
    depth = (r + g * 256 + b * 256 * 256).astype(F32)   # integer sum, then float
    return depth / F32(CODE_MAX)                         # normalized, float32

def logarithmic_linear(normalized):            # port: ColorConverter::LogarithmicLinear
    value = F32(1.0) + np.log(normalized) / F32(5.70378)
    return np.maximum(np.minimum(value, F32(1.0)), F32(0.005))

def to_radians(deg):                           # port: Math::ToRadians<float>
    return F32(deg) * (F32(np.pi) / F32(180.0))

def get_inverse_matrix(loc, pitch, yaw, roll):  # port: Transform::GetInverseMatrix, float32
    cy, sy = np.cos(to_radians(yaw)), np.sin(to_radians(yaw))
    cr, sr = np.cos(to_radians(roll)), np.sin(to_radians(roll))
    cp, sp = np.cos(to_radians(pitch)), np.sin(to_radians(pitch))
    x, y, z = (F32(0) - F32(c) for c in loc)   # InverseTransformPoint(0): translate, then inverse-rotate
    a = (x * (cp * cy) + y * (cp * sy) + z * sp,
         x * (cy * sp * sr - sy * cr) + y * (sy * sp * sr + cy * cr) + z * (-cp * sr),
         x * (-cy * sp * cr - sy * sr) + y * (-sy * sp * cr + cy * sr) + z * (cp * cr))
    return [[cp * cy, cp * sy, sp, a[0]],
            [cy * sp * sr - sy * cr, sy * sp * sr + cy * cr, -cp * sr, a[1]],
            [-cy * sp * cr - sy * sr, -sy * sp * cr + cy * sr, cp * cr, a[2]],
            [0.0, 0.0, 0.0, 1.0]]

def build_projection_matrix(w, h, fov):        # port: tutorial
    focal = w / (2.0 * np.tan(fov * np.pi / 360.0))
    K = np.identity(3)
    K[0, 0] = K[1, 1] = focal
    K[0, 2], K[1, 2] = w / 2.0, h / 2.0
    return K

def get_image_point(loc, K, w2c):              # port: tutorial
    point_camera = np.dot(w2c, np.array([loc[0], loc[1], loc[2], 1]))
    point_camera = [point_camera[1], -point_camera[2], point_camera[0]]   # UE (x, y, z) -> (y, -z, x)
    point_img = np.dot(K, point_camera)
    return point_img[:2] / point_img[2], point_img[2]

# ---------- our code: encoder, 8-bit output, scene, independent projection ----------
def encode(depth_m):                           # nearest 24-bit code, R least significant, BGRA bytes
    code = np.rint(np.minimum(depth_m, 1000.0) / 1000.0 * CODE_MAX).astype(np.int64)
    bgra = np.empty(np.shape(depth_m) + (4,), np.uint8)
    bgra[..., 0], bgra[..., 1], bgra[..., 2], bgra[..., 3] = code >> 16, (code >> 8) & 255, code & 255, 255
    return bgra

def decode64(bgra):                            # the docs' formula in float64
    b, g, r = (bgra[..., i].astype(np.float64) for i in range(3))
    return 1000.0 * (r + g * 256 + b * 256 * 256) / CODE_MAX

def grey8(x):                                  # float in [0, 1] -> 8 bit, rounding to nearest (assumed)
    return np.rint(x.astype(np.float64) * 255).astype(np.uint8)

print("NumPy %s | SciPy %s" % (np.__version__, scipy.__version__))
W, H, FOV = 800, 600, 90.0                     # the depth camera's default attributes
K = build_projection_matrix(W, H, FOV)
f, cx, cy = K[0, 0], K[0, 2], K[1, 2]
boxes = {"A": ((8.0, -3.0, 0.0), (12.0, -1.0, 1.5)), "B": ((30.0, 1.0, 0.0), (34.0, 3.5, 3.0))}

# Part A scene: camera at (0, 0, 2) m, rotation (0, 0, 0); UE axes x forward, y right, z up; pixel centres at +0.5
u, v = np.meshgrid(np.arange(W) + 0.5, np.arange(H) + 0.5)
cam = np.array([0.0, 0.0, 2.0])
ray = np.stack([np.ones_like(u), (u - cx) / f, -(v - cy) / f], axis=-1)   # forward component 1
t = np.full(u.shape, np.inf)
down = ray[..., 2] < 0
t[down] = -cam[2] / ray[..., 2][down]                                     # ground plane z = 0
label = np.where(down, 1, 0)                                              # 0 sky, 1 ground, 2 A, 3 B
for k, (lo, hi) in enumerate(boxes.values()):
    t0, t1 = (np.array(lo) - cam) / ray, (np.array(hi) - cam) / ray
    tmin, tmax = np.minimum(t0, t1).max(-1), np.maximum(t0, t1).min(-1)
    hit = (tmax >= tmin) & (tmin > 0) & (tmin < t)
    t[hit], label[hit] = tmin[hit], 2 + k
planar = np.minimum(t, 1000.0)                                            # depth along x (assumed reading)
rng = np.minimum(t * np.linalg.norm(ray, axis=-1), 1000.0)                # Euclidean range (other reading)
sweep = (np.arange(1, 1_000_001) * 1e-3).reshape(1000, 1000)              # ramp image, 1 mm steps to 1000 m
print("D1 pixels sky/ground/A/B: %s | planar depth %.4f..%.1f m | pixels at 1000 m: %d" % (
    [int((label == i).sum()) for i in range(4)], planar.min(), planar.max(), int((planar == 1000).sum())))
px = encode(0x123456 / CODE_MAX * 1000.0)
print("D2 code 0x123456 -> bytes (B, G, R, A) = (%#04x, %#04x, %#04x, %d) -> code %#08x" % (
    *px[:3], px[3], int(round(decode64(px) / 1000.0 * CODE_MAX))))
print("q = %.6f mm" % (q * 1e3))

# H1: float64 round trip
for name, d in (("scene planar", planar), ("scene range", rng), ("sweep", sweep)):
    print("H1 %-12s max |decoded - true| = %.9f q" % (name, np.abs(decode64(encode(d)) - d).max() / q))

# H2: float32 decoding (port) against float64, on the sweep
bgra = encode(sweep)
d64 = decode64(bgra)
n32 = depth_converter(bgra)
far, near = (sweep >= 500) & (sweep < 1000), sweep < 10
for name, d32 in (("(i)  float32 normalized, x1000 in float64", n32.astype(np.float64) * 1000.0),
                  ("(ii) float32 throughout", (n32 * F32(1000.0)).astype(np.float64))):
    e = np.abs(d32 - d64) / q
    print("H2 %-42s far max %.4f q | near max %.4f q" % (name, e[far].max(), e[near].max()))

# H3: 8-bit grey outputs on the sweep
dl = sweep.ravel()
def steps(levels):
    tr = dl[1:][np.diff(levels.ravel().astype(np.int16)) != 0]           # first depth of each new level
    return tr, np.diff(tr)
lin_tr, lin_st = steps(grey8(n32))
logv = logarithmic_linear(n32)
log_tr, log_st = steps(grey8(logv))
print("H3a linear: %d transitions | step min %.4f max %.4f m | 1000/255 = %.4f m" % (
    len(lin_tr), lin_st.min(), lin_st.max(), 1000.0 / 255))
rel = log_st / log_tr[:-1]
near_idx = [int(np.argmin(np.abs(log_tr[:-1] - x))) for x in (10.0, 100.0)]
print("H3b log: %d transitions, first at %.4f m | relative step min %.5f max %.5f | exp(5.70378/255)-1 = %.5f" % (
    len(log_tr), log_tr[0], rel.min(), rel.max(), np.exp(5.70378 / 255) - 1))
print("    step at first transition %.4f m, near 10 m %.4f m, near 100 m %.4f m" % (
    log_st[0], log_st[near_idx[0]], log_st[near_idx[1]]))
print("H3c largest depth with LogarithmicLinear = 0.005: %.4f m | 1000 exp(-5.70378 * 0.995) = %.4f m" % (
    dl[logv.ravel() == F32(0.005)].max(), 1000.0 * np.exp(-5.70378 * 0.995)))
print("C-docs smallest step per grey level: linear %.1f mm, logarithmic %.1f mm (bar 1 mm)" % (
    1e3 * lin_st.min(), 1e3 * log_st.min()))

# Part B: projection of the 16 box corners, port against an independent derivation
loc, (pitch, yaw, roll) = (-2.0, 0.5, 2.5), (-6.0, 5.0, 8.0)             # carla.Rotation(pitch, yaw, roll), degrees
w2c = np.array(get_inverse_matrix(loc, pitch, yaw, roll), dtype=np.float64)
corners = np.array([c for lo, hi in boxes.values() for c in itertools.product(*zip(lo, hi))],
                   dtype=F32).astype(np.float64)                          # carla.Location holds float32
port = [get_image_point(p, K, w2c) for p in corners]
uv_port, z_port = np.array([p[0] for p in port]), np.array([p[1] for p in port])

S = np.diag([1.0, -1.0, 1.0])                  # UE -> right-handed (x forward, y left, z up)
R = SciRot.from_euler("xyz", [roll, -pitch, -yaw], degrees=True).as_matrix()   # roll, then pitch, then yaw
b = (R.T @ (S @ (corners - np.array(loc)).T)).T                          # (forward, left, up) in the camera
X, Y, Z = -b[:, 1], -b[:, 2], b[:, 0]                                     # OpenCV: right, down, forward
uv_ind = np.stack([f * X / Z + cx, f * Y / Z + cy], axis=1)
print("D3 corners: min Z %.3f m | u %.1f..%.1f | v %.1f..%.1f (image %d x %d)" % (
    z_port.min(), uv_port[:, 0].min(), uv_port[:, 0].max(), uv_port[:, 1].min(), uv_port[:, 1].max(), W, H))
print("H4 max |port - independent| over 16 corners = %.3e px (bar 1e-3 px)" % np.abs(uv_port - uv_ind).max())
M = np.array([[0, 1, 0], [0, 0, -1], [1, 0, 0]])                          # (x, y, z) -> (y, -z, x)
print("C-hand det of the UE -> OpenCV axis mapping = %.15f" % np.linalg.det(M))
NumPy 2.3.5 | SciPy 1.16.3
D1 pixels sky/ground/A/B: [239545, 229827, 9230, 1398] | planar depth 2.6711..1000.0 m | pixels at 1000 m: 240310
D2 code 0x123456 -> bytes (B, G, R, A) = (0x12, 0x34, 0x56, 255) -> code 0x123456
q = 0.059605 mm
H1 scene planar max |decoded - true| = 0.497797357 q
H1 scene range  max |decoded - true| = 0.499996610 q
H1 sweep        max |decoded - true| = 0.500000001 q
H2 (i)  float32 normalized, x1000 in float64  far max 0.5000 q | near max 0.0078 q
H2 (ii) float32 throughout                    far max 0.9997 q | near max 0.0082 q
H3a linear: 255 transitions | step min 3.9210 max 3.9220 m | 1000/255 = 3.9216 m
H3b log: 254 transitions, first at 3.4480 m | relative step min 0.02240 max 0.02283 | exp(5.70378/255)-1 = 0.02262
    step at first transition 0.0780 m, near 10 m 0.2280 m, near 100 m 2.2850 m
H3c largest depth with LogarithmicLinear = 0.005: 3.4290 m | 1000 exp(-5.70378 * 0.995) = 3.4298 m
C-docs smallest step per grey level: linear 3921.0 mm, logarithmic 78.0 mm (bar 1 mm)
D3 corners: min Z 9.709 m | u 222.1..400.9 | v 251.8..383.9 (image 800 x 600)
H4 max |port - independent| over 16 corners = 1.593e-05 px (bar 1e-3 px)
C-hand det of the UE -> OpenCV axis mapping = -1.000000000000000

Data. All three data checks held:

  • D1: the scene shows sky, ground and both boxes, and its depth runs from 2.6711 m at the bottom edge to the 1,000 m maximum.
  • D2: the code 0x123456 is stored as the bytes (B, G, R, A) = (0x12, 0x34, 0x56, 255) and decodes back to itself.
  • D3: all 16 corners lie at least 9.7 m in front of the part B camera and project inside the image.

H1, the 24-bit round trip. The 64-bit decode stays within half a quantum (bar 0.5q0.5q with a relative margin of 10−610^{-6}) on the planar reading, the range reading and the ramp. The ramp’s maximum of 0.500000001 qq lies below the bar of 0.5000005 qq. H1 held in all three. Rounding to the nearest code bounds the error by q/2q/2 by construction, and qq is 0.06 mm, so the stored value resolves depth far below a millimetre at every distance. The bound does not depend on whether the stored quantity is depth or range.

Depth map’s example stores depth in a single 16-bit channel at TUM RGB-D’s 5,000 units per metre. CARLA instead spreads a 24-bit code over three 8-bit channels of a BGRA image, scaled to a fixed range of 1,000 m.

H2, decoding in 32-bit floats. Between 500 and 1,000 m the extra error over the 64-bit decode is 0.5000 qq for variant (i) and 0.9997 qq for variant (ii), inside the bar of 0.25–2 qq. Below 10 m it is 0.0078 qq and 0.0082 qq, under the bar of 0.05 qq. H2 held for both variants, at both ranges. A 32-bit float’s spacing grows with its magnitude. For a normalized value between 0.5 and 1 the spacing is 2−242^{-24}, which is the size of the quantum itself, and a product near 1,000 has a spacing of 2−142^{-14} m, about one quantum (our arithmetic). Decoding far depths in 32-bit floats therefore adds an error as large as the encoding’s own; in our reading, code that needs the full 24-bit resolution at long range decodes in 64-bit floats.

H3, the 8-bit outputs.

  • H3a held. The linear grey output changes level 255 times along the ramp, every 3.9210–3.9220 m, within 3 mm of 1000/255=3.92161000/255 = 3.9216 m.
  • H3b held. For the logarithmic output, each step is a constant fraction of the distance at which it starts, 0.02240–0.02283 against the closed form e5.70378/255−1=0.02262e^{5.70378/255} - 1 = 0.02262 (bar ±0.001). The step grows from 0.0780 m at the first boundary to 0.2280 m near 10 m and 2.2850 m near 100 m. These are the properties of a geometric sequence, which ln⁡\ln produces.
  • H3c held. The clamp at 0.005 makes the logarithmic value constant below 1000 e−5.70378×0.995=3.42981000\,e^{-5.70378 \times 0.995} = 3.4298 m. The last ramp depth with the clamped value is 3.4290 m, within the 2 mm bar, and the first boundary of the 8-bit output lies at 3.4480 m.

So, in our reproduction, the logarithmic view resolves 10 m to about 0.23 m where the linear view resolves 3.9 m. Below 3.43 m, however, it gives the same grey value for every depth.

The documentation check. The sensor reference’s statement that the precision “is milimetric [sic] in both” was checked under our definition: “holds if the smallest step per grey level of each 8-bit output is ≤ 1 mm”. The smallest steps are 3,921.0 mm for the linear output and 78.0 mm for the logarithmic one. The check failed. The result is for our reproduction of the 0.9.16 conversions, with 8-bit rounding to nearest as our assumption, since boost::gil was not read. The Python API reference of the same version (python_api.md, carla.ColorConverter) itself states that depth conversions “cause a loss of accuracy” because depth becomes a grey value between 0 and 255.

H4 and the axis swap. The port and the independent route agree on all 16 corners to 1.593×10−51.593 \times 10^{-5} px, under the bar of 10−310^{-3} px, so H4 held. The two routes share no composite formula. Their agreement shows that the 32-bit GetInverseMatrix, the tutorial’s axis swap and KK match the composition roll, then pitch, then yaw with the single-angle signs, mapped into a right-handed frame. The swap (x,y,z)→(y,−z,x)(x, y, z) \to (y, -z, x) has determinant −1, so the check C-hand held. A map from a left-handed frame to a right-handed one must reverse orientation. Without the minus sign the swap would be a cyclic permutation with determinant +1 and would mirror the image vertically (our reasoning). Camera pose shows what axis mistakes do to a projection: a forgotten flip puts every point behind the camera, and negating a single axis mirrors the image.

Ecosystem and Integrations

  • ROS. CARLA 0.9.16 offers two connections (ecosys_ros.md):

    • The native interface, built into the server and recommended for its lower latency, supports ROS 2 only.
    • The ROS Bridge, a separate package with additional latency, supports ROS 1 and ROS 2. The bridge page lists depth, segmentation, RGB and DVS cameras, LiDAR, semantic LiDAR, GNSS, radar and IMU among the sensor data it publishes as ROS topics. It also accepts steering, throttle and brake commands and can set synchronous mode.

    Release 0.10.0’s notes list native server support for ROS 2, without the bridge.

  • Scenarios and evaluation. The README lists ScenarioRunner, an engine for traffic scenarios in CARLA 0.9.x, and the CARLA Autonomous Driving Leaderboard. The Leaderboard’s routes, scenarios and driving score are described in end-to-end autonomous driving, together with the 2017 benchmark and the studies run on it.

  • Co-simulation and road networks. The core-concepts page lists:

    • co-simulation with SUMO and PTV-Vissim traffic simulators;
    • an OpenDRIVE standalone mode that builds a road mesh from an OpenDRIVE file alone;
    • an integration of ad-rss-lib, a C++ library for Responsibility Sensitive Safety checks.

    The README also lists the Autoware bridge, the RoadRunner and Map Editor tools for maps, and repositories for conditional imitation learning and reinforcement learning.

  • Recent additions by the release notes.

    • 0.9.16 added integrations with NVIDIA Cosmos Transfer1 and the NVIDIA Neural Reconstruction Engine (NuRec), SimReady OpenUSD and MDL converters, and V2X sensors for vehicle-to-vehicle messages.
    • 0.10.0 added data-driven traffic generation with InvertedAI and support for Scenic 3.0.
  • Vision pipelines. Camera data reaches Python as flat BGRA byte arrays. The tutorial reshapes them with NumPy, and the depth encoding above turns them into metres. Labels from the segmentation cameras feed the uses discussed in semantic segmentation and instance segmentation, and frames rendered by CARLA appear as training data in video generation.

Alternatives

The sources do not rank simulators, and neither does this article.

  • Game-based data. Dosovitskiy et al. (2017) contrast CARLA with commercial games such as Grand Theft Auto V. They describe these as high-fidelity but closed, with limited scripting, sensor specification and feedback on traffic violations. They also describe racing simulators such as TORCS as lacking pedestrians, intersections and traffic rules.
  • General simulators and renderers. Synthetic data names Blender and Isaac Sim as further sources of rendered worlds; Gazebo is named here as well. Nothing about these tools is claimed in this article.
  • Indoor embodied simulators. Habitat and AI2-THOR serve indoor navigation and manipulation. Embodied AI and visual navigation describe them.
  • Logged real data. The end-to-end driving article describes benchmarks built on recorded drives, nuPlan and NAVSIM, whose sensors are real; NAVSIM’s title calls its simulation non-reactive. In our reading, they trade CARLA’s reactive world for real sensor data.

Versions and Compatibility

As of November 2026, the newest entry in the repository’s releases list is 0.9.16. The project maintains two lines:

  • 0.9.x on Unreal Engine 4 (4.26 from 0.9.12, by the rendering page).
    • 0.9.16 (16 September 2025) is the version the sources here were read from. Among many items, its notes list support for left-handed traffic maps, V2X sensors, type hints for the Python API, actor_id in bounding boxes and doors in the recorder. They also list a fix for frame, timestamp and transform of camera data not matching the image actually sent, and an option for whether depth and segmentation traverse translucent materials.
    • The 0.9.x documentation notes that the API “changed significantly” from 0.8.x.
  • 0.10.0 (19 December 2024) is, by its release notes, the migration from Unreal Engine 4.26 to 5.5. Physics moves from PhysX to Chaos, Town 10 is remodelled with Nanite enabled for all of it, Lumen provides new illumination, and the build system moves to CMake. The semantic and instance segmentation sensors are refactored, and the notes add under that item: “Only supports meshes with Nanite enabled”. The Python API supports Python 3.8 to 3.12, RSS is removed from the docs and the light manager from the API. A script, map or label set written for one line is therefore not guaranteed to work on the other; check each feature against its version’s documentation.

The 0.9.16 quick-start page lists the requirements:

  • Windows 10 or 11, or Ubuntu 20.04 or 22.04;
  • a dedicated GPU “equivalent to an NVIDIA 2070 or better” with at least 8 GB of memory;
  • about 20 GB of disk;
  • the two TCP ports;
  • Python 3.7 to 3.12.

The client library installs with pip install carla. The README’s recommended system is more demanding: at least 32 GB of RAM and an RTX 3070, 3080 or 4090 class GPU.

Licences. By the 0.9.16 README and LICENSE:

  • CARLA’s own code is under the MIT licence and its own assets under CC-BY.
  • Unreal Engine follows its own licence terms; the 2017 paper described the engine as free for non-commercial use at the time.
  • The RSS build variant links ad-rss-lib, which is LGPL-2.1-only.
  • The SUMO integration uses PROJ (X/MIT), SQLite (public domain) and Xerces-C (Apache 2.0), and the Chrono integration uses Eigen (MPL 2.0).
  • Map import can optionally use the Autodesk FBX SDK.

Limitations

  • Simulated sensors are not real ones. Rendered images, physics and traffic differ from the real world in ways a model can exploit. Sim-to-real transfer covers the gap and the methods for closing it.
  • Reproducibility has to be set up. The defaults are asynchronous mode and a variable time step. Deterministic physics, deterministic traffic and precise replays need synchronous mode, a fixed step, the Traffic Manager in synchronous mode with a seed, and a world reload per run.
  • Timing between sensors. Camera data is usually generated a couple of frames late, by the synchrony page, and with several clients only one may tick. In our reading, multi-sensor data needs synchronous mode and a queue per sensor.
  • Depth is easy to misread. The docs do not say in prose whether the depth camera stores depth along the axis or range along the ray. The two pages of the 0.9.16 docs disagree about the precision of the converted output. In our reproduction, the 8-bit views keep metre-scale steps (linear) or nothing below 3.43 m (logarithmic), and 32-bit decoding adds up to about one quantum of error at long range. Raw images decoded with the formula keep the 24-bit resolution.
  • Left-handed coordinates. Data passed to right-handed tools needs an explicit axis change, and visualizers that invert yy show mirrored outputs. Some attributes are in centimetres.
  • Heavy and version-split. The simulator needs a dedicated GPU and a supported operating system. The UE4 and UE5 lines differ in physics, rendering and sensors.

Related

  • Sim-to-Real Transfer

    Training perception models and robot policies in simulation and making them work on real hardware, covering the reality gap, visual and dynamics randomization, system identification, pixel- and feature-level domain adaptation, translating real images into a canonical simulation, synthetic data for detection and pose estimation, and what the published results show, each within its own set-up.

  • Depth Map

    An image whose pixels store the distance to the scene, usually the depth Z along the optical axis, so that each pixel with a known camera can be lifted to a 3D point.

  • Camera Pose

    The position and orientation of a camera as a rigid transform in SE(3), the two directions it is stored in (world-to-camera and camera-to-world), the axis conventions of OpenCV, COLMAP, OpenGL and NeRF, and how pose errors are measured.

  • Rotation Representations

    The ways a 3D rotation can be written down (rotation matrices, Euler angles, rotation vectors, unit quaternions, 6D vectors and the Lie algebra), how they convert into each other, and where each one fails.

  • Autonomous Driving Perception

    How a self-driving vehicle turns camera, LiDAR, radar and map data into a 3D description of the road, its own position and the moving objects around it, from Dickmanns's vision-guided van and the DARPA challenges to today's learned 3D detectors, bird's-eye-view models and benchmarks.

  • Embodied AI

    The study of agents that perceive their surroundings through their own sensors and act in them, covering where the idea comes from, simulators used as research platforms, embodied tasks from navigation and question answering to instruction following and rearrangement, learning by imitation, reinforcement and large pretrained models including vision-language-action models, transfer from simulation to real robots, and how embodied agents are evaluated.

References

  1. Dosovitskiy, A., Ros, G., Codevilla, F., López, A. & Koltun, V. (2017). CARLA: An Open Urban Driving Simulator. Conference on Robot Learning (CoRL), PMLR 78, 1–16.
  2. CARLA 0.9.16 documentation: Sensors reference (Docs/ref_sensors.md), carla-simulator/carla at commit 294096e (read November 2026).
  3. CARLA 0.9.16 documentation: Python API reference (Docs/python_api.md), carla-simulator/carla at commit 294096e (read November 2026).
  4. CARLA 0.9.16 documentation: Synchrony and time-step (Docs/adv_synchrony_timestep.md), carla-simulator/carla at commit 294096e (read November 2026).
  5. CARLA 0.9.16 documentation: Foundations, Core concepts, World and client, Actors and blueprints, Sensors and data (Docs/foundations.md, core_concepts.md, core_world.md, core_actors.md, core_sensors.md), carla-simulator/carla at commit 294096e (read November 2026).
  6. CARLA 0.9.16 documentation: Bounding boxes tutorial (Docs/tuto_G_bounding_boxes.md), carla-simulator/carla at commit 294096e (read November 2026).
  7. CARLA 0.9.16 documentation: Traffic Manager (Docs/adv_traffic_manager.md), carla-simulator/carla at commit 294096e (read November 2026).
  8. CARLA 0.9.16 documentation: ROS (Docs/ecosys_ros.md, ros_documentation.md), Quick start (start_quickstart.md) and Rendering options (adv_rendering_options.md), carla-simulator/carla at commit 294096e (read November 2026).
  9. CARLA 0.9.16 source: LibCarla image/ColorConverter.h, image/ImageView.h, geom/Transform.h, geom/Rotation.h, geom/Math.h and geom/Math.cpp, carla-simulator/carla at commit 294096e (MIT; read November 2026).
  10. CARLA 0.9.16: README.md (licences) and LICENSE, carla-simulator/carla at commit 294096e (read November 2026).
  11. CARLA release notes for 0.9.16 (16 September 2025) and 0.10.0 (19 December 2024), GitHub releases of carla-simulator/carla (read November 2026).