Vision Systems & Engineering
FFmpeg
The open-source command-line tool and libraries that demux, decode, filter, encode and mux video, with packets and frames, timestamps and time bases, GOPs and frame types, seeking, frame extraction, pixel formats, CRF, ffprobe and licensing, checked on synthetic video whose frames carry their own index.
beginner
FFmpeg is an open-source collection of libraries and command-line programs for reading, converting and writing audio and video. Its documentation calls the ffmpeg program a universal media converter. ffmpeg reads any number of inputs (files, pipes, network streams, capture devices), filters and transcodes them, and writes any number of outputs. The companion program ffprobe reports what a file contains. The libraries it is built from, as ffmpeg -version lists them, are libavutil, libavcodec, libavformat, libavdevice, libavfilter, libswscale and libswresample. The project is developed by the FFmpeg developers, and most of its code is under the LGPL 2.1 or later.
In vision work FFmpeg can be the step between a video file or camera stream and an array of frames, or between rendered frames and a video. Datasets are prepared with it: the HMDB51 clips, for example, as that article reports, were compressed with the DivX 5.0 codec through ffmpeg. Libraries embed it: OpenCV’s wheels include FFmpeg for their video back end. This article covers what a user of the tool needs in order to get the frames they expect. That means the pipeline model, packets and frames, timestamps, keyframes and frame types, seeking, frame-rate conversion, pixel formats, the encoder’s quality setting, ffprobe, hardware decoding and the licence. The example runs ffmpeg and ffprobe 9.0.2 on synthetic video in which every frame carries its own index, so each claim about which frame comes out can be checked exactly. Several neighbours own nearby material, and this article links them. Codecs for still images and their containers belong to Image Compression and Image File Formats. The luma and chroma matrices and studio range belong to Color Spaces. Motion estimation inside video encoders, including FFmpeg’s own search code, belongs to Block Matching. Video coding itself, the theory of codecs and containers, goes beyond what a user of the tool needs.
Purpose
A video file is not a sequence of images. It is a container holding one or more compressed elementary streams, each a sequence of packets with timestamps, and, as the example shows, the order in which packets are decoded is not the order in which frames are shown. A vision pipeline wants something simpler: decoded frames, in display order, at known times, in a known pixel layout. FFmpeg does the conversion in both directions. It also converts between containers without decoding, changes frame rate and resolution, and re-encodes at a chosen quality. The same program handles files, pipes, devices and network protocols, and its demuxers include image sequences (image2) and raw video, so a script that calls it can read a clip, an image sequence or an RTSP camera with the same commands. The protocols documentation describes RTSP as a demuxer and muxer in libavformat rather than a plain protocol handler.
Core Capabilities
The pipeline. By its documentation, ffmpeg builds a transcoding pipeline from five kinds of component:
- Demuxers read an input and split it into elementary streams. They send encoded packets to decoders or muxers. One demuxer is created for each
-i. - Decoders turn packets into raw frames: arrays of pixels for video.
- Filtergraphs transform frames. A simple filtergraph (
-vf) has one input and one output and is tied to one output stream. A complex one (-filter_complex) can combine several inputs and outputs. - Encoders compress frames back into packets.
- Muxers interleave the packets of the output streams and write the container.
Options apply to the next input or output file named after them, so the same option means different things before and after -i. Streams are addressed by index (0:1 is the second stream of the first input) and chosen for each output with -map.
Stream copy and transcoding. With -c copy the packets go from demuxer to muxer untouched. The documentation describes this as very fast and without quality loss. It is how one changes the container or drops a stream. Nothing can be filtered on this path, since filters act on decoded frames. Everything else is transcoding: decode, optionally filter, encode. Encoding is usually lossy, so the documentation advises transcoding only when needed. Writing raw frames to a pipe (-f rawvideo -) also counts as transcoding: the frames are decoded and passed to the rawvideo encoder. It is one way to hand frames to NumPy, and the way the example reads every frame.
Timestamps and time base. In the example’s files every packet carries a presentation timestamp (PTS) and a decoding timestamp (DTS). Both are integers in the stream’s time base, a fraction of a second, which need not be the frame period. The example’s MP4 files have time base 1/12800, so one frame at 25 fps lasts 512 ticks. With B-frames the two orders differ. The encoder sends a later frame before the frames that are displayed ahead of it; in our reading, because those frames are predicted from it as well. The decoding timestamps then start below zero, which in our reading keeps every frame’s decoding time at or before its presentation time. Options that touch timestamps include:
-copyts, which keeps input timestamps unchanged.-fps_mode, which decides whether frames are passed through with their timestamps (passthrough), duplicated and dropped to a constant rate (cfr), passed through or dropped so that no two frames share a timestamp (vfr), or chosen betweencfrandvfrby the muxer’s capabilities (auto, the default).- The format option
avoid_negative_ts, which can shift all output timestamps.
GOPs and frame types. An encoder places keyframes at intervals. In our reading they are the seek points that the -ss documentation refers to, the frames from which decoding can start, and part (c) of the example behaves accordingly. The generic codec option -g sets the group of pictures (GOP) size, with a default of 12. That default belongs to the generic option: for libx264 the documentation maps -g to x264’s keyint, the maximum GOP size, and lists keyint_min and the scene-change threshold sc_threshold as related options. -bf sets the maximum number of B-frames between non-B frames; the documentation gives no default, and this build’s ffmpeg -h full lists 0. -force_key_frames places keyframes at given times or by an expression, for example at chapter starts. ffprobe -show_frames reports each decoded frame’s pict_type (I, P or B) and key_frame flag.
Seeking. -ss before -i seeks the input. The documentation notes that most formats cannot seek exactly, so the demuxer seeks to the closest seek point before the position. When transcoding, -accurate_seek is on by default, and the segment from the seek point to the requested position is decoded and discarded. With stream copy, or with -noaccurate_seek, that segment is kept. -ss after -i is an output option: it decodes but discards input until the timestamps reach the position. -sseof seeks relative to the end, and -t and -to limit duration and end position. Positions use FFmpeg’s time-duration syntax, either [-][HH:]MM:SS[.m...] or a number of seconds.
Frame extraction. These filters and options select frames:
- The
fpsfilter converts to a constant frame rate by duplicating or dropping frames. Its rate defaults to 25. Itsroundoption sets how timestamps are rounded (zero,inf,down,upornear, the default). selectevaluates an expression per frame and keeps frames where it is non-zero. Its variables includepict_type,keyand ascene-change score, soselect='eq(pict_type\,I)'keeps the I-frames.- The decoder option
-skip_frame nokeydiscards every frame except keyframes before decoding them. -frames:v Nstops after N frames.scaleresizes.
Pixel formats. The libx264 wrapper takes YCbCr or grey input (its formats are listed below); the separate libx264rgb encoder takes packed RGB. yuv420p keeps one Cb and one Cr sample per 2 × 2 block of luma samples (as the plane sizes in the example’s code show), yuv444p keeps all three planes at full resolution, and RGB formats such as rgb24 are packed. -pix_fmt chooses the output format. If the encoder cannot take the requested format, ffmpeg warns and picks the best one it supports. Prefixing the format with + makes that an error instead. Conversions are done by libswscale, whose flags -sws_flags sets. The libx264 wrapper in this build accepts, among others, yuv420p, yuv422p, yuv444p, nv12 and 10-bit variants. Which matrix and range a conversion uses is a colour question for Color Spaces. What changes in the samples themselves is shown in part (e) of the example.
Encoding quality. For libx264 the documentation describes -crf as the quality for constant quality mode and -preset as the encoding preset. This build reports medium as the default preset and −1 (unset) as the default crf. What each CRF value means is defined by x264, whose documentation the FFmpeg page points to and which was not read here. Part (b) of the example measures the direction of the setting on one clip.
ffprobe. ffprobe prints the streams (-show_streams), packets (-show_packets) and decoded frames (-show_frames) of an input. -show_entries selects fields, -select_streams selects streams, -of json gives machine-readable output, and -read_intervals limits reading to parts of the file. Nothing is claimed here about other versions’ field names, so a script should read them from the installed version’s output, as the example does with 9.0.2.
Hardware decoding. -hwaccel selects hardware-accelerated decoding. The 9.0.2 documentation lists vdpau, dxva2, d3d11va, vaapi, qsv and videotoolbox, plus auto and the default none. -hwaccels lists the methods compiled into a build, and whether they run depends on the hardware and driver. The documentation warns that most methods are meant for playback and are not faster than software decoding on modern CPUs. It adds that ffmpeg usually has to copy the decoded frames from GPU memory to system memory, so it calls the option mainly useful for testing. This build was configured with --enable-videotoolbox. The example decodes in software.
Threads. The codecs documentation gives auto as the default of the generic -threads option, while this build’s ffmpeg -h full lists the library option’s default as 1; which applies when the option is omitted was not tested. FFmpeg’s multithreading notes describe two methods: slice threading, which decodes parts of one frame in parallel, and frame threading, which decodes several frames at once and adds one frame of delay for each thread beyond the first. The example passes -threads 1 to the encoder, a choice made for reproducibility.
Minimal Example
The example checks five behaviours against frames whose content says which frame they are.
The source is 300 frames of 320 × 240 video at 25 fps (12 s), generated directly as raw yuv420p. Luma holds two moving sinusoidal gratings. The top 32 rows hold the frame index as ten black-or-white 32 × 32 cells. Chroma holds two slow sinusoids. A frame’s index is read back from the means of the cell centres. Because the source is already yuv420p, luma PSNR against it has exact ground truth, with no colour conversion inside the pipeline. ffmpeg encodes the source with libx264 to MP4 at -preset medium -crf 23 -bf 2 -threads 1 with two GOP sizes, -g 50 and -g 10. Every decode writes rawvideo yuv420p with -fps_mode passthrough, so no frame is duplicated or dropped on the way out.
The five parts are:
- (a) Packet timestamps, frame types and keyframes, from
ffprobe. - (b) Luma PSNR by frame type, file size and PSNR at CRF 18, 23 and 28, and a repeated encode.
- (c) The first frame produced by
-ss Tfor four values of T:- before
-i(accurate seek, transcoding to rawvideo); - after
-i; - before
-iwith-noaccurate_seek; - for a stream copy (
-ss T -i g50.mp4 -c copy) decoded twice: once with the MP4 demuxer’s defaults and once with-ignore_editlist 1.
- before
- (d) Which source frames
fps=10keeps withround=nearandround=down. - (e)
rgb24→yuv444pagainstrgb24→yuv420pon a separate still image. The still has a 64 × 64 patch of alternating one-pixel red and blue columns and a flat-coloured patch, measured on their interiors.
Hypotheses. They were written before ffmpeg processed any video (only its version and help commands had run), from the documentation’s text, with targets computed from the construction by a planning script that did not run ffmpeg:
- H1: keyframes fall exactly at frames 0, g, 2g, …, and each is an I-frame.
- H1b: no run of B-frames is longer than 2.
- H2: accurate seeking, and
-ssafter-i, start at the first frame whose timestamp is at or after T. That is : 115, 116, 175 and 254 for T = 4.6, 4.61, 7 and 10.13. The batch plan worded H2 as “accurate seeking lands on round(T × 25)”: 115, 115, 175 and 253. The literal reading of the documentation (decode and discard up to the position) gives . Both readings were in the example’s plan before the pilot, and both are reported. The T values were chosen so that two are exact frame times, where the readings agree, and two fall between frames, where they differ. - H3:
-noaccurate_seek, and the stream copy read with its edit list ignored, start at the preceding keyframe: 100, 100, 150 and 250. - H4:
fps=10keeps exactly K = 120 frames, and the kept set equals one of two closed forms. The documentation gives only the rounding rule, so the closed forms are our derivation. Input frame gets output tick (or fordown), and tick then shows either the last frame with or the first with . The two forms differ at every one of the 120 ticks, and the example made no prediction between them. - H5: in part (e):
- the luma planes of the two formats are identical (maximum difference 0);
- the mean chroma error of 4:2:0, upsampled by ffmpeg, against 4:4:4 is at least 10 code values on the stripe patch and at most 1 on the flat patch;
- the RGB round trip of the patch is worse through 4:2:0 than through 4:4:4.
- H6: from CRF 18 to 23 to 28, file size and mean luma PSNR both fall strictly. This is our own expectation, not a statement of the FFmpeg documentation.
Checks. C1 and C2 require the index codes to decode correctly in 300 of 300 source and decoded frames. C3 requires decoding timestamps to increase and presentation timestamps to hit each frame slot once. C4 requires ffprobe’s frames to come in index order. C5 requires the repeated encode to decode to identical bytes.
Bars set after the planning script. The planning script printed the values from which these bars and targets were written:
- C2’s 300/300.
- H5’s bars of 10 and 1. The script’s model of 4:2:0 gave a patch error of 70.000 and a flat error of 0.000. Our derivation, with the BT.601 limited-range matrix the script assumed, gives 70 on the patch for any chroma filter with non-negative weights summing to one, because the patch’s chroma is constant after subsampling.
- The H2 and H3 target frames.
- H4’s K and closed forms.
- H1’s keyframe lists.
No ffmpeg ran in that script.
Departures from the batch plan. The example departs from the batch plan’s description in several ways, all approved before the first run:
- The source is generated in
yuv420pinstead of RGB, and the red–blue patch is in a separate still. - Part (e) converts with libswscale alone, without an encoder.
- “Re-encoding” after seeking is done as transcoding to rawvideo.
- H2 is given in both forms.
- Several conditions were added:
-noaccurate_seek, the two decodes of the stream copy,round=down, H1b, C3–C5, H6, and the explicit-preset medium.
Runs. A pilot of the same script printed the same output as the first run shown here, and the script was not changed between them. It runs in about 5 seconds; two runs printed identical output. It calls ffmpeg and ffprobe from the PATH and writes its files into a work folder. FFmpeg is only called, and none of its code is reproduced.
import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
os.environ[v] = "1" # one BLAS thread, set before NumPy is imported
import hashlib, json, math, subprocess
from fractions import Fraction
import numpy as np
# ffmpeg and ffprobe 9.0.2 (Homebrew build: --enable-gpl --enable-version3 --enable-libx264)
N, W, H, FPS = 300, 320, 240, 25 # 12 s of 320 x 240 video at 25 fps
NBITS, CELL = 10, 32 # frame index as 10 black/white 32 x 32 cells
WORK = "work" # every file is written here
os.makedirs(WORK, exist_ok=True)
P = lambda name: os.path.join(WORK, name)
def run(args, data=None):
return subprocess.run(args, input=data, capture_output=True, check=True).stdout
def ff(*args, data=None):
return run(["ffmpeg", "-hide_banner", "-v", "error", "-y", *args], data)
def probe(*args):
return json.loads(run(["ffprobe", "-v", "error", "-of", "json", *args]))
def make_frame(i):
"""yuv420p frame i: moving gratings in Y and in chroma, the code of i in the top 32 rows."""
y, x = np.mgrid[0:H, 0:W]
Y = 120 + 40 * np.sin(2 * np.pi * (x + 3 * i) / 80) + 30 * np.sin(2 * np.pi * (y - 2 * i) / 60)
for b in range(NBITS):
Y[0:CELL, CELL * b:CELL * (b + 1)] = 200 if (i >> (NBITS - 1 - b)) & 1 else 50
Y = np.clip(np.rint(Y), 16, 235).astype(np.uint8)
yc, xc = np.mgrid[0:H // 2, 0:W // 2]
U = np.rint(128 + 20 * np.sin(2 * np.pi * (xc + i) / 40)).astype(np.uint8)
V = np.rint(128 + 20 * np.cos(2 * np.pi * (yc + i) / 30)).astype(np.uint8)
return Y, U, V
def read_code(Y):
"""Decode the frame index from the centres of the 10 cells."""
bits = [Y[8:24, CELL * b + 8:CELL * b + 24].mean() > 125 for b in range(NBITS)]
return sum(int(bit) << (NBITS - 1 - k) for k, bit in enumerate(bits))
def luma(frames):
return frames[:, :W * H].reshape(-1, H, W)
def decode(path, before=(), after=()):
"""Decode to raw yuv420p frames; before/after are options placed before/after -i."""
raw = ff(*before, "-i", path, *after, "-fps_mode", "passthrough",
"-f", "rawvideo", "-pix_fmt", "yuv420p", "-")
return np.frombuffer(raw, np.uint8).reshape(-1, W * H * 3 // 2)
def codes(frames):
return [read_code(Y) for Y in luma(frames)]
def psnr(a, b):
mse = np.mean((a.astype(float) - b.astype(float)) ** 2)
return 10 * np.log10(255 ** 2 / mse)
# ---- source: raw yuv420p, so luma has exact ground truth -------------------------------
src = [make_frame(i) for i in range(N)]
with open(P("src.yuv"), "wb") as f:
for Y, U, V in src:
f.write(Y.tobytes() + U.tobytes() + V.tobytes())
srcY = np.stack([Y for Y, _, _ in src])
c = [read_code(Y) for Y in srcY]
print("C1 source codes == index:", sum(a == i for i, a in enumerate(c)), "of", N,
"| Y range", srcY.min(), srcY.max())
RAW_IN = ["-f", "rawvideo", "-pix_fmt", "yuv420p", "-s", f"{W}x{H}", "-r", str(FPS), "-i", P("src.yuv")]
def encode(name, g, crf):
ff(*RAW_IN, "-c:v", "libx264", "-preset", "medium", "-crf", str(crf), "-g", str(g),
"-bf", "2", "-threads", "1", P(name))
return P(name)
# ---- (a) packets, frame types and keyframes ---------------------------------------------
print("\n(a) packets and frames")
files = {"g50": encode("g50.mp4", 50, 23), "g10": encode("g10.mp4", 10, 23)}
types = {}
for name, path in files.items():
g = int(name[1:])
pk = probe("-select_streams", "v:0", "-show_entries", "stream=time_base:packet=pts,dts", path)
tb = Fraction(pk["streams"][0]["time_base"])
pts = [int(p["pts"]) for p in pk["packets"]]; dts = [int(p["dts"]) for p in pk["packets"]]
step = Fraction(1, FPS) / tb
grid = sorted(pts) == [min(pts) + k * int(step) for k in range(N)]
print(f"{name}: time_base {tb}, ticks per frame {step}, packets {len(pts)}, pts != dts in "
f"{sum(a != b for a, b in zip(pts, dts))} | C3 dts increasing {all(b > a for a, b in zip(dts, dts[1:]))}, "
f"pts one per frame on the grid {grid}")
print(" first packets (dts, pts) in decode order:", list(zip(dts[:6], pts[:6])))
fr = probe("-select_streams", "v:0", "-show_frames",
"-show_entries", "frame=pict_type,key_frame,best_effort_timestamp_time", path)["frames"]
t = [f["pict_type"] for f in fr]; types[name] = np.array(t)
keys = [k for k, f in enumerate(fr) if int(f["key_frame"]) == 1]
order = sum(round(float(f["best_effort_timestamp_time"]) * FPS) == k for k, f in enumerate(fr))
runs = max(len(r) for r in "".join(t).replace("I", " ").replace("P", " ").split() + [""])
print(f" frames {len(fr)} (C4 timestamp order == index: {order}) | I {t.count('I')} P {t.count('P')} "
f"B {t.count('B')} | longest B run {runs}")
print(f" keyframes {keys if len(keys) <= 8 else keys[:6] + ['...'] + keys[-2:]} "
f"| H1 == multiples of {g}: {keys == list(range(0, N, g))} | all I: {all(t[k] == 'I' for k in keys)}")
# ---- (b) luma PSNR by frame type, CRF, determinism --------------------------------------
print("\n(b) luma PSNR against the source (dB)")
for name, path in files.items():
dec = decode(path)
p = np.array([psnr(a, b) for a, b in zip(luma(dec), srcY)])
ok = sum(a == i for i, a in enumerate(codes(dec)))
row = " | ".join(f"{ty}: mean {p[types[name] == ty].mean():.2f} min {p[types[name] == ty].min():.2f}"
for ty in "IPB" if (types[name] == ty).any())
print(f"{name}: C2 codes {ok}/{N} | {row}")
for crf in (18, 23, 28):
path = files["g50"] if crf == 23 else encode(f"crf{crf}.mp4", 50, crf)
dec = decode(path)
p = np.mean([psnr(a, b) for a, b in zip(luma(dec), srcY)])
print(f"crf {crf}: {os.path.getsize(path)} bytes, mean PSNR {p:.2f}, C2 codes "
f"{sum(a == i for i, a in enumerate(codes(dec)))}/{N}")
again = encode("g50_again.mp4", 50, 23)
h = [hashlib.sha256(decode(q).tobytes()).hexdigest()[:12] for q in (files["g50"], again)]
same_file = open(files["g50"], "rb").read() == open(again, "rb").read()
print(f"C5 repeated encode: decoded sha256 {h[0]} vs {h[1]}, equal {h[0] == h[1]} | file bytes equal {same_file}")
# ---- (c) seeking on g50 (keyframes every 50 frames) -------------------------------------
print("\n(c) first frame after -ss T on g50.mp4")
print("T ceil round key | in+acc out in+noacc | copy: default (first, n) ignore_editlist (first, n)")
for t in ["4.6", "4.61", "7", "10.13"]:
T = Fraction(t)
ce, ro, key = math.ceil(T * FPS), math.floor(T * FPS + Fraction(1, 2)), 50 * math.floor(T * FPS / 50)
first = lambda before=(), after=(): codes(decode(files["g50"], before, ["-frames:v", "1", *after]))[0]
acc, out, noacc = first(["-ss", t]), first((), ["-ss", t]), first(["-noaccurate_seek", "-ss", t])
ff("-ss", t, "-i", files["g50"], "-c", "copy", P("cut.mp4"))
d0, d1 = codes(decode(P("cut.mp4"))), codes(decode(P("cut.mp4"), ["-ignore_editlist", "1"]))
print(f"{t:6} {ce:4} {ro:5} {key:3} | {acc:6} {out:4} {noacc:8} | {d0[0]:5} {len(d0):4}"
f" {d1[0]:5} {len(d1):4}")
# ---- (d) fps=10: which source frames are kept --------------------------------------------
print("\n(d) fps=10 on g50.mp4")
K = N * 10 // FPS # 12 s at 10 fps
for rnd in ("near", "down"):
r = [math.floor(Fraction(10 * i, FPS) + (Fraction(1, 2) if rnd == "near" else 0)) for i in range(N)]
last = [max(i for i in range(N) if r[i] <= k) for k in range(K)]
first = [min(i for i in range(N) if r[i] >= k) for k in range(K)]
kept = codes(decode(files["g50"], (), ["-vf", f"fps=10:round={rnd}"]))
print(f"round={rnd}: {len(kept)} frames, kept[:8] {kept[:8]} | equal to 'last' closed form "
f"{kept == last} ({sum(a == b for a, b in zip(kept, last))}/{K}), to 'first' {kept == first} "
f"({sum(a == b for a, b in zip(kept, first))}/{K})")
# ---- (e) pixel formats: 4:4:4 against 4:2:0 on an RGB still -----------------------------
print("\n(e) rgb24 still -> yuv444p / yuv420p (swscale defaults)")
rgb = np.full((H, W, 3), 128, np.uint8)
rgb[32:96, 32:96] = (255, 0, 0)
rgb[32:96, 33:96:2] = (0, 0, 255) # 1-pixel red/blue columns
rgb[32:96, 160:224] = (40, 160, 90) # flat colour
open(P("still.rgb"), "wb").write(rgb.tobytes())
RGB_IN = ["-f", "rawvideo", "-pix_fmt", "rgb24", "-s", f"{W}x{H}", "-i", P("still.rgb")]
conv = lambda inp, fmt: ff(*inp, "-f", "rawvideo", "-pix_fmt", fmt, "-")
y444 = np.frombuffer(conv(RGB_IN, "yuv444p"), np.uint8).reshape(3, H, W).astype(int)
raw420 = conv(RGB_IN, "yuv420p")
open(P("still420.yuv"), "wb").write(raw420)
IN420 = ["-f", "rawvideo", "-pix_fmt", "yuv420p", "-s", f"{W}x{H}", "-i", P("still420.yuv")]
up = np.frombuffer(conv(IN420, "yuv444p"), np.uint8).reshape(3, H, W).astype(int)
Y420 = np.frombuffer(raw420[:W * H], np.uint8).reshape(H, W).astype(int)
open(P("still444.yuv"), "wb").write(y444.astype(np.uint8).tobytes())
IN444 = ["-f", "rawvideo", "-pix_fmt", "yuv444p", "-s", f"{W}x{H}", "-i", P("still444.yuv")]
back = {k: np.frombuffer(conv(i, "rgb24"), np.uint8).reshape(H, W, 3).astype(int)
for k, i in (("444", IN444), ("420", IN420))}
patch, flat = (slice(36, 92), slice(36, 92)), (slice(36, 92), slice(164, 220))
cerr = lambda reg: np.abs(up[1:] - y444[1:])[:, reg[0], reg[1]].mean()
rerr = lambda k, reg: np.abs(back[k] - rgb.astype(int))[reg].mean()
print(f"max |Y(420) - Y(444)| over the image: {np.abs(Y420 - y444[0]).max()}")
print(f"mean |CbCr(420, upsampled) - CbCr(444)|: patch {cerr(patch):.3f}, flat {cerr(flat):.3f}")
print(f"RGB round trip mean |error|: patch 444 {rerr('444', patch):.3f}, 420 {rerr('420', patch):.3f} | "
f"flat 444 {rerr('444', flat):.3f}, 420 {rerr('420', flat):.3f}")
C1 source codes == index: 300 of 300 | Y range 50 200
(a) packets and frames
g50: time_base 1/12800, ticks per frame 512, packets 300, pts != dts in 300 | C3 dts increasing True, pts one per frame on the grid True
first packets (dts, pts) in decode order: [(-1024, 0), (-512, 1536), (0, 512), (512, 1024), (1024, 2560), (1536, 2048)]
frames 300 (C4 timestamp order == index: 300) | I 6 P 109 B 185 | longest B run 2
keyframes [0, 50, 100, 150, 200, 250] | H1 == multiples of 50: True | all I: True
g10: time_base 1/12800, ticks per frame 512, packets 300, pts != dts in 300 | C3 dts increasing True, pts one per frame on the grid True
first packets (dts, pts) in decode order: [(-1024, 0), (-512, 1536), (0, 512), (512, 1024), (1024, 2560), (1536, 2048)]
frames 300 (C4 timestamp order == index: 300) | I 30 P 103 B 167 | longest B run 2
keyframes [0, 10, 20, 30, 40, 50, '...', 280, 290] | H1 == multiples of 10: True | all I: True
(b) luma PSNR against the source (dB)
g50: C2 codes 300/300 | I: mean 50.29 min 49.34 | P: mean 49.95 min 47.60 | B: mean 46.41 min 44.01
g10: C2 codes 300/300 | I: mean 49.66 min 48.80 | P: mean 49.13 min 46.79 | B: mean 46.13 min 43.92
crf 18: 426992 bytes, mean PSNR 50.44, C2 codes 300/300
crf 23: 278761 bytes, mean PSNR 47.77, C2 codes 300/300
crf 28: 179675 bytes, mean PSNR 44.32, C2 codes 300/300
C5 repeated encode: decoded sha256 1e306ebdcdbf vs 1e306ebdcdbf, equal True | file bytes equal True
(c) first frame after -ss T on g50.mp4
T ceil round key | in+acc out in+noacc | copy: default (first, n) ignore_editlist (first, n)
4.6 115 115 100 | 115 115 100 | 115 185 100 200
4.61 116 115 100 | 116 116 100 | 116 184 100 200
7 175 175 150 | 175 175 150 | 175 125 150 150
10.13 254 253 250 | 254 254 250 | 254 46 250 50
(d) fps=10 on g50.mp4
round=near: 120 frames, kept[:8] [1, 3, 6, 8, 11, 13, 16, 18] | equal to 'last' closed form True (120/120), to 'first' False (0/120)
round=down: 120 frames, kept[:8] [2, 4, 7, 9, 12, 14, 17, 19] | equal to 'last' closed form True (120/120), to 'first' False (0/120)
(e) rgb24 still -> yuv444p / yuv420p (swscale defaults)
max |Y(420) - Y(444)| over the image: 0
mean |CbCr(420, upsampled) - CbCr(444)|: patch 70.000, flat 0.000
RGB round trip mean |error|: patch 444 0.167, 420 88.667 | flat 444 0.333, 420 0.333
The checks held. All 300 index codes decoded correctly from the source (C1) and from every encoded file at every CRF (C2). The cells are large and far apart in grey level, so the lossy codec never moved a cell mean across the threshold. In both files the decoding timestamps increased and the presentation timestamps filled the 300 slots of 512 ticks once each (C3), and ffprobe listed the frames in index order (C4). The repeated encode with one thread decoded to the same bytes and even wrote the same file (C5).
(a) Frame types. H1 held for both GOP sizes: keyframes at 0, 50, …, 250 for -g 50 and at every multiple of 10 for -g 10, all of them I-frames. x264 inserted no keyframe of its own on this clip. H1b held: the longest run of B-frames was 2 in both files, the -bf 2 limit. The packet list shows the reordering that B-frames require. The first packets in decoding order present frames 0, 3, 1, 2, 5, 4 (PTS 0, 1536, 512, 1024, 2560, 2048, divided by 512). Their DTS begins two frames (1,024 ticks) below zero, so PTS and DTS differ in every packet, not only in the B-frames.
(b) Quality. The I-frames have the highest mean PSNR and the B-frames the lowest in both files. These absolute values belong to this build of x264 and this clip. With five times as many I-frames, -g 10 had a lower mean PSNR than -g 50 at the same CRF for every frame type. H6 held: raising CRF from 18 to 23 to 28 shrank the file from 426,992 to 278,761 to 179,675 bytes and lowered mean luma PSNR from 50.44 to 47.77 to 44.32 dB.
(c) Seeking. The batch plan’s reading of H2, , held at T = 4.6 and 7 and failed at T = 4.61 and 10.13. There both accurate input seeking and output seeking printed 116 and 254, where the plan’s reading gives 115 and 253. The documentation-literal reading, , held at all four T in both conditions. It was in the plan before the pilot, as stated above. Frame 115 is shown at 4.60 s, before 4.61 s, so it falls in the discarded segment. H3 held at all four T: with -noaccurate_seek, and in the stream copy read with its edit list ignored, the first frame was the preceding keyframe (100, 100, 150, 250), and the copied file held 200, 200, 150 and 50 frames, every frame from that keyframe to the end. In our reading, a stream copy can only start at a keyframe, since no packet is decoded, so a copy always carries the pre-roll back to it.
The default decode of the copied files had no hypothesis. On the MP4 demuxer’s edit lists, the demuxers documentation says only this: by default the demuxer modifies the stream index to reflect the timeline that an edit list describes, ignore_editlist makes it ignore edit list atoms, and advanced_editlist (on by default) controls whether the whole timeline or only its start is reflected. In the printed output, the default decodes started at 115, 116, 175 and 254, the same frames as accurate seeking, and returned 185, 184, 125 and 46 frames. The pre-roll frames were in the file, as the decode with the edit list ignored shows, but the default decode did not return them.
(d) Frame-rate conversion. H4 held for both rounding modes. Each kept 120 frames, and each matched the “last” closed form at all 120 ticks and the “first” form at none. With the default near, a tick shows the last source frame whose rounded timestamp is not later than the tick: frames 1, 3, 6, 8, … rather than 0, 2, 4, 7, …. With down, each of the first eight kept frames is one frame later; in our arithmetic on the two “last” closed forms, which the output matched at all 120 ticks, the same holds at every tick. In our reading, for datasets whose annotations refer to source frame numbers, the rounding mode therefore decides which annotated frame each extracted image is.
(e) Pixel formats. H5 held in all four parts. The luma planes of yuv420p and yuv444p were identical. On the stripe patch the 4:2:0 chroma differed from the 4:4:4 chroma by 70.000 code values on average, the value our derivation gives for any filter with non-negative weights under the matrix it assumed, and on the flat patch by 0.000. Through RGB, the stripes came back with a mean error of 88.667 levels from 4:2:0 and 0.167 from 4:4:4. In 4:2:0 every 2 × 2 block shares one chroma pair, so one-pixel colour detail is replaced by its average while brightness detail is untouched. The flat patch’s error of 0.333 is the same in both formats; in our reading it is the rounding of the conversion itself. Also in our reading, colour features computed at full resolution from decoded yuv420p video, such as colour edges or small coloured markers, see averaged chroma at every edge.
Ecosystem and Integrations
FFmpeg’s libraries are the decoding layer of other tools. The OpenCV article reports that OpenCV’s wheels include FFmpeg under the LGPL 2.1 and that cv2.getBuildInformation() names it among the video back ends of the wheel described there. Block Matching describes FFmpeg’s motion-estimation code and its minterpolate filter, which converts frame rates by motion-compensated interpolation instead of dropping and repeating frames as fps does. A dataset paper can record FFmpeg settings as part of its protocol: as the HMDB51 article reports, that dataset’s clips were resized, converted to 30 fps and DivX-compressed through ffmpeg. In a live system FFmpeg can be the ingest stage, reading an RTSP camera through its RTSP demuxer. The latency and buffering of such a pipeline are the subject of Real-Time Vision Pipelines, and decoding many streams at once is a concern of video analytics pipelines. Video models that sample clips, described in Video Understanding, depend on which frames such a pipeline returns.
Alternatives
- GStreamer, described in Real-Time Vision Pipelines for its queues, clocks and latency model, is an open-source multimedia framework whose pipelines are made of elements, some of which start their own streaming threads, with the
queueelement as the explicit thread boundary. - OpenCV’s
videoiomodule reads video files and camera streams and writes video, through the back ends a build was compiled with. For the wheel the OpenCV article describes, FFmpeg is among them. - Python bindings and dataset loaders that wrap FFmpeg’s libraries also exist. None was read for this article, and nothing is claimed about their behaviour.
Versions and Compatibility
The example ran with ffmpeg and ffprobe 9.0.2, a Homebrew build (libavutil 61.1.102, libavcodec 63.1.102, libavformat 63.1.102, libavfilter 12.1.102, libswscale 10.1.102), with NumPy 2.3.5. The documentation cited is the repository’s doc/ folder at tag n9.0.2 (commit 946fcce). The repository’s tag list, read in November 2026, also has a tag n9.1-dev. Nothing is claimed here about versions not read. Options and ffprobe field names change between releases, and the documentation is generated per version, so scripts should state the version they were tested with. An encoder is available only if the build includes it. libx264, for instance, needs --enable-libx264, and ffmpeg -version prints the configure line.
Licence. FFmpeg’s LICENSE.md states the following:
- Most of FFmpeg is under the LGPL 2.1 or later, with some files under MIT/X11/BSD-style licences, and in combination the LGPL 2.1+ applies.
- Optional parts are under the GPL 2 or later and are enabled only with
--enable-gpl, which turns the whole build into GPL 2+. They include some x86 optimizations, test tools and a list of filters. - External libraries can change the licence of a binary. libx264 and libx265 are GPL 2 libraries that require
--enable-gpl.--enable-version3upgrades to version 3 of the (L)GPL.--enable-nonfreemakes a binary unredistributable.
The build used here is configured with --enable-gpl and --enable-version3, so by LICENSE.md it is under the GPL version 3, and its ffmpeg -L prints the GPL version 3 or later. The same build also passes --enable-openssl, and LICENSE.md lists OpenSSL among the libraries whose licences are incompatible with the GPL v2 and v3; what that means for redistributing this binary is not settled here. The licence of a binary is therefore a property of how it was built, and the configure line in ffmpeg -version shows it. This article only calls the programs; no FFmpeg code is reproduced.
Limitations
- Seeking depends on the mode. Input seeking with stream copy or
-noaccurate_seekstarts at a keyframe. Accurate seeking starts at the first frame at or after the position, not at the nearest frame. And an MP4 written by a stream copy can hold more frames than a default decode returns. - Frame-rate conversion is a sampling rule.
fpsdrops or repeats whole frames according to its rounding mode. It does not interpolate, andround=nearandround=downkeep different frames. - Pixel-format conversion can lose chroma silently.
yuv420paverages colour over 2 × 2 blocks. RGB output decoded from it cannot recover the lost colour detail, and nothing in the frame says it was lost. - Encoder results are build-specific. Frame-type placement, quality at a given CRF and file sizes come from the encoder library and its version, here x264 in this build, not from FFmpeg’s documentation.
- Defaults differ by layer. The generic
-gdefault of 12 does not describe what libx264 does when-gis unset, and the example fixed-threads 1for reproducibility; other thread counts were not tested. Scripts that need exact frames should state the options they rely on. - The command line is option-order sensitive. An option placed before
-iapplies to the input and after it to the output, as-ssshows. A misplaced option changes the result without an error.
Related
- OpenCV
The open-source C++ library of computer vision and image processing functions, with Python bindings that work on NumPy arrays; its modules, conventions, versions from 1.0 to 5.0, contrib modules and licence.
- Block Matching
Estimating motion by dividing a frame into blocks and finding, for each block, the displacement whose block in a reference frame matches it best, by exhaustive search or by fast search patterns, as video encoders do.
- Color Spaces
A color space assigns numbers to colors; choosing and converting between RGB, CIE XYZ, CIELAB, YCbCr, and HSV correctly matters for both image processing and learning.
- Peak Signal-to-Noise Ratio
The mean squared error between an image and its reference expressed in decibels relative to the largest possible pixel value; how the peak value, colour channels, border cropping and averaging over a test set change the number, and why a higher PSNR does not always mean a better-looking image.
- Real-Time Vision Pipelines
How a vision system turns a stream of camera frames into timely decisions, covering frame budgets, latency against throughput, pipelining and threads, queues and dropped frames, timestamps and clock synchronization, and how end-to-end latency is measured.
- Video Understanding
Recognizing, locating and describing what happens in video, and how it differs from understanding single images; the overview of the tasks, the model eras from space-time features to video transformers, the datasets and how results are measured.
References
- FFmpeg developers. ffmpeg Documentation, doc/ffmpeg.texi (GitHub repository FFmpeg/FFmpeg at tag n9.0.2, commit 946fcce; read November 2026).
- FFmpeg developers. FFmpeg Filters Documentation, doc/filters.texi (tag n9.0.2, commit 946fcce; read November 2026).
- FFmpeg developers. ffprobe Documentation, doc/ffprobe.texi (tag n9.0.2, commit 946fcce; read November 2026).
- FFmpeg developers. FFmpeg Codecs Documentation, doc/codecs.texi (tag n9.0.2, commit 946fcce; read November 2026).
- FFmpeg developers. FFmpeg Codecs Documentation: encoders, libx264 section, doc/encoders.texi (tag n9.0.2, commit 946fcce; read November 2026).
- FFmpeg developers. FFmpeg Formats Documentation: demuxers (mov, mp4 options) and format options, doc/demuxers.texi and doc/formats.texi (tag n9.0.2, commit 946fcce; read November 2026).
- FFmpeg developers. FFmpeg Protocols Documentation (rtsp) and FFmpeg Utilities Documentation (time duration syntax), doc/protocols.texi and doc/utils.texi (tag n9.0.2, commit 946fcce; read November 2026).
- FFmpeg developers. FFmpeg multithreading methods, doc/multithreading.txt (tag n9.0.2, commit 946fcce; read November 2026).
- FFmpeg developers. LICENSE.md (tag n9.0.2, commit 946fcce; read November 2026).