Vision Systems & Engineering
LiteRT (TensorFlow Lite)
Google's on-device runtime for machine-learning models, formerly TensorFlow Lite, covering the .tflite FlatBuffer format, the Interpreter and CompiledModel APIs, CPU, GPU and NPU acceleration, converters and the arena planner that packs a network's tensors into one buffer, with a NumPy port of that planner run on a small CNN.
intermediate
LiteRT is Google’s runtime for running machine-learning models on phones, embedded boards, desktops and browsers. Its repository describes it as “LiteRT (short for Lite Runtime), formerly known as TensorFlow Lite” (litert/README.md at v2.3.2). A trained model is converted into a .tflite file, a FlatBuffer that holds the graph and its constant weights, and the runtime executes that file on a CPU, a GPU or a neural processing unit. The TensorFlow article tells the history of TensorFlow Lite, its converter and the renaming; edge inference covers the hardware it runs on and how on-device speed is measured. This article covers the runtime itself: the file format, the two programming interfaces, the accelerators and tools as the repository describes them, and how the interpreter plans memory. The worked example ports that memory planner to NumPy and uses its plan to run a small convolutional network from a single byte buffer. Versions are as of November 2026: LiteRT v2.3.2, released 26 October 2026, with NumPy 2.3.5 for the example.
Purpose
A model leaving a training framework is a program written for a workstation: Python code, a framework’s operator library, dynamic memory allocation. In our framing, a phone application needs something else: a self-contained file, a small runtime library, predictable memory use and access to whatever accelerator the device has. LiteRT is Google’s answer. The README at v2.3.2 describes it as continuing “the legacy of TensorFlow Lite” and lists, for its second major version, a Compiled Model API with automatic accelerator selection, unified access to NPUs from several chipset vendors, and GPU acceleration through a component called ML Drift. Models come from PyTorch, TensorFlow and JAX.
For vision this is the path that takes a classifier, detector or segmenter trained on a server into a camera application. Published latency figures can depend on it: the MobileNet article reports MobileNetV3 latencies measured on a single large core of Pixel phones with TFLite.
Core Capabilities
The .tflite file
The file format is defined by a FlatBuffers schema, schema.fbs, which in v2.3.2 sits under tflite/converter/schema/. The schema declares the file identifier "TFL3" (“This corresponds to the version”) and the extension tflite. The root table is Model:
Modelholds a schemaversion, a list ofoperator_codes, a list ofsubgraphs(the first is the main model), a list ofbuffers, metadata andsignature_defs. Buffer 0 must be an empty sentinel, so that a tensor with no data can point to it.SubGraphholds thetensors, the indices of itsinputsandoutputs, and itsoperators, which the schema stores “in execution order”. In our reading, the order of execution is therefore fixed when the file is written, which is what lets the runtime plan memory ahead of time.Tensorhas ashape, atype(FLOAT32,INT8,FLOAT16,INT4,BFLOAT16and others, up to the 8-bit floating-point typesFLOAT8_E4M3FNandFLOAT8_E5M2), abufferindex and optional quantization and sparsity parameters. For builtin operators the shape is[batch, height, width, channels], which the schema calls “Tensorflow’s NHWC”. Intermediate tensors point to buffer 0; constants such as weights point to a buffer holding their bytes, assumed little-endian and stored row-major.QuantizationParametersgivesscaleandzero_point, with a float value for scale and zero point , and an optionalquantized_dimensionfor per-channel scales. The quantization article covers what these numbers mean and how integer-only arithmetic uses them.Operatorrefers to its operator code by index, lists input and output tensor indices (−1 marks an optional input that is absent), and carries options such asConv2DOptions(paddingSAMEorVALID, strides, dilation and afused_activation_functionsuch asRELUorRELU6).
The schema’s revision history records how the format grew: subgraphs were added in version 1, buffers moved from tensors to the model in version 3, and later revisions added constant buffers stored outside the FlatBuffer, for models larger than 2 GB, and external buffer tables referring to separate files.
The Interpreter
The original C++ entry point is tflite::Interpreter. Its header at v2.3.2 shows the sequence: build a FlatBufferModel from a file, create an interpreter with an InterpreterBuilder and an operator resolver (BuiltinOpResolver for the builtin operators), call AllocateTensors(), fill the input tensors, call Invoke(), which runs “the whole graph in dependency order”, and read the outputs. AllocateTensors() “must be called” before inference and again whenever an input tensor is resized; it fails if the model contains an operator the resolver does not support, other than those a delegate has taken over. A delegate takes part of the graph and runs it elsewhere; ModifyGraphWithDelegate installs one.
AllocateTensors() is where memory planning happens. Each tensor has an allocation type (common.h): kTfLiteMmapRo for read-only data such as weights, kTfLiteArenaRw for the arena that holds inputs, outputs and intermediates, kTfLiteArenaRwPersistent for a second arena whose contents persist across invocations, kTfLiteDynamic for tensors allocated during evaluation, and a few others. In subgraph.cc, the first allocation creates an ArenaPlanner, calls its PlanAllocations() once and then ExecuteAllocations() over the nodes prepared so far; preparation stops at a node whose outputs are dynamic, since their sizes are unknown until it runs.
The arena planner
arena_planner.h at v2.3.2 states the idea: before the model runs, the planner determines when each tensor must be allocated and deallocated, preallocates the memory, and assigns portions of one buffer so that a tensor allocated after another has been released can reuse its bytes. Inference optimization places this among memory-planning methods in general; the details below come from arena_planner.cc and simple_memory_arena.cc at v2.3.2.
Lifetimes. PlanAllocations walks the operators in execution order and gives each arena tensor an interval of nodes: it is allocated at the node that writes it and deallocated at the node whose read brings its reference count to zero, its last reader. Graph outputs get one extra reference, so they are never deallocated; graph inputs also get one, and are allocated at node 0. Constants are never allocated in the arena. The intervals are inclusive, so the input and output of the same node always count as live together. With preserve_all_tensors set, nothing is deallocated, so that intermediate tensors can be read after a run.
In-place operators. Before lifetimes are fixed, IdentifyInPlaceTensors lets an operator’s first output reuse one of its input buffers. A kernel’s registration declares which inputs it may share (kTfLiteInplaceOpInput0Shared and so on, in common.h). Sharing is refused if the input is a graph input or the output a graph output, if their allocation types are incompatible, or if preserve_all_tensors is set; for an operator that changes the data, it is also refused if the two tensors differ in size, if the size is at most 4 bytes, or if anything else still reads the input. In v2.3.2’s kernels, add.cc registers ADD with kTfLiteInplaceOpInput0Shared | kTfLiteInplaceOpInput1Shared, while conv.cc, pooling.cc, reduce.cc and fully_connected.cc set no such flag.
Order. CreateTensorAllocationVector sorts the tensors to place. Tensors that live for the whole inference (allocated at node 0 and never deallocated) come first, by index. The rest follow from largest to smallest, and tensors of equal size in the order of their allocation node.
Placement. SimpleMemoryArena::Allocate places each tensor in turn. It keeps the tensors already placed sorted by offset and walks through them, skipping any whose interval does not intersect the new tensor’s. At each one that does intersect, it looks at the gap between the end of the previous intersecting allocation and this one’s start. If the new tensor fits there, after rounding the start up to the tensor alignment, and the gap is the tightest seen so far, it becomes the best candidate. If no gap fits, the tensor goes at the furthest end of the intersecting allocations, rounded up to the alignment. The arena’s size, its high-water mark, is the largest end offset of any tensor. Zero-size tensors get offset 0. One detail of the code: the fit is tested against the aligned start, but the gap stored for later comparison is measured from the unaligned end, alloc.offset - current_offset. The example ports this exactly as written.
Alignment. Two constants are involved. kDefaultArenaAlignment = 64 (arena_planner.h) aligns the arena buffers themselves. The tensor alignment passed to Allocate is a separate constructor parameter, which subgraph.cc sets to kDefaultTensorAlignment, defined as 64 in util.h. Allocate requires the tensor alignment to be no larger than the arena’s.
Kernels can also ask for temporaries, scratch tensors that live only during one node: conv.cc allocates an im2col buffer and transposed weights “if necessary”, depending on the kernel variant. ExecuteAllocations gives such tensors the one-node interval of their node.
Edge Inference describes a related design for microcontrollers: TensorFlow Lite Micro allocates every tensor from a single arena that the application provides, because dynamic allocation cannot be assumed there.
The CompiledModel API
LiteRT’s second major version adds CompiledModel, which tflite/README.md calls the “modern” API, next to the “legacy” Interpreter API. In the C header at v2.3.2, a compiled model is created from a loaded model and optional compilation options; internally it “instantiates runtime and applies Delegates mapped to the compilation options”. The application queries the buffer requirements of each input and output, creates tensor buffers that meet them, fills the inputs and runs the model synchronously, or asynchronously where the operators, the chosen accelerator and the device allow it, in which case only some parts of the model may run asynchronously. Per tflite/README.md, the interpreter runtime in the tflite/ directory “backs the LiteRT compiled model runtime”, so, in our reading, the planner above still sits underneath.
Accelerators
The README’s tool diagram ends in three back ends: XNNPack on CPU, ML Drift on GPU, and supported TPUs and NPUs. Its platform table, at v2.3.2, lists:
| Platform | GPU APIs | NPUs and other accelerators |
|---|---|---|
| Android | OpenCL, OpenGL | Broadcom, Google Tensor, Intel, MediaTek, Qualcomm; S.LSI marked “coming soon” |
| iOS | Metal | ANE, “coming soon” |
| Linux | WebGPU | Broadcom, Intel |
| macOS | WebGPU, Metal | ANE, “coming soon” |
| Windows | WebGPU | Intel |
| Web | WebGPU | WebNN, “coming soon” |
| IoT | WebGPU | Raspberry Pi, “coming soon” |
Every platform has CPU support. The XNNPACK document in the tflite/ tree describes how XNNPACK, “a highly optimized library of neural network inference operators” for ARM, x86 and WebAssembly, attaches to the interpreter as a delegate. It supports a limited set of operators, and unsupported operators fall back to the default kernels, so a model mixing the two still runs. It enables half-precision inference automatically on hardware with native FP16 arithmetic when the model’s reduced_precision_support metadata allows it. Quantized inference through XNNPACK is off by default and needs extra build flags. Which operators an accelerator supports, and what falling back costs, is covered in general in edge AI accelerators.
The release notes date the accelerator changes. Version 2.2.0 (13 August 2026) added an experimental CPU accelerator called YNNPACK, native support for the two 8-bit floating-point element types, and FP16 support in XNNPACK for several convolution and matrix operators. Version 2.3.0 (7 October 2026) released ML Drift as a standalone open-source GPU accelerator and shipped the GPU and vendor NPU libraries as separate Android packages. Version 2.3.2 lists minor bug fixes.
Minimal Example
LiteRT is not installed on the machine used for this article, and nothing could be installed, so the example does not call LiteRT. Instead it ports the arena planner, PlanAllocations (with its in-place step), CreateTensorAllocationVector, CalculateAllocations and SimpleMemoryArena::Allocate, from v2.3.2’s C++ into Python, and uses the plan to run a network. The outcomes test our port of the planner, not LiteRT. The convolution, pooling, mean and fully connected kernels are our own NumPy code, not LiteRT’s kernels, so the example says nothing about LiteRT’s arithmetic.
Set-up. The image is a rendered 48 × 48 RGB picture: a red disk of radius 9 centred at row 16, column 16 and a blue 12 × 12 square at rows 28–39, columns 26–37, on grey. Divided by 255, it is the float32 input tensor of shape [1, 48, 48, 3], in the schema’s NHWC order. The network is laid out the way a SubGraph stores one, as a tensor list and an operator list in execution order: three 3 × 3 convolutions with 8 channels (ReLU fused into the first two), an ADD with fused ReLU that brings back the first convolution’s output as a residual skip, 2 × 2 max pooling, a 3 × 3 convolution to 12 channels, a MEAN over height and width, and a fully connected layer to 5 logits. The weights are untrained, drawn from a seeded generator; they are constants, outside the arena, as kTfLiteMmapRo tensors are. ADD is marked as sharing inputs 0 and 1, the others as not sharing, following the kernel files above. Both alignments are 64.
Lifetimes of inputs and outputs. The batch plan for this article described graph inputs and outputs as living throughout the inference. arena_planner.cc at v2.3.2 does something narrower: inputs are allocated at node 0 and never deallocated, while outputs are never deallocated but are allocated at the node that writes them. The port follows the source.
What is not modelled. Taken from the source and not run: kernel temporaries such as conv.cc’s im2col buffer, the persistent arena (this graph has no persistent tensors), dynamic tensors and incremental planning, delegates, variable tensors, optional inputs and preserve_all_tensors. The port makes one ExecuteAllocations call over all nodes, which in our reading is what subgraph.cc does for a graph without dynamic tensors.
Measures. The script prints a table of every arena tensor with its size, its interval of nodes, its offset and, for a shared tensor, the tensor whose buffer it uses, plus the order of placement. It compares the arena’s high-water mark with two references. The naive sum gives every non-constant tensor its own buffer, each rounded up to 64 bytes. The lower bound is the largest total of 64-byte-rounded buffers that are live at the same node; in our reasoning, no placement that keeps every buffer 64-byte aligned can use less, except by the rounding of the last buffer’s end, which is zero when every buffer live at that node is a multiple of 64 bytes. An overlap check counts pairs of buffers whose intervals intersect and how many of those share any byte. The network then runs twice, once with each tensor as a NumPy view into one 64-byte-aligned uint8 arena at its planned offset and once with plain separate arrays, and the script compares every tensor bit for bit as it is written, and the final logits. A negative control puts every tensor at offset 0, an allocator that ignores lifetimes entirely, and must corrupt the result. A data check confirms the image: its colours, the disk’s pixel count against the number of integer points within radius 9 computed from a formula independent of the mask, the square’s 144 pixels, and no overlap between them.
Checks. Before any script ran, we traced the port’s rules through this graph by hand. That trace is the test (C-trace). It predicts that ADD’s output shares the third convolution’s buffer, the order and offset of every tensor, and an arena of 248,832 bytes, which equals the lower bound. The batch plan did not predict whether best fit would reach the lower bound; our hand trace predicted that it would here. The hypotheses: H1, lower bound ≤ arena < naive sum; H2, no two buffers with intersecting intervals share a byte; H3, every tensor is bitwise equal to the plain run when written, and so are the logits; C-neg, the control’s logits differ and its overlap check finds conflicts. Every check is an exact integer or bitwise comparison. A planning script, run before the pilot and independent of the port, recomputed the naive sum and the lower bound, which the hand trace had already fixed, computed the integer-point count, and repeated the plain run. The disk-count bar of 253 pixels was written down after that script printed it. Before its first execution, two format-only edits were made to the pilot script: a comment, and a dash printed in the last-node column for a shared tensor. The script below is the pilot script unchanged. The run takes well under a second.
# LiteRT's arena planner, ported to NumPy, planning the memory of a small CNN.
#
# PlanAllocations, IdentifyInPlaceTensors, CreateTensorAllocationVector and
# CalculateAllocations are ported from tflite/arena_planner.cc, and Allocate from
# tflite/simple_memory_arena.cc, google-ai-edge/LiteRT at tag v2.3.2:
# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
# Licensed under the Apache License, Version 2.0.
# Modified (ported from C++ to Python, reduced to the parts this graph uses).
# Licence: https://github.com/google-ai-edge/LiteRT/blob/v2.3.2/LICENSE
import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
os.environ[v] = "1"
import math
import numpy as np
END = 2**31 - 1 # kNodeNotAssigned: never deallocated
TENSOR_ALIGN = ARENA_ALIGN = 64 # kDefaultTensorAlignment (util.h), kDefaultArenaAlignment
# ---- the image and the graph, laid out like a .tflite SubGraph -------------
img = np.full((48, 48, 3), 128, np.uint8)
r, c = np.mgrid[0:48, 0:48]
img[(r - 16) ** 2 + (c - 16) ** 2 <= 81] = (220, 60, 40) # disk, radius 9
img[28:40, 26:38] = (40, 90, 210) # 12 x 12 square
rng = np.random.default_rng(0) # untrained weights
W = lambda *s: (rng.standard_normal(s) * np.sqrt(2.0 / math.prod(s[1:]))).astype(np.float32)
B = lambda k: (rng.standard_normal(k) * 0.1).astype(np.float32)
const = {"w0": W(8, 3, 3, 3), "b0": B(8), "w1": W(8, 3, 3, 8), "b1": B(8),
"w2": W(8, 3, 3, 8), "b2": B(8), "w5": W(12, 3, 3, 8), "b5": B(12),
"w7": W(5, 12), "b7": B(5), "axes": np.array([1, 2], np.int32)}
act = {"image": (1, 48, 48, 3), "a0": (1, 48, 48, 8), "a1": (1, 48, 48, 8),
"a2": (1, 48, 48, 8), "a3": (1, 48, 48, 8), "a4": (1, 24, 24, 8),
"a5": (1, 24, 24, 12), "a6": (1, 12), "logits": (1, 5)}
names = list(act) + list(const) # tensor index = position
T = {n: i for i, n in enumerate(names)}
nbytes = [4 * math.prod(act[n]) if n in act else const[n].nbytes for n in names]
is_const = [n in const for n in names] # buffer >= 1: kTfLiteMmapRo
ops = [ # (opcode, inputs, output, fused ReLU); operators in execution order
("CONV_2D", ["image", "w0", "b0"], "a0", True),
("CONV_2D", ["a0", "w1", "b1"], "a1", True),
("CONV_2D", ["a1", "w2", "b2"], "a2", False),
("ADD", ["a2", "a0"], "a3", True),
("MAX_POOL_2D", ["a3"], "a4", False),
("CONV_2D", ["a4", "w5", "b5"], "a5", True),
("MEAN", ["a5", "axes"], "a6", False),
("FULLY_CONNECTED", ["a6", "w7", "b7"], "logits", False)]
ops = [(o, [T[x] for x in i], T[y], f) for o, i, y, f in ops]
graph_in, graph_out = [T["image"]], [T["logits"]]
INPLACE = {"ADD": (0, 1)} # add.cc: Input0Shared | Input1Shared; others None
# ---- PlanAllocations (arena_planner.cc) -------------------------------------
def plan_allocations(preserve_all_tensors=False):
n_t = len(names)
alloc, dealloc, refc = [END] * n_t, [END] * n_t, [0] * n_t
nodes_to_tensors = [set() for _ in ops]
def allocate(node, t):
if alloc[t] == END:
alloc[t] = node
def deallocate(node, t):
if alloc[t] != END: # constants are never allocated
dealloc[t] = node
for t in graph_out:
refc[t] += 1
for t in graph_in:
refc[t] += 1
allocate(0, t)
nodes_to_tensors[0].add(t)
refcounts = list(refc) # copy before sharing
for _, ins, _, _ in ops:
for t in ins:
refc[t] += 1
# IdentifyInPlaceTensors / InputTensorCanBeShared (data-modifying ops only)
shared = {}
find = lambda t: shared.get(t, t)
for op, ins, out, _ in ops:
for i in INPLACE.get(op, ()):
t = ins[i]
ok = (nbytes[t] == nbytes[out] and nbytes[t] > 4 and refc[t] <= 1
and t not in graph_in and out not in graph_out
and not is_const[t] and not preserve_all_tensors)
if ok: # first shareable input wins
root = find(t)
if refc[root] <= 1:
shared[out] = root
break
for _, ins, _, _ in ops:
for t in ins:
refcounts[find(t)] += 1
for node, (_, ins, out, _) in enumerate(ops):
nodes_to_tensors[node].add(out)
allocate(node, out)
if not preserve_all_tensors:
for t in ins:
t = find(t)
refcounts[t] -= 1
if refcounts[t] == 0:
deallocate(node, t)
return alloc, dealloc, shared, nodes_to_tensors
# ---- SimpleMemoryArena::Allocate (simple_memory_arena.cc) -------------------
def align_to(a, x):
return x if x % a == 0 else x + a - x % a
def arena_allocate(active, alignment, size, tensor, first, last):
if size == 0:
return 0
best_offset = best_fit = None # kOffsetNotAssigned
current = 0
for a in active: # sorted by offset
if a["last"] < first or a["first"] > last:
continue # lifetimes do not intersect
aligned_current = align_to(alignment, current)
if aligned_current + size <= a["offset"] and (
best_fit is None or a["offset"] - aligned_current < best_fit):
best_offset = aligned_current
best_fit = a["offset"] - current # stored unaligned, as in v2.3.2
current = max(current, a["offset"] + a["size"])
if best_fit == 0:
break
if best_offset is None:
best_offset = align_to(alignment, current)
new = {"offset": best_offset, "size": size, "tensor": tensor,
"first": first, "last": last}
k = sum(a["offset"] <= best_offset for a in active) # std::upper_bound
active.insert(k, new)
return best_offset
# ---- CreateTensorAllocationVector + CalculateAllocations --------------------
def calculate_allocations(alloc, dealloc, shared, nodes_to_tensors):
todo = [t for s in nodes_to_tensors for t in s]
whole = lambda t: alloc[t] == 0 and dealloc[t] == END
todo.sort(key=lambda t: (0, t) if whole(t) else (1, -nbytes[t], alloc[t]))
offsets, active, order = {}, [], []
for t in todo:
order.append(t)
if t in shared:
continue # shares its input's buffer
offsets[t] = arena_allocate(active, TENSOR_ALIGN, nbytes[t], t,
alloc[t], dealloc[t])
for t, root in shared.items():
offsets[t] = offsets[root]
high_water = max(a["offset"] + a["size"] for a in active)
return offsets, high_water, order
# ---- kernels (ours, NumPy), plain run and arena run -------------------------
def conv(x, w, b, relu):
n, h, wd, ci = x.shape
p = np.pad(x, ((0, 0), (1, 1), (1, 1), (0, 0))) # SAME, 3 x 3
cols = np.stack([p[:, i:i + h, j:j + wd, :] for i in range(3) for j in range(3)], 3)
y = (cols.reshape(n * h * wd, 9 * ci) @ w.reshape(len(w), 9 * ci).T + b)
y = y.reshape(n, h, wd, len(w))
return np.maximum(y, 0) if relu else y
def kernel(op, xs, relu):
if op == "CONV_2D":
return conv(xs[0], xs[1], xs[2], relu)
if op == "ADD":
y = xs[0] + xs[1]
return np.maximum(y, 0) if relu else y
if op == "MAX_POOL_2D":
n, h, w, ch = xs[0].shape
return xs[0].reshape(n, h // 2, 2, w // 2, 2, ch).max(axis=(2, 4))
if op == "MEAN":
return xs[0].mean(axis=tuple(xs[1]))
return xs[0] @ xs[1].T + xs[2] # FULLY_CONNECTED
x_in = img[None].astype(np.float32) / np.float32(255)
def run_plain():
v = {T["image"]: x_in.copy()}
v.update({T[n]: a for n, a in const.items()})
for op, ins, out, relu in ops:
v[out] = kernel(op, [v[t] for t in ins], relu)
return v
def run_arena(offsets, size, ref):
raw = np.zeros(size + ARENA_ALIGN, np.uint8)
start = (-raw.ctypes.data) % ARENA_ALIGN
arena = raw[start:start + size] # 64-byte aligned
view = lambda t: arena[offsets[t]:offsets[t] + nbytes[t]].view(np.float32).reshape(act[names[t]])
v = {T[n]: a for n, a in const.items()}
v.update({t: view(t) for t in offsets})
v[T["image"]][...] = x_in
equal_at_write = [np.array_equal(v[T["image"]], ref[T["image"]])]
for op, ins, out, relu in ops:
y = kernel(op, [v[t] for t in ins], relu)
v[out][...] = y
equal_at_write.append(np.array_equal(v[out], ref[out]))
return v[T["logits"]].copy(), equal_at_write
def conflicts(offsets, first, last, owners):
pairs = bad = 0
for i, s in enumerate(owners):
for t in owners[i + 1:]:
if first[s] <= last[t] and first[t] <= last[s]:
pairs += 1
bad += (offsets[s] < offsets[t] + nbytes[t] and
offsets[t] < offsets[s] + nbytes[s])
return pairs, bad
# ---- run -------------------------------------------------------------------
print("data: shape", img.shape, img.dtype, "| colours", len(np.unique(img.reshape(-1, 3), axis=0)))
disk = (r - 16) ** 2 + (c - 16) ** 2 <= 81
square = np.zeros_like(disk); square[28:40, 26:38] = True
lattice = sum(2 * math.isqrt(81 - x * x) + 1 for x in range(-9, 10))
print(f"data: disk {int(disk.sum())} px (lattice count {lattice}), square {int(square.sum())} px "
f"(144), overlap {int((disk & square).sum())}")
alloc, dealloc, shared, n2t = plan_allocations()
offsets, high_water, order = calculate_allocations(alloc, dealloc, shared, n2t)
owners = [t for t in offsets if t not in shared]
fmt = lambda n: "end" if n == END else str(n)
al = lambda b: align_to(64, b)
print("\ntensor bytes aligned first last offset shares")
for t in sorted(offsets):
root = shared.get(t)
last = "-" if root is not None else fmt(dealloc[t]) # an alias frees nothing
print(f"{names[t]:7s}{nbytes[t]:6d} {al(nbytes[t]):8d} {alloc[t]:6d} {last:>5s}"
f" {offsets[t]:7d} {names[root] if root is not None else '-'}")
print("allocation order:", " ".join(names[t] + ("(shared)" if t in shared else "") for t in order))
naive = sum(al(nbytes[t]) for t in offsets)
lower = max(sum(al(nbytes[t]) for t in owners if alloc[t] <= n <= dealloc[t]) for n in range(len(ops)))
print(f"\nlower bound {lower} arena {high_water} naive {naive} "
f"H1 lower<=arena<naive: {lower <= high_water < naive}")
pairs, bad = conflicts(offsets, alloc, dealloc, owners)
print(f"overlap check: {pairs} pairs with intersecting lifetimes, {bad} share memory")
ref = run_plain()
logits, eq = run_arena(offsets, high_water, ref)
print("\nplain logits:", " ".join(f"{v:.8e}" for v in ref[T["logits"]].ravel()))
print("arena logits:", " ".join(f"{v:.8e}" for v in logits.ravel()))
print(f"equal at write: {sum(eq)} of {len(eq)}; max|diff| {np.abs(logits - ref[T['logits']]).max():.3e};"
f" bitwise equal: {np.array_equal(logits, ref[T['logits']])}")
zero = {t: 0 for t in offsets} # ignores lifetimes
logits0, eq0 = run_arena(zero, max(nbytes[t] for t in offsets), ref)
pairs0, bad0 = conflicts(zero, alloc, dealloc, owners)
print(f"\ncontrol (all at offset 0): equal at write {sum(eq0)} of {len(eq0)}; max|diff| "
f"{np.abs(logits0 - ref[T['logits']]).max():.6e}; bitwise equal: "
f"{np.array_equal(logits0, ref[T['logits']])}; overlap check: {bad0} of {pairs0} share memory")
Output (NumPy 2.3.5, Python 3, one BLAS thread; two runs gave identical output):
data: shape (48, 48, 3) uint8 | colours 3
data: disk 253 px (lattice count 253), square 144 px (144), overlap 0
tensor bytes aligned first last offset shares
image 27648 27648 0 end 0 -
a0 73728 73728 0 3 27648 -
a1 73728 73728 1 2 101376 -
a2 73728 73728 2 4 175104 -
a3 73728 73728 3 - 175104 a2
a4 18432 18432 4 5 55296 -
a5 27648 27648 5 6 27648 -
a6 48 64 6 7 55296 -
logits 20 64 7 end 27648 -
allocation order: image a0 a1 a2 a3(shared) a5 a4 a6 logits
lower bound 248832 arena 248832 naive 368768 H1 lower<=arena<naive: True
overlap check: 14 pairs with intersecting lifetimes, 0 share memory
plain logits: -7.22289681e-01 -2.46019006e-01 -3.74724902e-02 -2.81569719e-01 -2.57816732e-01
arena logits: -7.22289681e-01 -2.46019006e-01 -3.74724902e-02 -2.81569719e-01 -2.57816732e-01
equal at write: 9 of 9; max|diff| 0.000e+00; bitwise equal: True
control (all at offset 0): equal at write 4 of 9; max|diff| 8.991737e-01; bitwise equal: False; overlap check: 14 of 14 share memory
Data. The data check held: three colours, a disk of 253 pixels matching the integer-point count, a 144-pixel square and no overlap.
C-trace held. Every alias, interval, offset and the placement order match the hand trace, and the arena is 248,832 bytes. The table shows why. The input image, allocated at node 0 and never freed, is the only tensor in the first class and sits at offset 0. The three 73,728-byte feature maps a0, a1 and a2 are live together at node 2, so they get three separate slots after the image. ADD’s output a3 is shared with a2: the ADD kernel may share input 0, a2 has no other reader, and a3 is not a graph output, so the search stops at that first input. a2’s interval therefore runs to node 4, where pooling reads a3. The second input, a0, would not have qualified, since node 1 also reads it. The smaller tensors then reuse freed space. a5 (node 5–6) takes a0’s old slot at 27,648, because only the image is still live. a4 (node 4–5) must avoid a2 and a5 and fits in the gap between them at 55,296. The logits, a graph output allocated at node 7, are not in the first class, because the rule is “allocated at node 0 and never deallocated”. They go last by size, into the gap at 27,648 that a5 has vacated.
H1 held: 248,832 ≤ 248,832 < 368,768. The arena equals the lower bound, as the hand trace predicted: the peak is at node 2, where the image and three feature maps are live, and best fit packs those four end to end with nothing wasted. Part of the difference from the naive sum is a3, which has no buffer of its own.
H2 held: of the 14 buffer pairs whose intervals intersect, none share a byte.
H3 held: all 9 non-constant tensors matched the plain run bit for bit when written, and the logits are identical, with a maximum difference of 0. Bitwise equality is expected here, because both runs use the same kernels on the same values and differ only in where the results are stored.
C-neg held: with everything at offset 0, all 14 intersecting pairs share memory, only 4 of the 9 tensors are correct when written, and the logits differ by up to 0.899. In our reading, the first wrong tensor is ADD’s output, which reads a0 after the second and third convolutions have overwritten it.
Ecosystem and Integrations
The README’s tool chain at v2.3.2 runs from model to device. The LiteRT Torch converter turns PyTorch models into .tflite files, and its generative API produces .litertlm files for large language models. The AI Edge Quantizer quantizes converted models. The LiteRT runtime has C++, Kotlin and JavaScript bindings, and LiteRT-LM, for language models, feeds the same runtime in the README’s diagram. TensorFlow models go through the converter in tflite/converter/. Its README describes it as MLIR-based and reached through tf.lite.TFLiteConverter, with passes that import a TensorFlow graph, canonicalize it (including constant folding), legalize TensorFlow operations into TensorFlow Lite ones, leaving unmappable operations as TensorFlow operations, optimize, and export the FlatBuffer. The TensorFlow article runs that converter and describes the TF Select fallback for unmapped operations.
The README also lists LiteRT.js for browsers (WebGPU and WebAssembly), MediaPipe, which it describes as a framework for building machine-learning solutions for live streaming media, and XNNPACK. Ready-made .tflite and .litertlm models, image classifiers among them, are collected on a Hugging Face community page that the README links. The 2.3.0 release added a Swift package for iOS and macOS. For conversion between formats, ONNX describes tf2onnx, which also reads TensorFlow Lite models. Packaging the converted file with its preprocessing belongs to model deployment.
The tflite/ directory inside the LiteRT repository holds “the subset of TensorFlow Lite that LiteRT depends on”: the interpreter runtime, kernels and delegates; the converter; Java bindings; and the Python interpreter code packaged in the ai-edge-litert wheel. Standalone TensorFlow Lite deliverables that LiteRT does not need have been removed. tflite/README.md states that TensorFlow Lite packages and tensorflow/lite/ are in maintenance mode and receive only critical security and stability updates, and that new development happens in LiteRT.
Alternatives
- ExecuTorch, PyTorch’s on-device runtime, captures a model with
torch.exportand partitions it ahead of time into a.ptefile, as edge inference describes. - ONNX Runtime executes ONNX files, as the ONNX article describes.
- Core ML on Apple devices and OpenVINO on Intel hardware are vendor runtimes, named in model deployment and edge inference.
- TensorRT runs models on NVIDIA GPUs.
- TensorFlow Lite Micro targets microcontrollers with a single application-provided arena, as edge inference describes.
In our reading, the choice follows the training framework and the target hardware: LiteRT’s strengths are its converters from TensorFlow, PyTorch and JAX and its single file format for Android’s CPU, GPU and NPU paths.
Versions and Compatibility
- Releases. v2.2.0 on 13 August 2026, v2.3.0 on 7 October 2026 and v2.3.2 on 26 October 2026 (GitHub releases). The README states a target of stable releases every 6–8 weeks, with nightly builds in between. This article describes v2.3.2, as of November 2026.
- Migration. The README points users of TensorFlow Lite or LiteRT 1.x to a migration guide on ai.google.dev, which was not read. The
InterpreterAPI remains in LiteRT as the legacy interface, alongsideCompiledModel. - The format.
schema.fbskeeps the identifier"TFL3"and requires new table fields to be added at the end for backward compatibility. Its revision notes mark versions 3a, 3b and 3c as backward compatible with version 3. EachOperatorCodecarries aversionthat is “bumped whenever new parameters are introduced into an op”; in our reading this lets a runtime recognise an operator version it does not implement. - Android. Version 2.3.0 raised the minimum Android SDK to 24.
- Licence. Apache License 2.0, per the repository’s LICENSE and README. Source files carry “The TensorFlow Authors” or “Google LLC” copyright headers.
Limitations
- The plan is static. The planner computes offsets from a fixed execution order and fixed sizes. A dynamic-sized output stops preparation at that node, and resizing an input requires
AllocateTensors()again (interpreter.h). Models whose shapes change from frame to frame pay that cost. - Best fit is a heuristic. The planner sorts tensors by size and places each one in the tightest gap it finds. In the example it reached the lower bound. Nothing in the source claims that it always does, and in our reading the stored unaligned gap makes the comparison between candidate gaps not exactly a comparison of aligned gaps.
- Intermediates are overwritten. With planning on, an intermediate tensor’s memory is reused as soon as it is dead, so reading it after
Invoke()can return another tensor’s data. The negative control shows what reading through the wrong offsets does.preserve_all_tensorskeeps them; the planner’s header says that no memory areas then overlap. - Operator coverage. XNNPACK and the GPU and NPU paths each support a subset of operators, and unsupported ones fall back to the CPU kernels. A model converted with TensorFlow operations left in place needs the TF Select fallback that the TensorFlow article describes.
Related
- Edge Inference
Running a vision model on the device that captures the images, such as a phone, camera, robot or microcontroller, instead of in a data centre, covering why it is done, the limits of compute, memory, power and heat, the classes of hardware and runtimes, the roofline model for finding memory-bound layers, split and offloaded computation, and how latency and energy are measured.
- Model Quantization
Storing and computing a trained network's weights and activations in low-precision formats such as INT8, INT4 or FP8, with a scale and zero point per tensor, channel or group; the integer arithmetic behind it, post-training calibration, quantization-aware training with the straight-through estimator, and what it costs in accuracy.
- Deploying Vision Models
Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.
- Edge AI Accelerators
The inside of the neural processing units and other fixed-function accelerators that run vision models on devices, covering arrays of multiply-accumulate units, systolic arrays, weight- and output-stationary dataflows, the on-chip memory hierarchy, tiling a convolution onto a fixed array, utilization lost to shape mismatch and fill and drain, operator coverage, and why a peak TOPS figure is not a throughput.
- ONNX
An open file format and operator standard for trained machine-learning models, used to move a vision model from the framework that trained it, such as PyTorch, to the runtimes and compilers that execute it.
- MobileNet
Google's family of convolutional networks for phones and other small devices, from MobileNet v1 (2017), built from depthwise separable convolutions with width and resolution multipliers, through MobileNetV2's inverted residuals with linear bottlenecks and MobileNetV3's latency-driven architecture search, to MobileNetV4's universal inverted bottleneck and mobile attention (2024).
References
- Google AI Edge. LiteRT README, README.md, google-ai-edge/LiteRT at tag v2.3.2 (26 October 2026; Apache License 2.0; read November 2026).
- Google AI Edge. LiteRT Overview, litert/README.md, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- Google AI Edge. TensorFlow Lite (runtime dependency of LiteRT), tflite/README.md, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The TensorFlow Authors. TensorFlow Lite FlatBuffer schema, tflite/converter/schema/schema.fbs, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The TensorFlow Authors. Arena planner, tflite/arena_planner.cc and tflite/arena_planner.h, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The TensorFlow Authors. Simple memory arena, tflite/simple_memory_arena.cc and tflite/simple_memory_arena.h, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The TensorFlow Authors. Subgraph, tflite/core/subgraph.cc, and utilities, tflite/util.h, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The TensorFlow Authors. Common C types, tflite/core/c/common.h, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The TensorFlow Authors. Interpreter, tflite/core/interpreter.h, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The TensorFlow Authors. ADD kernel registrations, tflite/kernels/add.cc, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The TensorFlow Authors. CONV_2D kernel, tflite/kernels/conv.cc (with pooling.cc, reduce.cc and fully_connected.cc in the same directory), google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- Google LLC. LiteRT compiled model C API, litert/c/litert_compiled_model.h, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- XNNPACK backend for TensorFlow Lite, tflite/delegates/xnnpack/README.md, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- The new MLIR based TensorFlow to TensorFlow Lite converter, tflite/converter/README.md, google-ai-edge/LiteRT at tag v2.3.2 (read November 2026).
- Google AI Edge. LiteRT release notes, versions 2.2.0 (13 August 2026), 2.3.0 (7 October 2026) and 2.3.2 (26 October 2026). GitHub releases.