Edge Inference

Edge inference runs a trained model on or next to the device that produces the data: a phone, a smart camera, a drone, a robot, a car, an industrial sensor or a battery-powered microcontroller. It is also called on-device inference or edge AI, and at the scale of microcontrollers TinyML. The alternative is to send each image or frame over a network to a server, run the model there and send the answer back. For vision the choice matters especially, because images are large and arrive continuously, and because they often show people and places.

The model that runs at the edge is usually smaller than the one that would run on a server, made so by a compact architecture such as MobileNet, by quantization or by the other methods of model compression. This article is about the place where it runs: what the device limits are, which kinds of processors and runtimes do the work, how to reason about whether a layer is limited by arithmetic or by memory traffic, when to split work between the device and a server, and how latency and energy on a device are measured. How a whole camera-to-decision chain meets its frame budget is the subject of real-time vision pipelines.

Definition

Edge inference is the execution of a model’s forward pass on hardware located where the data is captured or used, under that hardware’s limits on computation, memory, storage, power and heat, rather than on remote, shared servers. “Edge” is a relative term. Matsubara, Levorato and Restuccia (2022) use local computing for running the whole model on the mobile device and edge computing for offloading it to a server close to the device, at the edge of the network; both are edge inference in the broad sense used here, as opposed to a distant data centre. Between them sits split computing, in which the device runs the first layers and a server the rest.

Intuition

The reasons to run a model at the edge follow from what a network round trip costs:

• Latency. A local result needs no transmission, queueing at a server or return trip, and its timing does not depend on the network’s load.
• Bandwidth. A camera produces far more data than a decision. Sending one $224 \times 224$ RGB image with 8 bits per channel is 150,528 bytes; over a LoRa link, which Matsubara et al. cite as limited to 37.5 kbit/s by duty-cycle rules, that takes about 32 s (our arithmetic), while the class label it produces fits in a few bytes.
• Privacy. Images that never leave the device cannot be intercepted or retained elsewhere. Apple’s Core ML documentation gives this as a reason for running models on the device, together with not needing a network connection, and Banbury et al. (2021) list privacy and responsiveness among the reasons for TinyML.
• Connectivity. A robot in a warehouse, a drone or a car must keep working when the link is slow or absent.
• Energy and cost. For tiny battery-powered devices, Banbury et al. note that wireless communication costs far more energy than computation. In principle, running locally also moves the cost of computation from a shared server to hardware that has already been paid for, although whether that is cheaper depends on the device and the workload.

The price is a much smaller computer. A microcontroller has kilobytes of memory where a server accelerator has gigabytes, and a phone must stay cool in a hand and run on a battery. Edge inference is the engineering of making a model’s demands fit those limits.

Formal Definition

A simple model of how fast a layer can run on a device is the roofline model of Williams, Waterman and Patterson (2009). It bounds the attainable performance of a computation by two hardware limits, the peak arithmetic rate $P$ and the peak memory bandwidth $B$:

$$\text{attainable performance} = \min\left(P,\; B \cdot I\right),$$

where $I$ is the computation’s operational intensity, the number of operations per byte of traffic to main memory (DRAM). Williams et al. chose this term over arithmetic intensity on purpose: arithmetic intensity, in their usage, counts traffic between the processor and its caches, whereas operational intensity counts the traffic that reaches main memory after the cache hierarchy has filtered it. They measured $B$ with their own microbenchmarks, as the sustained bandwidth a program can actually reach, not the pin bandwidth of the memory chips.

Plotted on log–log axes, $B \cdot I$ is a diagonal line and $P$ a horizontal one; they meet at the ridge point

$$I^* = \frac{P}{B},$$

the minimum intensity needed to reach peak performance. A computation with $I < I^*$ is memory-bound: the arithmetic units wait for data, and only less traffic or more bandwidth makes it faster. One with $I > I^*$ is compute-bound. Williams et al. add ceilings below the roof for performance that needs a particular optimization, such as SIMD instructions or software prefetching, to reach.

For a neural network, the same bound can be applied layer by layer. With $M_\ell$ multiply-accumulate operations (MACs) and $D_\ell$ bytes of weights and activations moved in layer $\ell$, and layers run one after another, a lower bound on the time per image is

$$T \geq \sum_\ell \max\left(\frac{M_\ell}{P},\; \frac{D_\ell}{B}\right),$$

which is the cost model that MobileNetV4 used to design for many devices at once, as the MobileNet article describes. Counting MACs alone is the special case of an infinitely fast memory. $D_\ell$ depends on how the runtime schedules the layer: if each weight and each input and output activation crosses the memory bus exactly once, $D_\ell$ is the compulsory traffic, while fusing layers so that intermediate results stay on chip lowers it, and spilling to memory raises it.

Energy is the time integral of power. Over a measurement window of $N$ inferences, the energy per inference is

$$E = \frac{1}{N}\int_{t_0}^{t_1} p(t)\,dt,$$

which is what benchmark suites report in joules or microjoules per inference. Because it depends on which components’ power $p(t)$ includes, the measurement boundary has to be stated with the number.

Constraints at the Edge

Memory capacity

A model must fit in two kinds of memory: storage for its weights and code, which also sets the size of app downloads and over-the-air updates, and working memory for activations. Lin et al. (2020) give an example of how small the second can be: a then state-of-the-art Arm Cortex-M7 microcontroller with 320 kB of SRAM and 1 MB of flash, three orders of magnitude less on-chip memory than a mobile device and five to six less than a cloud GPU. By their count, ResNet-50 exceeds that flash by 100 times, MobileNetV2 exceeds the SRAM’s peak-memory limit by 22 times (20 times in the caption of their Table 1), and even its INT8 version by 5.3 times, before counting runtime buffers. In the arXiv version of the paper (v2), they also found that at similar ImageNet accuracy MobileNetV2 has a 4.6 times smaller model than ResNet-18 but a 1.8 times larger peak activation size, so a model designed for few parameters or few operations can still be the wrong model for a device whose limit is working memory. Lin et al. describe such microcontrollers as bare-metal devices without an operating system.

Memory bandwidth

Even where a model fits, moving its tensors costs time and energy. The model compression article quotes energy figures for a 45 nm process in which reading a 32-bit word from off-chip DRAM costs hundreds of times more energy than a 32-bit floating-point addition. Bandwidth is the denominator of the roofline’s diagonal: on a device whose ridge point is high, layers with few operations per byte, such as depthwise convolutions, element-wise operations and fully connected layers at batch size 1, run at a small fraction of peak (see the example below).

Power and thermal limits

A battery limits the energy per inference and per day; a power supply or a thermal design limits the sustained power. When a chip produces heat faster than its package and case can remove it, its temperature rises until the operating system slows it down. The Linux kernel’s thermal framework describes the mechanism in general terms: thermal zones report temperatures and carry trip points, cooling devices, which may be fans or processors, have a range of throttle states, and thermal management, through a governor chosen for each zone, decides from the current temperatures and the trip points which devices to throttle. A zone can also declare a sustainable power that it can dissipate. For a processor, throttling in practice means lowering its clock frequency and voltage (dynamic voltage and frequency scaling; our summary, since the kernel document describes throttle states only abstractly). The practical consequence, in our reasoning, is that a model’s speed on a device is not a constant: it can be faster in the first seconds of a run, on a cool device, than once the device has heated up, and faster on a bench than inside a sealed enclosure in the sun.

Hardware Classes

Edge processors fall into a few classes, which differ in their peak rates, quoted per number format, their ridge points, the formats they accelerate and how easy they are to program. A system-on-chip often contains several of them, and a runtime decides which part of a model runs where.

• CPUs with SIMD. Every device has a CPU, and every runtime can fall back to it. Single-instruction, multiple-data (SIMD) vector instructions apply one operation to several values at once, and since a vector register of fixed width holds four times as many 8-bit values as 32-bit ones, low-precision arithmetic multiplies throughput on the same register (our arithmetic). CPUs have a relatively low ridge point, so memory-light layers are cheap on them and large dense convolutions expensive, as MobileNetV4’s analysis found. Phone CPUs often combine fast, power-hungry cores with slower, efficient ones; the TFLite benchmark tool’s documentation warns that this can cause significant variance between runs and suggests pinning the benchmark to one kind of core.
• Mobile and embedded GPUs. Many parallel arithmetic lanes give a GPU a high peak rate for convolutions and matrix products, often in 16-bit floating point; MobileNetV4 was benchmarked with 16-bit floats on GPUs and Apple’s Neural Engine and with INT8 on CPUs, DSPs and Google’s EdgeTPU, as the MobileNet article records.
• NPUs, DSPs and other accelerators. Neural processing units and digital signal processors with machine-learning extensions are built around arrays of multiply-accumulate units for low-precision tensor arithmetic, which gives them high peak rates in those formats and therefore high ridge points (our characterization, which the example illustrates). They also accelerate the fewest operations and formats: Ignatov et al. (2019) describe a 2019 phone NPU that supported only quantized inference, and phone chips typical of 2018 whose accelerators might handle only floating-point or only quantized models. An operation an accelerator does not support runs elsewhere, usually on the CPU, and each switch between processors costs a transfer. FPGAs, configurable logic, are a further class that the MLPerf Inference paper lists among inference targets.
• Microcontrollers. At the low end, a microcontroller runs a model in kilobytes of SRAM, from flash, at very low power; Banbury et al. describe TinyML as inference below a milliwatt. The software must do without features that larger systems take for granted. David et al. (2021) list dynamic memory management, virtual memory, an operating system, a file system and floating-point hardware among the features embedded platforms may lack.

The classes are principles, not products: specific chips mix them, change every year and are documented by their makers.

On-Device Runtimes

A runtime loads a model file, plans its memory, chooses kernels for each operation and dispatches subgraphs to the processors it supports. The main ones as of October 2026, described from their own documentation:

• LiteRT, Google’s on-device runtime, formerly TensorFlow Lite (the TensorFlow article describes the renaming and the .tflite format). Its README for version 2.3.2 (26 October 2026) describes CPU execution through XNNPACK and GPU and NPU acceleration, and converters from PyTorch, TensorFlow and JAX models.
• ExecuTorch, PyTorch’s on-device runtime. A model is captured with torch.export, quantized, optimized and partitioned ahead of time into a .pte file, and a C++ runtime executes it; partitioners delegate subgraphs to NPU or GPU back ends, with CPU fallback for the rest (README, version 1.5.1). Its first numbered release, 0.1.0, appeared on 17 October 2023, after dated snapshot releases from August 2023, version 1.0.0 on 18 October 2025, and 1.5.1 on 23 September 2026 (GitHub releases).
• ONNX Runtime, which runs ONNX files by partitioning the graph among execution providers for the hardware libraries present, including mobile ones such as Core ML and Android NNAPI; the ONNX article lists them, together with TensorRT for NVIDIA GPUs and OpenVINO for Intel hardware.
• Core ML, Apple’s framework, which according to its documentation uses the CPU, GPU and Neural Engine of the device. An app can restrict a model to a subset of them through MLComputeUnits, for example the CPU only or the CPU and Neural Engine. Models from other libraries are converted with coremltools, whose version 9.0 was released on 10 November 2025.
• TensorFlow Lite Micro for microcontrollers. David et al. (2021) describe an interpreter that takes a contiguous memory arena from the application and allocates every tensor from it during initialization, because dynamic memory allocation cannot be assumed and allocations after start-up could fragment the heap of a long-running device; an operator resolver links into the binary only the operators a model uses.
• ML compilers generate code for a model and a target rather than calling a fixed library of kernels. Apache TVM, which its README describes as an open machine learning compilation framework that turns models into minimal deployable modules, released version 0.27.0 on 25 September 2026.

Runtimes differ in which operators they support, how they implement them, and which accelerators they reach, and a converted model is not always numerically identical. Reddi et al. (2020) give an example from MLPerf Inference: SSD-MobileNet-v1 in TensorFlow used a non-maximum suppression variant that TensorFlow Lite did not implement, and with TensorFlow Lite’s fast NMS instead, its COCO mAP fell from 23.1% to 22.3% after conversion. Choosing among these runtimes for a product, packaging the model and keeping pre- and post-processing consistent are part of deploying a model rather than of the device.

On-Device, Offloaded and Split Computation

Matsubara et al. (2022) compare four ways of serving a mobile device:

• Local computing. The whole model runs on the device; no data crosses the wireless channel, but the model must be small enough for the device’s compute and energy, which usually costs accuracy.
• Edge computing (full offloading). The input is sent to a nearby server that runs the full model. Accuracy is preserved, but the whole input must be transmitted, and wireless links that are fast on average can fluctuate sharply, which hurts latency-bound applications.
• Split computing. The device runs a head, the first layers, and sends the intermediate tensor to a server that runs the tail. Matsubara et al. describe Kang et al.’s Neurosurgeon (2017), which, without modifying the network, searched for the split layer that minimized latency or energy; as they summarize its results, for most models it considered the best split was at the input or the output layer, that is, fully on the server or fully on the device. In our reading, one reason is that the early feature maps of a CNN are often larger than the input image. Later work adds a bottleneck, a narrow or quantized layer trained into the network at the split point, so that less data is sent.
• Early exiting. Classifier heads at several depths let the device stop as soon as one is confident enough, trading accuracy for time according to current conditions.

Privacy differs between them. Sending a compressed image, such as a JPEG, sends a representation designed to be reconstructed; Matsubara et al. note that this may reduce privacy compared with a task-oriented bottleneck representation, which may lack the information needed to reconstruct the input. A bottleneck is not a guarantee: whether an intermediate representation can be inverted depends on the model, and on-device inference protects the images only if the results and logs sent off the device do not reveal what the images showed (our reasoning).

Measuring Latency and Energy

A latency or energy figure for a device is only meaningful with its conditions: the device, runtime and version, number format, batch size, threads or processors used, input size, and the thermal and power state. The benchmark suites encode the practices that make numbers comparable.

Warm-up. The first inference includes one-time work. In MLPerf Inference, the system under test may load data into memory and perform untimed operations, which Reddi et al. list as including compilation, cache warm-up and preprocessing, before signalling that it is ready for the first query. The TFLite benchmark tool performs one warm-up run by default before its 50 timed runs (warmup_runs and num_runs). Start-up cost is real for an application that loads a model on demand, so it is worth reporting separately rather than discarding.

Percentiles, not means. A frame-by-frame system is limited by its slow inferences, so latency is reported as a high percentile. MLPerf Inference’s single-stream scenario issues one query of one sample at a time, the next as soon as the previous completes, and reports the 90th-percentile latency. The current rules (August 2026) compute it with an early-stopping criterion that allows shorter runs but raises the reported percentile slightly to offset their larger variance. The original paper derived the number of queries for its latency-bounded scenarios from the percentile, a 99% confidence level and a margin of error (0.5% for the 90th percentile, 0.05% for the 99th; one-twentieth of the distance to 100%, in our arithmetic), which gave 24,576 inferences for a 90th percentile and 270,336 for a 99th, while single-stream runs needed only 1,024 queries, so that phones and embedded platforms could finish quickly.

Duration and equilibrium. Reddi et al. required each run to last at least 60 s so that power management and dynamic frequency scaling reach equilibrium, particularly for the short single-stream runs; the current rules list 600 s for each scenario. On Linux, a cooling device’s stats/time_in_state_ms records the time spent in each throttle state, which shows whether a benchmark ran throttled.

Variance control. Pinning the benchmark to a fixed set of cores, as the TFLite tool’s documentation shows for phones with fast and slow cores, reduces run-to-run variance. Since a deployed application does not control its scheduling in that way, the pinned figure describes the hardware rather than the application’s experience (our reasoning).

Power and energy. Three methods are common, at different boundaries:

• Wall power. MLPerf Inference measures the power of the whole system under test at the AC input with an external power analyzer logged by SPEC’s PTDaemon, from the same run as the performance result, so that a submitter cannot report the best performance and the lowest power from different runs. Tschand et al. (2025) note that for low-power edge devices behind adapters under 75 W such meters show larger errors because of the adapters’ high crest factor, and that full-system measurement includes subsystems, such as radios and idle interfaces, that do no machine-learning work. As of their paper, MLPerf did not support power measurement on mobile devices.
• Supply measurement of a board or core. MLPerf Tiny measures energy with an energy monitor that supplies the device under test; only the power delivered to the core is counted, a single supply must power it (no batteries or supercapacitors that could hide consumption), and the board’s level shifters are excluded. Each of latency and energy is measured five times, each time running inference for at least 10 s and at least 10 iterations, and the median of the five is reported, as inferences per second and microjoules per inference.
• On-chip telemetry. Many processors report their own power estimates, which are convenient but cover only the components their sensors see (our characterization). Tschand et al. describe software telemetry as MLPerf’s method for large training clusters, where external meters are impractical, not for edge devices.

Benchmark suites. MLPerf Inference, from MLCommons, defines its tasks, reference models, quality targets and four scenarios (single-stream, multistream, server and offline); its closed division requires a model equivalent to the reference, while the open division allows changes to the model (Reddi et al., 2020). As of the August 2026 rules, its edge suite includes image classification with ResNet-50 v1.5 on ImageNet (quality target 99% of FP32 accuracy) and object detection with YOLO11l on a 1,525-image MLPerf subset of COCO 2017, each required in the single-stream, multistream and offline scenarios. MLPerf Tiny (Banbury et al., 2021) targets microcontroller-class systems with four benchmarks, two of them visual: Visual Wake Words, detecting whether a person is present in a $96 \times 96$ image with a MobileNetV1 of 325 kB, and CIFAR-10 classification with a ResNet of 96 kB. These suites report results under stated rules; they do not rank devices for a given application, whose model, input and duty cycle differ.

Examples

A roofline for two classifiers

The example counts, for every convolution and fully connected layer of torchvision’s MobileNetV2 and ResNet-18 at $224 \times 224$, the MACs, weights and activations for one rendered image, and places each layer on the roofline of two hypothetical devices. The devices’ numbers are illustrative, chosen to show the mechanism, and do not describe any product: both have 20 GB/s of memory bandwidth; the “CPU-like” device performs 0.2 tera-MACs per second in FP32 and 0.8 in INT8, and the “accelerator” 8 tera-MACs per second in INT8 only. Traffic is the compulsory amount: every weight and every input and output activation moves once per layer, with weights read once per batch. Batch normalization is assumed folded into the convolutions, and element-wise operations (activations, residual additions, pooling) are left out, which understates memory traffic. The models are built with weights=None, since only their shapes matter (torchvision 0.29.2, PyTorch 2.14.1, NumPy 2.5.4).

import numpy as np
import torch
from torchvision.models import mobilenet_v2, resnet18

torch.manual_seed(0)
torch.set_num_threads(1)

# A rendered 224x224 RGB image: a bright disk and a dark square on grey.
yy, xx = np.mgrid[:224, :224]
img = np.full((224, 224, 3), 0.5, np.float32)
img[(yy - 80) ** 2 + (xx - 140) ** 2 < 40 ** 2] = (0.9, 0.8, 0.2)
img[130:190, 30:90] = (0.1, 0.2, 0.6)
x = torch.from_numpy(img).permute(2, 0, 1)[None]          # 1 x 3 x 224 x 224

def layer_costs(model):
    """MACs, weight elements and activation elements of every conv and linear
    layer, for one image (each tensor counted once, as a lower bound on traffic)."""
    rows = []
    def hook(m, inp, out):
        if isinstance(m, torch.nn.Conv2d):
            k = m.kernel_size[0] * m.kernel_size[1]
            macs = out.numel() * (m.in_channels // m.groups) * k
            kind = ("depthwise" if m.groups == m.in_channels > 1 else
                    "pointwise" if k == 1 else "standard")
        else:
            macs, kind = out.numel() * m.in_features, "linear"
        rows.append((kind, macs, m.weight.numel(), inp[0].numel(), out.numel()))
    hs = [m.register_forward_hook(hook) for m in model.modules()
          if isinstance(m, (torch.nn.Conv2d, torch.nn.Linear))]
    with torch.no_grad():
        model.eval()(x)
    for h in hs:
        h.remove()
    return rows

# Two hypothetical devices (illustrative numbers, not real products), each with
# 20 GB/s of off-chip memory bandwidth and a peak MAC rate per number format.
BW = 20e9                                                   # bytes/s
DEVICES = {"CPU-like": {"fp32": 0.2e12, "int8": 0.8e12},    # MAC/s
           "accelerator": {"int8": 8e12}}
BYTES = {"fp32": 4, "int8": 1}

def roofline(rows, peak, prec, batch):
    """Per-image time of each layer: max(compute time, memory time)."""
    out = []
    for kind, macs, w, a_in, a_out in rows:
        nbytes = (w + batch * (a_in + a_out)) * BYTES[prec]   # weights read once per batch
        intensity = batch * macs / nbytes                      # MAC per byte
        t = max(batch * macs / peak, nbytes / BW) / batch
        out.append((kind, intensity, t, intensity < peak / BW))
    return out

CASES = [("CPU-like", "fp32", 1), ("CPU-like", "int8", 1), ("CPU-like", "fp32", 8),
         ("accelerator", "int8", 1), ("accelerator", "int8", 8)]

for name, net in [("MobileNetV2", mobilenet_v2(weights=None)),
                  ("ResNet-18", resnet18(weights=None))]:
    rows = layer_costs(net)
    macs = sum(r[1] for r in rows)
    print(f"{name}: {len(rows)} conv/linear layers, {macs / 1e6:.0f}M MACs, "
          f"{sum(r[2] for r in rows) / 1e6:.2f}M weights")
    res = roofline(rows, DEVICES["CPU-like"]["fp32"], "fp32", 1)
    for kind in ["standard", "depthwise", "pointwise", "linear"]:
        sel = [(r, q) for r, q in zip(rows, res) if r[0] == kind]
        if sel:
            ai = [q[1] for _, q in sel]
            print(f"  {kind:9s} x{len(sel):2d}: {sum(r[1] for r, _ in sel) / macs:5.1%} of MACs, "
                  f"fp32 batch-1 intensity {min(ai):5.2f} to {max(ai):5.2f} MAC/B")
    for dev, prec, batch in CASES:
        peak = DEVICES[dev][prec]
        res = roofline(rows, peak, prec, batch)
        t = sum(r[2] for r in res)
        t_mem = sum(r[2] for r in res if r[3])
        print(f"  {dev:11s} {prec} batch {batch}: ridge {peak / BW:5.1f} MAC/B, "
              f"{t * 1e3:5.2f} ms/image (compute bound alone {macs / peak * 1e3:4.2f}), "
              f"{sum(r[3] for r in res):2d} memory-bound layers, {t_mem / t:4.0%} of the time")

Output (identical in two runs):

MobileNetV2: 53 conv/linear layers, 301M MACs, 3.47M weights
  standard  x 1:  3.6% of MACs, fp32 batch-1 intensity  4.90 to  4.90 MAC/B
  depthwise x17:  6.9% of MACs, fp32 batch-1 intensity  0.43 to  1.12 MAC/B
  pointwise x34: 89.1% of MACs, fp32 batch-1 intensity  2.66 to 14.49 MAC/B
  linear    x 1:  0.4% of MACs, fp32 batch-1 intensity  0.25 to  0.25 MAC/B
  CPU-like    fp32 batch 1: ridge  10.0 MAC/B,  3.49 ms/image (compute bound alone 1.50), 38 memory-bound layers,  81% of the time
  CPU-like    int8 batch 1: ridge  40.0 MAC/B,  0.87 ms/image (compute bound alone 0.38), 38 memory-bound layers,  81% of the time
  CPU-like    fp32 batch 8: ridge  10.0 MAC/B,  3.22 ms/image (compute bound alone 1.50), 31 memory-bound layers,  73% of the time
  accelerator int8 batch 1: ridge 400.0 MAC/B,  0.85 ms/image (compute bound alone 0.04), 53 memory-bound layers, 100% of the time
  accelerator int8 batch 8: ridge 400.0 MAC/B,  0.69 ms/image (compute bound alone 0.04), 53 memory-bound layers, 100% of the time
ResNet-18: 21 conv/linear layers, 1814M MACs, 11.68M weights
  standard  x17: 98.9% of MACs, fp32 batch-1 intensity 11.52 to 83.01 MAC/B
  pointwise x 3:  1.1% of MACs, fp32 batch-1 intensity  5.19 to  8.76 MAC/B
  linear    x 1:  0.0% of MACs, fp32 batch-1 intensity  0.25 to  0.25 MAC/B
  CPU-like    fp32 batch 1: ridge  10.0 MAC/B,  9.21 ms/image (compute bound alone 9.07),  4 memory-bound layers,   3% of the time
  CPU-like    int8 batch 1: ridge  40.0 MAC/B,  2.30 ms/image (compute bound alone 2.27),  4 memory-bound layers,   3% of the time
  CPU-like    fp32 batch 8: ridge  10.0 MAC/B,  9.11 ms/image (compute bound alone 9.07),  2 memory-bound layers,   1% of the time
  accelerator int8 batch 1: ridge 400.0 MAC/B,  0.82 ms/image (compute bound alone 0.23), 21 memory-bound layers, 100% of the time
  accelerator int8 batch 8: ridge 400.0 MAC/B,  0.33 ms/image (compute bound alone 0.23), 15 memory-bound layers,  74% of the time

The counts agree with the published figures for these networks (about 300 million MACs and 3.4 million parameters for MobileNetV2), and the per-layer intensities show the structure of each design. MobileNetV2’s depthwise layers do about one MAC or less per byte moved, and even its pointwise layers, which hold 89% of its MACs, range from 2.7 to 14.5 MAC/B, so that more than half of them sit below the CPU-like device’s ridge of 10 MAC/B, because they work on thick, expanded activation tensors. ResNet-18’s standard convolutions do 12 to 83 MACs per byte. The fully connected classifier of both does 0.25 MAC/B at batch 1: in FP32 each 4-byte weight is used once.

Three consequences follow. First, on the CPU-like device MobileNetV2 needs six times fewer MACs than ResNet-18 but runs only 2.6 times faster, and spends 81% of its time in memory-bound layers, more than twice its compute bound; ResNet-18 is within 2% of its compute bound. Second, on the accelerator, whose ridge is 400 MAC/B, every layer of both networks is memory-bound at batch 1, and the two models take nearly the same time, 0.85 and 0.82 ms, despite the six-fold difference in MACs: the accelerator’s arithmetic is idle most of the time. Third, batching helps only where weights dominate traffic. At batch 8 ResNet-18 on the accelerator falls to 0.33 ms per image, because its large late-layer weights are read once for eight images, while MobileNetV2, whose traffic is mostly activations that grow with the batch, improves only to 0.69 ms.

INT8 on the CPU-like device divides every time by four without changing which layers are memory-bound, because it divides the bytes and multiplies the peak by the same factor, moving each layer and the ridge point by the same amount. It follows from the formula that a device whose INT8 rate rose by less than four times would see more layers become compute-bound after quantization, and one whose INT8 rate rose by more would see more become memory-bound. The model is a bound under stated assumptions, not a prediction: a runtime that fuses a depthwise layer with the pointwise layers around it, or a cache that holds a small tensor, removes traffic counted here, and the element-wise layers left out add traffic. Measuring a real device, as described above, is what tells which effect dominates.

Common Misconceptions

• “Peak operations per second predict speed.” A device’s peak rate applies only above its ridge point. In the example, the accelerator has ten times the CPU-like device’s INT8 rate, yet runs MobileNetV2 in INT8 in 0.85 ms against 0.87 ms, because the network’s layers are memory-bound on both.
• “A model that runs on the device is private.” Only the images stay on the device. Results, logs and, in split computing, intermediate features that leave it can still carry information about them.
• “Edge inference is always faster than the cloud.” It removes the network, but a small device running a large model can be slower than a server plus a round trip; the split-computing literature finds the best placement to depend on the model, the device and the link.
• “The first inference shows the latency.” It includes one-time costs, and a few runs on a cool device do not show throttling.

Where It Is Used

Edge inference is used wherever images must be interpreted close to the camera: image classification and face detection in phone cameras, person detection in battery-powered cameras of the kind MLPerf Tiny’s Visual Wake Words task models, object detection and semantic segmentation on robots and vehicles, and tracking for augmented reality, where a round trip to a server would add latency to every frame. The models are usually compact architectures such as MobileNet and EfficientNet, compressed with the methods of model compression and quantization, converted through formats such as ONNX, and run inside a real-time vision pipeline that sets their frame budget.