Neural Architecture Search

Neural architecture search (NAS) automates the design of a network’s architecture: which layers it has, how they are connected and how wide and deep it is. Elsken, Metzen and Hutter (2019) describe every NAS method along three dimensions. A search space defines which architectures can be represented at all; a search strategy chooses which ones to try; and a performance estimation strategy says how good each tried architecture is, from full training on held-out data down to cheap proxies. Li and Talwalkar (2019) call NAS a specialized instance of hyperparameter tuning, whose search spaces usually involve discrete hyperparameters represented as a directed acyclic graph (§1, §2.2); the hyperparameter tuning article describes grid and random search, Bayesian optimization, Hyperband and population-based training. Searched networks in computer vision include NASNet, AmoebaNet, MnasNet and FBNet for classification, the baseline network of EfficientNet, the larger MobileNet models, NAS-FPN for detection and Auto-DeepLab for segmentation.

Purpose

Most architectures in vision were designed by hand, by trial and error over repeated motifs (the Inception module, the residual block, the dense block). Elsken et al. present NAS as the logical next step after deep learning automated feature engineering, since increasingly complex architectures were still designed by hand. The goal is an architecture that does well on held-out data for a given task, often under a cost constraint such as multiply-adds, parameters or measured latency on a target device.

The obstacle is cost. Evaluating an architecture properly means training it, so a search that tries thousands of candidates costs thousands of training runs. Elsken et al. (§4) note that training each candidate from scratch led to costs “in the order of thousands of GPU days” in early work, and much of the field’s later history is about reducing that number: through the search space, through cheaper estimates of performance, and through sharing weights between candidates.

When to Use It

NAS pays off when:

• the deployment target imposes a trade-off that hand design handles poorly, such as accuracy against latency on a particular phone; MnasNet and FBNet were built for this case, and FBNet argues that the optimal architecture changes with resolution and device (§1);
• a family of models will be reused widely, so one expensive search is amortized over many uses, as with the searched baselines of MobileNetV3 and EfficientNet;
• a good search space can be written down from prior designs, which is what makes cell-based search work for image classification.

It helps little, or misleads, when:

• the compute available is small compared with training one model many times, and no weight sharing or tabular benchmark applies;
• the evaluation protocol is noisy relative to the differences between candidates: Li and Talwalkar found that rankings of DARTS architectures after 100 epochs of training did not match their rankings after 600 (§4.1.3, Table 3);
• the gains claimed come with a different training recipe; Elsken et al. (§5) consider it “conceivable” that improvements from learning-rate schedules, augmentation and regularization have a larger effect on reported numbers than the searched architectures.

An alternative to search is to encode expert rules: nnU-Net configures U-Nets for each medical dataset by fixed rules rather than by architecture search.

How It Works

Search spaces

Chain-structured (macro) spaces. The simplest space is a sequence of $n$ layers in which layer $i$ takes the output of layer $i-1$. Elsken et al. (§2) parametrize it by the number of layers, the operation in each (convolution, pooling, depthwise separable or dilated convolution) and that operation’s hyperparameters (filters, kernel size, stride), which exist only once the operation is chosen, so the space is conditional. Multi-branch spaces let the input of a layer be any function of earlier outputs, which includes plain chains, residual connections (sums) and DenseNet-style concatenations as special cases. ENAS’s macro space for CIFAR-10 is of this kind: each of $L$ layers picks one of 6 operations and any subset of earlier layers as skip inputs, giving $6^L \times 2^{L(L-1)/2}$ networks, $1.6 \times 10^{29}$ for $L = 12$ (Pham et al., §2.3).

Cell-based (micro) spaces. NASNet (Zoph et al., 2018) searched for a repeated module, a cell, instead of a whole network. The outer network is fixed by hand: a stack of normal cells, which keep the spatial size, interleaved with reduction cells, which halve it and double the filters. A cell takes the outputs of the two previous cells and builds $B = 5$ blocks; for each block a controller makes five choices: two inputs from the hidden states available so far, an operation for each from a list of 13 (identity, $1\times1$ and $3\times3$ convolutions, $1\times3$-then-$3\times1$ and $1\times7$-then-$7\times1$ convolutions, $3\times3$ dilated convolution, $3\times3$ average pooling, $3\times3$, $5\times5$ and $7\times7$ max pooling and $3\times3$, $5\times5$ and $7\times7$ depthwise separable convolutions) and a combination by addition or concatenation. Unused hidden states are concatenated into the cell’s output (§3). The number of cell repeats $N$ and the number of filters are free parameters chosen afterwards for the target scale. Because the cell’s complexity does not depend on depth or image size, the authors searched on CIFAR-10 and transferred the cell to ImageNet, stacking more copies of it; they estimate the search was about seven times more efficient than the whole-network search of Zoph and Le (2017) after discounting faster hardware (§4, footnote 1). Elsken et al. (§2) list the advantages: a much smaller space, transfer across datasets by changing $N$ and the filter count, and repetition as a proven design principle. The cost is that the macro-architecture is still designed by hand.

Hierarchical spaces. Elsken et al. (§2) describe the hierarchical space of Liu et al. (2018) as several levels of motifs, primitive operations at the first level and graphs of lower-level motifs above, of which the cell-based space is the three-level special case. Auto-DeepLab (Liu et al., 2019) argues that dense prediction is sensitive to how the network changes resolution, which cell search leaves to hand design, and searches both levels for semantic segmentation: a trellis of possible resolution paths at the network level and a NASNet-style cell, relaxed together so that gradients reach both, with the network path decoded by the Viterbi algorithm (§1, Table 1).

Factorized and layer-wise spaces. Repeating one cell everywhere rules out layers that differ by depth in the network, which matters for latency because early layers process more pixels. MnasNet’s factorized hierarchical space (Tan et al., 2019, §4.1) divides the network into a sequence of blocks with fixed resolutions; each block searches one layer (convolution type, kernel size $3$ or $5$, a squeeze-and-excitation ratio, a skip operation, the filter size relative to MobileNetV2) and the number of times it repeats. With $B$ blocks, a per-block sub-space of size $S$ and $N$ layers per block, the space has $S^B$ members against $S^{BN}$ for a flat per-layer space; for their “typical case” $S = 432$, $B = 5$, $N = 3$ the authors give about $10^{13}$ against $10^{39}$. FBNet (Wu et al., 2019, §3.1) fixes a macro-architecture of 22 searchable layers and lets each choose one of 9 blocks ($1 \times 1$, depthwise and $1 \times 1$ convolutions with expansion 1, 3 or 6, kernel 3 or 5 and optional group convolution, or skip), $9^{22} \approx 10^{21}$ architectures.

Bias in the space. The space encodes prior knowledge, which makes the search easier but, as Elsken et al. (§2, §5) put it, introduces a human bias that may prevent finding building blocks beyond current knowledge; a space built from image-classification motifs also transfers poorly to tasks where repeated cells in a chain do not fit. Real et al. note that their mutations cover the whole NASNet space; NASNet’s authors take the small gap they found between random and reinforcement-learning search as a sign that the space itself is well constructed (§3).

Search strategies

Reinforcement learning. Zoph and Le (2017), as NASNet’s §3 and Elsken et al. (§3) describe it, used a recurrent controller that emits an architecture as a sequence of tokens; each sampled child network is trained and its validation accuracy is the reward for a policy-gradient update of the controller. NASNet’s controller is a one-layer LSTM with 100 units and $2 \times 5B$ softmax outputs, trained with Proximal Policy Optimization; 450 workers trained child networks concurrently, the search stopped after 20,000 child models, each trained for 20 epochs, and the top 250 were then trained to convergence to pick the best (Appendix A). Elsken et al. (§3) point out that since the controller receives no intermediate rewards and observes no external state, the problem can be read as a stateless multi-armed bandit over whole architectures.

Evolution. Neuro-evolution is older than the deep-learning revival; Elsken et al. (§3) trace it to genetic algorithms that proposed architectures trained by backpropagation in 1989, and note that recent methods evolve only the architecture and train weights by gradient descent. Real et al. (2019) introduced aging evolution, or regularized evolution, a variant of tournament selection (their Algorithm 1). A population of $P$ trained models is initialized at random. Each cycle samples $S$ models from the population uniformly with replacement, takes the most accurate as the parent, applies one random mutation (rewiring one input of one block, replacing one operation, or, with probability 0.05, nothing), trains and evaluates the child, adds it, and removes the oldest model. Standard tournament selection would remove the worst instead. The authors speculate that aging helps with noisy training: a model that was lucky once cannot stay in the population, so an architecture survives only by retraining well repeatedly through its descendants, which they call a form of regularization (Discussion). With $P = 100$ and $S = 25$, each experiment used 450 K40 GPUs for about 7 days to evaluate 20,000 small models (Methods Details). In their controlled comparison in the NASNet space, with five repeats each, evolution reached accurate models earlier than reinforcement learning, ended with similar accuracy, and found models with lower FLOPs; both beat random search (Figures 4 and 5). The two papers that compare aging with non-aging evolution may define the non-aging variant differently: Real et al. remove the worst model of the $S$-sample, while NAS-Bench-101 (Ying et al., §4.2) says only that it removes “the lowest fitness individual”, without restricting it to the sample.

Random search. Uniform sampling from the space is the baseline every strategy must beat. NASNet’s authors found random search “slightly worse” than reinforcement learning in their space (§3); in their Figure 6 the best model found by RL is over one point better at 20 epochs. ENAS’s authors trained randomly sampled architectures from their spaces, which did far worse than the searched ones (6.77% against 3.54% CIFAR-10 test error for cells; §3.3). Li and Talwalkar (2019) argued that published random baselines were too weak: ENAS evaluated one random architecture and DARTS 24, while random search with early stopping (ASHA) evaluated over 700 in 9 GPU days (§4.1.1).

Bayesian optimization. Elsken et al. (§3) note that few groups had applied it to NAS, since typical toolboxes are based on Gaussian processes and focus on low-dimensional continuous problems (some works derive kernels for architecture spaces), while tree-based models (tree Parzen estimators, random forests) handle high-dimensional conditional spaces. On NAS-Bench-101, the random-forest method SMAC was among the best searchers, while TPE fell back to the level of random search (Ying et al., §4.1).

Gradient-based search. DARTS (Liu, Simonyan and Yang, 2019), as Elsken et al. (§3, §4) describe it, makes the choice of operation continuous: the output of an edge is a convex combination $y = \sum_{i=1}^m \alpha_i o_i(x)$ of all candidate operations, with $\alpha_i \ge 0$ and $\sum_i \alpha_i = 1$, so the architecture weights $\alpha$ can be learned by gradient descent on validation data, alternating with steps on the network weights on training data; the final architecture keeps the operation with the largest $\alpha_i$ on each edge. Elsken et al. also describe SNAS, which optimizes a distribution over operations through the concrete (Gumbel-softmax) relaxation, and ProxylessNAS, which “binarizes” the architecture weights, masking out all but one edge, so that the whole one-shot model need not be kept in GPU memory. FBNet (Wu et al., §3.3) samples one block per layer from a softmax over learnable logits $\theta_{l,i}$, relaxes the sampling mask with the Gumbel softmax so that the loss is differentiable in $\theta$, and trains weights and $\theta$ in alternation; its supernet was trained for 90 epochs on 100 randomly chosen ImageNet classes (§4.1).

Performance estimation

Every strategy needs an estimate of how good an architecture is. Elsken et al. (§4, Table 1) group the ways to make it cheaper into four families.

• Lower fidelity. Train for fewer epochs, on a subset of the data, at lower resolution or with fewer cells and filters. NASNet trained each child for 20 epochs with $N = 2$ during search (Appendix A.6); Real et al. trained small models ($N = 3$, $F = 24$) for 25 epochs and then applied the model augmentation trick, enlarging the best architectures and training them for 600 epochs. Low fidelity biases the estimate downwards, which matters only if it changes the ranking; Elsken et al. report evidence that rankings can change dramatically when the gap between the cheap and the full evaluation is too big, and argue for increasing the fidelity gradually, citing Hyperband and BOHB. MnasNet searched on ImageNet itself for 5 epochs rather than on CIFAR-10, because the authors found that small proxy tasks do not work once latency is part of the objective (§5).
• Learning-curve extrapolation, which stops runs whose early curves predict poor final performance, and surrogate models that predict an architecture’s performance from its structure.
• Weight inheritance and network morphisms, which warm-start a child from its parent’s weights; morphisms change the architecture without changing the function it computes (Elsken et al., §4).
• One-shot models and weight sharing. All architectures are treated as subgraphs of one supergraph (the one-shot model), whose weights are trained once and shared; evaluating an architecture then needs only a validation pass.

ENAS (Pham et al., 2018) shares one set of weights among all architectures in its space. It alternates two phases: the shared weights $\omega$ are trained by SGD on the training data with an architecture $m$ sampled from the controller’s policy $\pi(m;\theta)$ for each minibatch, and the controller’s parameters $\theta$ are trained by REINFORCE to maximize the validation accuracy of sampled architectures under the current shared weights (§2.2). On one GTX 1080Ti, the cell search took 0.45 days (Table 2; the text of §3.2 says 11.5 hours) and gave 2.89% CIFAR-10 test error with Cutout, against 2.65% for NASNet-A, whose search the authors put at 450 GPUs for 3–4 days; they report the reduction in GPU-hours as more than 1000 times (§1, Table 2). Bender et al. (2018) showed that neither a controller nor a hypernetwork is needed: they trained one-shot models with path dropout, removing incoming paths at random with a rate increased linearly during training, then evaluated about 20,000 randomly sampled architectures on the one-shot model in around 80 GPU-hours and found a “near-monotonic” relation between one-shot accuracy and the accuracy of the same architectures trained alone for about 28 epochs (§4.1, Figure 5). The dropout rate had to be tuned: with too little dropout most architectures received very low one-shot accuracies, and with too much the one-shot accuracies were compressed into a narrow band (§4.1). Li and Talwalkar (2019) went further and combined plain random sampling with weight sharing, reaching 2.85 ± 0.08% on CIFAR-10 over 10 seeds (Table 1).

Weight sharing has known weaknesses. Elsken et al. (§4) note that the one-shot model underestimates the performance of the best architectures, that it is not clear how well its ranking correlates with stand-alone training, that the supergraph restricts the space to its subgraphs and must often fit in GPU memory, and that a search distribution trained jointly with the shared weights may reinforce its own early bias, with weights adapted to the architectures it already prefers.

Hardware-aware search

Multiply-adds are a poor proxy for speed on real devices. MnasNet’s authors cite MobileNet and NASNet: similar multiply-adds (575M against 564M) but 113 ms against 183 ms on a Pixel phone (§1, Table 1). MnasNet therefore measures latency on the phone during search and maximizes the reward

$$\mathrm{ACC}(m) \times \left[\frac{\mathrm{LAT}(m)}{T}\right]^{w}, \qquad w = \begin{cases} \alpha, & \mathrm{LAT}(m) \le T \\ \beta, & \text{otherwise,} \end{cases}$$

with target latency $T$ (Eqs. 2–3). Setting $\alpha = 0$, $\beta = -1$ makes latency a hard constraint; the authors use $\alpha = \beta = -0.07$, a soft constraint, from the empirical rule that doubling latency brings about 5% relative accuracy gain (§3). They used NASNet’s RNN controller, updated with PPO; each search took 4.5 days on 64 TPUv2 devices and sampled about 8,000 models, of which 15 were trained fully on ImageNet. MnasNet-A1 reached 75.2% top-1 on ImageNet at 78 ms on a Pixel 1 big CPU core, against 74.0% at 183 ms for NASNet-A in the same table (Table 1). The EfficientNet article describes how its baseline B0 was found in the MnasNet space with FLOPS in place of latency, and the MobileNet article how MobileNetV3 reused MnasNet-A1 and searched again for its small model. FBNet (Wu et al., §3.2–3.3) made latency differentiable instead: a lookup table of measured per-operator latencies gives the network’s latency as a sum over layers, weighted by the relaxed sampling masks, and the loss multiplies the cross-entropy by $\alpha \log(\mathrm{LAT})^{\beta}$. The authors estimate the search cost of FBNet-B at 216 GPU-hours, “420x” (abstract) or “421x” (§1) smaller than MnasNet’s. Elsken et al. (§5) note that multi-objective NAS is closely related to model compression, since both seek accurate but efficient networks.

Search beyond classification

NAS-FPN (Ghiasi et al., 2019) applied a reinforcement-learning controller to the cross-scale connections of a feature pyramid inside RetinaNet; the feature pyramid network article summarizes the resulting top-down and bottom-up cell. Auto-DeepLab searched its segmentation network directly on Cityscapes images in 3 P100 GPU days, and its final models were trained without ImageNet pretraining (abstract). Search has also been applied to other parts of the pipeline, such as augmentation policies (AutoAugment, in data augmentation) and activation functions (Swish, in activation function).

How to Apply It

1. Fix the protocol and the objective. Decide the training recipe used for every candidate, the validation split and the cost measure (parameters, multiply-adds or latency measured on the target device). Keep the test set out of the search.
2. Design the search space first. Start from a space that contains good known designs, as NAS-Bench-101 does with ResNet-like and Inception-like cells (§2.1), and count its size; a cell-based or block-factorized space keeps the search tractable.
3. Run random search as the baseline, with the same budget and the same performance estimator as the method being tested; random search with early stopping (see the hyperparameter tuning article) is a stronger baseline than a handful of fully trained random samples.
4. Choose the estimator for the budget. Full training is reliable but costly; lower fidelity and weight sharing are cheap but must be checked: train a sample of architectures alone and compare their ranking with the cheap estimate before trusting it.
5. Repeat the search with several seeds, retrain the final architecture several times, and report the search cost, the spread and the training recipe.

Example

The script builds a small tabular benchmark in the style of NAS-Bench-101, in which every architecture of a search space has been evaluated in advance, so that search strategies can be compared over hundreds of repetitions at no training cost. Architectures here are pipelines of fixed, untrained feature blocks with a closed-form ridge-regression head, so every evaluation is exact.

The images are $24 \times 24$ grayscale Gabor patches in four classes, the orientation of the grating (0°, 45°, 90° or 135°), with the patch shifted by up to 4 pixels in each direction, a wavelength of 4.5 to 5.5 pixels, random phase, contrast 0.4 to 1.0 and Gaussian noise of standard deviation 1.0; intensities are floating-point values around 0. There are three training sets of 400 images (the three “trials”, like NAS-Bench-101’s three training runs per architecture), 500 validation images and 1,000 test images.

An architecture chooses a pre-blur ($\sigma = 0$, 1 or 2 pixels), a filter bank (none, a Sobel pair, a Laplacian of Gaussian, Gabor filters of wavelength 5 or 10 at four orientations in quadrature pairs, or 8 fixed random $7 \times 7$ filters), a nonlinearity (none, ReLU, absolute value, square), a pooling (none, $3 \times 3$ or $6 \times 6$ average, $6 \times 6$ max, global average) and whether to append the pooled blurred image as a skip path: $3 \times 6 \times 4 \times 5 \times 2 = 720$ architectures. The head is one-vs-rest ridge regression with a fixed strength, fitted in closed form on centred features scaled to unit mean squared norm. A query of an architecture returns its validation error on one of the three training sets, drawn at random, as NAS-Bench-101 queries a random training run (Ying et al., §2.6). A search returns the architecture with the lowest validation error it has seen, which is scored by its test error averaged over the three training sets and by its percentile, the share of the 720 architectures with a strictly lower mean validation error (0 is the best); the top set is the architectures tied at the lowest mean validation error.

Three strategies are compared over 500 runs each, at budgets of 10 to 200 queries: random search, regularized evolution as in Real et al.‘s Algorithm 1 ($P = 20$, $S = 5$, identity mutation with probability 0.05, otherwise one of the five choices changed to a different value), and non-aging evolution as Real et al. define it (removing the worst model of the sample instead of the oldest). The script checks the simulated random search against its exact expectation, computed from the table, and runs a noise-free version (queries return the mean over the three training sets) with 720 queries to check that every method can reach the top set. A proxy evaluation trains the head on only the first 40 or 100 images of each training set; its rank correlation with the full evaluation is the same kind of measure as the early-rung rank correlation in the hyperparameter tuning example, here over discrete architectures and with fewer images instead of fewer iterations. Part B computes the sizes of published search spaces from the papers’ descriptions. The script uses NumPy 2.3.5 and SciPy 1.16.3, takes about 40 seconds to 2 minutes depending on the load of the shared machine (approximate), and two runs give identical output.

import os
os.environ["OMP_NUM_THREADS"] = "1"                 # identical sums on every run
import itertools, math
import numpy as np
from scipy.ndimage import gaussian_filter, gaussian_laplace
from scipy.signal import fftconvolve
from scipy.stats import spearmanr

# ---------- data: four orientations of a Gabor patch in noise, 24 x 24 ----------
S, NTR, NVA, NTE, TRIALS = 24, 400, 500, 1000, 3
yy, xx = np.mgrid[0:S, 0:S].astype(float)

def make(n, rng):
    y = np.arange(n) % 4                       # classes cycle, so every prefix is balanced
    X = np.empty((n, S, S)); info = []
    for i in range(n):
        th = y[i] * np.pi / 4
        dy, dx = rng.integers(-4, 5, size=2)
        lam, ph, c = rng.uniform(4.5, 5.5), rng.uniform(0, 2 * np.pi), rng.uniform(0.4, 1.0)
        u = (xx - 11.5 - dx) * np.cos(th) + (yy - 11.5 - dy) * np.sin(th)
        env = np.exp(-((xx - 11.5 - dx) ** 2 + (yy - 11.5 - dy) ** 2) / (2 * 4.0 ** 2))
        X[i] = c * env * np.cos(2 * np.pi * u / lam + ph)
        info.append((dy, dx, lam, c))
    return X + 1.0 * rng.standard_normal(X.shape), y, np.array(info)

rng = np.random.default_rng(0)
tr = [make(NTR, rng) for _ in range(TRIALS)]   # three training sets = three "training runs"
Xva, yva, iva = make(NVA, rng); Xte, yte, ite = make(NTE, rng)
X = np.concatenate([t[0] for t in tr] + [Xva, Xte])
inf = np.vstack([t[2] for t in tr] + [iva, ite])
print("data: class counts", np.bincount(tr[0][1]), np.bincount(yva), np.bincount(yte),
      f"| shift {inf[:,:2].min():.0f}..{inf[:,:2].max():.0f}, wavelength {inf[:,2].min():.2f}..{inf[:,2].max():.2f},"
      f" contrast {inf[:,3].min():.2f}..{inf[:,3].max():.2f}")

# ---------- the search space: blur x filter bank x nonlinearity x pooling x skip ----------
def gabor(lam):
    k = np.arange(-7, 8.0); Y, Xg = np.meshgrid(k, k, indexing="ij"); out = []
    for t in range(4):
        u = Xg * np.cos(t * np.pi / 4) + Y * np.sin(t * np.pi / 4)
        for f in (np.cos, np.sin):
            w = np.exp(-(Xg**2 + Y**2) / (2 * (0.6 * lam) ** 2)) * f(2 * np.pi * u / lam)
            w -= w.mean(); out.append(w / np.linalg.norm(w))
    return out

d = np.zeros((15, 15)); d[7, 7] = 1; lg = gaussian_laplace(d, 1.5)
rf = np.random.default_rng(123).standard_normal((8, 7, 7))
rf -= rf.mean(axis=(1, 2), keepdims=True); rf /= np.linalg.norm(rf, axis=(1, 2), keepdims=True)
sob = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]) / 4.0
BANKS = {"identity": [np.ones((1, 1))], "sobel": [sob, sob.T], "LoG": [lg / np.linalg.norm(lg)],
         "gabor5": gabor(5.0), "gabor10": gabor(10.0), "random": list(rf)}
BLURS = [0.0, 1.0, 2.0]
NONLIN = {"none": lambda z: z, "relu": lambda z: np.maximum(z, 0), "abs": np.abs, "square": np.square}
POOLS = ["none", "avg3", "avg6", "max6", "global"]
CHOICES = [list(range(len(BLURS))), list(BANKS), list(NONLIN), POOLS, [False, True]]
SPACE = list(itertools.product(*CHOICES)); A = len(SPACE); pos = {a: i for i, a in enumerate(SPACE)}
print(f"space: {A} architectures = " + " x ".join(str(len(c)) for c in CHOICES))

def pool(Z, p):                                  # Z: (images, channels, S, S) -> (images, features)
    n, C = Z.shape[:2]
    if p == "none": return Z.reshape(n, -1)
    if p == "global": return Z.mean(axis=(2, 3))
    k = 3 if p == "avg3" else 6
    Z = Z.reshape(n, C, S // k, k, S // k, k)
    return (Z.mean(axis=(3, 5)) if p.startswith("avg") else Z.max(axis=(3, 5))).reshape(n, -1)

def ridge_errors(F, itr, ytr, evals, lam=1e-2):  # closed-form ridge head, one-vs-rest, dual form
    Atr = F[itr]; mu = Atr.mean(0); Atr = Atr - mu
    sc = np.sqrt((Atr**2).sum(1).mean()); Atr = Atr / sc
    Y = -np.ones((len(itr), 4)); Y[np.arange(len(itr)), ytr] = 1; ym = Y.mean(0)
    W = Atr.T @ np.linalg.solve(Atr @ Atr.T + len(itr) * lam * np.eye(len(itr)), Y - ym)
    return [np.mean(np.argmax((F[ix] - mu) / sc @ W + ym, 1) != yv) for ix, yv in evals]

off = np.cumsum([0] + [NTR] * TRIALS + [NVA, NTE])
itr = [np.arange(off[t], off[t] + NTR) for t in range(TRIALS)]
ev = [(np.arange(off[3], off[4]), yva), (np.arange(off[4], off[5]), yte)]
PROXY = [40, 100]
VAL, TEST, PVAL = np.zeros((A, TRIALS)), np.zeros((A, TRIALS)), np.zeros((A, len(PROXY), TRIALS))
for b, s in enumerate(BLURS):
    Xb = X if s == 0 else gaussian_filter(X, (0, s, s), mode="nearest")
    for bank, ws in BANKS.items():
        conv = np.stack([fftconvolve(Xb, w[None], mode="same", axes=(1, 2)) for w in ws], 1)
        for nl, f in NONLIN.items():
            Z = f(conv)
            for p in POOLS:
                Fp = pool(Z, p)
                for sk in (False, True):
                    F = np.hstack([Fp, pool(Xb[:, None], p)]) if sk else Fp
                    i = pos[(b, bank, nl, p, sk)]
                    for t in range(TRIALS):
                        VAL[i, t], TEST[i, t] = ridge_errors(F, itr[t], tr[t][1], ev)
                        for j, n in enumerate(PROXY):
                            PVAL[i, j, t] = ridge_errors(F, itr[t][:n], tr[t][1][:n], ev[:1])[0]

mv, mt = VAL.mean(1), TEST.mean(1)
pct = np.array([np.mean(mv < mv[a]) for a in range(A)])     # share of architectures strictly better
top = np.flatnonzero(mv == mv.min())
name = lambda a: f"blur {BLURS[SPACE[a][0]]:.0f}, {SPACE[a][1]}, {SPACE[a][2]}, pool {SPACE[a][3]}, skip {'yes' if SPACE[a][4] else 'no'}"
print(f"\ntable: best mean val error {mv.min():.3f} ({len(top)} tied); top 3:")
for a in np.argsort(mv, kind="stable")[:3]:
    print(f"   {name(a):48s} val {mv[a]:.3f}  test {mt[a]:.3f}")
raw = pos[(0, "identity", "none", "none", False)]
print(f"median {np.median(mv):.3f}; raw pixels, linear {mv[raw]:.3f}; within 0.05 of best: {np.mean(mv <= mv.min() + 0.05):.3f}")
print("best per filter bank:", "  ".join(f"{bk} {min(mv[i] for i, a in enumerate(SPACE) if a[1] == bk):.3f}" for bk in BANKS))
ok1 = mv.min() <= 0.25 and mv[raw] >= 0.65 and all(np.all(np.bincount(y) == len(y) // 4) for y in (tr[0][1], yva, yte))
print("check B1 (learnable, not trivial, balanced):", ok1)

# ---------- proxy: fewer training images ----------
print("\nproxy vs full evaluation (720 architectures, means over the 3 training sets)")
for j, n in enumerate(PROXY):
    pm = PVAL[:, j].mean(1); first = np.argmin(pm)
    print(f"   n = {n:3d}: Spearman {spearmanr(pm, mv).correlation:.3f}; proxy's first choice is at the "
          f"{100*pct[first]:.1f}th percentile of the full evaluation")
print(f"   reference, two full training sets (trial 0 vs 1): Spearman {spearmanr(VAL[:,0], VAL[:,1]).correlation:.3f}")

# ---------- search strategies on the table ----------
def query(a, rg, noisy):
    return VAL[a, rg.integers(TRIALS)] if noisy else mv[a]

def mutate(a, rg):
    arch = list(SPACE[a])
    if rg.random() < 0.05: return a                          # identity mutation
    d = rg.integers(len(CHOICES)); arch[d] = rg.choice([c for c in CHOICES[d] if c != arch[d]])
    return pos[tuple(arch)]

def random_search(budget, rg, noisy=True, marks=()):
    best, out = (np.inf, None), {}
    for q in range(1, budget + 1):
        a = int(rg.integers(A)); v = query(a, rg, noisy)
        if v < best[0]: best = (v, a)
        if q in marks: out[q] = best[1]
    return out

def evolution(budget, rg, aging, noisy=True, marks=(), P=20, Sm=5):
    pop, best, out = [], (np.inf, None), {}               # pop: list of (val, arch), oldest first
    for q in range(1, budget + 1):
        if q <= P:
            a = int(rg.integers(A))
        else:
            idx = rg.integers(len(pop), size=Sm)
            par = min(idx, key=lambda i: (pop[i][0], list(idx).index(i)))
            a = mutate(pop[par][1], rg)
        v = query(a, rg, noisy); pop.append((v, a))
        if v < best[0]: best = (v, a)
        if q > P:
            if aging: pop.pop(0)                           # remove the oldest
            else:                                          # remove the worst of the sample
                worst = max(idx, key=lambda i: (pop[i][0], -list(idx).index(i)))
                pop.pop(worst)
        if q in marks: out[q] = best[1]
    return out

K, RUNS = (10, 20, 50, 100, 200), 500
methods = {"random search": lambda rg, **kw: random_search(200, rg, marks=K, **kw),
           "regularized evolution": lambda rg, **kw: evolution(200, rg, True, marks=K, **kw),
           "non-aging evolution": lambda rg, **kw: evolution(200, rg, False, marks=K, **kw)}
res = {m: [f(np.random.default_rng(seed0 + r)) for r in range(RUNS)]
       for (m, f), seed0 in zip(methods.items(), (10_000, 20_000, 30_000))}

cv, ct, ca, cp = VAL.ravel(), np.repeat(mt, TRIALS), np.repeat(np.arange(A), TRIALS), np.repeat(pct, TRIALS)
def exact_rs(k):                                 # exact expectation of random search from the table
    e, M = 0.0, len(cv)
    for v in np.unique(cv):
        lo, g = np.sum(cv < v), cv == v
        e += (((M - lo) / M) ** k - ((M - lo - g.sum()) / M) ** k) * ct[g].mean()
    return e

print(f"\nsearch on the table, {RUNS} runs each (P = 20, S = 5); returned architecture scored by its mean test error")
print("queries  method                   test error (s.e.)   percentile   top set")
ok2 = True
for k in K:
    for m in methods:
        a = np.array([r[k] for r in res[m]]); te = mt[a]
        print(f"{k:5d}    {m:22s}   {te.mean():.4f} ({te.std(ddof=1)/np.sqrt(RUNS):.4f})    {100*pct[a].mean():5.2f}      {np.isin(a, top).mean():.3f}")
        if m == "random search":
            ex = exact_rs(k); ok2 &= abs(te.mean() - ex) <= 3 * te.std(ddof=1) / np.sqrt(RUNS)
    print(f"         random search, exact     {exact_rs(k):.4f}")
print("check B2 (random-search simulation matches exact expectation):", ok2)

print("\nnoise-free queries (mean validation error), 720 queries, share of runs that found the top set:")
nf = {}
for (m, seed0), f in zip((("random search", 40_000), ("regularized evolution", 50_000), ("non-aging evolution", 60_000)),
                         (lambda rg: random_search(720, rg, False, (720,)), lambda rg: evolution(720, rg, True, False, (720,)),
                          lambda rg: evolution(720, rg, False, False, (720,)))):
    nf[m] = np.mean([f(np.random.default_rng(seed0 + r))[720] in top for r in range(RUNS)])
p_rs = 1 - (1 - len(top) / A) ** 720
print("   " + "; ".join(f"{m} {v:.3f}" for m, v in nf.items()) + f"; random search exact {p_rs:.3f}")
print("check B3 (reachability):", abs(nf["random search"] - p_rs) <= 3 * np.sqrt(p_rs * (1 - p_rs) / RUNS)
      and nf["regularized evolution"] > 0 and nf["non-aging evolution"] > 0)

# ---------- Part B: search-space sizes from the papers' descriptions ----------
print("\nPart B: search-space sizes")
nas_cell = math.prod((b + 1) ** 2 * 13**2 * 2 for b in range(1, 6)); nas_cell_sum = math.prod((b + 1) ** 2 * 13**2 for b in range(1, 6))
print(f"NASNet cell, 5 blocks: {nas_cell:.3e} with add/concat, {nas_cell_sum:.3e} sum only; pair of cells {nas_cell**2:.3e} / {nas_cell_sum**2:.3e}"
      f"; categorical dimensions 2 x 5 x (2 inputs + 2 ops) = {2*5*4} (Elsken et al.: 40)")
print(f"ENAS recurrent cell, N = 12: 4^N (N-1)! = {4**12 * math.factorial(11):.2e} (paper: approximately 1e14)")
print(f"ENAS macro, L = 12: 6^L 2^(L(L-1)/2) = {6**12 * 2**66:.2e} (paper: 1.6e29)")
ordered = math.prod((5 * (i - 1)) ** 2 for i in range(3, 8)); unordered = math.prod((5*(i-1)) * (5*(i-1) + 1) // 2 for i in range(3, 8))
print(f"ENAS micro, B = 7: paper's (5 (B-2)!)^4 = {(5 * math.factorial(5))**4:.2e} (paper: 1.3e11); from the description, "
      f"pair of cells {ordered**2:.2e} (ordered choices) or {unordered**2:.2e} (unordered)")
print(f"MnasNet: 432^5 = {432**5:.2e} (paper: about 1e13), 432^15 = {432**15:.2e} (paper: 1e39); "
      f"listed options per block 3 x 2 x 2 x 3 x 3 x 3 = {3*2*2*3*3*3} (paper's S: 432)")
print(f"FBNet: searchable layers 1+4+4+4+4+4+1 = {1+4*5+1}; 9^22 = {9**22:.2e} (paper: approximately 1e21); "
      f"VGG16 example (3 x 5)^16 = {15**16:.2e} (paper: approximately 6e18)")
print(f"NAS-Bench-101: 2^(7*6/2) x 3^5 = {2**21 * 3**5:,} (paper: approximately 510M)")

data: class counts [100 100 100 100] [125 125 125 125] [250 250 250 250] | shift -4..4, wavelength 4.50..5.50, contrast 0.40..1.00
space: 720 architectures = 3 x 6 x 4 x 5 x 2

table: best mean val error 0.189 (2 tied); top 3:
   blur 0, gabor5, square, pool none, skip no       val 0.189  test 0.167
   blur 0, gabor5, square, pool none, skip yes      val 0.189  test 0.164
   blur 0, gabor5, square, pool avg3, skip no       val 0.193  test 0.169
median 0.739; raw pixels, linear 0.728; within 0.05 of best: 0.068
best per filter bank: identity 0.719  sobel 0.569  LoG 0.717  gabor5 0.189  gabor10 0.719  random 0.433
check B1 (learnable, not trivial, balanced): True

proxy vs full evaluation (720 architectures, means over the 3 training sets)
   n =  40: Spearman 0.665; proxy's first choice is at the 0.0th percentile of the full evaluation
   n = 100: Spearman 0.817; proxy's first choice is at the 0.3th percentile of the full evaluation
   reference, two full training sets (trial 0 vs 1): Spearman 0.715

search on the table, 500 runs each (P = 20, S = 5); returned architecture scored by its mean test error
queries  method                   test error (s.e.)   percentile   top set
   10    random search            0.3463 (0.0085)    10.10      0.018
   10    regularized evolution    0.3565 (0.0084)    10.09      0.030
   10    non-aging evolution      0.3298 (0.0080)     8.95      0.026
         random search, exact     0.3382
   20    random search            0.2441 (0.0054)     4.89      0.054
   20    regularized evolution    0.2541 (0.0060)     5.38      0.062
   20    non-aging evolution      0.2423 (0.0055)     4.81      0.054
         random search, exact     0.2411
   50    random search            0.1852 (0.0016)     2.02      0.112
   50    regularized evolution    0.1923 (0.0029)     1.80      0.246
   50    non-aging evolution      0.1868 (0.0023)     1.44      0.236
         random search, exact     0.1849
  100    random search            0.1765 (0.0005)     1.17      0.196
  100    regularized evolution    0.1743 (0.0013)     0.53      0.576
  100    non-aging evolution      0.1728 (0.0009)     0.41      0.590
         random search, exact     0.1768
  200    random search            0.1737 (0.0005)     0.71      0.302
  200    regularized evolution    0.1682 (0.0003)     0.10      0.866
  200    non-aging evolution      0.1685 (0.0003)     0.10      0.870
         random search, exact     0.1734
check B2 (random-search simulation matches exact expectation): True

noise-free queries (mean validation error), 720 queries, share of runs that found the top set:
   random search 0.842; regularized evolution 1.000; non-aging evolution 1.000; random search exact 0.865
check B3 (reachability): True

Part B: search-space sizes
NASNet cell, 5 blocks: 2.287e+18 with add/concat, 7.147e+16 sum only; pair of cells 5.230e+36 / 5.107e+33; categorical dimensions 2 x 5 x (2 inputs + 2 ops) = 40 (Elsken et al.: 40)
ENAS recurrent cell, N = 12: 4^N (N-1)! = 6.70e+14 (paper: approximately 1e14)
ENAS macro, L = 12: 6^L 2^(L(L-1)/2) = 1.61e+29 (paper: 1.6e29)
ENAS micro, B = 7: paper's (5 (B-2)!)^4 = 1.30e+11 (paper: 1.3e11); from the description, pair of cells 2.56e+25 (ordered choices) or 4.39e+22 (unordered)
MnasNet: 432^5 = 1.50e+13 (paper: about 1e13), 432^15 = 3.41e+39 (paper: 1e39); listed options per block 3 x 2 x 2 x 3 x 3 x 3 = 324 (paper's S: 432)
FBNet: searchable layers 1+4+4+4+4+4+1 = 22; 9^22 = 9.85e+20 (paper: approximately 1e21); VGG16 example (3 x 5)^16 = 6.57e+18 (paper: approximately 6e18)
NAS-Bench-101: 2^(7*6/2) x 3^5 = 509,607,936 (paper: approximately 510M)

The checks hold: the task can be learned but not with raw pixels (the linear model on pixels is close to the chance error of 0.75), the simulated random search matches its exact expectation at every budget, and every strategy can find the top set once queries are noise-free.

The space. Most of the 720 architectures are useless: the median validation error is near chance and only 6.8% come within 0.05 of the best. The filter bank decides: only the Gabor bank whose wavelength matches the gratings gets below 0.4, and the top three architectures all use it with a squaring nonlinearity. In our reading, this follows from the construction: the class is the orientation of a grating with random phase and position, and filters tuned to its wavelength and orientation, followed by a nonlinearity or pooling that discards phase, are what a linear head needs. The two tied architectures differ only in the skip path. This is a needle-in-a-haystack space with a strong structure along one choice.

Proxies. Training the head on 40 or 100 images instead of 400 keeps much of the ranking (Spearman 0.665 and 0.817), and both proxies put an architecture at or near the top of the full evaluation first. The planned noise reference, the correlation between two single training sets, is not comparable with these numbers: it correlates single training sets, while the proxies correlate means over three, so it averages away less noise. That was a design error in the plan, and the hypothesis that both proxies would correlate less with the full evaluation than the reference failed in part (the 100-image proxy came out above it); no conclusion is drawn from comparing them.

Search. Up to 20 queries, the population size, both evolutionary methods have only drawn their random initial population, so all three strategies are random search run with different seeds, and their differences are sampling noise. From 50 queries on, both evolutionary methods return the top set far more often than random search (0.25 against 0.11 at 50 queries, 0.87 against 0.30 at 200) and better-ranked architectures. In test error the result is mixed: at 50 queries regularized evolution’s mean test error was worse than random search’s, with a standard error almost twice as large, so its runs are more spread; at 100 and 200 queries both evolutionary methods are better. The hypothesis written into the plan before the pilot, that regularized evolution would beat random search at 50, 100 and 200 queries, held at 100 and 200 and failed at 50 for test error. Aging did not help: non-aging evolution was equal to or slightly better than regularized evolution at every budget from 50 queries on (at 200 queries the two are within one standard error), so the planning hypothesis that aging would do at least as well, Real et al.’s conjecture for noisy evaluation, was not supported in this space. The test-error differences are small because many architectures near the top have similar errors, while the percentile and the top-set share separate the methods more clearly.

Search-space sizes. NASNet states no size for its space. From its description, one cell has about $2.3 \times 10^{18}$ variants with the addition-or-concatenation choice and $7.1 \times 10^{16}$ with sums only, which Elsken et al. say the experiments always used; the 40 categorical dimensions agree with Elsken et al.’s count. ENAS’s macro space, MnasNet’s $S^B$ and $S^{BN}$, FBNet’s $9^{22}$ over 22 searchable layers, its VGG example and NAS-Bench-101’s encoding all agree with the papers. Three counts do not, as we read the descriptions. ENAS’s recurrent-cell count is $6.7 \times 10^{14}$, closer to $10^{15}$ than to the stated “approximately $10^{14}$”. ENAS’s micro-space formula $(5 \times (B-2)!)^4$ reproduces the stated $1.3 \times 10^{11}$, but the description it is meant to count (two inputs from the $i-1$ previous nodes and two of five operations at each node) gives $2.6 \times 10^{25}$ for the pair of cells if the choices are ordered and $4.4 \times 10^{22}$ if they are not. MnasNet’s listed per-block options multiply to 324, while the paper’s typical case uses $S = 432$ without saying how it arises (its Figure 4 ends some option lists with an ellipsis).

Variants

• Macro, cell-based, hierarchical and layer-wise spaces, as above; MnasNet’s factorized space sits between the cell-based and the fully per-layer space.
• Multi-objective search, which optimizes accuracy together with latency, FLOPs or parameters, through a reward (MnasNet), a differentiable loss term (FBNet) or by sampling parents from a Pareto front, as Elsken et al. (§3) describe for one evolutionary method.
• Network morphisms, which grow or shrink a trained network by function-preserving changes and continue training it (Elsken et al., §4).
• Weight-sharing searches with a controller (ENAS), with a fixed sampling distribution (Bender et al.; Li and Talwalkar’s random search with weight sharing) or with a relaxed distribution (DARTS, SNAS, ProxylessNAS and FBNet, as described above).
• Joint search of architectures and training hyperparameters; Elsken et al. (§5) suggest evaluating NAS inside full AutoML systems that also tune hyperparameters and augmentation, and an earlier evolutionary search by Real et al. (2017), as Elsken et al. describe it, let mutations change the learning rate.
• Searching for other components: feature pyramids (NAS-FPN), segmentation networks (Auto-DeepLab), augmentation policies and activation functions.

Best Practices

• Compare search strategies in the same space with the same training recipe. Real et al.’s case study and NAS-Bench-101 compare methods in one space under one pipeline; comparisons across papers mix spaces, budgets and recipes (Elsken et al., §5).
• Report the cost of the whole search, including the final training of several candidates and any preliminary tuning, in hardware-specific units; Real et al. excluded their tuning runs from their plots, a choice made in advance (Methods Details).
• Use a strong random baseline, with as many evaluated architectures as the budget allows.
• Validate cheap estimates against full training on a sample of architectures before relying on them, and increase fidelity gradually.
• Repeat searches and report the spread over independent runs, as Li and Talwalkar did for random search with weight sharing.
• Release search and evaluation code with seeds; Li and Talwalkar found that none of 12 NAS papers at NeurIPS, ICML and ICLR since 2018 was exactly reproducible (§1, Table 5).
• Use a tabular benchmark such as NAS-Bench-101 to develop and compare search algorithms cheaply, keeping its test accuracies for offline evaluation only (Ying et al., §2.5).

Pitfalls

• Crediting the search with the recipe. A searched architecture trained with Cutout, ScheduledDropPath and a cosine schedule against a baseline trained without them measures the recipe as well as the architecture (Elsken et al., §5).
• Weak baselines: one or a few random architectures, or random search without early stopping, against a tuned search method (Li and Talwalkar, §2.1).
• Unstable selection. Choosing the final architecture after short training can pick a different winner than longer training would (Li and Talwalkar, §4.1.3).
• Trusting low-fidelity or one-shot rankings blindly. The one-shot model underestimates the best architectures and its correlation with stand-alone accuracy is not guaranteed (Elsken et al., §4); in NAS-Bench-101 the low rank correlation between short and long training budgets is the authors’ explanation for why Hyperband and BOHB did not give their usual speed-ups there (§4.1).
• Counting FLOPs as speed. Similar multiply-adds can mean very different latencies (MnasNet, §1), and searched cells with many branches can be slow on mobile CPUs (FBNet, §3.1).
• Single runs. A search reported from one run gives an extreme value of a noisy procedure (Li and Talwalkar, §2.3).
• Search on the test set, directly or by choosing among final architectures by test error.

Evidence and Impact

Accuracy. NASNet-A, searched on CIFAR-10 and transferred, reached 82.7% top-1 and 96.2% top-5 on ImageNet with 88.9M parameters and 23.8B multiply-adds at $331 \times 331$, the same top-1 as SENet with 145.8M parameters and 42.3B multiply-adds (Table 2), and 2.40% CIFAR-10 test error with Cutout, averaged over five runs (Table 1). Its features in Faster R-CNN gave 43.1% mAP on COCO test-dev at $1200 \times 1200$ (Table 4). AmoebaNet-A matched it at similar size (82.8% top-1 with 86.7M parameters) and reached 83.9% top-1 with 469M parameters (Real et al., Table 2), the first evolved model to surpass hand designs on ImageNet by the authors’ account.

Search against search. Elsken et al. (§3) put the margin over random search in perspective: in Real et al.’s case study random search reached about 4% CIFAR-10 test error against about 3.5% for reinforcement learning and evolution after model augmentation, and in Liu et al.’s hierarchical search 3.9% against 3.75%, and 21.0% against 20.3% ImageNet top-1 validation error. On NAS-Bench-101, regularized evolution, BOHB and SMAC performed best, overtaking random search after roughly 50,000 TPU seconds (about 25 architectures) and reaching random search’s final performance about five times faster; reinforcement learning converged much more slowly (§4.1, Figure 7). On a smaller version of the space with at most 6 vertices and on the full benchmark, the order was the same: random search, then non-aging evolution, then regularized evolution (§4.2).

Cost. Reported search costs fell by orders of magnitude, though they are measured on different hardware and are not directly comparable:

Search	Reported cost
Zoph and Le (2017), as NASNet reports it	800 GPUs for 28 days
NASNet (Zoph et al., 2018)	500 GPUs for 4 days (§4)
AmoebaNet (Real et al., 2019)	450 K40 GPUs for about 7 days
MnasNet (Tan et al., 2019)	4.5 days on 64 TPUv2 devices
Auto-DeepLab (Liu et al., 2019)	3 P100 GPU days
ENAS cells (Pham et al., 2018)	0.45 days on one GPU
FBNet-B (Wu et al., 2019)	216 GPU-hours

NASNet’s own figures are not consistent. Section 4 gives a pool of 500 GPUs while Appendix A.3 gives 450 concurrent workers, the figure ENAS and Bender et al. quote; and footnote 1 calls its totals “22,400 GPU-hours” and “2,000 GPU-hours”, although $800 \times 28 = 22{,}400$ and $500 \times 4 = 2{,}000$ are GPU-days. ENAS restates NASNet’s search as 32,400–43,200 GPU-hours (§1), and Li and Talwalkar’s table lists NASNet-A at 2,000 GPU days (Table 1).

Baselines and reproducibility. Li and Talwalkar’s random search with early stopping reached 3.03 ± 0.13% CIFAR-10 test error, comparable to first-order DARTS, and random search with weight sharing 2.85 ± 0.08%, comparable to SNAS and DARTS (Table 1). Their independent DARTS run reached 2.78 ± 0.12% against the published 2.76 ± 0.09%, but across six independent runs of their own method the averages ranged from 2.85% to 3.00% (Table 4). They concluded that the gap between leading NAS methods and standard hyperparameter optimization was much smaller than earlier random baselines had suggested. NAS-Bench-101 answered the cost of comparison: 423k unique cells, each trained three times at four budgets, over 5 million trained models and more than 100 TPU years of computation (§1, §2.4). Its best cell reached 94.32% mean CIFAR-10 test accuracy, against 93.12% and 92.95% for the ResNet-like and Inception-like cells in the same pipeline (§3.1).

In our reading, NAS changed how efficient vision networks are designed: the mobile baselines that later families were scaled or adapted from, MnasNet, MobileNetV3 and EfficientNet-B0, came from latency- or FLOPS-aware search. Elsken et al. (§5) also note what it has not yet done: it gives little insight into why the architectures it finds work, or into how similar the architectures found in independent runs would be.