Embodied AI

Embodied AI studies agents that have a body, physical or simulated, perceive their surroundings through sensors carried on that body, and act to change what they will perceive next. The field draws on computer vision, language, reinforcement learning, graphics and robotics. Its working tools are simulators fast enough to give agents millions or billions of steps of experience, standard tasks with fixed evaluation protocols, and, since about 2022, large models pretrained on internet data and adapted to control robots.

This article is the overview of the subject for the robotics category. Robot perception is the hub for how robots sense and estimate their state, and visual navigation owns navigation tasks, their methods and the SPL metric; this article links to both rather than repeating them. A small example shows the idea behind domain randomization on rendered images.

Definition

Several definitions are in use, and they differ in emphasis. Batra et al. (2020) define embodied AI as the study and development of intelligent systems with a physical or virtual embodiment. The BEHAVIOR benchmark’s authors add that such agents perceive, reason and interact with the environment with the capabilities and limitations of a physical body (Srivastava et al., 2021). Habitat’s authors define it by contrast with what they call “internet AI”: pattern recognition in images, videos and text from datasets curated from the internet, against action by an embodied agent in an environment, which brings to the fore active perception, long-term planning, learning from interaction and dialog grounded in an environment (Savva et al., 2019). Duan et al. (2022) frame the same shift from learning on curated datasets to learning through interaction with an egocentric view.

Three properties follow from these definitions and separate embodied problems from the image and video tasks elsewhere on this site:

• Egocentric, partial observation. The agent sees through its own cameras and depth sensors from where its body is, never the whole scene.
• Closed loop. The agent’s actions decide what it sees next. The data distribution depends on the policy that collects it.
• Consequences. Success is judged by the state of the world or the agent at the end of an episode, not by a label for an input.

“Embodied” also has an older meaning in cognitive science, which several of these papers cite as motivation. Habitat and Embodied Question Answering both open with Smith and Gasser’s statement of the embodiment hypothesis, that intelligence emerges in an agent’s interaction with an environment through sensorimotor activity (Savva et al., 2019; Das et al., 2018). In current machine learning usage the term is broader and more practical: an agent trained in a simulator counts as embodied when it receives egocentric observations and acts through a modelled body, whatever one believes about the hypothesis.

History

Turing’s two paths. Brooks (1991) reads Alan Turing’s 1950 paper on machine intelligence as already proposing two routes: an unembodied one, programming intellectual activities such as chess, and an embodied one, giving a computer the best sense organs available and teaching it to understand and speak English. In Brooks’s account, artificial intelligence followed the first and all but ignored the second.

Early robots. SRI’s Shakey (1966–1972) and Moravec’s Stanford Cart (thesis 1980) perceived and acted in the real world, with perception organized as modules that built a world model for a planner; the robot perception article describes both.

Situatedness and embodiment. Brooks’s 1991 IJCAI paper made embodiment one of two cornerstones of an approach to AI that had come out of work with mobile robots. He defined situatedness as dealing with the here and now of the world rather than with abstract descriptions, and embodiment as having a body and experiencing the world directly, so that actions are part of a dynamic with the world and give immediate feedback on the robot’s own sensations. He argued that intelligence also comes from the situation, the sensors and the physical coupling with the world, not only from the computation (Brooks, 1991).

Simulators and benchmarks (2017–2019). Learning-based embodied AI grew around shared simulators. AI2-THOR, first released on arXiv in December 2017, offered near-photorealistic interactive indoor scenes in a game engine (Kolve et al., 2017); Gibson and Habitat followed in 2018 and 2019, and the visual navigation article describes them. The same period defined the first embodied tasks with protocols: navigation with its evaluation recommendations of 2018, vision-and-language navigation (2018), and Embodied Question Answering, posted in November 2017 and published at CVPR 2018 (Das et al., 2018). Duan et al. (2022) surveyed nine simulators released in the four preceding years and selected them from those used in the challenges of the annual Embodied AI workshop at CVPR.

Interaction and rearrangement (2020–2021). As navigation matured, attention moved to agents that change their environment. Batra et al. (2020) proposed rearrangement as a canonical task, and a new generation of simulators added physics, articulated objects and object states: iGibson 1.0 and 2.0, Habitat 2.0 and the BEHAVIOR benchmark, all of 2021, and ManiSkill for manipulation. RoboTHOR (2020) paired simulated apartments with physical copies to measure the gap between them (Deitke et al., 2020).

Large pretrained models (2022 onward). From 2022, robot learning began to borrow models and data scale from vision and language. SayCan (2022) used a large language model to choose among a robot’s learned skills (Ahn et al., 2022); RT-1 (December 2022) trained one transformer policy on about 130,000 real demonstrations (Brohan et al., 2023a); RT-2 (July 2023) fine-tuned vision-language models to output robot actions and named them vision-language-action models (Brohan et al., 2023b). Open X-Embodiment (2023), OpenVLA (2024) and π0 (2024) followed. These are recent works, and their claims are best read as the authors’ reports of their own experiments.

Intuition

A model trained on a fixed dataset of photographs sees whatever the photographers chose. An embodied agent chooses its own data: if it turns left, the next image shows what is to the left. Two consequences follow, and they shape the methods.

First, perception can be active. An agent that cannot tell whether a mug is behind the sofa can walk around the sofa. Embodied Question Answering makes this the task: to answer “What color is the car?”, the agent must first find the car (Das et al., 2018).

Second, errors compound. A policy trained to imitate an expert sees the states the expert visited. Once it makes a mistake it drifts into states the expert never visited, where its predictions are untrained, and mistakes breed further mistakes. Ross, Gordon and Bagnell (2011), citing earlier work by Ross and Bagnell, state the consequence for a learner that errs with probability $\epsilon$ on the expert’s state distribution: over $T$ steps, on the states it reaches itself, it can make as many as $T^2\epsilon$ mistakes in expectation, where errors that did not compound would give $T\epsilon$ (our gloss); methods that let the learner influence the states it is trained on bring the guarantee back to linear, or nearly linear, in $T$ and $\epsilon$. The same coupling is why simulators matter: a simulated agent can try, fail and be reset millions of times, which a real robot cannot.

Formal Definition

Most embodied tasks fit a partially observable Markov decision process, the formalism BEHAVIOR, for example, adopts for its activities (Srivastava et al., 2021). The environment has a state $s_t$ (the agent’s pose and the poses and states of objects), which the agent never sees directly. At each step it receives an observation $o_t$ (images, depth, sometimes its own joint positions) and a goal specification $g$, and chooses an action $a_t$ from a policy conditioned on its history,

$$a_t \sim \pi(a \mid o_{1:t}, a_{1:t-1}, g),$$

after which the environment moves to $s_{t+1}$. An episode ends when the agent declares it has finished or exhausts a budget of steps, and is scored by a function of the final state and the trajectory: success, the fraction of goal conditions met, path length, time or energy. The parts vary by task. In point-goal navigation $g$ is a vector to the target and the actions are a few discrete moves; in rearrangement $g$ describes a goal state of several objects and the actions move an arm and a gripper; in instruction following $g$ is a sentence. A reward $r_t$ is added when the policy is trained by reinforcement learning, but evaluation uses the task’s own score, which need not equal the reward.

Simulators as Research Platforms

Training in the physical world, Habitat’s authors argue, is slow, since the real world runs no faster than real time and cannot be parallelized, and it is also dangerous, resource intensive, hard to control and hard to reproduce (Savva et al., 2019). Simulators address all five, at the price of a gap to reality discussed below. They differ in what they simulate, and the choice limits which tasks can be studied.

Simulator	First paper	What it adds
AI2-THOR	Kolve et al., arXiv 2017	Interactive indoor scenes in the Unity engine; object state changes (opening, slicing, toggling); by the paper’s 2022 revision (arXiv v4), 120 single-room scenes, 89 RoboTHOR apartments, 10 ArchitecTHOR houses and procedural generation
Habitat	Savva et al., ICCV 2019	Fast rendering of scanned buildings; a simulator (Habitat-Sim) separate from a library for tasks and training (Habitat-API)
RoboTHOR	Deitke et al., CVPR 2020	89 simulated apartments, of which 14 test scenes also exist as physical rooms; remote testing on a real robot
iGibson 1.0	Shen et al., IROS 2021	15 fully interactive replicas of real homes with 108 rooms; randomization of object materials and shapes; motion planners
iGibson 2.0	Li et al., CoRL 2021	Object states such as temperature, wetness, cleanliness and sliced; predicate logic over states; virtual-reality demonstrations
Habitat 2.0	Szot et al., NeurIPS 2021	Rigid-body and articulated physics for mobile manipulators; ReplicaCAD apartments
BEHAVIOR	Srivastava et al., CoRL 2021	100 household activities defined in a logic language and instantiated in iGibson 2.0; 500 human demonstrations in virtual reality
ManiSkill	Mu et al., NeurIPS 2021	Manipulation of articulated objects with large shape variation; about 36,000 demonstrations

The first generation rendered static scans and moved the agent through them, fast enough for the billion-step navigation experiments described in the visual navigation article.

Interaction raises the cost. Habitat 2.0 simulates a Fetch mobile manipulator in ReplicaCAD apartments, 111 layouts of one apartment with 92 artist-authored objects and cabinets and drawers that open and close. A step renders 128 × 128 RGB-D observations and simulates 1/30 s of rigid-body dynamics. In the authors’ benchmark (Table 2), with the robot moving but not touching objects, it ran at about 1,200 steps per second in one process and 25,734 on an 8-GPU node; with the robot interacting with objects, the figures were 510 and about 7,700. The introduction rounds the 8-GPU figure to 26,000 and the abstract to over 25,000; for existing simulators the authors give 10 to 400 steps per second (Szot et al., 2021). To reach that speed it deliberately leaves out deformable objects, fluids, cutting and tactile sensing. iGibson 2.0 made the opposite choice, adding object states so that tasks such as cleaning or cooking can be defined at all (Li et al., 2021). ManiSkill3, posted as a preprint in 2024 and published in 2025, runs simulation and rendering in parallel on the GPU and reports simulation plus rendering at up to 30,000 frames per second and more, depending on the task (Tao et al., 2025).

Two abstractions recur and should be noted when reading results. Some benchmarks let an agent grasp an object by pointing at it, the “magic pointer” of Habitat and AI2-THOR that Batra et al. (2020) describe, instead of simulating a gripper. And some provide action primitives that internally use privileged information such as object identities and poses; BEHAVIOR offers them for prototyping while its standard setting does not (Srivastava et al., 2021). A result obtained with such abstractions measures planning and perception, not manipulation.

Embodied Tasks

Navigation. Point-goal, object-goal, image-goal and area-goal navigation, the first tasks with shared protocols, are the subject of the visual navigation article, which covers map-based and learned approaches, the success and SPL metrics and the simulators’ role in them.

Embodied question answering. In EQA an agent is spawned at a random location in a house and asked a question such as “What color is the car?”; it must navigate to gather the information through a single RGB camera and then answer (Das et al., 2018). Its question templates test location, color, spatial prepositions, existence, counting and logical combinations; the first dataset, EQA v1, uses four of them (location, color, color in a named room and preposition), with over 5,000 questions in over 750 environments from the SUNCG collection, rendered with the House3D renderer and split so that no environment appears in two splits. The task ties the task to visual question answering with the image replaced by exploration.

Instruction following. Vision-and-language navigation, in which an agent follows directions through a building, is described in the visual navigation article. ALFRED extends instruction following to interaction: 25,743 English directives for expert demonstrations in 120 AI2-THOR scenes, with high-level goals such as “Rinse off a mug and place it in the coffee maker” and step-by-step instructions, long action sequences and irreversible state changes (Shridhar et al., 2020). A sequence-to-sequence baseline with progress monitoring completed 4.0% of tasks in test scenes seen during training and 0.4% in unseen ones, while five people controlling the agent through the same actions completed 91.0% of 100 unseen-test directives (Table 3). The authors note that the agents took more steps than the expert, sometimes because they lost track of state changes, such as heating an egg in the microwave several times, and that they generalized poorly to unseen scenes.

Rearrangement. Batra et al. (2020) proposed rearrangement as a canonical task: given a physical environment, bring it into a specified goal state. Setting a table, loading a dishwasher and tidying a room are instances, as are many earlier robotics settings. The goal can be given as object poses (GeometricGoal), as an image of the goal state (ImageGoal), in language (LanguageGoal), as a set of logical predicates such as on(plate1, table1) (PredicateGoal), or by letting the agent explore the environment in the goal state first and asking it to restore it (ExperienceGoal). The authors limit the framework to rigid and articulated objects, leaving out liquids, cloth and cutting, and recommend acting from realistic onboard sensing, moving objects by forces in a physics simulation, and testing on objects and environments never seen in training, without privileged information such as ground-truth poses or perfect localization.

Mobile manipulation. Habitat 2.0’s Home Assistant Benchmark instantiates rearrangement for a mobile manipulator: tidying a house, preparing groceries and setting a table, by picking objects from receptacles, placing them and opening drawers and fridges as needed (Szot et al., 2021). BEHAVIOR’s 100 activities span cleaning, maintenance and food preparation (Srivastava et al., 2021). Grasping and the perception that manipulation needs are covered in robotic grasping and vision-based robotic manipulation; embodied AI adds the combination with navigation over long horizons in large scenes.

Learning

Embodied agents are built in three broad ways, which are often combined: as modular pipelines of perception, mapping and planning, the classical approach that the navigation and manipulation articles describe; as policies learned end to end; and, recently, around large pretrained models. This section outlines the learning approaches.

Imitation learning

Imitation learning trains a policy from demonstrations, pairs of observations and the expert’s actions. The simplest form, behavior cloning, is supervised learning on those pairs. It needs no reward and learns quickly from good data, but it suffers from the compounding errors described above. DAgger (Ross et al., 2011) addresses them by iterating: run the current policy, ask the expert which action it would have taken in the states the policy reached, add those labels to the dataset and retrain, so that the learner is trained on the distribution of states it induces itself. Demonstrations come from human teleoperation, motion planners with privileged information (ALFRED’s demonstrations were produced by a planner with access to the full state), virtual reality (iGibson 2.0, BEHAVIOR) or scripted controllers. The large models below are trained mainly by imitating demonstrations, and the manipulation article covers visuomotor policies trained this way, including diffusion policies.

Reinforcement learning

Reinforcement learning trains a policy by trial and error to maximize a reward. It needs no demonstrations and can discover behavior an expert would not show, but it needs far more experience, which is why it is mostly done in simulation. On point-goal navigation, policies trained by reinforcement learning with billions of simulated steps reached near-perfect scores under idealized sensing (described in the visual navigation article). On long-horizon interaction the picture is different. In Habitat 2.0’s benchmark, monolithic reinforcement-learning policies learned individual skills such as picking, placing, navigating and opening drawers, but the authors could not get them to chain skills over the long tasks. A hierarchy of separately trained skills, sequenced by a perfect task planner, did much better, but suffered from “hand-off” problems, where one skill leaves the next badly set up, for example by navigating to a poor spot for grasping or opening a drawer only partly. Classical sense-plan-act pipelines matched learned policies on simple skills but were more brittle, since they must map every obstacle to plan (Szot et al., 2021). In BEHAVIOR, soft actor-critic agents with realistic actions and onboard sensing satisfied no goal condition in any of the 12 simplest activities, and with action primitives that shorten the horizon a PPO agent made progress on only one (Srivastava et al., 2021, Table 2). Started from a demonstration’s state 1 s before its successful end, agents succeeded in seven of the 12 activities in that table, although the text says most failed; started 10 s before, they succeeded in none.

Large pretrained models

Large pretrained models enter embodied systems in two roles. As planners, a language model breaks an instruction into steps that a robot’s existing skills can execute. SayCan scores each candidate skill by the language model’s estimate that it helps complete the instruction and by a learned value function’s estimate that it will succeed from the current state, so that the plan is grounded in what the robot can do; on 101 instructions for a mobile manipulator in a mock kitchen it reported a planning success rate of 84% and an execution success rate of 74%, and 81% and 60% in a real office kitchen (Ahn et al., 2022, Table 2). As policies, a pretrained vision-language model is fine-tuned to output actions directly, the vision-language-action models of the next section. Pretrained visual encoders are also used on their own as the perception part of a policy, as in RT-1, whose image encoder starts from an ImageNet-pretrained network (Brohan et al., 2023a); foundation models describes the families these encoders come from.

Vision-Language-Action Models

A vision-language-action (VLA) model takes camera images and a language instruction and outputs robot actions, using a model pretrained on internet-scale image and text data. Brohan et al. (2023b) introduced the term for RT-2, and later papers adopted it. The table lists the models named in this article, each from its own paper; the parameter counts and data are as the authors report them.

Model	First version	Built on	Actions	Robot data
RT-1	arXiv Dec 2022; RSS 2023	FiLM-conditioned EfficientNet-B3 and a decoder-only transformer; 35M parameters	11 dimensions (arm, base, mode), each in 256 bins; 3 Hz	About 130,000 episodes, over 700 tasks, 13 robots, 17 months
RT-2	arXiv Jul 2023; CoRL 2023	PaLI-X (5B and 55B) or PaLM-E (12B), co-fine-tuned with web data	Bins written as text tokens; 1–3 Hz for the 55B model	RT-1’s data
RT-1-X, RT-2-X	arXiv Oct 2023; ICRA 2024	RT-1 and RT-2 architectures	Discretized end-effector actions	Open X-Embodiment mixture
OpenVLA	arXiv Jun 2024; CoRL 2024	Llama 2 7B with DINOv2 and SigLIP image features; 7B parameters	256 bins per dimension	970,000 episodes from Open X-Embodiment
π0	arXiv Oct 2024; RSS 2025	PaliGemma (3B) plus a 300M action expert; 3.3B parameters	Continuous chunks of 50 actions by flow matching; up to 50 Hz	Over 10,000 hours from 7 robot configurations and 68 tasks, plus open datasets

RT-1 (Brohan et al., 2023a) is a robot policy rather than a VLA in the later sense, but it set the pattern: a transformer that reads a short history of images and an instruction and outputs discretized actions as tokens, trained by imitation on a large real dataset. In the authors’ evaluation it performed 97% of more than 200 seen instructions, 76% of 21 unseen ones, and 83% and 59% of tasks with distractors and new backgrounds, ahead of their baselines, Gato (reduced to a size similar to RT-1’s) and BC-Z (Table 2 of arXiv v2).

RT-2 (Brohan et al., 2023b) represents each action as a string of integers, the indices of 256 bins per dimension, and trains a vision-language model to produce them like text. The key detail, the authors report, is co-fine-tuning on robot data together with the original web vision-language data rather than on robot data alone. Over about 6,000 real evaluation trials, the RT-2 models matched RT-1 on seen tasks and averaged 62% on unseen objects, backgrounds and environments, against 32% for RT-1 and 35% for MOO in the same evaluation (Appendix Table 5 of the CoRL version). They also followed instructions that need knowledge from web data, such as placing an object on a particular number or icon. The authors list two limits: the robot acquires no new motions from web data, only new ways to use the skills in its robot data, and the largest models are expensive enough that real-time inference may become a bottleneck.

Open X-Embodiment (Open X-Embodiment Collaboration, 2024) pooled 60 existing datasets from 34 laboratories into over a million real trajectories from 22 robot embodiments, in one format. RT-1’s architecture trained on a mixture of nine embodiments’ data, RT-1-X, had a mean success rate 50% higher than either the original method of each of five small datasets or RT-1 trained on that dataset alone, and beat the original method on four of the five. On the two large datasets, Bridge and RT-1’s own, RT-1-X underfit and performed worse than RT-1. The authors write that the larger RT-2-X outperformed both baselines there, but their Table I shows it ahead of RT-1 at one of two Bridge evaluation sites (50% against 40%), level at the other (30%) and one point behind on RT-1’s tasks (91% against 92%). They read the results as transfer across robots requiring enough model capacity.

OpenVLA (Kim et al., 2024) is an open 7B-parameter VLA, built on a vision-language model that combines DINOv2 and SigLIP image features (DINO and its successors are self-supervised encoders) with the Llama 2 language model. The authors report 16.5 points higher absolute success than RT-2-X (55B) across 29 tasks on several robots with seven times fewer parameters, and that low-rank adaptation (LoRA) of the model to a new robot matched full fine-tuning. The authors released its weights and training code, citing as motivation that existing VLAs were largely closed to the public.

π0 (Black et al., 2025) replaces discretized action tokens with continuous actions. A separate “action expert” attached to a PaliGemma vision-language model generates chunks of 50 future actions by flow matching, a relative of the diffusion models that the manipulation article’s Diffusion Policy also uses, which the authors use for dexterous tasks such as folding laundry, at up to 50 Hz. They list as open questions how pretraining data should be composed and weighted, how much data a task needs to work reliably, and whether the transfer extends to driving, navigation or legged locomotion.

These models are, as of October 2026, a fast-moving research area. Their evaluations are run by their authors on their own robots and tasks, and the numbers are not directly comparable across papers. The same model can score differently in different evaluations: RT-1 performed 97% of seen instructions in its own paper and 92% in RT-2’s evaluation of seen tasks.

Sim-to-Real Transfer

The differences between a simulator and the real world, in rendering, sensor noise, physics, actuation and latency, are collectively called the reality gap (Tobin et al., 2017). A policy that exploits a simulator’s peculiarities can score well there and fail on a robot. There are three broad responses: make the simulator more faithful, for example by system identification, which tunes simulation parameters to match the physical system and which Tobin et al. call time-consuming and error-prone; make the policy insensitive to the differences; or train on real data. Driving research met the problem early: ALVINN (1988) was trained on simulated road images, as the autonomous-driving perception article describes.

Domain randomization. Tobin et al. (2017) proposed randomizing the simulator’s rendering so widely that “the real world may appear to the model as just another variation”. They trained an object detector for a robot arm on images rendered in MuJoCo with random textures of three kinds (a random color, a gradient between two random colors, a checker pattern), random lighting, camera pose and field of view, distractor objects and noise, deliberately not photorealistic. On real images it located objects to within 1.5 cm on average, robust to distractors and partial occlusions, and was good enough to grasp objects in clutter; the authors describe it as, to their knowledge, the first transfer of a deep network trained only on simulated RGB images, without pretraining on real images, to the real world for robotic control. In their ablations, performance degraded significantly with fewer than 1,000 unique textures, distractors in training were critical for robustness to real distractors, and, against their expectation, initializing from ImageNet weights was not essential given enough simulated data. In the same period Sadeghi and Levine (2017) trained a quadrotor’s collision-avoidance policy entirely on rendered 3D CAD models with highly randomized rendering settings, and flew it in real indoor environments without a single real training image. Randomization is a form of data augmentation aimed at the gap between two domains rather than at variation within one.

Dynamics randomization. The gap in physics can be treated the same way. Peng et al. (2018) randomized 95 dynamics parameters, including link masses, joint damping, the friction and mass of a pushed puck, controller gains, the time between actions and observation noise, while training a robot arm to push a puck in simulation. Their recurrent policy succeeded in 0.89 of 28 real trials, against 0.91 in simulation; a feedforward policy trained without randomization succeeded in none of 10 real trials. Holding the action timestep fixed or removing observation noise in training cut real success to 0.29 and 0.25, the largest drops in their ablation, from which the authors conclude that coping with controller latency and sensor noise is important for adapting to the physical system.

Measuring the gap. Simulators that are paired with real copies let the gap be measured. RoboTHOR’s object-navigation agent, trained by reinforcement learning in simulation, succeeded in 55.3% of easy test episodes in simulation and 33.3% of a subset of them on the real robot, with SPL falling from 38.1 to 3.5 (Deitke et al., 2020, Tables 1 and 2). Images of the same places in simulation and reality, which look similar to the eye, formed separate clusters in a t-SNE embedding of an ImageNet-pretrained ResNet-18’s features, and a COCO-trained detector scored 0.338 mAP at an IoU of 0.5 on real images and 0.255 on simulated ones. Changing the camera’s field of view from 90° in training to 42.5° at test dropped real success to 16% on easy episodes and 0% on medium and hard ones, with the robot rotating in place instead of moving. Kadian et al. (2020) asked a different question, whether rankings of methods in simulation predict rankings on a robot, and found that for point-goal navigation the answer depended on simulator settings such as sliding on collision; the visual navigation article summarizes the study. For manipulation policies trained on real data, SIMPLER (Li et al., 2024) goes the other way: it builds simulated copies of real evaluation setups, matching the visuals by pasting real backgrounds behind simulated objects and tuning textures, and reports, from over 1,500 paired simulated and real evaluations, a strong correlation between policies’ performance in these copies and in the real world.

Example: domain randomization on rendered shapes

The script renders 28 × 28 grayscale images of three shapes (a disc, a square and an upright triangle) at random sizes and positions, and trains a small multilayer perceptron to classify them under three rendering styles:

• fixed: a white shape on a black background, one clean style;
• lighting: random brightness of shape and background, so the shape may be darker or brighter than its surroundings, times a random illumination level and horizontal gradient;
• randomized: the lighting changes plus Tobin et al.’s three texture types (flat, gradient, checker) drawn independently for shape and background.

Each classifier is tested on 600 images in its own training style and on 600 images in a “real” style that no training set contains: a smoothly textured background with a darker, textured shape, a vertical illumination gradient and a slight blur. All images get sensor noise. The images, the labels and the network’s initialization come from seeded generators; the script loops over seeds 0 to 4 and prints the mean and range. It runs with NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2 in 35 to 37 s across our runs on an Apple M5 laptop CPU.

import warnings
import numpy as np
from scipy.ndimage import gaussian_filter
from sklearn.exceptions import ConvergenceWarning
from sklearn.neural_network import MLPClassifier

warnings.filterwarnings("ignore", category=ConvergenceWarning)  # training is capped at 300 epochs

N = 28                                    # image size (pixels)
yy, xx = np.mgrid[0:N, 0:N].astype(float)
SHAPES = ["disc", "square", "triangle"]

def mask(shape, cy, cx, r):
    if shape == "disc":
        return (yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2
    if shape == "square":
        return (abs(yy - cy) <= 0.8 * r) & (abs(xx - cx) <= 0.8 * r)
    # upright triangle: apex at the top, base at the bottom
    return (yy >= cy - r) & (yy <= cy + r) & (abs(xx - cx) <= (yy - (cy - r)) / 2)

def texture(rng):
    """Tobin et al.'s three random texture types: flat, gradient, checker."""
    a, b = rng.uniform(0, 1, 2)
    kind = rng.integers(3)
    if kind == 0:
        return np.full((N, N), a)
    if kind == 1:
        t = np.cos(rng.uniform(0, 2 * np.pi)) * yy + np.sin(rng.uniform(0, 2 * np.pi)) * xx
        t = (t - t.min()) / (t.max() - t.min())
        return a + (b - a) * t
    k = rng.integers(2, 7)
    return np.where(((yy // k) + (xx // k)) % 2 == 0, a, b)

def render(label, style, rng):
    r = rng.uniform(5, 8)
    cy, cx = N / 2 + rng.uniform(-3, 3, 2)
    m = mask(SHAPES[label], cy, cx, r)
    if style == "fixed":                  # one clean rendering style
        img = np.where(m, 1.0, 0.0)
    elif style == "lighting":             # random brightness and illumination only
        img = np.where(m, rng.uniform(0, 1), rng.uniform(0, 1))
        img = img * (rng.uniform(0.5, 1.5) + rng.uniform(-0.4, 0.4) * (xx - N / 2) / N)
    elif style == "randomized":           # random textures and illumination
        img = np.where(m, texture(rng), texture(rng))
        img = img * (rng.uniform(0.5, 1.5) + rng.uniform(-0.4, 0.4) * (xx - N / 2) / N)
    else:                                 # "real": a style never rendered in training
        bg = 0.6 + 0.6 * gaussian_filter(rng.normal(0, 1, (N, N)), 2.0)
        fg = 0.3 + 0.4 * gaussian_filter(rng.normal(0, 1, (N, N)), 1.5)
        img = np.where(m, fg, bg) * (0.9 + 0.3 * (yy - N / 2) / N)
        img = gaussian_filter(img, 0.7)   # lens blur
    return np.clip(img + rng.normal(0, 0.03, (N, N)), 0, 1).ravel()

def dataset(style, n, rng):
    y = rng.integers(0, 3, n)
    return np.array([render(c, style, rng) for c in y]), y

print("accuracy (%), mean over seeds 0-4 (min-max)")
print(f"{'trained on':>12} {'test: same style':>20} {'test: real style':>20}")
for style in ["fixed", "lighting", "randomized"]:
    acc = []
    for seed in range(5):
        rng = np.random.default_rng(seed)
        X, y = dataset(style, 3000, rng)
        Xs, ys = dataset(style, 600, rng)
        Xr, yr = dataset("real", 600, rng)
        clf = MLPClassifier(hidden_layer_sizes=(64,), max_iter=300, random_state=seed).fit(X, y)
        acc.append((clf.score(Xs, ys), clf.score(Xr, yr)))
    a = 100 * np.array(acc)
    cells = [f"{m:5.1f} ({lo:4.1f}-{hi:5.1f})" for m, lo, hi in zip(a.mean(0), a.min(0), a.max(0))]
    print(f"{style:>12} {cells[0]:>20} {cells[1]:>20}")

Output (two runs gave identical output):

accuracy (%), mean over seeds 0-4 (min-max)
  trained on     test: same style     test: real style
       fixed    99.7 (99.3-100.0)    36.0 (33.2- 38.5)
    lighting    83.7 (81.0- 87.5)    69.2 (66.8- 72.2)
  randomized    74.9 (72.3- 78.3)    70.2 (66.5- 73.2)

The classifier trained on one clean style is nearly perfect in that style and close to chance, one in three, in the real style. Reading the setup rather than the output: in every fixed training image the shape is the bright region, while in the real style the shape is darker than its background, so what the network learned to look for is no longer the shape. Both randomized classifiers recover about 70% on the real style, with every seed between 66.5% and 73.2%; by the same reading, this is because darker-than-background shapes and uneven illumination occur in their training data. The real style then lies within the range of brightness and contrast the model has seen, although its textures were never rendered.

Two further results are worth noting. Randomization has a cost: the randomized classifiers do worse on their own style than the fixed one does on its own, presumably because their task is harder and a network of this size does not fully solve it. And in this toy, textures add nothing measurable over lighting alone, since the seed ranges overlap. Our reading is that the real style differs from the fixed one mainly in brightness and contrast, which the lighting randomization already covers, so the example shows that randomization must span the factors that actually change; it does not reproduce Tobin et al.’s finding that many textures were needed, which concerned a detector facing real textures and clutter.

Evaluation

Embodied agents are evaluated by running episodes and scoring their outcomes. Four kinds of measure recur.

Success. Whether the episode reached its goal: stopping near the target in navigation, answering correctly in EQA, all goal conditions met in ALFRED or BEHAVIOR. Binary success is easy to interpret and noisy to estimate, and it gives no credit for partial progress. Rearrangement and household benchmarks therefore use graded scores. Batra et al. (2020) propose completion, the percentage of per-object tests passed, each with its own tolerance (a book anywhere on a table, a flower in a narrow vase), with a score of zero unless the agent also passes a “do no harm” test on the rest of the scene. BEHAVIOR’s success score $Q$ generalizes this to the maximum fraction of satisfied goal predicates (Srivastava et al., 2021), and ALFRED reports a goal-condition success alongside task success (Shridhar et al., 2020).

Efficiency. How much it cost to succeed. Navigation weights success by path length (SPL, defined in the visual navigation article); ALFRED weights its scores by the ratio of the expert’s number of actions to the agent’s, so that an agent taking twice as long as the expert gets half credit (Shridhar et al., 2020). Batra et al. (2020) suggest simulated time, simulated energy, the integral of the physical work the agent does, which they consider a powerful general measure, and the computation the agent needs per step, reported separately rather than merged into one number. BEHAVIOR adds efficiency measures relative to its human demonstrations.

Generalization. Embodied benchmarks split environments, not episodes, between training and test, as a train–validation–test split at the level of scenes. ALFRED’s validation and test sets each have a seen fold, with scenes from training, and an unseen fold (Shridhar et al., 2020); RoboTHOR has train and validation scenes, test-dev scenes for validating in the real world and a blind test-standard set (Deitke et al., 2020). Real-robot VLA evaluations instead define held-out categories, such as unseen objects, backgrounds and environments with easy and hard cases in RT-2 (Brohan et al., 2023b). The gap between seen and unseen results is often the main finding: ALFRED’s baseline fell from 4.0% to 0.4% task success.

Agreement with reality. For work done in simulation, a further question is whether the conclusions hold on robots. Kadian et al. (2020) measure it as the correlation between methods’ scores in simulation and reality; SIMPLER uses the Pearson correlation and a rank-based measure, the mean maximum rank violation, which penalizes pairs of policies whose order flips between simulation and reality, weighted by how far apart they are in the real world (Li et al., 2024).

Benchmark suites bundle these choices: challenge tracks of the Embodied AI workshop and the Habitat and RoboTHOR challenges for navigation, ALFRED for instruction following, the Home Assistant Benchmark and BEHAVIOR for household tasks, ManiSkill for manipulation. Real-robot results are much harder to compare. They come from different robots, objects, rooms and lighting, with small numbers of trials, which is one motivation for shared physical setups such as RoboTHOR’s remotely accessible rooms and for simulated evaluation of real policies. Apart from tests such as Batra et al.’s do-no-harm condition, these benchmarks do not measure safety, and this article draws no conclusions about the safety of deployed systems.

Open Problems

• Long horizons. Long tasks with many subgoals remain hard for learned agents in simulation, as the ALFRED and BEHAVIOR baselines show (Shridhar et al., 2020; Srivastava et al., 2021).
• The reality gap. Measured drops from simulation to real robots are large in RoboTHOR’s experiments, and they come from appearance, camera parameters and control dynamics together (Deitke et al., 2020).
• What simulators can simulate. Fast simulators leave out deformable objects, fluids and touch; richer ones run more slowly (Szot et al., 2021; Li et al., 2021).
• Data for large policies. The VLA papers themselves name the open questions: which data to collect and how to weight it (Black et al., 2025), how to acquire new motions rather than new uses of known ones (Brohan et al., 2023b), and how much transfer there is across robots (Open X-Embodiment Collaboration, 2024).
• Inference cost. RT-2’s 55B model runs at 1–3 Hz, and its authors expect real-time inference to become a bottleneck for tasks that need fast control (Brohan et al., 2023b); π0 reaches higher rates by generating chunks of actions (Black et al., 2025); model compression and edge inference describe the general tools.
• Comparable evaluation. Shared protocols exist in simulation, while real-world evaluation of policies, SIMPLER’s authors write, is not scalable and faces reproducibility challenges (Li et al., 2024).

Common Misconceptions

• “Embodied AI means robots.” Much of the work described above runs in simulators, and the definitions include virtual bodies (Batra et al., 2020). Results in simulation are evidence about simulation until they are tested on robots.
• “A photorealistic simulator closes the gap.” RoboTHOR’s real and simulated images look alike to people and still separate in a network’s features (Deitke et al., 2020), and in navigation a physics detail, sliding on collision, was the main reason simulation failed to predict reality (Kadian et al., 2020). Domain randomization succeeds with deliberately unrealistic renderings (Tobin et al., 2017).
• “Vision-language-action models are general-purpose robot controllers.” Their authors describe skills limited to those in the robot data (Brohan et al., 2023b), tasks that do not all work reliably, and open questions about whether the transfer extends to driving, navigation or legged locomotion (Black et al., 2025).
• “Imitation learning is just supervised learning.” It uses the same losses, but the policy’s actions change the inputs it will see, which breaks the independence assumption of supervised learning and makes errors compound (Ross et al., 2011).
• “A higher score on one benchmark means a better agent.” Scores depend on sensors, action abstractions such as magic pointers, simulator settings and split; the conditions are part of the result.

Where It Is Used

Embodied AI is the research setting for household and service robots, which motivate most of the benchmarks above, and for warehouse robots: Batra et al. (2020) count picking and placing orders in a fulfillment center among the instances of rearrangement. The perception components are the ones covered across this site: object detection and semantic segmentation to recognize targets, depth estimation and SLAM to map and localize, visual grounding to connect words to image regions, and the robot-specific tasks of visual navigation, robotic grasping, vision-based robotic manipulation and visual servoing. Autonomous driving is an embodied problem with its own sensors, datasets and safety requirements, and has its own overview. The large pretrained models connect the field to vision-language models and foundation models, and the training methods to transfer learning.