Field notes · data collection modalities
Robot manipulation training data spans five genuinely different collection methods — from passive internet video down to a human physically teleoperating the exact robot arm. Scale drops by orders of magnitude at each step down this ladder. This page lays out what's actually disclosed, by whom, per modality — see the companion Robot Learning at Scale link map for the underlying sources.
Striped fill = derived estimate, not the source's own stated number (RT-1: 130,000 episodes × 20s, the midpoint of the paper's own stated 10–30s/episode range at 3Hz control; UMI: no paper-wide total exists — bar shown is the single largest reported task-level collection, ~1,400 episodes at 12 person-hours, extrapolated only as an upper illustrative bound, not a claim about UMI's typical scale). BridgeData V2 (60,096 trajectories) is omitted from this hours-based chart — no reliable episode-length figure could be confirmed to convert it honestly; see its demo-count entry in the table below instead.
No robot, no action labels, not necessarily egocentric — third-person web-scale video, used to pretrain a general visual/world model before any robotics-specific data is introduced.
| Source | Scale | What it's used for |
|---|---|---|
| CosmosNVIDIA | 20,000,000 hrs | World Foundation Models pretraining corpus: driving + robotics + human-interaction + industrial video. Feeds synthetic-data generation for GR00T and other manipulation policies. |
| V-JEPA 2Meta | 1,000,000+ hrs | Action-free internet video pretraining. The robot-capable variant (V-JEPA 2-AC) is then post-trained on under 62 hrs of unlabeled DROID robot video — a small, separate second stage. |
| GR00T N1NVIDIAnot quantified | "internet-scale" | The Eagle-2 VLM backbone is described as pretrained on internet-scale data, but no dataset size is given in the paper — the web tier of GR00T's four-tier pyramid is stated qualitatively only. |
| Genie / Genie 2 / Genie 3DeepMindnot disclosed | — | None of the three Genie blog posts or papers give a training-data hour count — scale is described only as "internet-scale video," with no figure across any of the three generations. |
A person wears a head- or chest-mounted camera doing everyday or task-specific activity. No robot, no handheld tool — the camera rides the body. This is the tier GR00T N1 names as the base of its data pyramid (Ego4D, Ego-Exo4D, EPIC-KITCHENS, Assembly-101, HOI4D, HoloAssist are all cited as sources) without giving an aggregate figure of its own.
| Source | Scale | Note |
|---|---|---|
| Ego4DMeta | 3,670 hrs | 3,025 hours of daily-life egocentric video from 74 locations across 9 countries. |
| Ego-Exo4DMeta | 1,422 hrs | 221 hrs egocentric + matched exocentric (third-person) footage of skilled human activities, paired for cross-view learning. |
| EgoDexApple | 829 hrs | 338k demonstrations, 194 tasks, captured via Vision Pro hand-tracking — headset-based, not a worn camera per se; the closest bridge to the wrist-cam category below. |
| EPIC-KITCHENS-100Bristol et al. | 100 hrs | Kitchen-activity footage, extended version of the original EPIC-KITCHENS release. |
A camera worn on the wrist or hand itself — distinct from a head-mounted rig and distinct from UMI's handheld gripper-jaw tool. This is the category the trajectory implies should exist between "egocentric video" and "UMI," and it largely doesn't as a standalone public dataset.
A human holds a physical gripper-jaw tool with a mounted camera and performs the task by hand — no robot present during collection, but the resulting trajectory is directly usable as robot action data. UMI's paper (Columbia/TRI/MIT, 2024) never states a corpus-wide total — every number in it is per task.
| Task | Episodes | Note |
|---|---|---|
| Cup arrangement, in-the-wild | 1,400 episodes | 12 person-hours, 3 demonstrators, 30 locations — the largest single collection reported in the paper. |
| Cup arrangement, narrow domain | 305 episodes | 2 demonstrators, controlled setting. |
| Dynamic tossing | 280 episodes | — |
| Bimanual cloth folding | 250 episodes | 2 demonstrators. |
| Dish washing | 258 episodes | 1 demonstrator. |
A human directly drives the real robot — leader-follower rigs (ALOHA-style), VR teleop, joysticks. Highest fidelity (the action labels are the exact robot's own joint commands, no retargeting needed) and by far the most expensive per hour: the robot hardware, the space, and a skilled operator all have to be present simultaneously.
| Source | Scale | Note |
|---|---|---|
| RT-1Googlederived | 130,000 episodes ≈722 hrs (est.) |
700+ tasks, 13 robots, 17 months of collection. Hour figure derived from 130k episodes × 20s (midpoint of the paper's own stated 10–30s/episode range at 3Hz control) — not the paper's own stated total. |
| DROIDBerkeley | 76,000 trajectories 350 hrs |
564 scenes, 84 tasks, 50 collectors across 3 continents, 12 months. Author-stated figure, not derived. |
| BridgeData V2Berkeleyhrs not disclosed | 60,096 trajectories | 24 environments, 13 skills, low-cost hardware. The paper reports trajectory count, not aggregate hours, and no reliable per-episode duration could be confirmed to convert it honestly — left as a demo-count rather than a guessed hour figure. |
| RoboTurkStanford | 111+ hrs | 54 crowdsourced users, 3 tasks, 1 week — the only crowdsourced (vs. single-operator) figure in this table. |
| Mobile ALOHAStanford | 50 demos/task | No aggregate hours stated — demo-count is the unit the paper reports in. |
| ACT / ALOHAStanford | ~10 min/task | The paper's headline result: strong performance from roughly 10 minutes of demonstration per task — the smallest figure in this entire page. |