What is egocentric data, and why do robots need it?

A plain definition

Egocentric data (also called first-person data) is video and sensor data captured from the point of view of the person doing the work. The core stream is a head-mounted camera at eye height. Serious datasets add a camera on one or both wrists, an inertial measurement unit (IMU) on the head and wrist, and sometimes eye tracking, all recorded on one shared clock.

The term comes from computer vision research. The best-known academic example is Ego4D, which gathered more than 3,000 hours of everyday first-person video from hundreds of participants.¹ What changed in 2025 and 2026 is that robotics labs started using this kind of data to train manipulation policies, not only to study perception.

Why robot learning uses first-person video

A humanoid robot sees the world from roughly where a person’s eyes are, and its grippers enter the frame from below, like hands. Footage recorded from a worker’s head therefore has the same geometry as the robot’s own camera: the same reach distances, the same occlusions and the same view of the work surface.

The evidence that this transfers is recent. Physical Intelligence reported that once a vision-language-action model had been pretrained on enough robot data, adding human video recorded with head and wrist cameras nearly doubled its performance on generalization tasks that appeared only in the human data.² NVIDIA’s EgoScale work trained on 20,854 hours of action-labeled first-person video and found performance rising log-linearly with hours, from an average task completion of 0.30 at 1,000 hours to 0.71 at 20,000.³

What an egocentric episode contains

An episode is one continuous recording of a task, from the first reach to the last release. A training-grade episode carries raw streams and the labels derived from them:

Where first-person data falls short

• Hands are not grippers. Wrist pose and grip aperture retarget well; fingertip force and contact detail do not.
• A head camera cannot see under the palm. That is why careful capture adds wrist and fixed cell cameras.
• Transfer needs a base. The reported gains came on models already pretrained on diverse robot data.
• Some tasks need the robot’s own body. Tight insertions and forceful tool use still call for teleoperated or on-robot data during fine-tuning.

Volume is solved. Skill is not.

Open first-person datasets have grown fast. Build AI released 10,000 hours of factory video under an open license,⁴ then 100,000 hours from 14,228 workers,⁵ and announced a million hours in April 2026.⁶ Generalist trains on more than 270,000 hours of manipulation data from wearable devices.⁷

When hours are plentiful, a dataset is worth what others do not have: expert operators doing skilled work, calibrated multi-camera capture that allows 3D hand tracking, labels a policy can consume, and a consent record a buyer can audit. That last point is no longer optional: in August 2026 Bloomberg reported that workers paid about $2.62 an hour to wear head cameras were often not told the footage would train robots.⁸

If you are evaluating datasets, our twelve-point checklist covers what to measure before you buy, and teleoperation vs. human video covers when each is the right tool.