Specified like a sensor.
What a neocognix episode contains, how each layer is labeled, the thresholds every delivery is measured against, and the formats it ships in.
- Streams
- 9, synchronized
- Hand pose
- 21 keypoints × 2
- Label layers
- 8
- Formats
- 5
- QA report
- every delivery
One episode, played back
Twelve seconds of two-handed kitting, drawn from its label files. Switch layers on and off, or scrub the timeline.
ep_0412/kitting, tray_B/head cam
/ 00:00:11:59
R handL handGazeStepIMU
Nine streams, pixels to steps
Raw sensor streams first, then every layer derived from them. Select a stream to see the fields it delivers.
Hand pose
- hands.{l,r}.keypoints
- float32[21, 3], meters, head camera frame, 60 Hz
- r[8] = [0.071, 0.442, 0.286]
- hands.{l,r}.keypoints_world
- float32[21, 3], meters, world frame, 60 Hz
- r[8] = [0.212, 0.031, 0.118]
- hands.{l,r}.mano
- pose float32[48], shape float32[10], 60 Hz
- shape fixed per operator
- hands.{l,r}.visible
- bool, 60 Hz
- true
| Field | Type | Rate | Example |
|---|---|---|---|
| hands.{l,r}.keypoints | float32[21, 3], meters, head camera frame | 60 Hz | r[8] = [0.071, 0.442, 0.286] |
| hands.{l,r}.keypoints_world | float32[21, 3], meters, world frame | 60 Hz | r[8] = [0.212, 0.031, 0.118] |
| hands.{l,r}.mano | pose float32[48], shape float32[10] | 60 Hz | shape fixed per operator |
| hands.{l,r}.visible | bool | 60 Hz | true |
How labels are checked
- Labeled, then reviewed
- Every segment is checked by a second person. A lead who knows the task settles each disagreement.
- Agreement, measured blind
- One episode in ten is labeled twice, independently. Each delivery reports agreement on that sample: boundary timing, class κ and keypoint error.
- Failures are dropped
- An episode that fails review is re-labeled or left out. It never ships with a caveat attached.
Label layers
Each layer is optional. You choose the ones your policy trains on, and we label only those.
| Layer | Contents | Rate | How it is made |
|---|---|---|---|
| Hand pose | 21 keypoints per hand in camera and world frames, plus MANO pose and shape | 60 Hz | Triangulated across head, wrist and cell views, then corrected by hand where views disagree |
| Objects | Track ID, a box per camera and a 6-DoF pose for every part and tool the hands touch | 60 Hz | Tracked between reviewed keyframes, with masks on every tenth frame |
| Contacts | Which hand touches what, at every onset and release, with grasp type from a 33-type taxonomy | per event | Derived from pose and depth, then reviewed event by event |
| Actions | reach, grasp, transport, place, release, retract, plus task verbs from your schema | per segment | Labeled, then checked by a second person; a task lead settles disagreements |
| Gaze | Fixation target and saccades | 60 Hz | Eye-tracking glasses, on sessions where you need them |
| Steps | The line of the site’s work instruction each segment carries out, and whether it was done, redone or skipped | per step | Aligned against the SOP document with the site’s engineer |
| Narration | One line per action in the Ego4D convention (“C places bracket in slot 4”), plus the operator’s own commentary where given | ~12 a minute | Written by annotators against the video; operator commentary is recorded after each take, transcribed and aligned |
| Skill | Trade, years on the job, expert or trainee; errors and recoveries as actions of their own | per episode | From site records and review of every flagged segment |
- Hand pose
- Contents: 21 keypoints per hand in camera and world frames, plus MANO pose and shape
- Rate: 60 Hz
- How it is made: Triangulated across head, wrist and cell views, then corrected by hand where views disagree
- Objects
- Contents: Track ID, a box per camera and a 6-DoF pose for every part and tool the hands touch
- Rate: 60 Hz
- How it is made: Tracked between reviewed keyframes, with masks on every tenth frame
- Contacts
- Contents: Which hand touches what, at every onset and release, with grasp type from a 33-type taxonomy
- Rate: per event
- How it is made: Derived from pose and depth, then reviewed event by event
- Actions
- Contents: reach, grasp, transport, place, release, retract, plus task verbs from your schema
- Rate: per segment
- How it is made: Labeled, then checked by a second person; a task lead settles disagreements
- Gaze
- Contents: Fixation target and saccades
- Rate: 60 Hz
- How it is made: Eye-tracking glasses, on sessions where you need them
- Steps
- Contents: The line of the site’s work instruction each segment carries out, and whether it was done, redone or skipped
- Rate: per step
- How it is made: Aligned against the SOP document with the site’s engineer
- Narration
- Contents: One line per action in the Ego4D convention (“C places bracket in slot 4”), plus the operator’s own commentary where given
- Rate: ~12 a minute
- How it is made: Written by annotators against the video; operator commentary is recorded after each take, transcribed and aligned
- Skill
- Contents: Trade, years on the job, expert or trainee; errors and recoveries as actions of their own
- Rate: per episode
- How it is made: From site records and review of every flagged segment
One step, every layer
Step 4.2 of the work instruction as it was carried out: actions, gaze, contact and narration on one timeline. The eyes reach the slot 0.6 s before the hand does.
Work instruction
SOP 4.2 bracket to s1
Left hand
reachgrasptransportplacereleaseretract
Gaze
bin_Ltray_B/s1tray_B
Contact
grasp 5.24release 7.00
Narration
reaches into the left bin with the left handplaces a bracket in slot 1
Eyes on slot s1 0.60 s before the hand starts to place.
Thresholds we publish
Every delivery reports its measured value against each row. An episode that fails is dropped, not patched.
| Check | Threshold | How it is measured |
|---|---|---|
| Stream sync | ≤ 2 ms head–wrist, ≤ half a frame to cell | One hardware clock per station, checked with a coded LED sequence every camera sees at the start and end of each session |
| Calibration | ≤ 0.5 px mean reprojection error | ChArUco board at the start and end of every session, per camera |
| Hand keypoints | ≤ 12 mm mean error | Against expert keypoints on held-out frames, placed in all three views and triangulated |
| Action boundaries | ≤ 100 ms between annotators | One episode in ten labeled twice, blind, in every batch |
| Action classes | Cohen’s κ ≥ 0.8 | Per batch, on the same double-labeled sample |
| De-identification | 0 unblurred faces, in any view | Automatic pass, a person reviews every flagged frame, and a random 2% of frames is audited in full; the miss rate is reported |
- Stream sync
- Threshold: ≤ 2 ms head–wrist, ≤ half a frame to cell
- How it is measured: One hardware clock per station, checked with a coded LED sequence every camera sees at the start and end of each session
- Calibration
- Threshold: ≤ 0.5 px mean reprojection error
- How it is measured: ChArUco board at the start and end of every session, per camera
- Hand keypoints
- Threshold: ≤ 12 mm mean error
- How it is measured: Against expert keypoints on held-out frames, placed in all three views and triangulated
- Action boundaries
- Threshold: ≤ 100 ms between annotators
- How it is measured: One episode in ten labeled twice, blind, in every batch
- Action classes
- Threshold: Cohen’s κ ≥ 0.8
- How it is measured: Per batch, on the same double-labeled sample
- De-identification
- Threshold: 0 unblurred faces, in any view
- How it is measured: Automatic pass, a person reviews every flagged frame, and a random 2% of frames is audited in full; the miss rate is reported
Thresholds apply per session. Tighter limits for your task can be written into the capture plan.
Loads straight into your training loop.
LeRobot v3, RLDS, MCAP, HDF5 or WebDataset. The same episodes and labels, in whichever reader your stack already uses.
load_lerobot.py
import torch
from lerobot.datasets.lerobot_dataset import LeRobotDataset
ds = LeRobotDataset("neocognix/kitting-sample", root="./kitting-sample")
frame = ds[0]
frame["observation.images.head"].shape # torch.Size([3, 1080, 1920])
frame["observation.state"].shape # torch.Size([126]), 2 × 21 × 3
frame["action"].shape # torch.Size([14]), ee pose + grip, per hand
frame["task"] # "kit brackets into tray B"
loader = torch.utils.data.DataLoader(ds, batch_size=32, shuffle=True)
batch = next(iter(loader))One delivery, as it lands in your bucket
- kitting-sample/
- meta/
- info.json
- stats.json
- tasks.parquet
- episodes/
- data/chunk-000/
- videos/
- observation.images.head/
- observation.images.wrist/
- observation.images.cell/
Added by us, in the same folder
- calibration/
- session_027.yaml
- provenance/
- ep_0412.json
- qa/
- report.json
Start with one episode.
We send a sample episode with every stream and label layer, in the format you train on. No NDA needed.