label

Episode annotation and metadata schema template

Decide the shape of an episode's labels before labeling starts, and every tool, labeler and training script reads the same thing. This schema keeps time in integer nanoseconds from episode start, uses controlled vocabularies where labels are counted, and records where every label came from, so you can tell a threshold detection from a person's judgment.

Who fills it: The person who owns the data pipeline, agreed with whoever labels and whoever trains.

Other formats

When to use it

  • Before the first labeling pass, or before a second team starts labeling.
  • Episodes from different sessions or sites need to be merged into one dataset.
  • You want contact events or phases as training targets, not only success labels.

When something else fits better

  • You are using an existing dataset format end to end and its metadata already covers these fields: map to it instead of adding a parallel schema.

Preview

episode-annotation-schema.md

1.Episode

One record per attempt, successful or not.

1.1episode_id

Globally unique, built from site, date, session and sequence number.

Example: lab_a_2026-11-02_s03_ep0042

Why this fieldReadable IDs make it possible to trace an episode back to its session log without a lookup table.
1.2task_id, session_id, environment_id

Keys into the spec and the session log.

Why this fieldSplits (train/held-out) and per-environment analysis are joins on these keys.
1.3operator_id

Pseudonymous operator ID.

Why this fieldNeeded for per-operator analysis and for deleting a person's data on withdrawal. Never a name.
1.4collection_method

teleoperation, kinesthetic, human_demonstration, scripted or autonomous_rollout.

Why this fieldMethods differ in embodiment and quality; training code often weights or filters by them.
1.5start_time_utc, duration_ns

Absolute start time and length.

Why this fieldAll other times in the record are offsets from start, which keeps them exact and small.
1.6calibration_id

The calibration the episode was recorded under.

Why this fieldLets you reprocess or exclude episodes when a calibration turns out to be wrong.
1.7variation

The values of each variation factor from the spec.

Example: { "lighting": "dim", "bin_fill": 9 }

Why this fieldCoverage of the variation grid is computed from this, not reconstructed from video.

2.Segments

Phases of the task, non-overlapping, covering the episode.

2.1phase

A label from the task's controlled vocabulary.

Example: reach, grasp, lift, transport, place, release

Why this fieldFree-text phases can't be counted or trained on. A fixed list per task, agreed upfront, can.
2.2start_ns, end_ns

Boundaries in nanoseconds from episode start.

Why this fieldInteger time avoids rounding mismatches between streams recorded at different rates.
2.3subgoal

Optional natural-language description of the segment.

Why this fieldUseful for language-conditioned policies, without replacing the countable phase label.

3.Contact events

Instants, not intervals: when contact starts, slips or ends.

3.1type

contact_onset, release, slip, collision or regrasp.

Why this fieldThese are the events video struggles to time precisely and that touch or force sensing measures directly.
3.2t_ns

Time of the event from episode start.

Why this fieldThe accuracy of this field is bounded by the sync tolerance in the spec. Report both together.
3.3effector, object_id

Which finger, pad or gripper jaw, and which object.

Why this fieldMulti-finger contact and multi-object scenes need both to be unambiguous.
3.4source, confidence

How the event was found (tactile_threshold, force_threshold, manual, model) and how sure.

Why this fieldAutomatic and manual labels have different error profiles. Keeping the source lets you filter or audit either.

4.Success label

4.1label

success, failure, partial or invalid.

Why this field'invalid' separates hardware or protocol problems from task outcomes, so they don't distort success rates.
4.2partial_score

0–1 when the task has defined partial credit; otherwise null.

Why this fieldOnly meaningful if partial credit is defined in the spec. Otherwise leave it empty.
4.3source, labeler_id

Who decided: operator, human_review, automatic or consensus, and which labeler.

Why this fieldLets you measure agreement between labelers and catch a labeler who drifts.
4.4failure_mode

From the protocol's failure taxonomy.

Why this fieldThe same taxonomy in collection, labeling and evaluation makes failure statistics comparable end to end.

5.Sensor metadata

5.1stream_id, modality, device

One entry per stream in the episode.

Why this fieldRecorded per episode, not per dataset, because devices get swapped mid-collection.
5.2rate_hz, units, frame

Actual rate, units and coordinate frame.

Why this fieldActual rates can differ from nominal. Units and frames prevent silent errors when streams are combined.
5.3clock, sync_offset_ms

Clock source and the offset measured at the session's sync check.

Why this fieldLets training code exclude episodes outside tolerance, or correct a known constant offset.

Copy or download

episode-annotation-schema.md

The whole template with prompts, examples and the reason for each field.

# Episode annotation and metadata schema

Decide the shape of an episode's labels before labeling starts, and every tool, labeler and training script reads the same thing. This schema keeps time in integer nanoseconds from episode start, uses controlled vocabularies where labels are counted, and records where every label came from, so you can tell a threshold detection from a person's judgment.

Who fills it: The person who owns the data pipeline, agreed with whoever labels and whoever trains.

How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.

## 1. Episode

One record per attempt, successful or not.

### 1.1 episode_id

Globally unique, built from site, date, session and sequence number.

Example: lab_a_2026-11-02_s03_ep0042

Answer: 

> Why: Readable IDs make it possible to trace an episode back to its session log without a lookup table.

### 1.2 task_id, session_id, environment_id

Keys into the spec and the session log.

Answer: 

> Why: Splits (train/held-out) and per-environment analysis are joins on these keys.

### 1.3 operator_id

Pseudonymous operator ID.

Answer: 

> Why: Needed for per-operator analysis and for deleting a person's data on withdrawal. Never a name.

### 1.4 collection_method

teleoperation, kinesthetic, human_demonstration, scripted or autonomous_rollout.

Answer: 

> Why: Methods differ in embodiment and quality; training code often weights or filters by them.

### 1.5 start_time_utc, duration_ns

Absolute start time and length.

Answer: 

> Why: All other times in the record are offsets from start, which keeps them exact and small.

### 1.6 calibration_id

The calibration the episode was recorded under.

Answer: 

> Why: Lets you reprocess or exclude episodes when a calibration turns out to be wrong.

### 1.7 variation

The values of each variation factor from the spec.

Example: { "lighting": "dim", "bin_fill": 9 }

Answer: 

> Why: Coverage of the variation grid is computed from this, not reconstructed from video.

## 2. Segments

Phases of the task, non-overlapping, covering the episode.

### 2.1 phase

A label from the task's controlled vocabulary.

Example: reach, grasp, lift, transport, place, release

Answer: 

> Why: Free-text phases can't be counted or trained on. A fixed list per task, agreed upfront, can.

### 2.2 start_ns, end_ns

Boundaries in nanoseconds from episode start.

Answer: 

> Why: Integer time avoids rounding mismatches between streams recorded at different rates.

### 2.3 subgoal

Optional natural-language description of the segment.

Answer: 

> Why: Useful for language-conditioned policies, without replacing the countable phase label.

## 3. Contact events

Instants, not intervals: when contact starts, slips or ends.

### 3.1 type

contact_onset, release, slip, collision or regrasp.

Answer: 

> Why: These are the events video struggles to time precisely and that touch or force sensing measures directly.

### 3.2 t_ns

Time of the event from episode start.

Answer: 

> Why: The accuracy of this field is bounded by the sync tolerance in the spec. Report both together.

### 3.3 effector, object_id

Which finger, pad or gripper jaw, and which object.

Answer: 

> Why: Multi-finger contact and multi-object scenes need both to be unambiguous.

### 3.4 source, confidence

How the event was found (tactile_threshold, force_threshold, manual, model) and how sure.

Answer: 

> Why: Automatic and manual labels have different error profiles. Keeping the source lets you filter or audit either.

## 4. Success label

### 4.1 label

success, failure, partial or invalid.

Answer: 

> Why: 'invalid' separates hardware or protocol problems from task outcomes, so they don't distort success rates.

### 4.2 partial_score

0–1 when the task has defined partial credit; otherwise null.

Answer: 

> Why: Only meaningful if partial credit is defined in the spec. Otherwise leave it empty.

### 4.3 source, labeler_id

Who decided: operator, human_review, automatic or consensus, and which labeler.

Answer: 

> Why: Lets you measure agreement between labelers and catch a labeler who drifts.

### 4.4 failure_mode

From the protocol's failure taxonomy.

Answer: 

> Why: The same taxonomy in collection, labeling and evaluation makes failure statistics comparable end to end.

## 5. Sensor metadata

### 5.1 stream_id, modality, device

One entry per stream in the episode.

Answer: 

> Why: Recorded per episode, not per dataset, because devices get swapped mid-collection.

### 5.2 rate_hz, units, frame

Actual rate, units and coordinate frame.

Answer: 

> Why: Actual rates can differ from nominal. Units and frames prevent silent errors when streams are combined.

### 5.3 clock, sync_offset_ms

Clock source and the offset measured at the session's sync check.

Answer: 

> Why: Lets training code exclude episodes outside tolerance, or correct a known constant offset.

Use it with