Episode annotation and metadata schema template
Decide the shape of an episode's labels before labeling starts, and every tool, labeler and training script reads the same thing. This schema keeps time in integer nanoseconds from episode start, uses controlled vocabularies where labels are counted, and records where every label came from, so you can tell a threshold detection from a person's judgment.
Who fills it: The person who owns the data pipeline, agreed with whoever labels and whoever trains.
When to use it
- Before the first labeling pass, or before a second team starts labeling.
- Episodes from different sessions or sites need to be merged into one dataset.
- You want contact events or phases as training targets, not only success labels.
When something else fits better
- You are using an existing dataset format end to end and its metadata already covers these fields: map to it instead of adding a parallel schema.
Preview
1.Episode
One record per attempt, successful or not.
Globally unique, built from site, date, session and sequence number.
Example: lab_a_2026-11-02_s03_ep0042
Keys into the spec and the session log.
Pseudonymous operator ID.
teleoperation, kinesthetic, human_demonstration, scripted or autonomous_rollout.
Absolute start time and length.
The calibration the episode was recorded under.
The values of each variation factor from the spec.
Example: { "lighting": "dim", "bin_fill": 9 }
2.Segments
Phases of the task, non-overlapping, covering the episode.
A label from the task's controlled vocabulary.
Example: reach, grasp, lift, transport, place, release
Boundaries in nanoseconds from episode start.
Optional natural-language description of the segment.
3.Contact events
Instants, not intervals: when contact starts, slips or ends.
contact_onset, release, slip, collision or regrasp.
Time of the event from episode start.
Which finger, pad or gripper jaw, and which object.
How the event was found (tactile_threshold, force_threshold, manual, model) and how sure.
4.Success label
success, failure, partial or invalid.
0–1 when the task has defined partial credit; otherwise null.
Who decided: operator, human_review, automatic or consensus, and which labeler.
From the protocol's failure taxonomy.
5.Sensor metadata
One entry per stream in the episode.
Actual rate, units and coordinate frame.
Clock source and the offset measured at the session's sync check.
Copy or download
The whole template with prompts, examples and the reason for each field.
# Episode annotation and metadata schema
Decide the shape of an episode's labels before labeling starts, and every tool, labeler and training script reads the same thing. This schema keeps time in integer nanoseconds from episode start, uses controlled vocabularies where labels are counted, and records where every label came from, so you can tell a threshold detection from a person's judgment.
Who fills it: The person who owns the data pipeline, agreed with whoever labels and whoever trains.
How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.
## 1. Episode
One record per attempt, successful or not.
### 1.1 episode_id
Globally unique, built from site, date, session and sequence number.
Example: lab_a_2026-11-02_s03_ep0042
Answer:
> Why: Readable IDs make it possible to trace an episode back to its session log without a lookup table.
### 1.2 task_id, session_id, environment_id
Keys into the spec and the session log.
Answer:
> Why: Splits (train/held-out) and per-environment analysis are joins on these keys.
### 1.3 operator_id
Pseudonymous operator ID.
Answer:
> Why: Needed for per-operator analysis and for deleting a person's data on withdrawal. Never a name.
### 1.4 collection_method
teleoperation, kinesthetic, human_demonstration, scripted or autonomous_rollout.
Answer:
> Why: Methods differ in embodiment and quality; training code often weights or filters by them.
### 1.5 start_time_utc, duration_ns
Absolute start time and length.
Answer:
> Why: All other times in the record are offsets from start, which keeps them exact and small.
### 1.6 calibration_id
The calibration the episode was recorded under.
Answer:
> Why: Lets you reprocess or exclude episodes when a calibration turns out to be wrong.
### 1.7 variation
The values of each variation factor from the spec.
Example: { "lighting": "dim", "bin_fill": 9 }
Answer:
> Why: Coverage of the variation grid is computed from this, not reconstructed from video.
## 2. Segments
Phases of the task, non-overlapping, covering the episode.
### 2.1 phase
A label from the task's controlled vocabulary.
Example: reach, grasp, lift, transport, place, release
Answer:
> Why: Free-text phases can't be counted or trained on. A fixed list per task, agreed upfront, can.
### 2.2 start_ns, end_ns
Boundaries in nanoseconds from episode start.
Answer:
> Why: Integer time avoids rounding mismatches between streams recorded at different rates.
### 2.3 subgoal
Optional natural-language description of the segment.
Answer:
> Why: Useful for language-conditioned policies, without replacing the countable phase label.
## 3. Contact events
Instants, not intervals: when contact starts, slips or ends.
### 3.1 type
contact_onset, release, slip, collision or regrasp.
Answer:
> Why: These are the events video struggles to time precisely and that touch or force sensing measures directly.
### 3.2 t_ns
Time of the event from episode start.
Answer:
> Why: The accuracy of this field is bounded by the sync tolerance in the spec. Report both together.
### 3.3 effector, object_id
Which finger, pad or gripper jaw, and which object.
Answer:
> Why: Multi-finger contact and multi-object scenes need both to be unambiguous.
### 3.4 source, confidence
How the event was found (tactile_threshold, force_threshold, manual, model) and how sure.
Answer:
> Why: Automatic and manual labels have different error profiles. Keeping the source lets you filter or audit either.
## 4. Success label
### 4.1 label
success, failure, partial or invalid.
Answer:
> Why: 'invalid' separates hardware or protocol problems from task outcomes, so they don't distort success rates.
### 4.2 partial_score
0–1 when the task has defined partial credit; otherwise null.
Answer:
> Why: Only meaningful if partial credit is defined in the spec. Otherwise leave it empty.
### 4.3 source, labeler_id
Who decided: operator, human_review, automatic or consensus, and which labeler.
Answer:
> Why: Lets you measure agreement between labelers and catch a labeler who drifts.
### 4.4 failure_mode
From the protocol's failure taxonomy.
Answer:
> Why: The same taxonomy in collection, labeling and evaluation makes failure statistics comparable end to end.
## 5. Sensor metadata
### 5.1 stream_id, modality, device
One entry per stream in the episode.
Answer:
> Why: Recorded per episode, not per dataset, because devices get swapped mid-collection.
### 5.2 rate_hz, units, frame
Actual rate, units and coordinate frame.
Answer:
> Why: Actual rates can differ from nominal. Units and frames prevent silent errors when streams are combined.
### 5.3 clock, sync_offset_ms
Clock source and the offset measured at the session's sync check.
Answer:
> Why: Lets training code exclude episodes outside tolerance, or correct a known constant offset.