specify

Robot training data requirements spec template

Write this before the first recording session. It turns "we need demonstrations of the task" into numbers a collection team can hit and a reviewer can check: which streams, at what rate, synchronized how tightly, in which environments, and how many episodes. Each field carries a note on why it is there, so the spec can be argued with rather than just filled in.

Who fills it: The technical lead for the dataset, with the person who will train the policy and the person who runs collection.

Other formats

When to use it

  • You are starting a new dataset for a manipulation or mobile-manipulation policy.
  • Collection is about to move from one person's bench to a team, a vendor or a second site.
  • A trained policy underperforms and you need to write down what the data was supposed to contain.

When something else fits better

  • You are deciding which sensors to use at all: fill the sensing modality selection worksheet first, then bring its answers here.
  • You are documenting a dataset that already exists for other people: use the dataset datasheet.

Preview

data-requirements-spec.md

1.Task definition

Define the task as the policy will face it at deployment, not as it looks in a demo video.

1.1Task name

A short, unique name used in file names and episode metadata.

Example: bin_pick_soft_pouch

Why this fieldThe name becomes a key in every episode record. Changing it later breaks joins between episodes, labels and evaluation logs.
1.2Task statement

One sentence: the start state, the action, and the end state.

Example: From a bin of 5–15 mixed soft pouches, pick one pouch and place it flat in the tote within 20 s.

Why this fieldA start and end state make episode boundaries unambiguous, which annotators and the success label both depend on.
1.3Success criterion

An observable condition someone other than the operator can judge from the recording.

Example: Pouch fully inside tote, not folded, gripper open and clear of tote at episode end.

Why this fieldIf success is judged by feel, labels drift between labelers and between data collection and evaluation. Use the same criterion in the policy evaluation plan.
1.4Embodiment

Robot model, end effector, degrees of freedom, and any mounted sensors.

Example: 6-DoF arm, parallel-jaw gripper with fingertip tactile pads, wrist force/torque sensor.

Why this fieldData collected on, or retargeted to, a different embodiment needs a mapping step. Writing the target down makes that gap visible early.
1.5Action space

What the policy outputs and how often.

Example: Delta end-effector pose (6) + gripper width (1) at 10 Hz.

Why this fieldThe recorded actions must be convertible to this representation at this rate, or the data cannot supervise the policy directly.
1.6Observations available at deployment

The sensors the policy will actually have on the robot in production.

Example: Wrist RGB, head RGB-D, joint states, fingertip tactile. No EMG, no gloves.

Why this fieldStreams recorded only during collection (EMG, a glove, motion capture) can still be valuable as training targets or auxiliary signals, but the plan for using them has to be explicit.
1.7Out of scope

Situations the dataset deliberately does not cover.

Example: Torn pouches, pouches heavier than 1 kg, human in the workcell.

Why this fieldSaying what is excluded stops collection from drifting into edge cases and tells evaluators what not to test.

2.Modalities and sampling rates

List every stream that will be recorded, including the ones only used during collection.

StreamSensorRateUnits / resolutionFrameAt deployment?
wrist_rgbWrist cameraCamera's native frame rateWidth × height, 8-bit RGBcamera_wristYes
joint_stateRobot controllerController's state raterad, rad/sbaseYes
ft_wristWrist force/torque sensorFast enough to resolve contact onsetN, N·mtool0Yes
tactile_leftFingertip tactile padFast enough to resolve slipAs the sensor reports itfinger_leftYes / No
emg_forearmForearm EMG bandThe band's native rateµV or normalizedn/aNo
2.1Stream list

Fill the table above: one row per stream, with sensor model, rate, units, coordinate frame, and whether it exists at deployment.

Why this fieldA stream without units or a frame is hard to use a year later. The deployment column separates policy inputs from supervision-only signals.
2.2Rate rationale

For each stream, why this rate: the fastest event it must resolve.

Example: Tactile at the sensor's native rate because slip onset is the event we label; RGB at the camera's native rate because motion blur, not frame rate, limits us.

Why this fieldRates are often set to whatever the driver defaults to. Tying each to an event makes undersampling (missed contact transients) and oversampling (storage cost) both arguable.
2.3Rate at training time

How streams will be resampled or windowed for the policy, if differently from capture.

Example: All streams aligned to 10 Hz action steps; tactile and F/T summarized over each 100 ms window.

Why this fieldCapturing fast and training slow is common. Writing the reduction down keeps the training pipeline and the labels consistent.

3.Timing and synchronization

3.1Clock source

Which clock every timestamp comes from, and how devices are synchronized to it.

Example: Host monotonic clock; cameras hardware-triggered; tactile and EMG timestamped on arrival with measured offset.

Why this fieldMixed clocks are the most common reason contact signals line up with the wrong video frame.
3.2Timestamp unit

The unit and type used everywhere.

Example: Integer nanoseconds since episode start, plus absolute start time in UTC.

Why this fieldFloats in seconds lose precision over long sessions; a single unit avoids silent conversions.
3.3Synchronization tolerance

The maximum allowed offset between each pair of streams that are used together.

Example: A maximum offset in milliseconds for each pair: actions ↔ joint state, camera ↔ joint state, tactile and F/T ↔ joint state.

Why this fieldSet the tolerance from the shortest event you need to align (a contact onset, a release). If streams drift more than that event lasts, labels attach to the wrong moment.
3.4Sync check method

How the offset is measured in every session.

Example: Operator taps the fingertip on a fixed plate at session start and end; the tap must appear in tactile, F/T and video within tolerance.

Why this fieldA tolerance nobody measures is a hope. A physical event visible in every stream gives a number per session.

4.Calibration

4.1Camera intrinsics and extrinsics

How and how often they are calibrated, and where the calibration files live.

Example: Checkerboard intrinsics per camera at install; extrinsics via hand-eye calibration weekly and after any bump.

Why this fieldPolicies that use 3D information inherit every calibration error. Each episode must reference the calibration it was recorded under.
4.2Force and tactile tare

When force and tactile sensors are zeroed.

Example: Tare with gripper open and unloaded at the start of every session and after any collision.

Why this fieldForce/torque and many tactile sensors drift with temperature and load history; untared offsets look like contact.
4.3Recalibration triggers

Events that force recalibration before collection continues.

Example: Camera moved, sensor swapped, sync check out of tolerance, collision.

Why this fieldExplicit triggers stop operators from carrying on with a known-bad setup to hit a daily target.

5.Data source

5.1Collection method

Teleoperation, kinesthetic teaching, human demonstration with wearable sensors, scripted, autonomous rollouts, or a mix (with shares).

Example: Human demonstrations with wearable sensing for contact phases; teleoperation on the target robot for the full task.

Why this fieldEach method has a different embodiment gap, throughput and cost. Mixing is fine if every episode records which method produced it.
5.2Operators

How many people demonstrate, and what they must be able to do.

Example: Four operators, each passing a 10-episode qualification run before their data counts.

Why this fieldOne operator gives consistent but narrow data; several give variety but need a qualification bar. Either way, operator ID goes in every episode.

6.Environments and variation

6.1Environments

Each place data is collected, with an ID and whether it is used for training or held out.

Example: lab_a (train), lab_b (train), customer_site_1 (held-out).

Why this fieldA held-out environment is the cheapest honest test of generalization. Decide it before collection, not after results are in.
6.2Variation factors

What changes between episodes, and the range of each.

Example: Lighting (3 settings), bin fill level (5–15 items), pouch pose (random), distractor objects (0–3).

Why this fieldPolicies generalize along the axes the data varies on. Listing them turns 'diverse data' into a grid you can check coverage against.
6.3Object set

The objects used, split into train and held-out.

Example: Train: 24 pouch SKUs. Held-out: 6 SKUs never shown in training.

Why this fieldHeld-out objects measure what the policy learned about the category rather than the specific items.

7.Volume targets

7.1Pilot size

How many episodes the pilot collects before the full target is set.

Example: Enough episodes to train one small policy end to end, across 2 environments and 2 operators.

Why this fieldPublished dataset sizes vary by orders of magnitude between tasks. A pilot plus a small training run shows how performance moves with data for your task, which is a better basis for the target than another team's number.
7.2Episodes per condition

Target episodes for each cell of the variation grid.

Example: The same target for every (environment × lighting × fill level) cell, so no cell stays empty.

Why this fieldA single total hides empty cells. Per-cell targets make coverage gaps visible while collection is still running.
7.3Total target

Total episodes and hours, and the date it should be reached.

Why this fieldConverts the plan into operator-hours and storage, so budget and schedule can be checked.
7.4Failures kept

Whether failed attempts are kept and labelled, and how they are used.

Example: Kept, labelled with failure mode; excluded from behaviour cloning, used for success classifiers and analysis.

Why this fieldFailed episodes are expensive to collect and useful for success detection and recovery behaviour. Deleting them loses that.

8.Acceptance and quality

8.1Acceptance checks

Automatic checks every episode must pass.

Example: All required streams present; dropped frames below the spec's threshold; sync offset within tolerance; episode length within bounds.

Why this fieldAutomated checks at ingest catch broken sessions the same day, while they can still be re-recorded.
8.2Human review

What share of episodes a person reviews, and what they look for.

Example: A fixed random share of each operator's episodes per day: task followed, success label correct, no PII in frame.

Why this fieldSome problems (wrong object, mislabelled success) only a person sees. A fixed sample rate keeps review from being skipped under deadline.

9.Storage and versioning

9.1Format

Container format for raw and training-ready data.

Example: Raw: MCAP per session. Training: converted to the training framework's format per dataset version.

Why this fieldKeeping raw data in a lossless, timestamped container means the training format can change without re-collecting.
9.2Naming and versioning

How sessions, episodes and dataset versions are named.

Example: {site}_{date}_{session}/ep_{0001}; dataset versions as semver with a changelog.

Why this fieldModels must be traceable to the exact data version they were trained on.
9.3Retention and access

Who can access raw data, and how long it is kept.

Why this fieldVideo and biosignals of people may be personal data. Retention and access rules must match what operators consented to.

10.Sign-off

10.1Owner and reviewers

One owner, the reviewers, and the date this version was approved.

Why this fieldA spec without an owner stops being updated the first time collection deviates from it.

Copy or download

data-requirements-spec.md

The whole template with prompts, examples and the reason for each field.

# Robot training data requirements spec

Write this before the first recording session. It turns "we need demonstrations of the task" into numbers a collection team can hit and a reviewer can check: which streams, at what rate, synchronized how tightly, in which environments, and how many episodes. Each field carries a note on why it is there, so the spec can be argued with rather than just filled in.

Who fills it: The technical lead for the dataset, with the person who will train the policy and the person who runs collection.

How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.

## 1. Task definition

Define the task as the policy will face it at deployment, not as it looks in a demo video.

### 1.1 Task name

A short, unique name used in file names and episode metadata.

Example: bin_pick_soft_pouch

Answer: 

> Why: The name becomes a key in every episode record. Changing it later breaks joins between episodes, labels and evaluation logs.

### 1.2 Task statement

One sentence: the start state, the action, and the end state.

Example: From a bin of 5–15 mixed soft pouches, pick one pouch and place it flat in the tote within 20 s.

Answer: 

> Why: A start and end state make episode boundaries unambiguous, which annotators and the success label both depend on.

### 1.3 Success criterion

An observable condition someone other than the operator can judge from the recording.

Example: Pouch fully inside tote, not folded, gripper open and clear of tote at episode end.

Answer: 

> Why: If success is judged by feel, labels drift between labelers and between data collection and evaluation. Use the same criterion in the policy evaluation plan.

### 1.4 Embodiment

Robot model, end effector, degrees of freedom, and any mounted sensors.

Example: 6-DoF arm, parallel-jaw gripper with fingertip tactile pads, wrist force/torque sensor.

Answer: 

> Why: Data collected on, or retargeted to, a different embodiment needs a mapping step. Writing the target down makes that gap visible early.

### 1.5 Action space

What the policy outputs and how often.

Example: Delta end-effector pose (6) + gripper width (1) at 10 Hz.

Answer: 

> Why: The recorded actions must be convertible to this representation at this rate, or the data cannot supervise the policy directly.

### 1.6 Observations available at deployment

The sensors the policy will actually have on the robot in production.

Example: Wrist RGB, head RGB-D, joint states, fingertip tactile. No EMG, no gloves.

Answer: 

> Why: Streams recorded only during collection (EMG, a glove, motion capture) can still be valuable as training targets or auxiliary signals, but the plan for using them has to be explicit.

### 1.7 Out of scope

Situations the dataset deliberately does not cover.

Example: Torn pouches, pouches heavier than 1 kg, human in the workcell.

Answer: 

> Why: Saying what is excluded stops collection from drifting into edge cases and tells evaluators what not to test.

## 2. Modalities and sampling rates

List every stream that will be recorded, including the ones only used during collection.

| Stream | Sensor | Rate | Units / resolution | Frame | At deployment? |
| --- | --- | --- | --- | --- | --- |
| wrist_rgb | Wrist camera | Camera's native frame rate | Width × height, 8-bit RGB | camera_wrist | Yes |
| joint_state | Robot controller | Controller's state rate | rad, rad/s | base | Yes |
| ft_wrist | Wrist force/torque sensor | Fast enough to resolve contact onset | N, N·m | tool0 | Yes |
| tactile_left | Fingertip tactile pad | Fast enough to resolve slip | As the sensor reports it | finger_left | Yes / No |
| emg_forearm | Forearm EMG band | The band's native rate | µV or normalized | n/a | No |

### 2.1 Stream list

Fill the table above: one row per stream, with sensor model, rate, units, coordinate frame, and whether it exists at deployment.

Answer: 

> Why: A stream without units or a frame is hard to use a year later. The deployment column separates policy inputs from supervision-only signals.

### 2.2 Rate rationale

For each stream, why this rate: the fastest event it must resolve.

Example: Tactile at the sensor's native rate because slip onset is the event we label; RGB at the camera's native rate because motion blur, not frame rate, limits us.

Answer: 

> Why: Rates are often set to whatever the driver defaults to. Tying each to an event makes undersampling (missed contact transients) and oversampling (storage cost) both arguable.

### 2.3 Rate at training time

How streams will be resampled or windowed for the policy, if differently from capture.

Example: All streams aligned to 10 Hz action steps; tactile and F/T summarized over each 100 ms window.

Answer: 

> Why: Capturing fast and training slow is common. Writing the reduction down keeps the training pipeline and the labels consistent.

## 3. Timing and synchronization

### 3.1 Clock source

Which clock every timestamp comes from, and how devices are synchronized to it.

Example: Host monotonic clock; cameras hardware-triggered; tactile and EMG timestamped on arrival with measured offset.

Answer: 

> Why: Mixed clocks are the most common reason contact signals line up with the wrong video frame.

### 3.2 Timestamp unit

The unit and type used everywhere.

Example: Integer nanoseconds since episode start, plus absolute start time in UTC.

Answer: 

> Why: Floats in seconds lose precision over long sessions; a single unit avoids silent conversions.

### 3.3 Synchronization tolerance

The maximum allowed offset between each pair of streams that are used together.

Example: A maximum offset in milliseconds for each pair: actions ↔ joint state, camera ↔ joint state, tactile and F/T ↔ joint state.

Answer: 

> Why: Set the tolerance from the shortest event you need to align (a contact onset, a release). If streams drift more than that event lasts, labels attach to the wrong moment.

### 3.4 Sync check method

How the offset is measured in every session.

Example: Operator taps the fingertip on a fixed plate at session start and end; the tap must appear in tactile, F/T and video within tolerance.

Answer: 

> Why: A tolerance nobody measures is a hope. A physical event visible in every stream gives a number per session.

## 4. Calibration

### 4.1 Camera intrinsics and extrinsics

How and how often they are calibrated, and where the calibration files live.

Example: Checkerboard intrinsics per camera at install; extrinsics via hand-eye calibration weekly and after any bump.

Answer: 

> Why: Policies that use 3D information inherit every calibration error. Each episode must reference the calibration it was recorded under.

### 4.2 Force and tactile tare

When force and tactile sensors are zeroed.

Example: Tare with gripper open and unloaded at the start of every session and after any collision.

Answer: 

> Why: Force/torque and many tactile sensors drift with temperature and load history; untared offsets look like contact.

### 4.3 Recalibration triggers

Events that force recalibration before collection continues.

Example: Camera moved, sensor swapped, sync check out of tolerance, collision.

Answer: 

> Why: Explicit triggers stop operators from carrying on with a known-bad setup to hit a daily target.

## 5. Data source

### 5.1 Collection method

Teleoperation, kinesthetic teaching, human demonstration with wearable sensors, scripted, autonomous rollouts, or a mix (with shares).

Example: Human demonstrations with wearable sensing for contact phases; teleoperation on the target robot for the full task.

Answer: 

> Why: Each method has a different embodiment gap, throughput and cost. Mixing is fine if every episode records which method produced it.

### 5.2 Operators

How many people demonstrate, and what they must be able to do.

Example: Four operators, each passing a 10-episode qualification run before their data counts.

Answer: 

> Why: One operator gives consistent but narrow data; several give variety but need a qualification bar. Either way, operator ID goes in every episode.

## 6. Environments and variation

### 6.1 Environments

Each place data is collected, with an ID and whether it is used for training or held out.

Example: lab_a (train), lab_b (train), customer_site_1 (held-out).

Answer: 

> Why: A held-out environment is the cheapest honest test of generalization. Decide it before collection, not after results are in.

### 6.2 Variation factors

What changes between episodes, and the range of each.

Example: Lighting (3 settings), bin fill level (5–15 items), pouch pose (random), distractor objects (0–3).

Answer: 

> Why: Policies generalize along the axes the data varies on. Listing them turns 'diverse data' into a grid you can check coverage against.

### 6.3 Object set

The objects used, split into train and held-out.

Example: Train: 24 pouch SKUs. Held-out: 6 SKUs never shown in training.

Answer: 

> Why: Held-out objects measure what the policy learned about the category rather than the specific items.

## 7. Volume targets

### 7.1 Pilot size

How many episodes the pilot collects before the full target is set.

Example: Enough episodes to train one small policy end to end, across 2 environments and 2 operators.

Answer: 

> Why: Published dataset sizes vary by orders of magnitude between tasks. A pilot plus a small training run shows how performance moves with data for your task, which is a better basis for the target than another team's number.

### 7.2 Episodes per condition

Target episodes for each cell of the variation grid.

Example: The same target for every (environment × lighting × fill level) cell, so no cell stays empty.

Answer: 

> Why: A single total hides empty cells. Per-cell targets make coverage gaps visible while collection is still running.

### 7.3 Total target

Total episodes and hours, and the date it should be reached.

Answer: 

> Why: Converts the plan into operator-hours and storage, so budget and schedule can be checked.

### 7.4 Failures kept

Whether failed attempts are kept and labelled, and how they are used.

Example: Kept, labelled with failure mode; excluded from behaviour cloning, used for success classifiers and analysis.

Answer: 

> Why: Failed episodes are expensive to collect and useful for success detection and recovery behaviour. Deleting them loses that.

## 8. Acceptance and quality

### 8.1 Acceptance checks

Automatic checks every episode must pass.

Example: All required streams present; dropped frames below the spec's threshold; sync offset within tolerance; episode length within bounds.

Answer: 

> Why: Automated checks at ingest catch broken sessions the same day, while they can still be re-recorded.

### 8.2 Human review

What share of episodes a person reviews, and what they look for.

Example: A fixed random share of each operator's episodes per day: task followed, success label correct, no PII in frame.

Answer: 

> Why: Some problems (wrong object, mislabelled success) only a person sees. A fixed sample rate keeps review from being skipped under deadline.

## 9. Storage and versioning

### 9.1 Format

Container format for raw and training-ready data.

Example: Raw: MCAP per session. Training: converted to the training framework's format per dataset version.

Answer: 

> Why: Keeping raw data in a lossless, timestamped container means the training format can change without re-collecting.

### 9.2 Naming and versioning

How sessions, episodes and dataset versions are named.

Example: {site}_{date}_{session}/ep_{0001}; dataset versions as semver with a changelog.

Answer: 

> Why: Models must be traceable to the exact data version they were trained on.

### 9.3 Retention and access

Who can access raw data, and how long it is kept.

Answer: 

> Why: Video and biosignals of people may be personal data. Retention and access rules must match what operators consented to.

## 10. Sign-off

### 10.1 Owner and reviewers

One owner, the reviewers, and the date this version was approved.

Answer: 

> Why: A spec without an owner stops being updated the first time collection deviates from it.

Use it with