collect

Human demonstration collection protocol template

A protocol makes session 40 look like session 1. Run it in order, every session, and log what happened: the logs are how you find out later why a batch of episodes looks different. Steps marked with a check must be completed before recording continues.

Who fills it: The collection lead writes it once per task; the session supervisor follows it and fills the logs every session.

Other formats

When to use it

  • Anyone other than the person who designed the task will record demonstrations.
  • Collection runs across several days, operators or sites.
  • Demonstrations are recorded from people wearing sensors, so consent and comfort are part of the session.

When something else fits better

  • You are still deciding what to record: write the data requirements spec first; this protocol carries out its decisions.
  • Data comes only from autonomous robot rollouts with no person in the loop: use the calibration, sync and logging steps and skip the consent and operator ones.

Preview

demonstration-collection-protocol.md

1.Roles and versions

1.1Protocol version

Version and date. Record it in every session log.

Example: v1.2, 2026-11-02

Why this fieldWhen the protocol changes mid-collection, the version in each log tells you which episodes were recorded under which rules.
1.2Roles

Who is the operator (demonstrates), who supervises (runs the checklist), and who to call when hardware fails.

Why this fieldAn operator can't watch sync dashboards while doing the task. Separating the roles is what makes the checks happen.
1.3Spec reference

The data requirements spec version this protocol implements.

Why this fieldRates, tolerances and acceptance checks live in the spec. The protocol points to them instead of copying numbers that will drift.

2.Consent and briefing

Before any sensor is put on or any recording starts.

2.1Informed consent

The consent form the operator signs: what is recorded (video, audio, face, hands, biosignals), what it's used for, who can access it, how long it's kept, and how to withdraw.

Why this fieldVideo and biosignals identify people. Check the requirements that apply to you (an ethics board, works council, or data-protection counsel) before the first session, not after.
2.2Withdrawal

How an operator stops a session or withdraws their data, and how you delete it.

Why this fieldWithdrawal is only real if deletion is possible: operator IDs must map to every episode they recorded.
2.3Pseudonymous operator ID

The ID used in all data instead of the operator's name.

Example: op_017

Why this fieldKeeps names out of the dataset while still letting you analyze per-operator differences.
2.4Briefing

What the operator is told: the task, the success criterion, what to do on failure, and that failures are expected and kept.

Why this fieldOperators who think failures are 'bad data' hide or abort them. Saying failures are wanted keeps the failure distribution honest.

3.Task script

3.1Start pose and scene

Exactly how the scene and the operator start each episode. Include a photo reference.

Example: Hands resting on the marked pads; bin at position A; tote empty.

Why this fieldA consistent start makes episode boundaries detectable and lets the variation you planned be the only variation.
3.2Steps

Numbered steps for the task, with what is allowed to vary.

Example: 1. Pick any pouch. 2. Place it flat in the tote. Any grasp is allowed; do not use two hands.

Why this fieldToo strict a script produces unnaturally uniform data; too loose a script produces multiple tasks. Say what's fixed and what's free.
3.3End condition

The observable state that ends an episode, and the time limit.

Why this fieldMust match the success criterion in the spec so the success label can be read off the recording.
3.4Reset

How the scene is reset between episodes, and how the next variation is chosen.

Example: Supervisor reshuffles the bin and draws the next lighting setting from the schedule.

Why this fieldUnplanned resets bias the data toward easy configurations. A schedule keeps coverage of the variation grid on track.

4.Setup and calibration

4.1Hardware check

✓ Every sensor powered, streaming and showing live data in the recorder.

Why this fieldA silent stream is the most common way to lose a whole session.
4.2Wearable fit

✓ For wearable sensors: placement per the device's instructions, fit checked, comfort confirmed with the operator.

Why this fieldPlacement changes signals, especially for EMG and gloves. Photograph placement so it can be repeated.
4.3Tare

✓ Force and tactile sensors zeroed in the unloaded state.

Why this fieldUntared offsets appear as contact in every episode of the session.
4.4Calibration ID

✓ The camera calibration in use, recorded in the session log.

Why this fieldEvery episode needs to reference the calibration it was recorded under.

5.Sync check

5.1Start-of-session sync event

✓ Perform the sync event (e.g. three taps on a fixed plate) visible to every stream. Measure the offset between streams.

Why this fieldA sharp physical event is the only way to measure true end-to-end alignment, including driver and USB latency.
5.2Tolerance gate

✓ If any offset exceeds the spec's tolerance, stop and fix before recording.

Why this fieldEpisodes recorded out of tolerance usually can't be repaired afterwards with confidence.
5.3End-of-session sync event

✓ Repeat the sync event at the end and log both offsets.

Why this fieldComparing start and end offsets reveals clock drift during the session.

6.Recording episodes

6.1One attempt, one episode

Start a new episode for every attempt, including retries after a failure.

Why this fieldRetries inside one episode mix a failure and a success, which corrupts the success label.
6.2Episode markers

How start and end are marked: supervisor key press, foot pedal, or automatic from the start pose.

Why this fieldMarked boundaries are cheaper than having annotators find them later.
6.3Live notes

The supervisor notes anything unusual with the episode ID: interruptions, dropped objects, sensor warnings.

Why this fieldNotes written during the session are far more reliable than recollection during labeling.

7.Failure logging

7.1Failure taxonomy

The fixed list of failure modes for this task.

Example: grasp_miss, slip, drop, collision, wrong_object, timeout, operator_error, sensor_dropout, other

Why this fieldA fixed list makes failure counts comparable across sessions and is reused by the annotation schema and the evaluation plan.
7.2Log every failure

Fill a row in the failure log for each failed episode, with the time of failure.

Why this fieldFailures are kept, not deleted. The time stamp lets annotators jump to the moment.
7.3Hardware failures

Sensor dropouts and recorder crashes are logged separately from task failures.

Why this fieldA dropout isn't a task outcome. Mixing them distorts success rates.

8.Breaks and fatigue

8.1Break schedule

How often operators rest, set during the pilot.

Why this fieldFatigue changes how people move and grip, and it shows in EMG. Scheduled breaks keep the data from drifting within a session.
8.2Stop conditions

When the session stops: discomfort, pain, hardware fault, or operator request.

Why this fieldOperator wellbeing comes before episode count, and it should be written down so nobody has to argue for it.

9.Close-out

9.1Offload and verify

✓ Copy data off the recorder, verify checksums, and run the spec's acceptance checks.

Why this fieldSame-day checks catch broken sessions while the setup is still in place to re-record.
9.2Session log

✓ Complete the session log row.

Why this fieldThe session log is the index of the dataset. Without it, raw files are hard to interpret.
9.3Incidents

Anything that went wrong and what was done about it.

Why this fieldIncidents explain anomalies that show up weeks later in training curves.

Copy or download

demonstration-collection-protocol.md

The whole template with prompts, examples and the reason for each field.

# Human demonstration collection protocol

A protocol makes session 40 look like session 1. Run it in order, every session, and log what happened: the logs are how you find out later why a batch of episodes looks different. Steps marked with a check must be completed before recording continues.

Who fills it: The collection lead writes it once per task; the session supervisor follows it and fills the logs every session.

How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.

## 1. Roles and versions

### 1.1 Protocol version

Version and date. Record it in every session log.

Example: v1.2, 2026-11-02

Answer: 

> Why: When the protocol changes mid-collection, the version in each log tells you which episodes were recorded under which rules.

### 1.2 Roles

Who is the operator (demonstrates), who supervises (runs the checklist), and who to call when hardware fails.

Answer: 

> Why: An operator can't watch sync dashboards while doing the task. Separating the roles is what makes the checks happen.

### 1.3 Spec reference

The data requirements spec version this protocol implements.

Answer: 

> Why: Rates, tolerances and acceptance checks live in the spec. The protocol points to them instead of copying numbers that will drift.

## 2. Consent and briefing

Before any sensor is put on or any recording starts.

### 2.1 Informed consent

The consent form the operator signs: what is recorded (video, audio, face, hands, biosignals), what it's used for, who can access it, how long it's kept, and how to withdraw.

Answer: 

> Why: Video and biosignals identify people. Check the requirements that apply to you (an ethics board, works council, or data-protection counsel) before the first session, not after.

### 2.2 Withdrawal

How an operator stops a session or withdraws their data, and how you delete it.

Answer: 

> Why: Withdrawal is only real if deletion is possible: operator IDs must map to every episode they recorded.

### 2.3 Pseudonymous operator ID

The ID used in all data instead of the operator's name.

Example: op_017

Answer: 

> Why: Keeps names out of the dataset while still letting you analyze per-operator differences.

### 2.4 Briefing

What the operator is told: the task, the success criterion, what to do on failure, and that failures are expected and kept.

Answer: 

> Why: Operators who think failures are 'bad data' hide or abort them. Saying failures are wanted keeps the failure distribution honest.

## 3. Task script

### 3.1 Start pose and scene

Exactly how the scene and the operator start each episode. Include a photo reference.

Example: Hands resting on the marked pads; bin at position A; tote empty.

Answer: 

> Why: A consistent start makes episode boundaries detectable and lets the variation you planned be the only variation.

### 3.2 Steps

Numbered steps for the task, with what is allowed to vary.

Example: 1. Pick any pouch. 2. Place it flat in the tote. Any grasp is allowed; do not use two hands.

Answer: 

> Why: Too strict a script produces unnaturally uniform data; too loose a script produces multiple tasks. Say what's fixed and what's free.

### 3.3 End condition

The observable state that ends an episode, and the time limit.

Answer: 

> Why: Must match the success criterion in the spec so the success label can be read off the recording.

### 3.4 Reset

How the scene is reset between episodes, and how the next variation is chosen.

Example: Supervisor reshuffles the bin and draws the next lighting setting from the schedule.

Answer: 

> Why: Unplanned resets bias the data toward easy configurations. A schedule keeps coverage of the variation grid on track.

## 4. Setup and calibration

### 4.1 Hardware check

✓ Every sensor powered, streaming and showing live data in the recorder.

Answer: 

> Why: A silent stream is the most common way to lose a whole session.

### 4.2 Wearable fit

✓ For wearable sensors: placement per the device's instructions, fit checked, comfort confirmed with the operator.

Answer: 

> Why: Placement changes signals, especially for EMG and gloves. Photograph placement so it can be repeated.

### 4.3 Tare

✓ Force and tactile sensors zeroed in the unloaded state.

Answer: 

> Why: Untared offsets appear as contact in every episode of the session.

### 4.4 Calibration ID

✓ The camera calibration in use, recorded in the session log.

Answer: 

> Why: Every episode needs to reference the calibration it was recorded under.

## 5. Sync check

### 5.1 Start-of-session sync event

✓ Perform the sync event (e.g. three taps on a fixed plate) visible to every stream. Measure the offset between streams.

Answer: 

> Why: A sharp physical event is the only way to measure true end-to-end alignment, including driver and USB latency.

### 5.2 Tolerance gate

✓ If any offset exceeds the spec's tolerance, stop and fix before recording.

Answer: 

> Why: Episodes recorded out of tolerance usually can't be repaired afterwards with confidence.

### 5.3 End-of-session sync event

✓ Repeat the sync event at the end and log both offsets.

Answer: 

> Why: Comparing start and end offsets reveals clock drift during the session.

## 6. Recording episodes

### 6.1 One attempt, one episode

Start a new episode for every attempt, including retries after a failure.

Answer: 

> Why: Retries inside one episode mix a failure and a success, which corrupts the success label.

### 6.2 Episode markers

How start and end are marked: supervisor key press, foot pedal, or automatic from the start pose.

Answer: 

> Why: Marked boundaries are cheaper than having annotators find them later.

### 6.3 Live notes

The supervisor notes anything unusual with the episode ID: interruptions, dropped objects, sensor warnings.

Answer: 

> Why: Notes written during the session are far more reliable than recollection during labeling.

## 7. Failure logging

### 7.1 Failure taxonomy

The fixed list of failure modes for this task.

Example: grasp_miss, slip, drop, collision, wrong_object, timeout, operator_error, sensor_dropout, other

Answer: 

> Why: A fixed list makes failure counts comparable across sessions and is reused by the annotation schema and the evaluation plan.

### 7.2 Log every failure

Fill a row in the failure log for each failed episode, with the time of failure.

Answer: 

> Why: Failures are kept, not deleted. The time stamp lets annotators jump to the moment.

### 7.3 Hardware failures

Sensor dropouts and recorder crashes are logged separately from task failures.

Answer: 

> Why: A dropout isn't a task outcome. Mixing them distorts success rates.

## 8. Breaks and fatigue

### 8.1 Break schedule

How often operators rest, set during the pilot.

Answer: 

> Why: Fatigue changes how people move and grip, and it shows in EMG. Scheduled breaks keep the data from drifting within a session.

### 8.2 Stop conditions

When the session stops: discomfort, pain, hardware fault, or operator request.

Answer: 

> Why: Operator wellbeing comes before episode count, and it should be written down so nobody has to argue for it.

## 9. Close-out

### 9.1 Offload and verify

✓ Copy data off the recorder, verify checksums, and run the spec's acceptance checks.

Answer: 

> Why: Same-day checks catch broken sessions while the setup is still in place to re-record.

### 9.2 Session log

✓ Complete the session log row.

Answer: 

> Why: The session log is the index of the dataset. Without it, raw files are hard to interpret.

### 9.3 Incidents

Anything that went wrong and what was done about it.

Answer: 

> Why: Incidents explain anomalies that show up weeks later in training curves.

Use it with