Sensing modality selection template
Every extra sensor costs synchronization work, calibration, storage and operator time. This worksheet makes you say, for one task, what each modality would tell the policy that the others don't, whether the robot will have it at deployment, and what it costs to capture. Score each modality on the same criteria, then record the decision and the reason.
Who fills it: The person specifying the dataset, ideally with someone who has run the task by hand.
When to use it
- Before writing the data requirements spec, when the stream list is still open.
- A vision-only policy fails at contact: slips, crushes, misses insertion, or can't tell when a grasp is secure.
- You are choosing between collection approaches that capture touch and force in different ways.
When something else fits better
- The sensors are fixed by hardware you can't change: go straight to the data requirements spec.
- The task is free-space motion with no meaningful contact (e.g. pointing, gross navigation): video and proprioception are usually enough.
Preview
1.Task profile
Answer these first: the scores below depend on them.
The task in one sentence, with start and end state.
Example: Insert a flexible connector into its socket until it clicks, from a loose grasp on the cable.
List the phases where the hand or gripper touches something, and what must be controlled in each.
Example: Grasp (grip force), alignment (lateral force), insertion (axial force until click), release.
If a policy or baseline exists, how it fails. Otherwise, how a novice human fails.
Example: Pushes too hard and bends pins; or stops before the click.
At the moment of contact, can a camera see it?
Example: No: the fingers cover the connector during insertion.
What the production robot will have.
Example: Wrist camera, joint torques, wrist F/T. No fingertip tactile yet.
2.RGB video
Adds information when
- Object identity, pose and scene layout matter.
- Large datasets or pretrained visual models are part of the plan.
- Human demonstrations will be retargeted from what the person did, seen from their viewpoint.
Adds little when
- The key event is hidden by the hand or gripper.
- Success depends on force (how hard, how long), which video shows only indirectly.
Which cameras: head or egocentric, wrist, fixed external. Why each.
Information for this task (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
3.Depth
Adds information when
- Geometry drives the action: grasp points on unknown objects, clearance, stacking.
- Lighting varies a lot between environments.
Adds little when
- Objects are transparent, reflective or very thin, where many depth sensors return holes.
- The camera is very close to the contact, inside the sensor's minimum range.
Working distance and the surfaces in the scene. Check both against the sensor's datasheet.
Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
4.Tactile
Adds information when
- Contact location, onset, slip or release must be detected precisely.
- Contact is occluded from every camera.
- Objects are deformable or fragile, so grasp quality isn't visible.
Adds little when
- The robot will never have tactile sensing and there is no plan to use it as a training target.
- Contact is brief and rigid, and the gripper's own position feedback already tells you it closed on something.
On the robot's fingers, on a glove worn by a human demonstrator, or on the object.
How finely you need to locate contact and how quickly you need to detect it.
Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
5.Force / torque
Adds information when
- The task is defined by force: insertion, wiping, pressing, tightening, peeling.
- Collisions and jams must be detected and labelled.
Adds little when
- Contact forces are tiny compared with the sensor's noise floor.
- Joint torque estimates on the robot already give the signal at the precision you need.
Wrist six-axis sensor, joint torques, fingertip force, or a force plate under the workpiece.
Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
6.EMG
Adds information when
- You record human demonstrations and want a signal of grip effort and intent from the demonstrator's muscles.
- Effort changes before motion does, e.g. bracing before a pull, and you want to label or predict that.
Adds little when
- Data comes from teleoperation or the robot itself: there is no human muscle to measure.
- You need absolute force in newtons: surface EMG relates to muscle activation, and mapping it to force needs per-person calibration.
Policy input (rare, since robots have no muscles), training target, auxiliary signal, or labeling aid.
Example: Auxiliary signal for grip-effort phases; helps label grasp tightening and release.
How biosignal data is covered by operator consent and access rules.
Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
7.Ways to capture touch and force
If tactile, force or EMG scored well above, choose how to capture them. These are common approaches, scored on the same criteria. Specs and prices change: check each one's current documentation.
| Approach | What it measures | Embodiment | Where it works | Watch for |
|---|---|---|---|---|
| Preload: head-mounted camera, tactile sensor glove and forearm EMG band, synchronized | Egocentric video, contact and pressure at the hand, muscle activity behind grip effort | Human hand; needs mapping to the robot gripper | Wherever a person does the task, including real workplaces, without instrumenting a robot | Embodiment gap shared by all human-demonstration data; check preload.ai for channel counts, rates and formats |
| Teleoperation with wrist F/T and fingertip tactile on the robot | Force and contact as the robot experiences them | Robot's own | Where the robot is installed | Operator throughput; limited haptic feedback to the operator can change how they grasp |
| Kinesthetic teaching with joint torque sensing | Joint torques and robot state while a person guides the arm | Robot's own | Robots that support hand-guiding | The person's hand on the arm adds forces to separate from contact forces |
| Instrumented objects or force plates | Forces on the object or work surface | Independent of who does the task | Fixed workstations with known objects | Only covers objects you instrument |
| Video only, force inferred | Nothing directly; force estimated from motion and deformation | Either | Anywhere a camera fits | Estimates can't be checked without a measured reference |
Which of the approaches above (or others) you considered, with a 0–2 score on each criterion.
8.Decision record
Each modality, marked as policy input, supervision-only, or skipped.
Example: Wrist RGB + joint state: input. Fingertip tactile: input. EMG: supervision-only. Depth: skipped (thin parts).
Each modality you skipped and the reason.
The result that would change this decision.
Example: If insertion success stays below target after the pilot, add wrist F/T.
Copy or download
The whole template with prompts, examples and the reason for each field.
# Sensing modality selection
Every extra sensor costs synchronization work, calibration, storage and operator time. This worksheet makes you say, for one task, what each modality would tell the policy that the others don't, whether the robot will have it at deployment, and what it costs to capture. Score each modality on the same criteria, then record the decision and the reason.
Who fills it: The person specifying the dataset, ideally with someone who has run the task by hand.
How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.
## 1. Task profile
Answer these first: the scores below depend on them.
### 1.1 Task
The task in one sentence, with start and end state.
Example: Insert a flexible connector into its socket until it clicks, from a loose grasp on the cable.
Answer:
> Why: Modality value is task-specific. A worksheet filled for 'manipulation in general' always concludes that everything helps.
### 1.2 Contact phases
List the phases where the hand or gripper touches something, and what must be controlled in each.
Example: Grasp (grip force), alignment (lateral force), insertion (axial force until click), release.
Answer:
> Why: Touch and force sensing earn their cost in contact phases. A task with one brief grasp needs them less than one with sustained, regulated contact.
### 1.3 Current failure mode
If a policy or baseline exists, how it fails. Otherwise, how a novice human fails.
Example: Pushes too hard and bends pins; or stops before the click.
Answer:
> Why: Failures caused by information the camera can't see (force, slip, hidden contact) point to touch or force sensing. Failures of perception or planning usually don't.
### 1.4 Occlusion at contact
At the moment of contact, can a camera see it?
Example: No: the fingers cover the connector during insertion.
Answer:
> Why: When the contact is occluded, video can only infer it from context; a contact sensor measures it.
### 1.5 Sensors at deployment
What the production robot will have.
Example: Wrist camera, joint torques, wrist F/T. No fingertip tactile yet.
Answer:
> Why: A modality the robot won't have can still be recorded for supervision or analysis, but it can't be a policy input. Mark the difference now.
## 2. RGB video
Adds information when:
- Object identity, pose and scene layout matter.
- Large datasets or pretrained visual models are part of the plan.
- Human demonstrations will be retargeted from what the person did, seen from their viewpoint.
Adds little when:
- The key event is hidden by the hand or gripper.
- Success depends on force (how hard, how long), which video shows only indirectly.
### 2.1 Viewpoints
Which cameras: head or egocentric, wrist, fixed external. Why each.
Answer:
> Why: Egocentric and wrist views see the contact region best; fixed views give stable context for scene state and success labels.
### 2.2 Score
Information for this task (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
Answer:
> Why: Same criteria for every modality, so the comparison is fair.
## 3. Depth
Adds information when:
- Geometry drives the action: grasp points on unknown objects, clearance, stacking.
- Lighting varies a lot between environments.
Adds little when:
- Objects are transparent, reflective or very thin, where many depth sensors return holes.
- The camera is very close to the contact, inside the sensor's minimum range.
### 3.1 Range and surfaces
Working distance and the surfaces in the scene. Check both against the sensor's datasheet.
Answer:
> Why: Depth quality depends on range and material. The datasheet's minimum range and the scene's worst surface decide whether it helps.
### 3.2 Score
Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
Answer:
> Why: Same criteria for every modality.
## 4. Tactile
Adds information when:
- Contact location, onset, slip or release must be detected precisely.
- Contact is occluded from every camera.
- Objects are deformable or fragile, so grasp quality isn't visible.
Adds little when:
- The robot will never have tactile sensing and there is no plan to use it as a training target.
- Contact is brief and rigid, and the gripper's own position feedback already tells you it closed on something.
### 4.1 Where the sensing sits
On the robot's fingers, on a glove worn by a human demonstrator, or on the object.
Answer:
> Why: Robot-mounted tactile gives data in the robot's own embodiment; a glove captures human touch where the skill is, and needs a mapping to the gripper; object-mounted sensing measures the object's side of the contact.
### 4.2 Spatial and temporal resolution needed
How finely you need to locate contact and how quickly you need to detect it.
Answer:
> Why: Slip detection needs fast sampling; contact localization needs taxel density. Check both against candidate sensors' datasheets.
### 4.3 Score
Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
Answer:
> Why: Same criteria for every modality.
## 5. Force / torque
Adds information when:
- The task is defined by force: insertion, wiping, pressing, tightening, peeling.
- Collisions and jams must be detected and labelled.
Adds little when:
- Contact forces are tiny compared with the sensor's noise floor.
- Joint torque estimates on the robot already give the signal at the precision you need.
### 5.1 Measurement point
Wrist six-axis sensor, joint torques, fingertip force, or a force plate under the workpiece.
Answer:
> Why: A wrist sensor sums every contact on the tool; fingertip or object-side sensing separates them. Pick the point where the force you care about is isolated.
### 5.2 Score
Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
Answer:
> Why: Same criteria for every modality.
## 6. EMG
Adds information when:
- You record human demonstrations and want a signal of grip effort and intent from the demonstrator's muscles.
- Effort changes before motion does, e.g. bracing before a pull, and you want to label or predict that.
Adds little when:
- Data comes from teleoperation or the robot itself: there is no human muscle to measure.
- You need absolute force in newtons: surface EMG relates to muscle activation, and mapping it to force needs per-person calibration.
### 6.1 What it will be used for
Policy input (rare, since robots have no muscles), training target, auxiliary signal, or labeling aid.
Example: Auxiliary signal for grip-effort phases; helps label grasp tightening and release.
Answer:
> Why: EMG is supervision-only for almost every robot policy. Writing its role down keeps it from being collected 'just in case'.
### 6.2 Consent and handling
How biosignal data is covered by operator consent and access rules.
Answer:
> Why: Muscle signals are recorded from a person's body. Treat them under the same consent and retention rules as video of the person.
### 6.3 Score
Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.
Answer:
> Why: Same criteria for every modality.
## 7. Ways to capture touch and force
If tactile, force or EMG scored well above, choose how to capture them. These are common approaches, scored on the same criteria. Specs and prices change: check each one's current documentation.
| Approach | What it measures | Embodiment | Where it works | Watch for |
| --- | --- | --- | --- | --- |
| Preload: head-mounted camera, tactile sensor glove and forearm EMG band, synchronized | Egocentric video, contact and pressure at the hand, muscle activity behind grip effort | Human hand; needs mapping to the robot gripper | Wherever a person does the task, including real workplaces, without instrumenting a robot | Embodiment gap shared by all human-demonstration data; check preload.ai for channel counts, rates and formats |
| Teleoperation with wrist F/T and fingertip tactile on the robot | Force and contact as the robot experiences them | Robot's own | Where the robot is installed | Operator throughput; limited haptic feedback to the operator can change how they grasp |
| Kinesthetic teaching with joint torque sensing | Joint torques and robot state while a person guides the arm | Robot's own | Robots that support hand-guiding | The person's hand on the arm adds forces to separate from contact forces |
| Instrumented objects or force plates | Forces on the object or work surface | Independent of who does the task | Fixed workstations with known objects | Only covers objects you instrument |
| Video only, force inferred | Nothing directly; force estimated from motion and deformation | Either | Anywhere a camera fits | Estimates can't be checked without a measured reference |
### 7.1 Approaches considered
Which of the approaches above (or others) you considered, with a 0–2 score on each criterion.
Answer:
> Why: Writing down the alternatives makes the choice reviewable and easy to revisit when the robot's sensors change.
## 8. Decision record
### 8.1 Modalities chosen
Each modality, marked as policy input, supervision-only, or skipped.
Example: Wrist RGB + joint state: input. Fingertip tactile: input. EMG: supervision-only. Depth: skipped (thin parts).
Answer:
> Why: This list becomes the stream table in the data requirements spec.
### 8.2 Rejected, and why
Each modality you skipped and the reason.
Answer:
> Why: Next year someone will ask why there's no depth. The reason saves re-running the analysis.
### 8.3 Revisit trigger
The result that would change this decision.
Example: If insertion success stays below target after the pilot, add wrist F/T.
Answer:
> Why: A decision with a trigger is a hypothesis you can test, not a permanent constraint.