Robot policy evaluation plan template
Real-robot evaluations are slow, so they tend to be small and informal, and small informal evaluations are easy to fool yourself with. Write this plan before running the policy: what question the evaluation answers, which trials, how success is judged and by whom, and how results will be reported with their uncertainty.
Who fills it: The person training the policy, reviewed by someone who didn't train it.
When to use it
- Comparing two checkpoints, two datasets, or a policy against a baseline on hardware.
- Deciding whether a policy is ready for a pilot deployment.
- Reporting results in a paper or to a customer, where others need to trust the number.
When something else fits better
- Quick smoke tests during development. Use the trial log alone and don't report those numbers as results.
Preview
1.Policy under test
Model checkpoint ID and training run.
Dataset version and any filtering applied.
Inputs, action representation, control rate.
2.Questions
One or two specific questions.
Example: Does adding fingertip tactile input improve insertion success on held-out connectors?
What you will do differently depending on the answer.
3.Trial design
The conditions tested: environments, objects, variation factors. Mark which are held out from training.
How each trial's starting state is set: a fixed list, or randomized from a seed.
Number of trials per condition and per policy, and the reasoning.
What the policy is compared with, run under identical conditions.
Example: Previous checkpoint; vision-only variant; scripted controller.
How trials of different policies are interleaved, and whether the evaluator knows which policy is running.
4.Success criteria
The same observable criterion as in the data requirements spec.
Stages that count as partial success, if any.
Example: Grasped (0.33), lifted (0.66), placed (1.0).
Maximum trial time, and whether any human intervention counts as failure.
Who decides the outcome, and how disagreements are resolved.
5.Ground-truth references
What independent measurements are recorded during trials.
Example: External video for every trial; wrist F/T for peak force; final object pose from a fixed camera.
Human demonstrations of the same trials, if recorded, for comparing force profiles or timing.
6.Safety
Force, speed or workspace limits that trigger a stop, and who can press the stop.
Who may be present during trials, and where.
7.Hardware test log
Fill one row per trial in the trial log (download below). Never delete a row; mark invalid trials as invalid with a reason.
Where the trial log and videos are stored.
What makes a trial invalid (e.g. setup error, hardware fault) and whether it is re-run.
8.Analysis and reporting
Success rate with confidence interval per condition, failure-mode breakdown, time to complete.
What counts as one policy beating another.
Who reviews the result and when.
Copy or download
The whole template with prompts, examples and the reason for each field.
# Robot policy evaluation plan
Real-robot evaluations are slow, so they tend to be small and informal, and small informal evaluations are easy to fool yourself with. Write this plan before running the policy: what question the evaluation answers, which trials, how success is judged and by whom, and how results will be reported with their uncertainty.
Who fills it: The person training the policy, reviewed by someone who didn't train it.
How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.
## 1. Policy under test
### 1.1 Checkpoint
Model checkpoint ID and training run.
Answer:
> Why: A result is only reproducible if the exact weights are named.
### 1.2 Training data version
Dataset version and any filtering applied.
Answer:
> Why: Connects evaluation results back to data decisions, which is how you learn which data helped.
### 1.3 Observation and action configuration
Inputs, action representation, control rate.
Answer:
> Why: Must match the spec's deployment observations. A mismatch here explains many lab-to-site drops.
## 2. Questions
### 2.1 Question this evaluation answers
One or two specific questions.
Example: Does adding fingertip tactile input improve insertion success on held-out connectors?
Answer:
> Why: The question decides the trials. 'How good is it?' leads to a set of trials that answers nothing in particular.
### 2.2 Decision it informs
What you will do differently depending on the answer.
Answer:
> Why: If no result would change a decision, the evaluation can be smaller.
## 3. Trial design
### 3.1 Conditions
The conditions tested: environments, objects, variation factors. Mark which are held out from training.
Answer:
> Why: Results on training conditions measure fit; held-out conditions measure generalization. Report them separately.
### 3.2 Initial conditions
How each trial's starting state is set: a fixed list, or randomized from a seed.
Answer:
> Why: Choosing start states by hand during the evaluation tends to drift toward ones the policy handles.
### 3.3 Trials per condition
Number of trials per condition and per policy, and the reasoning.
Answer:
> Why: Success rates from a handful of trials have wide confidence intervals. Compute the interval you'd get (e.g. a Wilson interval) for the planned count before running, and check it is narrow enough to answer the question.
### 3.4 Baselines
What the policy is compared with, run under identical conditions.
Example: Previous checkpoint; vision-only variant; scripted controller.
Answer:
> Why: Comparisons run on different days or setups confound the policy with the setup.
### 3.5 Order and blinding
How trials of different policies are interleaved, and whether the evaluator knows which policy is running.
Answer:
> Why: Interleaving controls for drift (lighting, wear, battery). Blinding the person judging success removes a known source of bias.
## 4. Success criteria
### 4.1 Success definition
The same observable criterion as in the data requirements spec.
Answer:
> Why: If training labels and evaluation use different definitions, you are measuring a different task.
### 4.2 Partial credit
Stages that count as partial success, if any.
Example: Grasped (0.33), lifted (0.66), placed (1.0).
Answer:
> Why: Partial credit shows where a policy fails, but report full success separately so it can't hide a low completion rate.
### 4.3 Time limit and interventions
Maximum trial time, and whether any human intervention counts as failure.
Answer:
> Why: Unbounded retries and resets inflate success. Decide what counts before the first trial.
### 4.4 Judge
Who decides the outcome, and how disagreements are resolved.
Answer:
> Why: A second reviewer on recorded video for a sample of trials catches inconsistent judging.
## 5. Ground-truth references
### 5.1 Reference measurements
What independent measurements are recorded during trials.
Example: External video for every trial; wrist F/T for peak force; final object pose from a fixed camera.
Answer:
> Why: Success alone hides how a policy succeeds. A measured reference (force, pose, time) shows whether it is gentle, precise or lucky.
### 5.2 Human reference
Human demonstrations of the same trials, if recorded, for comparing force profiles or timing.
Answer:
> Why: Where demonstrations included touch or force sensing, comparing the policy's contact profile with the human's is a direct check on whether it learned the physical part of the skill.
## 6. Safety
### 6.1 Stop conditions
Force, speed or workspace limits that trigger a stop, and who can press the stop.
Answer:
> Why: Untested policies do unexpected things. Stops protect people and hardware, and every stop is logged as an outcome.
### 6.2 People in the workcell
Who may be present during trials, and where.
Answer:
> Why: Follow your site's robot safety rules. The plan makes them explicit for each evaluation.
## 7. Hardware test log
Fill one row per trial in the trial log (download below). Never delete a row; mark invalid trials as invalid with a reason.
### 7.1 Log location
Where the trial log and videos are stored.
Answer:
> Why: Results tables are only as trustworthy as the log they were computed from.
### 7.2 Invalid trials
What makes a trial invalid (e.g. setup error, hardware fault) and whether it is re-run.
Answer:
> Why: Deciding upfront prevents quietly discarding failures as 'invalid'.
## 8. Analysis and reporting
### 8.1 Metrics
Success rate with confidence interval per condition, failure-mode breakdown, time to complete.
Answer:
> Why: An interval shows whether two policies actually differ. A failure breakdown shows what to fix next.
### 8.2 Comparison rule
What counts as one policy beating another.
Answer:
> Why: Set before the results come in, so the conclusion isn't chosen after seeing the data.
### 8.3 Sign-off
Who reviews the result and when.
Answer:
> Why: A second reader catches the most common errors: wrong checkpoint, wrong split, mislabelled trials.