Hmm… based only on what I could find so far:
Short version: I could not find an existing robot-policy/VLA attack reporting standard that exactly bundles those four items, but I did find fairly strong precedents for most of the pieces.
So I would not throw the list away. My default would be to keep the basic shape, but make #2 a little more method-agnostic, make #4 slightly more general than “query budget”, and add one very small piece of metadata: what exactly counts as an attack success, and what is in the denominator.
Something like:
- Matched benign control — same task and matched evaluation conditions, reported separately rather than subtracted.
- Uncertainty appropriate to the metric and sampling design — Wilson is a good simple-binomial default, but probably should not be the required method in every case.
- Evaluation provenance + attack relationship to the target — e.g. stub/simulation/local checkpoint/hosted endpoint/physical robot, plus whether the attack was optimized directly against that target or transferred from somewhere else.
- For adaptive/search attacks, the attack resource budget and stopping rule — target queries where that is the scarce resource; iterations/restarts/best-of-K where that is what actually controls search effort.
And I would probably record the attack event as x / n, not just ASR = x%.
For example:
attack_event: keep-out predicate fires at least once
attacked: 44 / 50
benign_same_measurement: 2 / 50
matched_on: task, initial state, environment randomness
uncertainty: ...
evaluation_target: local checkpoint <revision>
attack_mode: direct black-box search
budget: 100 target queries
stopping_rule: first success or budget exhausted
For a fixed non-adaptive attack, budget: N/A is perfectly informative.
So on question 4, I do not think the requirement is too strong. I think the awkward non-comparability is mostly real information rather than a defect in the table. A fixed attack and a search procedure that is allowed thousands of interactions with the evaluated policy are not the same experimental object. I would rather stratify them than erase that distinction.
Why I think the matched benign arm is the strongest part of the proposal
There is a fairly direct version of this problem already visible in VLA attack evaluation.
AttackVLA defines several different attack-success quantities. For attacks that disrupt task completion, it uses an untargeted attack success rate ASR_u = 1-SR. For static/non-responsive behavior it uses a different ASR_s, and for a targeted long-horizon sequence it uses ASR_t.
More importantly, the paper explicitly points out a problem with interpreting ASR_u: if clean task performance is already imperfect, a failed attacked rollout can be either an attack-induced failure or an ordinary policy failure. They describe clean performance as strongly affecting the interpretation of that ASR and call for more refined evaluation.
That seems very close to the problem you are describing.
The clean-control idea also appears in nearby forms, although not always with the same experimental meaning.
RoboJailBench pairs malicious and benign goals in the same scene so that security and benign utility can be evaluated together. That is not the same control you propose: their benign arm changes the instruction intent, whereas yours is closer to holding the task fixed and removing the attack. But the underlying motivation looks similar — an attack/defense number is much easier to interpret when the useful benign behavior is measured alongside it.
A recent VLA physical-safety example, RedVLA, separately reports:
- attack success rate,
- task success in the risk scenario,
- benign task success in the original benign scenario,
and additionally breaks unsafe rollouts into cases such as successful task execution with unsafe behavior, attempted task execution with unsafe behavior, and policy collapse.
That separation seems useful because these quantities answer different questions.
I would therefore keep at least three concepts distinct:
- clean competence: can the policy do the task at all?
- benign safety-event rate: how often does the same safety measurement fire without the attack?
- attacked safety-event rate: how often does it fire with the attack?
One small terminology caution: I would not automatically call 2/50 a false-positive rate unless there is independent ground truth showing that all 50 benign rollouts were actually safe. As you already noted, that 2/50 can contain both genuine policy failures and measurement/predicate errors.
So “benign rate”, “benign floor”, or simply “same predicate on the benign arm” seems safer until those causes are separated.
How I would interpret each of the four proposed fields
1. Matched benign control
I think this is the part I would preserve most directly.
I might make the wording slightly broader than “same (task, seed)”, though, because seed is really standing in for a set of things that can matter:
- task,
- initial state,
- environment randomness,
- reset behavior,
- policy randomness, if the policy is stochastic.
The current LeRobot LIBERO evaluation documentation gives a nice concrete example. For comparing policies on the same episodes, it recommends using the same seed and fixed initial states. It also notes that results can move by a few percentage points across evaluation seeds, recommends averaging the benchmark over multiple seeds, and warns that soft and hard resets are not bit-identical.
So perhaps the durable reporting requirement is not literally “same seed”, but:
state what was matched between benign and attacked trials.
That survives changes in simulator, policy, or RNG architecture better than assuming a single seed controls everything.
2. Wilson interval
I agree with the motivation, but I would probably not make Wilson specifically part of the standard.
If the reported quantity is just one binomial proportion — say 44 successes in 50 independent trials — Wilson is a very sensible default and much better than the naive normal interval.
But once the design says “matched benign and attacked trials”, there is more structure available.
For example, a pair can be:
| Benign |
Attack |
Interpretation |
| safe |
safe |
no violation in either arm |
| safe |
unsafe |
attack-associated transition |
| unsafe |
safe |
reverse transition |
| unsafe |
unsafe |
unsafe under both |
If the scientific claim is “the attack increases the probability of the event”, those paired discordant counts contain information that two separate Wilson intervals do not use. A paired binary analysis (for example an exact McNemar-style test, depending on the question) can directly use that structure.
And if many episodes are nested under a small number of tasks, treating every episode as an independent draw from one homogeneous population may answer yet another question.
This is why I would phrase the minimum more like:
report uncertainty appropriate to the metric, sampling unit, and pairing/clustering structure.
Wilson can still be the recommended simple case.
This is also consistent with the broader point in NIST AI 800-3, Expanding the AI Evaluation Toolbox with Statistical Models: benchmark uncertainty depends on what population/result you are trying to generalize to, and seemingly simple benchmark calculations can make hidden assumptions that produce invalid uncertainty estimates.
RedVLA is an interesting recent VLA-side example as well: its extended evaluation uses 10 independent runs and reports means, sample standard deviations, 95% CIs, and observed ranges rather than only one point estimate.
So I think the requirement should be uncertainty, not one universal formula.
3. Stub / local checkpoint / hosted model / transfer
I think the distinction you want is important, but I suspect there are actually two orthogonal questions hiding in this field.
One is where the result was measured:
- stub,
- simulator,
- local model/checkpoint,
- hosted endpoint,
- physical robot.
The other is how the attack relates to that evaluated target:
- optimized directly against the same target,
- optimized through black-box target queries,
- optimized on a surrogate and transferred,
- precomputed/universal and applied without target adaptation.
Those are different claims.
For example, “successful on a hosted endpoint” tells me where the measurement occurred, but not whether the attack used 5,000 adaptive calls to that endpoint or was generated elsewhere and transferred in one shot.
Conversely, “transfer attack” tells me something about the attacker-target relationship, but not whether the target was a local checkpoint, an API, or a physical deployment.
I would therefore be tempted to call the first field something like evaluation target/provenance, and keep direct vs transfer/access mode separate if the table has room.
I would not turn this into a giant threat-model taxonomy, though. The point of a minimum is probably just to stop materially different claims from collapsing into the same cell.
If useful for naming/search, the NIST adversarial-ML taxonomy separates concepts such as attacker goals, objectives, capabilities, and knowledge. I would use that as vocabulary/background rather than trying to import the whole taxonomy into a robotics results table.
4. Query budget
For genuinely adaptive search, I think this is quite defensible.
A very direct precedent is BlackboxBench. Its leaderboard fixes a maximum query count and reports attack success together with query-count statistics. In other words, “how often did it work?” and “how much target interaction did it need?” are both part of the comparison.
The same issue appears inside VLA red teaming itself.
RedVLA performs iterative optimization of a risk scenario by executing the policy, observing its trajectory, and updating the risk object pose. It also reports ASR@K, where success means at least one of K trials succeeded, and studies ASR across optimization iterations.
So the general principle seems broader than just API queries:
if attack effectiveness is a function of how much adaptive search was allowed, report the search budget.
I would probably use attack resource / optimization budget as the umbrella term, and let the concrete field depend on the attack:
- target queries,
- optimization iterations,
- restarts,
- attempts / best-of-K,
- possibly another clearly defined search resource.
I would also include the stopping rule. “100-query budget” can mean something different if one method stops at first success and another always consumes the full budget.
One small thing I might add: define the ASR event and denominator
After looking at a few of these evaluations, this may be the cheapest addition with the highest information value.
ASR is not one universal event.
Again, AttackVLA uses at least three materially different notions:
- failure to complete the task,
- remaining in a static/non-responsive state,
- producing a particular attacker-specified long-horizon action sequence.
All of those can reasonably be called an attack success in their respective threat models, but a bare ASR = 80% does not tell a future reader which one was measured.
The denominator can also change the interpretation.
Possible denominators include:
- all attacked episodes,
- only episodes eligible for a particular attack objective,
- only cases where the benign policy completed the task,
- only cases where the matched benign rollout was safe.
Those are not interchangeable.
So rather than adding another complicated metric, I would just make the primitive counts reconstructible:
attack_event:
numerator:
denominator:
eligibility:
Then the percentage is almost secondary.
This also makes later re-analysis much easier. If a community eventually decides on a different summary statistic, old results are still usable.
How I would handle fixed attacks vs search-based attacks
I do not think every attack has to fit one cost axis.
A small decision tree might be enough:
Is the attack adapted/search-optimized against the evaluated target?
No:
report attack event / x/n
report benign matched measurement
report uncertainty
report evaluation provenance
attack budget: N/A
Yes:
black-box target interaction
-> target-query budget + stopping rule
iterative optimization
-> iterations + restarts/constraints + stopping rule
repeated attempts / best-of-K
-> K or attempt budget
Transfer:
additionally state source and target,
and whether the target was queried during construction
One nuance: a perturbation can look “fixed” at evaluation time while having required substantial target-specific optimization beforehand. In that case I would report the construction/search budget, even if deployment itself is one-shot.
So I would not use “fixed vs search” purely as a runtime distinction. The useful question is whether substantial target-adaptive work was required to produce the reported artifact.
This is also why I do not think forced numerical comparability between every attack family should be a hard goal.
A fixed universal trigger, a 10,000-query black-box optimizer, and a surrogate-generated transfer attack may all be useful things to test, but a single sorted ASR column can hide more than it reveals.
A table can still compare their observed outcomes while clearly marking the regimes.
What seems established vs. what still looks like a design choice
Based on the examples I could find, I would separate these.
Seems fairly well supported
- Report a benign/clean side of the evaluation when baseline behavior affects interpretation.
- Record enough experimental context to reproduce the comparison.
- Treat adaptive attack effort as part of the attack specification.
- Distinguish simulation/checkpoint/API/physical validation rather than treating them as the same evidential claim.
- Report uncertainty rather than a point estimate alone when trial counts are small.
- Define the actual event behind a generic label such as ASR.
Still looks more like a design choice than a settled convention
- Wilson as the mandatory interval rather than one recommended default.
- Exactly which variables must be matched in every robotics environment.
- Whether provenance and transfer/access deserve separate columns in a minimum table.
- Whether paired statistical testing belongs in the minimum or only in fuller analyses.
- Whether all attack resources can usefully be collapsed into one scalar “budget”.
So I would keep the minimum fairly small and make the richer fields optional/conditional.
The standard is probably most useful if a reader can answer:
- What event was counted?
- Out of how many eligible trials?
- What happened without the attack under matched conditions?
- How uncertain is the estimate?
- What system/environment was actually evaluated?
- How much target-adaptive search was allowed?
If those six answers are recoverable, a lot of later comparison becomes possible without forcing every paper into the same attack design.
One final reason I like keeping the benign arm explicit rather than subtracting it: subtraction would compress several different phenomena into one number.
Suppose the attacked rate is 44/50 and the benign rate is 2/50.
The pair 44/50, 2/50 preserves useful information about both the policy/evaluator baseline and the attacked condition. A derived difference can always be computed later.
The reverse is not true: if only +84 percentage points survives in the table, a future reader cannot reconstruct either underlying rate, check a different uncertainty model, or tell whether the baseline itself was unstable.
So overall, my answer would be:
- Yes, there are adjacent subfields and recent VLA/robot-security papers already doing pieces of this, but I did not find one established reporting minimum that exactly matches your four.
- No, I do not think #4 is too strong for adaptive/search attacks. I would generalize it from “query budget” to “attack/search budget + stopping rule” and allow
N/A for attacks where it genuinely does not apply.
- I would keep the matched benign arm.
- I would make the uncertainty requirement method-agnostic rather than Wilson-specific.
- I would distinguish “where was this evaluated?” from “how was the attack constructed/transferred?” if space allows.
- And if I added only one small field, it would be the exact attack event / numerator / denominator / eligibility rule, because “ASR” already refers to materially different things across the literature.
That feels small enough to be a reporting minimum, while still preserving enough information for someone else to interpret or re-analyze the result later.