For now, I was able to reproduce the benchmark on my side as well:
My current take is that the review boundary looks genuinely promising, but the benchmark is currently strongest as evidence for semantic compression / review-surface reduction, not yet as evidence that humans actually review AAL more accurately or more safely than generated TypeScript.
I reran the repository benchmark and got the same basic result: the seven real submissions all passed the frozen 14/14 HTTP oracle with the same observed behavior fingerprint. For the three real Direct/AAL pairs, I also reproduced the reported reduction:
| pair |
non-empty lines |
bytes |
| Codex |
AAL 21.21% smaller |
AAL 46.97% smaller |
| OMP / DeepSeek |
AAL 10.34% smaller |
AAL 56.03% smaller |
| OpenCode / DeepSeek |
AAL 37.35% smaller |
AAL 58.19% smaller |
So I think the first benchmark does establish something useful: for these pairs, the same tested CRUD behavior can be represented behind a substantially smaller review surface.
I would just keep the next claim separate:
smaller review surface
≠ automatically easier human review
≠ automatically complete behavioral coverage
Those are now testable as separate questions, which is actually a nice place for the experiment to be.
The highest-information next route, in my opinion, would be:
If the goal is to strengthen this CRUD benchmark:
add a few survivor-derived regression cases, then freeze it.
If the goal is to test the central "human-auditable" claim:
run a tiny AAL-vs-generated-TypeScript human review pilot.
If the goal is to find AAL's readability/coverage limit:
move to one semantically denser task rather than making CRUD much larger.
I would probably take the second route next.
What I found when I tried to break the 14-case oracle
I tried a small targeted mutation probe against the Direct reference implementation. This was not a project-wide mutation score; I deliberately seeded 14 semantic faults around behavior already described by the task.
The frozen 14-case oracle killed 5/14.
9/14 survived, including changes such as:
- wrong exact error body;
- malformed JSON returning the wrong error body;
- missing/wrong create inputs returning the wrong error;
- removing PUT
name type validation;
- invalid decimal path IDs returning
404 rather than 400;
- unmatched item methods returning
405 rather than 404;
- a failed update secretly creating persistent state;
- removing JSON
Content-Type;
- returning
Content-Type on a successful 204.
A small supplemental oracle made only from rules already present in TASK.md distinguished all 14.
I would not interpret that as “the benchmark is bad”. A 14-case first benchmark is necessarily selective.
The useful part is that the survivors give you a very cheap regression backlog.
This is also exactly the useful interpretation of mutation testing: a surviving mutant means that changing some behavior did not cause the covering tests to fail. PIT’s documentation gives a concise description of that idea in its mutation-testing basic concepts.
One relational case was particularly illustrative.
The original benchmark already does something equivalent to:
PUT /items/999
-> 404
A deliberately faulty implementation can return the expected 404 and still secretly create item 999.
So the transcript still looks correct.
A later:
GET /items/999
is what distinguishes:
failed update -> no persistent state
That is a useful little example of the boundary between:
“these responses matched”
and
“the implementations are behaviorally equivalent”.
The former is directly supported by the original benchmark. The latter needs relational/stateful coverage too.
One survivor then appeared in a real Direct submission
This was the part I found most interesting.
One synthetic survivor was:
invalid decimal path-id
expected: 400 Invalid request input
mutant: 404 Not found
I later ran a corrected stateful differential probe over the actual bundled submissions.
That exact family appeared repeatedly in the real Direct Codex submission.
So there is a fairly clean chain here:
targeted mutation
↓
missing assertion identified
↓
cheap regression case
↓
same divergence observed in a real submission
That makes me think survivor-derived cases could be especially useful here: not as a giant mutation-testing program, but as an inexpensive way to decide which few cases deserve promotion into the next frozen oracle.
Corrected stateful differential result
I also expanded the test from isolated examples to repeated CRUD/state sequences.
One methodological note first: my first broad version had two probe-side problems involving HEAD response expectations and GET bodies. I discarded those raw scores rather than interpreting them and reran a corrected version.
The corrected run separated:
- 636 single-fault conformance cases, which are scored;
- 30 deliberately multi-fault PUT cases, which are observation-only;
- three fixed seeds;
- the same generated sequences across all nine bundled submissions.
The scored result was:
| submission |
scored single-fault result |
| AAL Codex |
636/636 |
| AAL OMP/DeepSeek |
636/636 |
| AAL OpenCode/DeepSeek |
636/636 |
| AAL OpenCode/GLM |
636/636 |
| AAL reference |
636/636 |
| Direct reference |
636/636 |
| Direct OMP/DeepSeek |
636/636 |
| Direct OpenCode/DeepSeek |
636/636 |
| Direct Codex |
594/636 |
All 42 Direct Codex failures were the same family:
invalid decimal path-id
expected 400
observed 404
There were no strict-204 header failures.
That is why I think this particular difference is worth turning into a permanent regression case: the task explicitly gives invalid input a 400 result, the original oracle did not exercise that path, the synthetic mutation survived, and a real Direct implementation independently made the same distinction.
A different 400/404 difference should not be called a bug yet
There was another split for requests that simultaneously had:
missing target item
+
invalid or malformed PUT body
The AAL implementations and Direct reference chose the 400 branch.
The three real Direct implementations chose 404.
But that combines two faults in the same request, and the current task does not appear to specify which one has precedence.
So I excluded those cases from pass/fail scoring.
I think that distinction is important because it turns what could look like a conformance failure into a much more useful design question:
Does AAL intend to own this observable decision?
If yes, the precedence can become an explicit, versioned rule.
If not, leaving it intentionally unspecified is also a perfectly coherent choice; it just should not be used as a conformance discriminator.
A semantic-ownership map may help keep AAL small
The 400/404 case made me think a small “who owns this semantic?” rule could become increasingly useful as AAL grows.
Something like:
observable decision
|
+-- application/business behavior?
| -> AAL
|
+-- HTTP / adapter policy?
| -> Binding or adapter contract
|
+-- code-generation choice with no observable semantic effect?
| -> compiler
|
+-- runtime/platform behavior?
| -> runtime contract/version
|
+-- intentionally unconstrained?
-> say so and do not score it
This seems compatible with the boundary you already describe rather than requiring a different architecture.
You already have:
AAL
-> Parser
-> AST
-> Semantic Check
-> Binding
-> Compiler
-> TypeScript
-> Node.js
and your determinism statement explicitly depends on the language/dialect, Binding, compiler, runtime, and dependencies.
So a useful long-term question may be less:
“Can AAL describe every observable detail?”
and more:
“Which observable decisions must remain visible at the human-review boundary?”
That gives you a way to expand coverage without automatically expanding AAL into another general-purpose implementation language.
There are mature DSLs with related separations, although obviously for different domains.
For example, Cedar separates policy semantics, schemas/validation, the production evaluator, and application responsibilities. Its documentation is explicit that validation can prove useful classes of consistency while still not knowing whether the author intended a particular well-formed policy. See Cedar policy validation and its security/shared-responsibility documentation.
I would not map Determinant directly onto Cedar, but I think the design lesson transfers well:
syntactic/semantic validity, implementation correctness, and author intent are different contracts.
That seems especially relevant for an audit language.
Semantic compression vs. semantic displacement
I think this is another useful distinction for the next benchmark.
Suppose:
130 lines of Direct code
↓
100 lines of AAL
There are at least two possible reasons.
Useful semantic compression
The 100 AAL lines retain the decisions a reviewer actually needs to inspect, while implementation machinery disappears behind a deterministic compiler.
That is exactly the mechanism Determinant is trying to test.
Semantic displacement
Some decisions disappear from AAL only because they have moved into:
- Binding;
- generated defaults;
- compiler conventions;
- runtime behavior;
- adapter behavior.
That may still be a good architecture, but it is a different reason for the line reduction.
So I think a helpful question for harder benchmarks is:
When AAL becomes smaller, where did each review-relevant decision go?
This does not need a sophisticated metric initially.
Even a manual table for one harder task could work:
| decision |
visible in AAL |
Binding |
fixed compiler semantics |
runtime |
intentionally unspecified |
If most business-relevant decisions stay in the first column while implementation details migrate rightward, that would be strong evidence that the reduction is real semantic compression rather than just displacement.
One current reason to test this on a harder task
The three real paired results are encouraging, but the current sample is still small.
Also, the four non-reference AAL submissions in this snapshot are byte-identical.
That is useful evidence that multiple tools converged on the same AAL for this task, but it also means this CRUD challenge is not yet strongly exercising variation in AAL design.
There is another useful boundary case in the repository itself: in the reference pair, AAL is about 39.8% smaller in bytes, but it has 104 non-empty lines versus 77 for Direct.
So I would avoid turning “AAL is shorter” into a universal invariant.
The more durable hypothesis looks like:
AAL can expose a smaller or simpler set of review-relevant semantic decisions than the implementation artifact.
That is a stronger and more interesting claim than line count alone, but it needs a different measurement.
The next experiment I would prioritize: a tiny human-review pilot
If the central question is:
“Is this review boundary actually useful in practice?”
then I think the highest-information next experiment is now very small.
Not a large user study.
Something closer to:
4 defective cases
2 correct controls
representation A:
AAL
representation B:
generated TypeScript
same requirement
same intended semantics
same semantic defect
For each review, record:
Was the defect detected?
Was a correct artifact incorrectly rejected?
Was the problematic behavior localized correctly?
How long did the review take?
How confident was the reviewer?
Defects could be things that are visible at both representation levels, for example:
< vs <= at a boundary;- missing state change;
- changing the wrong field;
- wrong output after a state change;
- missing positive-quantity guard;
- incorrect error/status policy;
- wrong composition target.
A few cheap controls would make even a tiny pilot much more informative:
- hide which model/tool produced the artifact;
- randomize or counterbalance AAL/TypeScript order;
- do not show the same person both representations of the same defective instance back-to-back;
- include correct controls so “flag everything” is not a successful strategy;
- record confidence separately from correctness.
The last point may be worth keeping even in a tiny pilot.
A recent controlled study of 86 Python programmers found a substantial asymmetry when people judged LLM-generated assertions: they were much better at judging correct assertions than incorrect ones, while confidence stayed fairly similar. That is not evidence about AAL specifically, but it is a useful warning that subjective confidence and actual defect detection should be measured separately. See Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions.
There is also older empirical code-review evidence that larger changesets impose review costs and tend to receive a lower density of useful review comments; the Microsoft study Characteristics of Useful Code Reviews is relevant background.
Again, neither result proves that AAL will win.
They just make the proposed mechanism plausible enough to test directly:
smaller / more semantically focused artifact
↓
less review effort?
better defect detection?
better localization?
That experiment would answer the “human-auditable” part much more directly than another thousand automated CRUD requests.
If the next target is language coverage instead, I would make the task denser rather than larger
The other obvious next branch is your second question:
can AAL remain readable while covering enough real application behavior?
For that, I would not simply add many more CRUD endpoints.
One task that combines two or three interacting semantic dimensions would probably be more informative.
For example:
Order
Inventory
Payment
with some combination of:
- multiple related objects;
- interacting guards;
- more than one state change;
- flow composition;
- money boundaries;
- authorization/ownership;
- rollback or partial failure;
- idempotency/retries;
- an external effect that must happen exactly once.
The useful property would be:
there are many reasonable TypeScript implementation structures, but the intended business behavior is still fairly crisp.
That is where AAL has a chance to demonstrate its actual advantage:
implementation variability
↓
removed from the review surface
business semantics
↓
remain explicit and stable
It would also be a better stress test for the current Object + Flow model than adding many more independent HTTP cases.
My default sequence would therefore be:
1. Freeze the current CRUD benchmark after a few cheap oracle fixes.
2. Run a tiny human-review pilot if "human-auditable" is the main claim to test.
3. Run one semantically denser benchmark if expressiveness/readability is the main claim to test.
4. Expand the language only in response to concrete missing semantics found there.
That last step seems important because every new language feature has two costs:
more expressiveness
vs.
more semantic surface for humans and tooling to reason about
Restricted languages can get real benefits from staying restricted.
For a different domain, CUE explicitly separates configuration from computation and uses restrictions/order-independent semantics to make configurations easier for both humans and machines to reason about; its language introduction is an interesting example of that design pressure.
I do not mean that AAL should copy CUE. The relevant point is simply that not exposing an implementation capability can itself be a language feature when analyzability is one of the goals.
One possible longer-term trust-boundary ladder
There is one more part of the architecture that I think becomes interesting if this works on harder tasks.
The human-review boundary is at AAL, but the guarantee still depends on:
AAL semantics
+ Binding
+ compiler
+ runtime/dependencies
That is normal; moving the review boundary does not make the downstream implementation disappear as a trust problem.
The nice thing is that the downstream path is deterministic, so its assurance can be strengthened independently of the LLM.
A relatively cheap progression could be:
now:
version pinning
semantic canaries
regression tests
differential/model-based tests
later, if multiple backends appear:
backend-to-backend differential tests
executable reference semantics
only if the assurance requirements justify it:
stronger formal verification / translation validation
Cedar again provides an interesting example of the middle of that ladder. The project has an executable Lean model, a production Rust implementation, property-based testing, and differential randomized testing between the formalization and production implementation. The cedar-spec repository describes that setup.
I would not jump anywhere near that level for an early experiment.
The reason I mention it is almost the opposite: because the LLM is already outside the accepted-AAL-to-code path, you can improve confidence in that path later using ordinary deterministic software-engineering techniques. You do not need to solve LLM reliability again at every stage.
That seems like one of the more attractive consequences of the architecture.
So after trying to reproduce and stress the first benchmark, I have become more interested in the boundary rather than less.
The evidence I would separate at this point is:
| question |
what the current experiment says |
| Is the first benchmark reproducible? |
Yes, in my rerun. |
| Can the tested CRUD behavior survive AAL → deterministic compilation? |
Yes for the observed oracle/stateful cases tested here. |
| Can AAL reduce review-surface size? |
Yes for the three real paired submissions, with the reference pair showing that this is not a universal line-count rule. |
| Does the original 14-case oracle cover every behavior stated in the task? |
No; targeted mutation found several cheap gaps. |
| Do any of those gaps matter outside synthetic mutants? |
At least one did: invalid path-id → 404 appeared repeatedly in a real Direct submission. |
| Is every Direct/AAL difference therefore a bug? |
No; the multi-fault PUT 400/404 case is currently a specification/precedence ambiguity. |
| Is AAL actually easier or safer for humans to review? |
Still open. This is now directly testable. |
| Can AAL stay readable on substantially richer applications? |
Still open. A semantically denser benchmark would tell us much more than simply enlarging CRUD. |
If I were choosing only one next experiment, I would freeze CRUD after the cheap survivor-derived regression cases and do the tiny human-review comparison.
That directly attacks the most distinctive part of the idea:
not whether an LLM can emit a DSL, and not whether a deterministic compiler can generate code, but whether moving the review boundary upward actually lets a human verify the intended behavior more effectively.
At that point, even a negative result would be useful: it would tell you whether the next design work belongs in AAL readability, semantic ownership, Binding visibility, compiler assurance, or simply the premise that a smaller executable review artifact is the right interface.