ALTK-Evolve lifts τ²-bench retail Pass^4 from 57.5% to 67.5% with no fine-tuning, but the leaderboard entries carry leakage caveats
IBM’s learned-guidelines library adds 12 to 65% to per-conversation cost. Here is what the held-out numbers show versus the unverified leaderboard claims.
no weight updates or human annotation are needed.
Comprehensive take
Worth a trial on your own task suite; do not switch on the leaderboard rank, which IBM’s own PRs flag as unverified.
Use cases
Customer-service agents on a fixed policy
The tau2-bench retail and airline domains are policy-bound tool-use tasks; guidelines learned from past attempts target exactly that pattern.
Coding agents with persistent lessons
The repo’s Lite version plugs into Claude Code, Codex, IBM Bob and Hermes to carry learned guidance between sessions.
Cheaper models closing the gap
The post argues guideline learning transfers across base models; test it with a smaller model before paying for a frontier tier.
Specs
| Retail test split, gpt-4.1 | Pass^4 57.5% → 67.5% (40 tasks) source |
|---|---|
| Airline test split, gpt-5.5 | Pass^4 55% → 70% (20 tasks) source |
| Retail leaderboard PR | Pass^4 67.54% (114 tasks × 4) source |
| Airline leaderboard PR | Pass^4 82.00% (50 tasks × 4) source |
| Retail cost per conversation | $0.084 baseline; $0.095 (support-3); $0.139 (lossless) source |
| Licence / install | Apache-2.0; pip install altk-evolve (Python 3.12+) source |
Insights
- Test-split and full-split numbers are different evaluations that happen to share 67.5% in retail; do not read them as one result.
- Both leaderboard PRs say the entries are expected to be listed as unverified because train and base eval splits overlap.
- Cost rises with guideline volume, so the filtering threshold is as much a budget knob as a quality knob.
Axion Scorecard
Overall: 6.8
Retail Pass^4 +10.0 points and airline +15.0 (gpt-5.5) on small held-out splits; IBM self-reported. source
Apache-2.0 library with public leaderboard PRs. source
pip install with optional pgvector or Milvus backends; repo notes a changed filter syntax and partial entity sharing. source
Cost per conversation up 12% to 65% on retail depending on guideline set. source
Not yet scored — no cited data.
Support-counted guidelines and conflict resolution around a frozen model. source
The pitch: learn around the model, not inside it
ALTK-Evolve is IBM Research's open-source library for agents that get better from their own attempts without touching model weights. In the project's words, "no weight updates or human annotation are needed." Instead, an LLM reviews the agent's attempts on training tasks, distils successes and failures into short behavioural guidelines, and appends them to the system prompt. Each guideline carries a support count: how many distinct training tasks it came from. The repo adds vector-backed memory and LLM-based conflict resolution, plus a "Lite" version that plugs into Claude Code, Codex, IBM Bob and Hermes.
The numbers, separated by what they actually measure
IBM reports two different kinds of result, and they should not be mixed.
Held-out test splits (small). With gpt-4.1 as the agent, retail accuracy rose from 78.1% to 82.5% and Pass^4 from 57.5% to 67.5% on 40 test tasks. On 20 airline tasks, the lossless guideline set moved accuracy from 57.5% to 65.0% and Pass^4 from 35% to 40%; a stricter support-3 filter gave 60.0% accuracy and Pass^4 of 45%. Swapping in gpt-5.5 for the airline domain lifted accuracy from 72.5% to 82.5% and Pass^4 from 55% to 70%.
Leaderboard submissions (full base splits). IBM's retail entry (gpt-4.1) is listed in a tau2-bench pull request at Pass^1 81.14% and Pass^4 67.54% over 114 tasks times 4 trials, against 61.4% for the RAFT-30B-A3B entry the blog compares to. The airline entry with gpt-5.5 shows Pass^1 88.50% and Pass^4 82.00% over 50 tasks times 4 trials, versus 66 to 72% for frontier entries.
Why the caveats matter more than the rank
Both pull requests state the entries are expected to be listed as unverified: the system prompt is modified, and the guidelines were learned on a training split that overlaps the base evaluation split, so leakage is possible. A single task is worth 2.5 to 5 points on the test splits, and the best guideline set differs by domain (support-3 for airline, lossless for retail). Everything is IBM self-reported, and telecom was not tested. We could not check the live leaderboard, so "number one" rests on IBM's own 12 October snapshot.
What it costs
Learned guidelines make every prompt longer. On retail, cost per conversation went from $0.084 to $0.095 with support-3 filtering (+12%) and to $0.139 with the lossless set (+65%). On airline it went from $0.081 to $0.097 (+19%) or $0.107 (+32%). IBM argues assistant turns fall slightly (10.59 to 10.16 on retail) and that prompt caching could offset the input-token increase; that is plausible but not demonstrated in the post.
How to use it today
The library is Apache-2.0, installs with pip install altk-evolve on Python 3.12 or newer, and optionally uses pgvector or Milvus for semantic search. Practical advice: start with the support-3 filter if you care about cost, collect at least a few dozen representative tasks with a clear success signal, and hold out a test split that is genuinely disjoint from training, which is exactly what the leaderboard entries did not guarantee. The repo flags a changed filter syntax (custom keys need the metadata. prefix) and partial entity sharing, so pin versions.
Axion take
The idea is sound and cheap to try: prompt-level memory is easy to audit, easy to roll back, and portable across base models, which the post demonstrates with gpt-4.1 and gpt-5.5. But treat the rank as a marketing number until the entries are verified. Try it on a task suite you own; switch only if you see gains on a split it never trained on.