A 14.8M-visitor trial says a sales-tuned agent lifts revenue per visitor 10.0% and ties human chat staff at 1.8% of the cost, but the vendor ran the study
Javo's field trial across 100 sites reports +21.6% engaged conversion over a frontier-model agent in identical scaffolding. The author runs the company, no independent review is claimed, and a new site should expect +2.3% to +17.7%.
Relative to no agent, Javo changed revenue per visitor by +10.0% (95% CI +5.1 to +14.9). Among engaged visitors it changed conversion by +21.6% relative to the frontier-model agent (95% CI +16.9 to +26.3).
Comprehensive take
The design is better than most vendor claims: randomized, a frontier arm in identical scaffolding, guardrails with margins. But it is one company's preprint on a hand-picked site network, so use +2.3% to +17.7% as your planning range and verify with your own holdout.
Use cases
Deciding on a chat agent for a considered-purchase site
Hotels, furniture and beauty sites dominated the battery; the lift is most relevant where shoppers ask questions before buying.
Writing a vendor evaluation plan
Copy the paper's structure: randomize to no agent, test against a frontier model in the same scaffold, and set return and complaint guardrails with margins up front.
Cost modeling for staffed live chat
The reported $0.06 versus $3.40 per engaged conversation is the number to sanity-check against your own staffing cost.
Specs
| Trial | 14.8M randomized visitors, 100 sites, 14 industries, May 1 to Sep 30, 2026 source |
|---|---|
| Revenue per visitor vs no agent | +10.0% (95% CI +5.1 to +14.9), 85 transactional sites source |
| Engaged conversion vs frontier agent | +21.6% (95% CI +16.9 to +26.3) source |
| Engaged conversion vs human associates | 11.26% vs 11.20%, non-inferior at -7% margin; CSAT 4.42 vs 4.43 source |
| Cost per engaged conversation | Javo $0.06; frontier $0.14; human $3.40 source |
| SalesLLM-EN (CustomerLM) | Javo 7.01; best frontier rerun 6.21; humans 5+ years 6.88 source |
Insights
- Axion reading: the frontier arm is the informative one, because it isolates the model from the interface; the no-agent arm mostly shows that any competent chat widget helps.
- Axion caution: the baseline model was frozen on April 24, 2026, so the 21.6% gap is a snapshot rather than a durable ranking.
- Axion caution: the paper discloses that the author is company-affiliated and claims no independent statistical review.
What the study claims
Javo is a sales chat agent fine-tuned on 4.6 million consented, de-identified commercial conversations. A new arXiv preprint tests it in the field: 14.8 million visitors across 100 live websites in 14 industries were randomized between May 1 and September 30, 2026 into four arms (no agent 2.6M, human chat associates 0.9M, Javo 5.6M, and a frontier-model agent 5.7M running in the same interface, catalog tools and guardrails). Against no agent, Javo changed revenue per visitor by +10.0% (95% CI +5.1 to +14.9) and conversion by +8.0% (+4.1 to +11.9), pooled over the 85 transactional sites.
The comparison that matters most
The frontier-model arm holds scaffolding fixed, so the gap is attributable to model weights. Among engaged visitors (about 8.07% of Javo arm traffic and 8.06% of the frontier arm, so engagement was balanced), Javo converted 21.6% better in relative terms (95% CI +16.9 to +26.3), and revenue per visitor was +16.9% higher (+11.2 to +22.6). The mechanism table is plausible: Javo elicited needs before its first recommendation in 82% of engaged conversations versus 61% for the frontier agent, and proposed a concrete next step in 88% versus 68%.
Against each site's own human chat staff (20 sites), engaged conversion was 11.26% versus 11.20%, non-inferior at a prespecified -7% margin but not significantly superior, with customer satisfaction at 4.42 versus 4.43. The paper reports cost per engaged conversation of $0.06 for Javo, $0.14 for the frontier agent and $3.40 for humans, which is where the "1.8% of the cost" figure comes from.
What to discount
Three things stop this from being a settled result. First, the author is the founder and CEO of the company that operates Javo, and the paper says plainly that no independent statistical review, study registration or external ethics approval is claimed. Second, many of the sites came through the author's client and agency network and over-represent heavy pre-purchase questions, so they are not a random sample of the web. Third, the frontier baseline is one model version frozen on April 24, 2026; a newer model could narrow the 21.6% gap, and the authors say so.
The human comparison also has a built-in confound: the human and AI widgets differ in disclosure as well as in who answers. And the paper's "engagement-scaled" +123.9% figure is, by its own description, not an identified causal effect among engagers; ignore it.
The benchmark side is mixed. On SalesLLM-EN Javo scored 7.01 against 6.21 for the best frontier model the authors reran (and 6.88 for human sellers with five or more years). But on tau-bench retail pass^1 it matched, not beat, the best frontier model at 73.0, and an ablation without constraint data scored higher on SalesLLM (7.13) while falling to 58.0 on tau-bench. Simulated buyers also reorder rankings; the paper notes two simulators correlate at only r = 0.24.
Guardrails
Return rate per order was 4.0% versus 4.1% with no agent; unsupported claims were 0.15% of audited turns versus 0.56% for the frontier agent; unauthorized offers were 0 versus 3. All met the prespecified margins, though the audit covered 5,000 conversations.
Should you act on it?
If you run chat on a high-consideration site, plan around the paper's own 95% prediction interval for a new site (+2.3% to +17.7% revenue per visitor), not the +10.0% headline, and run a holdout on your own traffic. Treat this as a well-documented vendor study with released harness claims, not as independent confirmation.