AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Simulated Shoppers Point to Website Tests Likely to Win, With Caveats

Companies that run website experiments face a triage problem: which proposed page changes deserve scarce live traffic? Testing a weak variant uses visitors that could go to a better idea, while setting aside a strong one can leave a real improvement undiscovered. A new preprint from Squoosh AI, whose authors also built the system under test, asks whether simulated shoppers can forecast which changes are worth testing.

The system, Squoosh UX-1, is an open-source base model fine-tuned on a dataset of how people browse websites. For each comparison, it simulates 1,000 shoppers on the original page and 1,000 on the variant. It then returns one of three forecasts (original win, variant win, or inconclusive), a signed estimate of relative lift, which is the percentage change in conversion rate, and a short explanation. The researchers say the training aimed to capture variation in decisions such as whether a shopper scrolls a listing, opens a product page, or leaves before converting.

The team screened 412 historical A/B tests from private partner sites and kept 318 that ran longer than 21 days. Each testโ€™s recorded outcome came from the 95% confidence interval on its primary conversion metric: a win if the interval lay entirely on one side of zero, and inconclusive if it included zero. The outcomes were 113 original wins, 137 variant wins, and 68 inconclusive results. The systems never saw these outcomes or which page was the variant.

UX-1 classified 267 of 318 outcomes correctly, an accuracy of 84.0% (95% Wilson interval 79.5โ€“87.6%, assuming experiments are independent). Qwen3 Next 80B, the strongest general-purpose configuration in the comparison, got 196 right (61.6%). GPT-OSS 120B got 187 (58.8%), and Gemini 3.5 Flash got 162 (50.9%). A rule that always predicted a variant win would have scored 43.1%. On the 250 tests with a recorded directional winner, UX-1 identified 212 winners.

The authors frame the most practical question as prioritization. UX-1 favored the variant in 126 experiments. In the live tests, 118 of those variants won (93.7%), four lost, and four were inconclusive. Those 118 wins amount to 86.1% of the 137 variants that won their tests. However, 19 variants outside that shortlist also won. Six came from experiments UX-1 predicted would favor the original, and 13 came from those it called inconclusive. The authors cite the 13 as a warning against dropping a candidate only because its forecast was inconclusive.

The paper also describes failures. In one popup experiment, UX-1 predicted the variant would outperform the original, but the live test favored the original. That was one of the four variants the shortlist contained that later lost.

The study has clear limits. Its duration filter excluded 94 shorter tests, which the authors note can include changes that show effects quickly. Ninety-seven of the screened tests had appeared in an earlier report by the same company, and the authors could not determine how much the sites overlapped with the modelโ€™s training data. The confidence intervals assume independence across experiments, and the authors warn that correlated errors within a site could make them too narrow. The comparison pits complete systems against one another under different evidence and compute, so it does not isolate the contribution of fine-tuning. An inconclusive label, moreover, does not prove that an effect is zero.

The authors suggest that teams could test the approach on their own sites by recording forecasts before launch and then testing candidates both inside and outside the shortlist. The work offers evidence that simulated forecasts can help rank candidates on one set of historical tests. Whether that holds for other companies and sites remains an open question.