WorldTest, a protocol for evaluating world-model learning via environment-level queries, instantiated as AutumnBench --- 43 interactive environments and 129 tasks.
We introduce WorldTest, a representation-agnostic protocol for evaluating world-model learning in AI agents. WorldTest moves beyond next-frame prediction by posing environment-level queries — asking whether an agent can predict unobserved states, plan action sequences toward goals, and detect changes in causal dynamics.
We instantiate WorldTest with AutumnBench, a suite of 43 interactive grid-world environments and 129 tasks across three families: masked-frame prediction, planning, and predicting changes to causal dynamics. We evaluated 517 human participants and three frontier reasoning models on AutumnBench. Humans outperform the models, and scaling compute improves performance only in some environments — exposing substantial headroom in world-model learning (Warrier et al., 2026).
PS: The games are fun to play — try them at autumn.basis.ai!
Try it yourself
Interactive task selector — play directly here.
Example human vs AI interactions
Human
Claude 4 Sonnet
Gemini 2.5 Pro
o3
References
2026
ICML
Benchmarking World-Model Learning with Environment-Level Queries
Archana Warrier, D. Nguyen, M. Naim, and 8 more authors
International Conference on Learning Representations, 2026
Model-learning agents should gather information to learn world models that support many downstream tasks and inferences, such as predicting unobserved states, estimating near- and far-term consequences of actions, planning action sequences, and detecting changes in dynamics. Current methods for learning and evaluating world models diverge from this goal: training and evaluation are anchored to next-frame prediction, and success is scored by reward maximization in the same environment. We propose WorldTest, a protocol to evaluate model-learning agents that separates reward-free interaction from a scored test phase in a different but related environment. WorldTest is open-ended—models should support many different tasks unknown ahead of time—and agnostic to model representation, allowing comparison across approaches. We instantiated WorldTest with AutumnBench, a suite of 43 interactive grid-world environments and 129 tasks across three families: masked-frame prediction, planning, and predicting changes to the causal dynamics. We compared 517 human participants and three frontier models on AutumnBench. We found that humans outperform the models, and scaling compute improves performance only in some environments but not others. WorldTest provides a novel template—reward-free exploration, derived tests, and behavior-based scoring—to evaluate what agents learn about environment dynamics, and AutumnBench exposes significant headroom in world-model learning.
@article{warrier2025benchmarking,title={Benchmarking World-Model Learning with Environment-Level Queries},author={Warrier, Archana and Nguyen, D. and Naim, M. and Jain, M. and Liang, Y. and Schroeder, K. and Yang, C. and Tenenbaum, J. and Vollmer, S. and Ellis, K. and Tavares, Z.},journal={International Conference on Learning Representations},year={2026},}