World Modeling in Transformers · paper demo

Pierre Beckmann, Matthieu Queloz, André Freitas Paper Code Homepage

TaxiGPT is a transformer trained on random walks through Manhattan. Using mechanistic interpretability, we find that it learns an internal map and develops world-modeling capacities for localizing itself and navigating.

move 0 of 0
origin goal model's move forced move illegal move (no street)