LeMario: Training a Joint-Embedding Predictive Architecture on Super Mario Bros

LeMario implements a Joint‑Embedding Predictive Architecture (JEPA) with 192‑dimensional latent representations for both frames and actions, feeding a six‑block transformer that uses AdaLN‑Zero to inject action‑dependent shift, scale, and gate controls into attention and MLP branches. The model is trained on 737,134 frames from 280 episodes across 32 Mario levels, with a loss that combines MSE prediction loss and a SIGReg regularizer to prevent representation collapse. One‑step error drops to 0.0137 and five‑step error to 0.0777, beating a persistence baseline and a shuffled‑action baseline by 45.5% and 47.5% respectively, proving that the model learns short‑horizon dynamics conditioned on player actions. Latent planning is performed with the Cross‑Entropy Method (CEM), which samples action sequences, rolls them forward through LeMario, and selects those that minimize the distance to a goal latent. Initial experiments with close goals failed because the latent space was too similar to the start, but a probe that recovers Mario’s horizontal position from the latent (MAE 9.30 px) allowed CEM to rank futures correctly and move Mario toward distant goals. The probe demonstrates that useful state information is encoded in the 192‑dimensional latent, and that latent planning can succeed when guided by such probes. The work shows that JEPA can learn dynamics from pixels and actions without reward signals, and that latent‑based planning is feasible with appropriate probing and optimization. The architecture and results provide a foundation for future research in reward‑free planning and model‑based RL in video games.