I was invited to give a talk in the AI4Science Talk Series on Recent Advances in ML for Science, organized by the Machine Learning for Simulation Science lab, University of Stuttgart and NEC Labs Europe, Heidelberg. In this talk, I presented our recent study on the challenges of accurate long rollouts in PDE solving when applying neural surrogates. We identify common neural operators to neglect spatial frequency information with low amplitude, which has low short-term, but high long-term impacts. To overcome these challenges, we propose PDE-Refiner. PDE-Refiner models an iterative refinement process to accurately predict the solution across the whole spatial frequency spectrum. For more details, check out our paper and project page.