In this contributed talk, I presented our recent study on the challenges of accurate long rollouts in PDE solving when applying neural surrogates. We identify common neural operators to neglect spatial frequency information with low amplitude, which has low short-term, but high long-term impacts. To overcome these challenges, we propose PDE-Refiner. PDE-Refiner models an iterative refinement process to accurately predict the solution across the whole spatial frequency spectrum. For more details, check out our paper and project page.