We consider the inverse problem of recovering a potential from the Dirichlet to Neumann map at a large fixed frequency on certain Riemannian manifolds. We extend the earlier result of [G. Uhlmann and Y. Wang, arXiv:2104.03477] to the case of simple manifolds, and more generally to manifolds where the geodesic ray transform is stably invertible. The argument involves an invariantly formulated construction of Gaussian beam quasimodes with uniform bounds for the underlying constants.