Traditional statistics give detailed information about the uncertainty of the processed results, are transparent, and are often transferable from one problem domain to another. Especially in these areas, we can find the challenges of using deep learning. In this work we review the deep learning literature for issues related to uncertainty and list tools for dealing with deep learning cautiously, with a focus on practicality of use in official statistics. We provide common methods for estimating uncertainty, obtaining probabilistic outputs and predicting when a model fails. Furthermore, we also explain caveats and problems related with these techniques from the literature.