Artificial Intelligence (AI) and Deep Learning (DL) solutions are becoming increasingly popular in our daily life. However, one of the main concerns that stakeholders face is the lack of accountability for the inevitable errors made by AI. In this paper, we build on the emerging research of developing DL solutions to predict combined sewer overflows, and present our vision of linking data engineering decisions to key requirements such as predictive accuracy and explainability.