A major challenge of agriculture is to improve the sustainability of food production systems in order to provide enough food for a growing human population. Pests and pathogens cause vast yield losses, while crop protection practices raise environmental and human health concerns. Decision support systems provide detailed information on optimal timing and necessity of crop protection interventions, but are often based on phenology models that are time-, cost-, and labor-intensive in development. Here, we aim to develop a data-driven approach for pest damage forecasting, relying on big data and deep learning algorithms. We present a framework for the development of deep neural networks for pest and pathogen damage classification and show their potential for predicting the phenology of damages. As a case study, we investigate the phenology of the pear leaf blister moth (Leucoptera malifoliella, Costa). We employ a set of 52,322 pictures taken during a period of 19 weeks and establish deep neural networks to categorize the images into six main damage classes. Classification tools achieved good performance scores overall, with differences between the classes indicating that the performance of deep neural networks depends on the similarity to other damages and the number of training images. The reconstructed damage phenology of the pear leaf blister moth matches mine counts in the field. We further develop statistical models to reconstruct the phenology of damages with meteorological data and find good agreement with degree-day models. Hence, our study indicates a yet underexploited potential for data-driven approaches to enhance the versatility and cost efficiency of plant pest and disease forecasting.