Coordinated large scale building retrofit is one of the most effective approaches not only to reduce the environmental footprint of existing buildings but also to assure that climate goals can be met. Existing bottom-up retrofit approaches become infeasible or lack in terms of accuracy and/or computational cost especially when the scale of application increases in size. Hence, the aim of this paper is to propose and develop a bottom-up approach for large-scale building retrofit with the use of a scalable building-level surrogate retrofit model. To ensure scalability and generalization to large areas, we use artificial neural networks. The developed approach consists of a set of artificial neural networks trained on building archetypes, aiming to predict near-optimal building retrofit solutions for all residential buildings in Switzerland. This is achieved with a small set of inputs, and within a short run time. In order to validate the developed approach, we compare it with a conventional mixed building simulation and optimization approach. The surrogate model computes near-optimal building retrofit solutions, with an average coefficient of determination of 0.91 and an average f1 score of 0.89. To illustrate the advantages of this model, we perform a large-scale retrofit analysis of a case study in Geneva municipality. Such a surrogate model is capable of providing insights into the overall retrofit potential of large areas and thus shall support reaching energy emission reduction targets of existing residential buildings.