Early Warning Systems (EWS) are increasingly applied to mitigate the risks posed by natural hazards. To compare the effect of EWS with alternative risk reduction measures and to optimize their design and operation, their reliability and effectiveness must be quantified. In the present contribution, a framework approach to the evaluation of threshold-based EWS for natural hazards is presented. The system reliability is classically represented by the Probability of Detection (POD) and Probability of False Alarms (PFA). We demonstrate how the EWS effectiveness, which is a measure of risk reduction, can be formulated as a function of POD and PFA. To model the EWS and compute the reliability, we develop a framework based on Bayesian Networks, which is further extended to a decision graph, facilitating the optimization of the warning system. In a case study, the framework is applied to the assessment of an existing debris flow EWS. The application demonstrates the potential of the framework for identifying the important factors influencing the effectiveness of the EWS and determining optimal warning strategies and system configurations.