This paper examines building energy flexibility at an aggregated level and addresses the main barriers and research gaps for the development of this resource across three design and development phases: market and policy, early planning and design, and operation. We review methodologies and tools and discuss barriers, challenges, and opportunities, incorporating policy, economic, technical, professional, and social perspectives. Although various legal and regulatory frameworks exist to foster the development of energy flexibility for small buildings, financing mechanisms are limited with a significant number of perceived risks undermining private investment. For the early planning and design phase, planners and designers lack appropriate tools and face interoperability challenges, which often results in insufficient consideration of demand response programs. The review of the operational phase highlighted the socio-technical challenges related to both the complexity of deployment and communication, as well as privacy and acceptability issues. Finally, the paper proposes a number of targeted research directions to address challenges and promote greater energy flexibility deployments, including capturing building demand side dynamics, improving baseline estimations and developing seamless connectivity between buildings and districts.