The increase in the renewable energy sources connected to the electricity grid has resulted in an increased need for frequency regulation. On the demand side, frequency regulation services can be provided by electrified heating/cooling systems exploiting the energy stored in thermal mass of buildings. To provide such services a first principles model of the building is needed, which is often difficult to obtain in practice. This issue can be overcome by using a buffer storage between the heating/cooling source and the building. Here, we present a solution based on robust optimization to offer frequency regulation reserves with such a system comprising a heat pump, a thermal storage in the form of a warm water buffer tank, and heating demand from a building that needs to be served. We mitigate the problem of limited thermal storage by introducing affine policies on uncertain variables. In three experiments with a real heat pump and warm water buffer storage and an emulated heating demand, we demonstrate that the system can indeed offer reserves and can successfully track a regulation signal while meeting the heating demand at all times.