The performance of X-ray and neutron grating interferometers is characterised by their visibility, which is a measure for the maximum achievable contrast. In this study we show how the real grating geometry in a grating interferometer with three gratings impacts the interference and self projection that leads to visibility in the first place. We quantify the individual contributions of wavelength distributions and grating shapes in terms of visibility reduction by determining the absolute as well as relative effect of each contribution. The understanding of the impact of changed geometry and wavelength distributions on the interference of neutrons/X-rays allows us to present the first fully quantitative model of a grating interferometer setup. We demonstrate the capability of the simulation framework by building a model of the neutron grating interferometer at the ICON beamline and directly comparing simulated and measured visibility values. The general nature of the model makes it possible to extend it to any given grating interferometer for both X-rays and neutrons.