We present high-resolution measurements of the electron beam energy spread at the SwissFEL injector as a function of the electron bunch charge, the optics, and the longitudinal dispersion of the lattice. The measured values are in general an order of magnitude higher than what is predicted by standard simulation codes. The measured dependences indicate that the energy spread blowup is caused primarily by intrabeam scattering and microbunching instabilities, effects not covered in the conventional modeling of radio-frequency injectors. We present further a numerical model that qualitatively reproduces the experimental data and an approach to mitigate the energy spread deterioration. The work underlines the importance of considering the energy spread in the optimization and design of high-brightness electron beam sources and the need to develop new models to adequately understand and simulate the observed physics effects.