The ultracold neutron (UCN) source at the Paul Scherrer Institute (PSI) provides high intensities of storable neutrons for fundamental physics experiments. The neutron velocity spectrum parallel to the beamline axis was determined by time-of-flight spectroscopy using a neutron chopper. In particular, the temporal evolution of the spectrum during neutron production and UCN storage in the source storage volume was investigated and compared to Monte Carlo simulation results. A softening of the measured spectrum from a mean velocity of 7.7(1)-5.1(1) m s^-1 occurred within the first 30 s after the proton beam pulse had impinged on the spallation target. A spectral hardening was observed over longer time scales of one measurement day, consistent with the effect of surface degradation of the solid deuterium moderator.