Spin waves in chiral magnetic materials are strongly influenced by the Dzyaloshinskii-Moriya interaction, resulting in intriguing phenomena like nonreciprocal magnon propagation and magnetochiral dichroism. Here, we study the nonreciprocal magnon spectrum of the archetypical chiral magnet MnSi and its evolution as a function of magnetic field covering the field-polarized and conical helix phase. Using inelastic neutron scattering, the magnon energies and their spectral weights are determined quantitatively after deconvolution with the instrumental resolution. In the field-polarized phase the imaginary part of the dynamical susceptibility χ′′(ɛ,q) is shown to be asymmetric with respect to wave vectors q longitudinal to the applied magnetic field H, which is a hallmark of chiral magnetism. In the helimagnetic phase, χ′′(ɛ,q) becomes increasingly symmetric with decreasing H due to the formation of helimagnon bands and the activation of additional spin-flip and non-spin-flip scattering channels. The neutron spectra are in excellent quantitative agreement with the low-energy theory of cubic chiral magnets with a single fitting parameter being the damping rate of spin waves.