We report on the structural and spectroscopic characterization of the multiferroic Fe₂Mo₃O₈. Synchrotron X-ray and neutron diffraction as well as thermal expansion measurements reveal a lattice anomaly at T_N≃60K but do not show any symmetry lowering in the magnetically ordered state. The lattice parameter c exhibits a nonmonotonic behavior with a pronounced minimum around 200 K, which is also reflected in an anomalous behavior of some of the observed infrared-active optical excitations and parallels the onset of short-range magnetic order. The infrared reflectivity spectra measured between 5 and 300 K in the frequency range of 100-8000cm^-1 reveal most of the expected phonon modes in comparison with the eigenfrequencies obtained by density-functional calculations. The A₁ phonons show an overall hardening upon cooling, whereas a nonmonotonic behavior is observed for some of the E₁ modes. These modes also show a strongly increased phonon lifetime below T_N, which we associate with the quenched direction of the orbital moment in the magnetically ordered state. A similar increase is observed in the lifetime of the higher-lying d-d excitations of the tetrahedral Fe²⁺ site that become clearly visible below T_N only.