MgV₂O₄ is a spinel based on magnetic V³⁺ ions, which host both spin (S = 1) and orbital (l_eff=1) moments. Owing to the underlying pyrochlore coordination of the magnetic sites, the spins in MgV₂O₄ only antiferromagnetically order once the frustrating interactions imposed by the Fd3¯m lattice are broken through an orbitally-driven structural distortion at T_S ≃ 60 K. Consequently, a Néel transition occurs at TN≃40 K. Low-temperature spatial ordering of the electronic orbitals is fundamental to both the structural and magnetic properties; however, considerable discussion on whether it can be described by complex or real orbital ordering is ambiguous. We apply neutron spectroscopy to resolve the nature of the orbital ground state and characterize hysteretic spin-orbital correlations using x-ray and neutron diffraction. Neutron spectroscopy finds multiple excitation bands and we parametrize these in terms of a multilevel (or excitonic) theory based on the orbitally degenerate ground state. Meaningful for the orbital ground state, we report an "optical-like"mode at high energies that we attribute to a crystal-field-like excitation from the spin-orbital j_eff = 2 ground-state manifold to an excited j_eff = 1 energy level. We parametrize the magnetic excitations in terms of a Hamiltonian with spin-orbit coupling and local crystalline electric field distortions resulting from deviations from perfect octahedra surrounding the V³⁺ ions. We suggest that this provides compelling evidence for complex orbital order in MgV₂O₄. We then apply the consequences of this model to understand hysteretic effects in the magnetic diffuse scattering where we propose that MgV₂O₄ displays a high-temperature orbital memory of the low-temperature spin order.