A detailed analysis of muon-spin rotation (μSR) spectra in the vortex state of type-II superconductors using different theoretical models is presented. Analytical approximations of the London and Ginzburg–Landau (GL) models, as well as an exact solution of the GL model were used. The limits of the validity of these models and the reliability for extracting parameters such as the magnetic penetration depth λ and the coherence length ξ from the experimental μSR spectra were investigated. The analysis of the simulated μSR spectra showed that at high magnetic fields there is a strong correlation between λ and ξ obtained for any value of the Ginzburg–Landau parameter κ = λ/ξ. The smaller the applied magnetic field, the smaller the possibility of finding the correct value of ξ. A simultaneous determination of λ and ξ without any restrictions is very problematic, regardless of the model used to describe the vortex state. It was found that for extreme type-II superconductors and low magnetic fields, the fitted value of λ is practically independent of ξ. The second-moment method frequently used to analyze μSR spectra by means of a multi-component Gaussian fit generally yields reliable values of λ over the whole range of applied fields H_c1 ≪ H ≲ H_c2 (H_c1 and H_c2 are the first and second critical fields, respectively). These results are also relevant for the interpretation of small-angle neutron scattering experiments on the vortex state in type-II superconductors