We study the microscopic origin of nonlocality in dense granular media. Discrete element simulations reveal that macroscopic shear results from a balance between microscopic elementary rearrangements occurring in opposite directions. The effective macroscopic fluidity of the material is controlled by these velocity fluctuations, which are responsible for nonlocal effects in quasistatic regions. We define a new micromechanically based unified constitutive law describing both quasistatic and inertial regimes, valid for different system configurations.