Magnetic damping plays a crucial role in the dynamics of magnetic systems. Hence, control over the damping is highly desirable for the development of magnetic devices. One of the possible ways to manipulate magnetic damping in a ferromagnetic material is the injection of spin currents generated by the spin Hall effect. The generated spin currents can be used to manipulate magnetic moments as well as the effective damping of the ferromagnetic material. In this paper, we demonstrate that the influence of the spin current on the damping is highly sensitive to the relative direction of the spin polarization and the precessional axis of the magnetization. The observations are important for various applications in magnetic storage devices and in microwave signal processing.