Ferrimagnetic alloys are model systems for understanding the ultrafast magnetization switching in materials with antiferromagnetically coupled sublattices. Here we investigate the dynamics of the rare-earth and transition-metal sublattices in ferrimagnetic GdFeCo and TbCo dots excited by spin-orbit torques with combined temporal, spatial and elemental resolution. We observe distinct switching regimes in which the magnetizations of the two sublattices either remain synchronized throughout the reversal process or switch following different trajectories in time and space. In the latter case, we observe a transient ferromagnetic state that lasts up to 2 ns. The asynchronous switching of the two magnetizations is ascribed to the master-agent dynamics induced by the spin-orbit torques on the transition-metal and rare-earth sublattices and their weak antiferromagnetic coupling, which depends sensitively on the alloy microstructure. Larger antiferromagnetic exchange leads to faster switching and shorter recovery of the magnetization after a current pulse. Our findings provide insight into the  dynamics of ferrimagnets and the design of spintronic devices with fast and uniform switching.