BaRFeO₄ compounds containing magnetic rare-earth ions (R = Yb, Tm) were prepared by a conventional solid-state method in air. We present detailed measurements of magnetic properties (specific heat, magnetic susceptibility) to demonstrate the presence of three successive magnetic phase transitions (at T_N1, T_N2, and T₃) in both compounds and employed neutron diffraction to determine the magnetic structures. Both compounds are isostructural (orthorhombic space group Pnma) and magnetic ions form rings and chains along the b direction. All magnetic structures are incommensurate with a propagation vector k = (0, 0, k_z). Magnetic ordering of Fe³⁺ ions occurs at T_N1 and T_N2, where two different irreducible representations (irreps) order. Below T_N1, there is a collinear spin-density wave with ordered Fe³⁺ moments along the chain direction (b axis). Below T_N2, this component remains stable and an additional component inside the ac plane appears. Each Fe chain adopts a collinear antiferromagnetic structure with a constant magnetic phase. The two Fe rings in the unit cell have a different chirality and a noncollinear coupling. The mixing of two irreps leads to a cycloidal spiral magnetic structure that allows spin-induced ferroelectric polarization at T_N2. With the presence of modulated components both perpendicular and along the propagation vector k, the magnetic structure can be viewed as a sum of a helix and a cycloid structure. For BaTmFeO₄, the magnetic structure has a larger cycloidal contribution and the dielectric constant ε exhibits a small peak at T_N2. Yb³⁺ moments order at T₃ with each Yb chain having a constant magnetic phase and a collinear antiferromagnetic structure stabilized by 4f - 4f electron-exchange interactions. In contrast, no constant magnetic phase is observed at the Tm chains. Below T₃, magnetic order of Tm2 ions is induced by 3d - 4f electron-exchange interactions and Tm1 ions remain disordered down to the lowest measured temperature T = 1.6 K due to frustration of magnetic exchange interactions. The obtained magnetic structures are compared with those of BaYFeO₄.