In the last two decades, carbon fiber reinforced polymer (CFRP) composites have been used for the strengthening of various civil structures. Since recently, iron-based shape memory alloy (Fe-SMA) has been introduced as a promising material, and Fe-SMA reinforcements have been utilized for prestressed strengthening of existing structures. This study aims to conduct a comparative analysis between the structural performance and cost of using Fe-SMA vs. CFRP reinforcements for the prestressed strengthening of existing civil structures. First, a comprehensive study is presented that compares Fe-SMA and CFRP at the material level, with the main focus on the relevant structural aspects, such as the static tensile behavior, high-cycle fatigue (HCF) performance, creep and relaxation, corrosion resistance, thermal compatibility with common construction materials (i.e., steel and concrete), and the behavior under high temperatures and fire exposure. Second, the application complexity and the structural performance of the two strengthening techniques, by using mechanically anchored prestressed CFRPs or thermally activated Fe-SMAs, are compared at the system level by performing sets of static and HCF tests on CFRP and Fe-SMA strengthened steel plates and girders. Finally, a cost comparison is carried out for a specific case, where normal modulus (NM) CFRP plates and Fe-SMA strips were used for the prestressed strengthening of a real-scale steel I-girder. The cost analysis revealed that although the current price of Fe-SMA strips is higher than that of the NM CFRP plates, both of the strengthening systems are financially equivalent, when the achievable prestressing force and the cost of clamping systems are taken into consideration.