In order to improve the sustainability of the low-noise Semi-Dense Asphalt (SDA), the aggregate fractions were replaced by recycled concrete aggregates (RCA), an abundant construction waste. The aggregate replacement was conducted by fractions of 2/4 and 0.125/2 mm corresponding to the 2/4 mm (coarse), 0.063/4 mm (sand) fractions of the control virgin aggregates, at about 15% replacement by volume of the control mixture for each fraction. Additionally, RCA filler replaced the control filler entirely, bringing the replacement level to over 20% of the total mixture. The binder was added at the same amount as the control binder and +0.5%, with additional binder compensating for the estimated increase in binder absorption in RCA. The wheel track rutting test at 60 °C indicated marginally improved performance with RCA sand replacement. However, low temperature semi-circular bending (SCB) test at 0 °C showed that there is a 10% reduction in cracking resistance using RCA coarse and 20% for RCA sand. The laser texture scanning indicated a small reduction in the texture level for the RCA samples before wheel tracking, and very similar texture profiles with the control after the wearing of the samples, indicating no significant effect in non-acoustic noise reduction properties. This study suggests that the use of RCA fractions in porous SDA surface mixtures is possible with particular attention to mixture optimization to improve rutting performance.