Black carbon (BC) is a major component in atmospheric particulate matter (PM), which causes adverse health impacts and contributes significantly to climate change. Without widespread and accurate BC measurements, it remains difficult to track incomplete combustion sources and reduce BC emissions. Currently commercial BC sensors remain too costly to be deployed widely. In this work, a fast, cost-effective, and easily accessible method based on a smartphone camera was used to quantify color information of PM collected on filters to estimate BC and elemental carbon (EC) loadings. A robust RGB (red, green, blue)-based linear interaction model was built and validated using 1878 PM samples collected in three different regions with collocated BC and EC measurements. After applying image correction methods, this model shows a good predictability with an R-squared (R²) of 0.904 with state-of-the-art BC measurement techniques, and a coefficient of variation of the root mean square error (CV(RMSE)) of 25.3% despite the complex sources and different reference measurement techniques. This work validates the viabilities of using smartphones to quantify BC or EC loading on PM filters with a unified model and track incomplete combustion sources.