A search is presented for the production of a pair of Higgs bosons, where one decays into two photons and the other one into a bottom quark–antiquark pair. The analysis is performed using proton–proton collision data at √s = 13 TeV recorded in 2016 by the CMS detector at the LHC, corresponding to an integrated luminosity of 35.9fb^-1. The results are in agreement with standard model (SM) predictions. In a search for resonant production, upper limits are set on the cross section for new spin-0 or spin-2 particles. For the SM-like nonresonant production hypothesis, the data exclude a product of cross section and branching fraction larger than 2.0fb at 95% confidence level (CL), corresponding to about 24 times the SM prediction. Values of the effective Higgs boson self-coupling κ_λ are constrained to be within the range -11 < κ_λ < 17 at 95% CL, assuming all other Higgs boson couplings are at their SM value. The constraints on κ_λ are the most restrictive to date.