Jet production in lead-lead (PbPb) and proton-proton (pp) collisions at a nucleon-nucleon center-of-mass energy of 5.02 TeV is studied with the CMS detector at the LHC, using PbPb and pp data samples corresponding to integrated luminosities of 404 μb^-1 and 27.4 pb^-1, respectively. Jets with different areas are reconstructed using the anti-k_T algorithm by varying the distance parameter R. The measurements are performed using jets with transverse momenta (p_T) greater than 200 GeV and in a pseudorapidity range of |η| < 2. To reveal the medium modification of the jet spectra in PbPb collisions, the properly normalized ratio of spectra from PbPb and pp data is used to extract jet nuclear modification factors as functions of the PbPb collision centrality, p_T and, for the first time, as a function of R up to 1.0. For the most central collisions, a strong suppression is observed for high-p_T jets reconstructed with all distance parameters, implying that a significant amount of jet energy is scattered to large angles. The dependence of jet suppression on R is expected to be sensitive to both the jet energy loss mechanism and the medium response, and so the data are compared to several modern event generators and analytic calculations. The models considered do not fully reproduce the data.