The universal theory of critical phase transitions describes the critical behavior at second-order phase transitions in infinitely large systems. With the increased contemporary interest in nanoscale materials, we investigated CoO nanoparticles by means of neutron scattering and found how the theory of critical phenomena breaks down in the nanoscale regime. Using CoO as a model system, we have identified a size-dependent nanocritical temperature region close to the antiferromagnetic phase transition where the magnetic correlation length of the nanoparticles converges to a constant value, which is significantly smaller than that of the saturated state found at low temperatures. This is in clear contrast to the divergence around T_N observed for bulk systems. Our finding of nanocriticality in the magnetic phase transition is of great importance for the understanding of phase transitions at the nanoscale.