For more than a century, abnormal nuclei in tumor cells, presenting subnuclear invaginations and folds on the nuclear envelope, have been known to be associated with high malignancy and poor prognosis. However, current nuclear morphology analysis focuses on the features of the entire nucleus, overlooking the malignancy-related subnuclear features in nanometer scale. The main technical challenge is to probe such tiny and randomly distributed features inside cells. We here employ nanopillar arrays to guide subnuclear features into ordered patterns, enabling their quantification as a strong indicator of cell malignancy. Both breast and liver cancer cells were validated as well as the quantification of nuclear abnormality heterogeneity. The alterations of subnuclear patterns were also explored as effective readouts for drug treatment. We envision that this nanopillar-enabled quantification of subnuclear abnormal features in tumor cells opens a new angle in characterizing malignant cells and studying the unique nuclear biology in cancer.