A penalization method to estimate the intrinsic dimensionality of data

Abstract

We propose a novel penalization method for estimating the intrinsic dimensionality of data within a Probabilistic Principal Components Model, extending beyond the Gaussian case. Unlike existing approaches, our method is designed to handle non-normal data, providing a flexible alternative to traditional factor models. Our procedure identifies the dimension at which the eigenvalues of a scatter matrix stabilize. We establish the consistency of the procedure under mild conditions and demonstrate its robustness across a range of data distributions. A comparative analysis highlights its advantages over existing techniques, making it a valuable tool for dimensionality estimation without relying on distributional assumptions.