Statistical breakthroughs and novel perspectives in deep learning theory

Since several years, deep learning has emerged as a transformative field, with its theory involving several disciplines such as approximation theory, statistics and optimization. In this talk we delve into key theoretical breakthroughs, with a particular focus on statistical results. We critically question prevailing frameworks and identify their key limitations. Central to the discussion is a novel statistical framework for image analysis that reinterprets images not as high-dimensional entities, but as structured objects shaped by geometric deformations such as translations, rotations, and scalings. Within this framework, classification is reframed as the task of learning uninformative deformations, leading to convergence rates with more favorable trade-offs between input dimension and sample size. This geometric-statistical perspective not only provides new guarantees for approximation and convergence in deep learning-based image classification but also prompts a rethinking of theoretical approaches for broader prediction problems. In the final part of the talk, we examine the expressive power of ReLU networks in comparison to networks with Heaviside activation functions. While ReLU-based models have become standard in deep learning theory, Heaviside networks offer a compelling alternative that aligns more closely with biologically inspired architectures. We conclude by outlining future research directions and reflecting on the role of theory in the field. 

This talk is based on joint work with Juntong Chen, Insung Kong and Johannes Schmidt-Hieber.