On the Approximation Power of Two-Layer Networks of Random ReLUs

Computing Systems
Written By Alexis Avedisian

Video

Team Information

Team Members

  • Clayton Sanford, PhD Student in Computer Science, Columbia Engineering

  • Rocco Servedio, Professor, Department of Computer Science, Columbia Engineering

  • Manolis Vlatakis-Gkaragkounis, PhD Student in Computer Science, Columbia Engineering

  • Faculty Advisor: Daniel Hsu, Associate Professor of Computer Science, Columbia Engineering

Abstract

In order to better understand the representational powers and limitations of neural networks and better quantify the Universal Approximation Theorem, we consider the following question: how well can depth-two ReLU networks with randomly initialized bottom-level weights represent smooth functions? We give near-matching upper and lower bounds for L2-approximation in terms of the Lipschitz constant, the desired accuracy, and the dimension of the problem. Our upper bounds employ tools from harmonic analysis and ridgelet representation theory, while our lower bounds are based on dimensionality arguments.

Team Lead Contact

Clayton Sanford: clayton@cs.columbia.edu

Alexis Avedisian
Previous

2D Convolutional Neural Network on FPGA for High-Resolution TPC Images
Next

Real-time Data Processing for High-rate 3D Particle Imaging