Analysis of one-hidden-layer Neural Networks via the Resolvent Method - ENS de Lyon - École normale supérieure de Lyon Accéder directement au contenu
Article Dans Une Revue Advances in Neural Information Processing Systems Année : 2021

Analysis of one-hidden-layer Neural Networks via the Resolvent Method

Résumé

In this work, we investigate the asymptotic spectral density of the random feature matrix M = Y Y * with Y = f (W X) generated by a single-hidden-layer neural network, where W and X are random rectangular matrices with i.i.d. centred entries and f is a non-linear smooth function which is applied entry-wise. We prove that the Stieltjes transform of the limiting spectral distribution approximately satisfies a quartic self-consistent equation, which is exactly the equation obtained by Pennington and Worah [22] and Benigni and Péché [6] with the moment method. We extend the previous results to the case of additive bias Y = f (W X + B) with B being an independent rank-one Gaussian random matrix, closer modelling the neural network infrastructures encountered in practice. Our key finding is that in the case of additive bias it is impossible to choose an activation function preserving the layer-to-layer singular value distribution, in sharp contrast to the bias-free case where a simple integral constraint is sufficient to achieve isospectrality. To obtain the asymptotics for the empirical spectral density we follow the resolvent method from random matrix theory via the cumulant expansion. We find that this approach is more robust and less combinatorial than the moment method and expect that it will apply also for models where the combinatorics of the former become intractable. The resolvent method has been widely employed, but compared to previous works, it is applied here to non-linear random matrices.
Fichier principal
Vignette du fichier
neurips_2021_paper.pdf (449.39 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

ensl-03475949 , version 1 (11-12-2021)

Identifiants

  • HAL Id : ensl-03475949 , version 1

Citer

Vanessa Piccolo, Dominik Schröder. Analysis of one-hidden-layer Neural Networks via the Resolvent Method. Advances in Neural Information Processing Systems, 2021. ⟨ensl-03475949⟩

Collections

ENS-LYON CNRS UDL
114 Consultations
64 Téléchargements

Partager

Gmail Facebook X LinkedIn More