When
Where
Speaker: Saad Qadeer, Pacific Northwest National Laboratory
Title: Accelerating Neural Network-based Regression Through Empirical Kernels
Abstract: Despite their promise in performing a variety of learning tasks, a theoretical understanding of the effectiveness of Deep Neural Networks (DNNs) has so far proven elusive, partly due to the difficulties inherent in studying their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed forms. This suggests, and experiments corroborate, that empirical kernel machines can also act as surrogates for finite width DNNs. The computational cost of assembling the full NTK, however, makes this approach practically infeasible.
In this talk, we will discuss the performance of the Conjugate Kernel (CK), a low-cost approximation to the NTK. For the regression problem of smooth functions and classification using logistic regression, we shall show that the CK performance is only marginally worse than that of the NTK and, in certain cases, much superior. In particular, we will present bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We also present demonstrations of this on large language models and physics-informed operator networks.