Neural Networks and Numerical Analysis

Bruno Després

Publisher:

De Gruyter

Publication Date:

2022

Number of Pages:

170

Format:

Hardcover

Price:

154.99

ISBN:

978-3110783124

Category:

Monograph

MAA Review
Table of Contents

[Reviewed by

Bill Satzer

, on

11/28/2022

]

The emerging technologies that are known collectively as neural networks, machine learning and deep learning can offer an efficient means of modeling nonlinear functions. This can be done with very good accuracy using an approach with just input and output data even in high-dimensional spaces. Yet, from the view of applied mathematics, these nonlinear functions are effectively unknowns, arising as the do purely from data. But results using these new methods have sometimes been compelling, and have included, for example, new insights in computational fluid dynamics and analysis of turbulence.

At the same time the relationship between practitioners of these new technologies and more traditional numerical analysts has been distant, often with little communication between the two. Part of this is likely due to their distinctly different terminologies, and perhaps concerns about lack of rigor from the more traditional numerical analysts.

The aim of the author in this book is to provide a unified framework for the foundations of numerical analysis of neural networks using some basic mathematical and algorithmic principles. Another goal is to introduce some related modern software and explain a connection with numerical discretization of simple partial differential equations.

The first three chapters – roughly half the book – are theoretical; the last three focus on practical problems needing computational treatment. The first part begins with basic principles for approximation of an objective function mapping a Euclidean space of one dimension to another, but with methods that can be implemented via neural networks. The author discusses the language of neural networks that frames the question, the parameters and hyperparameters involved, and the concept of a convolutive neural network. He also considers the existence and uniqueness issues for the parameters that minimize a cost function.

Important features of the first part of the book (not widely known among numerical analysts) are the theorems of Yarotsky and Cybenko that provide the foundation of most of the modern research in the numerical analysis of neural networks. These provide universal approximation theorems describing the ability of neural networks to describe functions implementable in neural networks.

The second part of the book, which considers computational questions, examines three distinct numerical analysis questions that arise in the application of neural network methods. The first looks at how to create a data set designed to handle numerical inverse problems. The primary issue here is dealing with very large amounts of input and output data by appropriately selecting a workable subset. The author considers the available sampling methods and recommends the Latin hypercube method as a good alternative to uniform sampling and the Monte Carlo method.

The other two questions explore stochastic gradient descent problems for estimating parameters in solving ordinary differential equations and applications of neural network and machine learning methods for the numerical approximation of solutions to partial differential equations. The first question is known as numerical optimization in the world of numerical analysts but called training of parameters in the machine learning community.

The author describes his intended audience as those applied scientists who are looking for a mathematical acquaintance with the general area in preparation for using neural networks and machine learning with their own projects. Prerequisites would include more than a basic knowledge of neural networks and a background in optimization and the general methods of numerical analysis. Some knowledge of functional analysis would also be useful. This is a specialized text primarily intended for researchers attempting to learn how to apply neural network methods to problems that would ordinarily fall into the domain of numerical analysis.

Bill Satzer (bsatzer@gmail.com), now retired from 3M Company, spent most of his career as a mathematician working in industry on a variety of applications. He did his PhD work in dynamical systems and celestial mechanics.

See the publisher's website.

Tags:

Numerical Analysis