When Differential Equations Meet Neural Networks: The Rise of Neural CDEs

In our paper club session on July 19, 2024, we discussed the paper "Neural Controlled Differential Equations for Irregular Time Series" by Kidger et al. This article is based on that discussion.

Differential equations and neural networks represent two dominant paradigms in mathematical modeling. While differential equations have long been the cornerstone of modeling dynamical systems in physics, engineering, and other sciences, neural networks have revolutionized machine learning and artificial intelligence. In 2018, researchers proposed an intriguing fusion of these approaches through Neural Ordinary Differential Equations (Neural ODEs) [1], which define the evolution of hidden states through ordinary differential equations parameterized by neural networks, reimagining neural computation as a continuous flow rather than a sequence of discrete transformations.

However, in Neural ODEs, the time dimension serves merely as an internal detail of the model, without direct connection to sequential data. Simply aligning this internal time with the natural ordering of sequential data isn't straightforward, since the solution to an ODE is determined entirely by its initial condition, with no mechanism to adjust the trajectory based on subsequent observations. This limitation led to the development of Neural Controlled Differential Equations (Neural CDEs) [2] in 2020.

Neural CDEs can be thought of as the continuous-time analog of Recurrent Neural Networks (RNNs). While RNNs process sequential data in discrete steps, updating their hidden state with each new input, Neural CDEs allow for continuous evolution of the hidden state, where the evolution is influenced not only by its initial condition but also by the incoming data, through a continuous path derived from the input. It's important to note that Neural ODEs and CDEs differ fundamentally from physics-informed neural networks (PINNs). While PINNs incorporate differential equations into the training process by embedding them in the loss function, Neural ODEs and CDEs introduce differential equations into the actual evolution of the neural network itself.

What makes Neural CDEs particularly powerful is their ability to handle irregularly-sampled and partially-observed time series data - a common challenge in real-world applications like healthcare, where measurements might be taken at irregular intervals and some variables might be missing. The model works by converting discrete observations into a continuous path using interpolation, then using this path to control the evolution of the system's hidden state. Benefiting from memory-efficient training through adjoint-based backpropagation, Neural CDEs have shown strong performance in tasks ranging from time-series forecasting to classification.

Naturally, these advantages come with some trade-offs. Neural CDEs tend to be computationally slower than traditional RNNs, and they can require careful consideration in their architecture design to manage the number of parameters. Despite these limitations, they represent a significant advancement in bringing together classical differential equations and modern deep learning approaches, offering new possibilities for modeling dynamic systems in machine learning.

References:
[1] Chen RTQ, Rubanova Y, Bettencourt J, Duvenaud D. Neural ordinary differential equations. Adv Neural Inf Process Syst. 2018;31:6571-83.
[2] Kidger P, Morrill J, Foster J, Lyons T. Neural controlled differential equations for irregular time series. Adv Neural Inf Process Syst. 2020;33:6696-707.