Abstract: Automatic differentiation (AD) is a method to compute gradients of numerical software with a long history in science, engineering, and recently machine learning. Because of the ongoing stall in processor clock speeds and the growing demand for energy-efficient computing, applications are increasingly using multithreading, SIMD vectorization, or GPU offloading. Current AD tools lack support for all these techniques, particularly in the adjoint or reverse mode, which is the basis for many real-world applications.
In this talk, I will discuss some of the technical difficulties that prevent AD tools from adapting to these user needs, particularly for unstructured programs whose memory access pattern is not known statically. Following this, I will present early success stories of code analysis and transformation methods that preserve a given parallelization during differentiation. The talk will conclude with a roadmap toward AD tools that can handle structured and unstructured multithreaded and vectorized programs in practice.