Much of this content is based on lecture slides from slides from Professor David Barber at University College London: resources relating to this can be found at: www.cs.ucl.ac.uk/staff/D.Barber/brml
What is Autodiff?
Autodiff, or Automatic Differentiation, is a method of determining the exact derivative of a function with respect to its inputs. It is widely used in machine learning- in this post I will give an overview of what autodiff is and why it is a useful tool.
The above is not a very helpful definition, so we can compare autodiff first to symbolic differentiation and numerical approximations before going into how it works.
Symbolic differentiation is what we do when we calculate derivatives when we do it by hand, i.e. given a function , we find a new function . This is really good when we want to know how functions behave across all inputs. For example if we had we can find the derivative as and then we can find the derivative of the function for all values of .…