## ch-2 How the backpropagation algorithm works

discuss **how to compute** the gradient ( $\nabla C$ ) of the cost function

Heart of backpropagation

- an expression for the partial derivative ∂ C /∂ w of the cost function C with respect to any weight w ( or bias b ) in the network
- tells us
**how quickly**the cost changes when

we change the weights and biases

### Warm up: a fast matrix-based approach to computing the output from a neural network

Notation

Vectorized form

- apply the weight matrix to the activations, then add the bias vector, and finally apply the σ function
- a^l=σ(w^la^{l−1}+b^l) =σ(z^l)

### The two assumptions we need about the cost function

- the cost function can be written as an average $ C=\frac{1}{n}∑_xC_x$ over cost functions $C_x$ for individual training examples, x.
- the cost can be written as a
**function of the outputs**from the neural network: $cost C = C ( a^L )$

### The four fundamental equations behind backpropagation

**DEF error ** $ δ^l_j $

**Error in the output layer** δ^l

**Error δ^l in terms of the error in the next layer, δ^l+1**

**Rate of change of the cost with respect to any bias**

**Rate of change of the cost with respect to any weight**

### Proof of the four fundamental equations

using chain rule

### The backpropagation algorithm

- correctness: because the cost is a function of outputs
- To understand how the cost varies with earlier weights and biases we need to repeatedly apply the chain rule, , working backward through the layers to obtain usable expressions

### Backpropagation: the big picture

a clever way of keeping track of small perturbations to the weights (and biases) as they propagate through the network, reach the output, and then affect the cost