4 General Fully Connected Neural Networks

4.1.1 The architecture of neural networks

Copy link

The ideal network architecture for a task must be found via experimentation guided by monitoring the validation set error.

Let’s consider the following architecture.

• The left most layer is called the input layer, and the neurons within the layer are called input neurons. A neuron of this layer is of a special kind since it has no input and it only outputs an $x_j$ value the $j$th features.

• The right most or output layer contains the output neurons (just one here). The middle layer is called a hidden layer, since the neurons in this layer are neither inputs nor outputs. We do not observe (directly) what goes out of this layer.

Somewhat confusingly, and for historical reasons, such multiple layer networks are sometimes called multilayer perceptrons or MLPs, despite being made up of sigmoid neurons, not perceptrons.

The design of the input and output layers in a network is often straightforward: as many neurons in the input layer than the number of explanatory/features variables; as many neurons in the output layer than the number of possible values for the response variable (if it is qualitative).

But there can be quite an art to the design of the hidden layers. Rectified linear units are an excellent default choice of hidden unit. Rectified linear units are easy to optimize because they are so similar to linear units.

When the output from one layer is used as input to the next layer (with no loops), we speak about feedforward neural networks. Other models are called recurrent neural networks.

Deep feedforward networks, also often called feedforward neural networks,or multilayer perceptrons (MLPs), are the quintessential deep learning models.

4.1.2 Goals

Copy link

• The goal of a feedforward network is to approximate some function $f^{\ast }$. A feedforward network defines a mapping $y=f(x;\theta )$ and learns the value of the parameters $\theta$ that result in the best function approximation.

• The function $f$ is composed of a chain of functions: $$f=f^{\left(k\right)}\left(f^{(k-1)}\right(\dots f^{\left(1\right)}\left)\right),$$ where $f^{\left(1\right)}$ is called the firstlayer, and so on. The depth of the network is $k$. The final layer of a feedforward network is called the output layer.

Rather than thinking of the layer as representing a single vector-to-vector function, we can also think of the layer as consisting of many unit that act in parallel, each representing a vector-to-scalar function.

Architecture of the network is an art:

• how many layers the network should contain

• how these layers should be connected to each other

• how many units should be in each layer.

4.1.3 One Layer NN

Copy link

In this shalow neural network, we have:

• $x_1,x_2,x_3$ are inputs of a Neural Network. These elements are scalars and they are stacked vertically. This also represents an input layer.

• Variables in a hidden layer are not seen in the input set.

• The output layer consists of a single neuron only $\hat{y}$ is the output of the neural network.

Let precise some notation that we will use

$$\{\begin{array}{ccc}{z}_1^{\left[1\right]}& =& {w_1^{\left[1\right]}}^{T}x+b_1^{\left[1\right]}{a}_1^{\left[1\right]}=\sigma \left(z_1^{\left[1\right]}\right)\\ z_2^{\left[1\right]}& =& {w_2^{\left[1\right]}}^{T}x+b_2^{\left[1\right]}{a}_2^{\left[1\right]}=\sigma \left(z_2^{\left[1\right]}\right)\\ z_3^{\left[1\right]}& =& {w_3^{\left[1\right]}}^{T}x+b_3^{\left[1\right]}{a}_3^{\left[1\right]}=\sigma \left(z_3^{\left[1\right]}\right)\\ z_4^{\left[1\right]}& =& {w_4^{\left[1\right]}}^{T}x+b_4^{\left[1\right]}{a}_4^{\left[1\right]}=\sigma \left(z_4^{\left[1\right]}\right)\end{array}$$

where $x=(x_1,x_2,x_3{)}^T$ and $w_j^{\left[1\right]}=(w_{j,1}^{\left[1\right]},w_{j,2}^{\left[1\right]},w_{j,3}^{\left[1\right]},w_{j,4}^{\left[1\right]}{)}^T$ (for $j=1,\dots ,4$).

Then, the output layer is defined by:

$$\begin{array}{ccc}{z}_1^{\left[2\right]}& =& {w_1^{\left[2\right]}}^T{a}^{\left[1\right]}+b_1^{\left[2\right]}{a}_1^{\left[2\right]}=\sigma \left(z_1^{\left[2\right]}\right)\end{array}$$

where $a^{\left[1\right]}=(a_1^{\left[1\right]},\dots ,a_4^{\left[1\right]}{)}^T$ and $w_1^{\left[2\right]}=(w_{1,1}^{\left[2\right]},w_{1,2}^{\left[2\right]},w_{1,3}^{\left[2\right]},w_{1,4}^{\left[2\right]}{)}^T$

One can use matrix representation for efficiency computation:

$$\begin{array}{}\left[\begin{array}{ccc}-& {w_1^{\left[1\right]}}^T& -\\ -& {w_2^{\left[1\right]}}^T& -\\ -& {w_3^{\left[1\right]}}^T& -\\ -& {w_4^{\left[1\right]}}^T& -\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]+\left[\begin{array}{c}{b}_1^{\left[1\right]}\\ b_2^{\left[1\right]}\\ b_3^{\left[1\right]}\\ b_4^{\left[1\right]}\end{array}\right]=\left[\begin{array}{c}{w_1^{\left[1\right]}}^{T}x+b_1^{\left[1\right]}\\ {w_2^{\left[1\right]}}^{T}x+b_2^{\left[1\right]}\\ {w_3^{\left[1\right]}}^{T}x+b_3^{\left[1\right]}\\ {w_4^{\left[1\right]}}^{T}x+b_4^{\left[1\right]}\end{array}\right]=\left[\begin{array}{c}{z}_1^{\left[1\right]}\\ z_2^{\left[1\right]}\\ z_3^{\left[1\right]}\\ z_4^{\left[1\right]}\end{array}\right]& \text{(1)}\end{array}$$

and by defining
$$W^{\left[1\right]}=\left[\begin{array}{ccc}-& {w_1^{\left[1\right]}}^T& -\\ -& {w_2^{\left[1\right]}}^T& -\\ -& {w_3^{\left[1\right]}}^T& -\\ -& {w_4^{\left[1\right]}}^T& -\end{array}\right]b^{\left[1\right]}=\left[\begin{array}{c}{b}_1^{\left[1\right]}\\ b_2^{\left[1\right]}\\ b_3^{\left[1\right]}\\ b_4^{\left[1\right]}\end{array}\right]z^{\left[1\right]}=\left[\begin{array}{c}{z}_1^{\left[1\right]}\\ z_2^{\left[1\right]}\\ z_3^{\left[1\right]}\\ z_4^{\left[1\right]}\end{array}\right]a^{\left[1\right]}=\left[\begin{array}{c}{a}_1^{\left[1\right]}\\ a_2^{\left[1\right]}\\ a_3^{\left[1\right]}\\ a_4^{\left[1\right]}\end{array}\right]$$ we can write $$z^{\left[1\right]}=W^{\left[1\right]}x+b^{\left[1\right]}$$ and then by applying Element-wise Independent activation function $\sigma (\cdot )$ to the vector $z^{\left[1\right]}$ (meaning that $\sigma (\cdot )$ are applied independently to each element of the input vector $z^{\left[1\right]}$) we get:

$$a^{\left[1\right]}=\sigma \left(z^{\left[1\right]}\right).$$ The output layer can be computed in the similar way:

$$z^{\left[2\right]}=W^{\left[2\right]}{a}^{\left[1\right]}+b^{\left[2\right]}$$

where

$$W^{\left[2\right]}=\left[\begin{array}{c}{w}_{1,1}^{\left[2\right]}\\ w_{1,2}^{\left[2\right]}\\ w_{1,3}^{\left[2\right]}\\ w_{1,4}^{\left[2\right]}\end{array}\right]b^{\left[2\right]}=\left[\begin{array}{c}{b}_1^{\left[2\right]}\\ b_2^{\left[2\right]}\\ b_3^{\left[2\right]}\\ b_4^{\left[2\right]}\end{array}\right]$$

and finally:

$$a^{\left[2\right]}=\sigma \left(z^{\left[2\right]}\right)⟶\hat{y}$$

So far we have used:

• The superscript number ${}^{\left[i\right]}$ for denoting the layer number and the subscript number ${}_j$ denotes the neuron number in a particular layer
• $x$ is the input vector consisting of 3 features.
• $W_j^{\left[i\right]}$ is the weight vector associated with neuron $j$ present in the layer $i$
• $b_j^{\left[i\right]}$ is the bias associated with neuron j present in the layer i.
• $z_j^{\left[i\right]}$ is the intermediate output associated with neuron $j$ present in the layer $i$.
• $a_j^{\left[i\right]}$ is the final output associated with neuron $j$ present in the layer $i$.
• As an example $\sigma (\cdot )$ is the sigmoid activation function

Using this notation the forward-propagation equations are:

$$\{\begin{array}{ccc}{z}^{\left[1\right]}& =& W^{\left[1\right]}x+b^{\left[1\right]}\\ a^{\left[1\right]}& =& \sigma \left(z^{\left[1\right]}\right)\\ z^{\left[2\right]}& =& W^{\left[2\right]}{a}^{\left[1\right]}+b^{\left[2\right]}\\ \hat{y}& =& a^{\left[2\right]}=\sigma \left(z^{\left[2\right]}\right)\end{array}$$

Why non-linear Activation is important. Consider this neural network without activation functions:

$$\{\begin{array}{ccc}{z}^{\left[1\right]}& =& W^{\left[1\right]T}x+b^{\left[1\right]}\\ \hat{y}& =& z^{\left[2\right]}=W^{\left[2\right]T}{z}^{\left[1\right]}+b^{\left[2\right]}\end{array}$$ Then, it follows

$$\{\begin{array}{ccc}{z}^{\left[1\right]}& =& W^{\left[1\right]}x+b^{\left[1\right]}\\ \hat{y}& =& z^{\left[2\right]}=W^{\left[2\right]}{W}^{\left[1\right]}x+W^{\left[2\right]}{b}^{\left[1\right]}+b^{\left[2\right]}\\ \hat{y}& =& z^{\left[2\right]}=W_{new}x+b_{new}\end{array}$$ The output is then a linear combination of a new weight matrix, input and a new bias. Thus if we use an identity activation function then the Neural Network will output linear output of the input. Indeed,a composition of two linear functions is a linear function and so we lose the representation power of a NN. However, a linear activation function is generally recommended and implemented in the output layer in case of regression.

4.1.4 N layers Neural Network

Copy link

We can generalize this simple previous neural network to a Multi-layer fully-connected neural networks by sacking more layers get a deeper fully-connected neural network defining by the following equations:

$$\{\begin{array}{ccc}{a}^{\left[1\right]}& =& g^{\left[1\right]}(W^{\left[1\right]}x+b^{\left[1\right]})\\ a^{\left[2\right]}& =& g^{\left[2\right]}(W^{\left[2\right]}{a}^{\left[1\right]}+b^{\left[2\right]})\\ \dots & =& \dots \\ a^{[r-1]}& =& g^{[r-1]}(W^{[r-1]}{a}^{[r-2]}+b^{[r-1]})\\ \hat{y}=a^{\left[r\right]}& =& g^{\left[r\right]}(W^{\left[r\right]}{a}^{[r-1]}+b^{\left[r\right]})\end{array}$$

This neural network is composed of $r$ layers based on $r$ weight matrices $W^{\left[1\right]},\dots ,W^{\left[r\right]}$ and $r$ bias vectors $b^{\left[1\right]},\dots ,b^{\left[r\right]}$. The $r$ activation functions noted $g^{\left[r\right]}$ might be different for each layer $r$. The number of neurons in each layer could be also be not equal and will be noted $m_r$. Using this notation:

• The total number of neurons in the network: $m_1+\dots +m_r$
• Total number of parameters in this network is $(d+1)m_1+(m_1+1)m_2+\dots +(m_{r-1}+1)\ast m_r$
• $W^{\left[1\right]}\in R^{m_1\times d}$, $W^{\left[k\right]}\in R^{m_k\times m_{k-1}}$ ($k=2,\dots ,r-1$) and $W^{\left[r\right]}\in R^{1\times m_{r-1}}$

How do we count layers in a neural network?:

When counting layers in a neural network we count hidden layers as well as the output layer, but we don’t count an input layer.

It is a four layer neural network with three hidden layers.

4.1.5 Vectorizing Across Multiple Training Examples

Copy link

So far we have defined our Neural Network using only one inpute feature vector $x$ to generate prediction $\hat{y}$ $$x⟶a^{\left[2\right]}=\hat{y}$$ Let consider $m$ training example $x^{\left[1\right]},\dots ,x^{\left[m\right]}$ and we can use our NN to get for each training example $x^{\left[i\right]}$ a prediction

$$x^{\left(i\right)}⟶a^{\left[2\right]\left(i\right)}=\hat{y}i=1,\dots m$$ One can use a loop for getting all the predictions. However, matrix representation will help us to overcome the computational issue of using loop strategy.

Let first define the matrix $\text{X}$ which every column is a feature vector for one training sample:

$$\text{X}=\left[\begin{array}{cccc}|& |& \dots & |\\ x^{\left(1\right)}& x^{\left(2\right)}& \dots & x^{\left(m\right)}\\ |& |& \dots & |\end{array}\right].$$

Then, we define the matrix ${\text{Z}}^{\left[1\right]}$ with columns $z^{\left[1\right]\left(1\right)}\dots z^{\left[1\right]\left(m\right)}$

$${\text{Z}}^{\left[1\right]}=\left[\begin{array}{cccc}|& |& \dots & |\\ z^{\left[1\right]\left(1\right)}& z^{\left[1\right]\left(2\right)}& \dots & z^{\left[1\right]\left(m\right)}\\ |& |& \dots & |\end{array}\right].$$ The activation matrix for the first hidden layer ${\text{A}}^{\left[1\right]}$ is defined similary by:

$${\text{A}}^{\left[1\right]}=\left[\begin{array}{cccc}|& |& \dots & |\\ a^{\left[1\right]\left(1\right)}& a^{\left[1\right]\left(2\right)}& \dots & a^{\left[1\right]\left(m\right)}\\ |& |& \dots & |\end{array}\right],$$ where for example the element in the first row and in the first column of a matrix ${\text{A}}^{\left[1\right]}$ is an activation of the first hidden unit and the first training example. For clarification:

$$A^{\left[1\right]}=\left[\begin{array}{cccc}{1}^{st}unitof1.tr.example& 1^{st}unitof{2}^{nd}tr.example& \dots & 1^{st}unitof{m}^{th}.tr.example\\ 2^{nd}unitof{1}^{st}tr.example& 2^{nd}unitof{2}^{nd}tr.example& \dots & 2^{nd}unitof{m}^{th}tr.example\\ thelastunitof{1}^{st}tr.example& thelastunitof{2}^{nd}tr.example& \dots & thelastunitof{m}^{th}tr.example\end{array}\right]$$ Based on this matrix representation we get:

$$\{\begin{array}{ccc}{Z}^{\left[1\right]}& =& W^{\left[1\right]}\text{X}+b^{\left[1\right]}\\ A^{\left[1\right]}& =& \sigma \left(Z^{\left[2\right]}\right)\\ Z^{\left[2\right]}& =& W^{\left[2\right]}{A}^{\left[1\right]}+b^{\left[2\right]}\\ A^{\left[2\right]}& =& \sigma \left(Z^{\left[2\right]}\right)\end{array}$$

One can notice that we add $b^{\left[1\right]}\in R^{4\times 1}$ to $W^{\left[1\right]}\text{X}\in R^{4\times m}$, which is strictly not allowed following the rules of linear algebra. In practice however, this addition is performed using broadcasting. By defining

$${\tilde{b}}^{\left[1\right]}=\left[\begin{array}{cccc}|& |& \dots & |\\ b^{\left[1\right]}& b^{\left[1\right]}& \dots & b^{\left[1\right]}\\ |& |& \dots & |\end{array}\right].$$ we can compute:

$$Z^{\left[1\right]}=W^{\left[1\right]}\text{X}+{\tilde{b}}^{\left[1\right]}$$

4.3.1 Sigmoid function

Copy link

$$\sigma \left(z\right)=g\left(z\right)=\frac{1}{1+e^{-z}}$$

Its derivative :

$$\frac{d}{dz}\sigma \left(z\right)=\sigma \left(z\right)(1-\sigma (z\left)\right)$$

f <- function(x){1 / (1 + exp(-x))}
df <- function(x){f(x)*(1-f(x))}
plotf <- plot_activation_function(f, 'Sigmoid', c(-4,4))
plotdf <- plot_activation_function(df, 'Derivative', c(-4,4))

library(ggpubr)
ggarrange(plotf,plotdf,nrow=1,ncol=2)

Figure 4.1: Sigmoid and derivative function

4.3.2 ReLU function

Copy link

A recent invention which stands for Rectified Linear Units.

$$ReLU\left(z\right)=max(0,z)$$

Despite its name and appearance, it’s not linear and provides the same benefits as Sigmoid but with better performance.

Its derivative :

$$\frac{d}{dz}ReLU\left(z\right)=\{\begin{array}{c}1ifz>0\\ 0ifz<0\\ undefinedifz=0\end{array}$$

rec_lu_func <- function(x){ ifelse(x < 0 , 0, x )}
drec_lu_func <- function(x){ ifelse(x < 0 , 0, 1)}
plotf <- plot_activation_function(rec_lu_func, 'ReLU', c(-4,4))
plotdf <- plot_activation_function(drec_lu_func, 'Derivative', c(-4,4))

ggarrange(plotf,plotdf,nrow=1,ncol=2)

Figure 4.2: ReLU and derivative function

4.3.3 LeakyRelu function

Copy link

Leaky Relu is a variant of ReLU. Instead of being 0 when $z<0$, a leaky ReLU allows a small, non-zero, constant gradient $\alpha$ (usually, $\alpha =0.01$). However, the consistency of the benefit across tasks is presently unclear.

$$LeaklyReLU\left(z\right)=max(\alpha z,z)$$

Its derivative :

$$\frac{d}{dz}LeaklyReLU\left(z\right)=\{\begin{array}{cc}\alpha & ifz<0\\ 1& ifz\ge 0\end{array}$$

rec_lu_func <- function(x){ ifelse(x < 0 , 0.01*x, x )}
drec_lu_func <- function(x){ ifelse(x < 0 , 0.01, 1)}
plotf <- plot_activation_function(rec_lu_func, 'LeaklyReLU', c(-4,4))
plotdf <- plot_activation_function(drec_lu_func, 'Derivative', c(-4,4))

ggarrange(plotf,plotdf,nrow=1,ncol=2)

Figure 4.3: LeaklyReLU and derivative function

4.3.4 Tanh function

Copy link

Tanh squashes a real-valued number to the range $[-1,1]$. It’s non-linear. But unlike Sigmoid, its output is zero-centered. Therefore, in practice the tanh non-linearity is always preferred to the sigmoid nonlinearity.

$$tanh\left(z\right)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$$

Its derivative :

$$\frac{d}{dz}tanh\left(z\right)=1-tanh(z{)}^2$$

tanh_func <- function(x){tanh(x)}
dtanh_func <- function(x){1-(tanh(x))**2}
plotf <- plot_activation_function(tanh_func, 'TanH', c(-4,4))
plotdf <- plot_activation_function(dtanh_func, 'Derivative', c(-4,4))

ggarrange(plotf,plotdf,nrow=1,ncol=2)

Figure 4.4: Tanh and derivative function

4.4.1 Simple Case with two layers

Copy link

Let derive the backpropagation algorithm for the following neural network by considering ReLu activations function for all the neurons of the first Layer and the identity function for the output layer.

Remind the main equations:

$$\{\begin{array}{ccc}{z}^{\left[1\right]}& =& W^{\left[1\right]}x+b^{\left[1\right]}\\ a^{\left[1\right]}& =& ReLu\left(Z^{\left[1\right]}\right)\\ z^{\left[2\right]}& =& W^{\left[2\right]}{a}^{\left[1\right]}+b^{\left[2\right]}\\ \hat{y}=a^{\left[2\right]}& =& g\left(z^{\left[2\right]}\right)\end{array}$$

Let consider a regression framework and consider the identity function for the output activation function: $g\left(x\right)=x$. In this framework it is natural to use the lesat square cost function:

$$J=\frac{1}{2}(y-\hat{y}{)}^2$$ Let precise some dimension of our objects:

• $d$ is the number of features and $x\in R^d$
• $m_1$ number of neurons in layer 1 and so $W^{\left[1\right]}\in R^{m_1\times d}$
• $m_2=1$ number of neurons in output layer and so $W^{\left[2\right]}\in R^{1\times m_1}$

Computing derivatives using Chain Rule using Backward strategy:

-(1) Compute $\frac{\partial J}{\partial W_i^{\left[2\right]}}$ then get vectorize version $\frac{\partial J}{\partial W^{\left[2\right]}}$

-(2) Compute $\frac{\partial J}{\partial W_{ij}^{\left[1\right]}}$ then get vectorize version $\frac{\partial J}{\partial W^{\left[1\right]}}$

-(3) Compute $\frac{\partial J}{\partial Z_i^{\left[1\right]}}$ then get vectorize version $\frac{\partial J}{\partial Z^{\left[1\right]}}$

-(4) Compute $\frac{\partial J}{\partial a_i^{\left[1\right]}}$ then get vectorize version $\frac{\partial J}{\partial a^{\left[1\right]}}$

Step 1:

$$\begin{array}{ccc}\frac{\partial J}{\partial W_i^{\left[2\right]}}& =& \frac{\partial J}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial W_i^{\left[2\right]}}\\ & =& (\hat{y}-y)\frac{\partial \hat{y}}{\partial W_i^{\left[2\right]}}\\ & =& (\hat{y}-y)a_i^{\left[1\right]}\end{array}$$ where $\hat{y}={\sum }_{i=1}^{m_1}{W}_i^{\left[2\right]}{a}_i^{\left[1\right]}+b^{\left[2\right]}$ $$\frac{\partial J}{\partial W^{\left[2\right]}}=(\hat{y}-y)a^{\left[1\right]T}\in R^{1\times m_1}$$

Similarly:

$$\frac{\partial J}{\partial b^{\left[2\right]}}=(\hat{y}-y)\in R$$

Step 2:

$$\begin{array}{ccc}\frac{\partial J}{\partial W_{ij}^{\left[1\right]}}& =& \frac{\partial J}{\partial z_i^{\left[1\right]}}\frac{\partial z_i^{\left[1\right]}}{\partial W_{ij}^{\left[1\right]}}\\ & =& \frac{\partial J}{\partial z_i^{\left[1\right]}}{x}_j\end{array}$$ where $z_i^{\left[1\right]}={\sum }_{k=d}^{m_1}{W}_{ik}^{\left[1\right]}{x}_k+b_i^{\left[1\right]}$

$$\frac{\partial J}{\partial W^{\left[1\right]}}=\frac{\partial J}{\partial z^{\left[1\right]}}{x}^T\in R^{m_1\times d}$$

Step 3:

$$\begin{array}{ccc}\frac{\partial J}{\partial z_i^{\left[1\right]}}& =& \frac{\partial J}{\partial a_i^{\left[1\right]}}\frac{\partial a_i^{\left[1\right]}}{\partial z_i^{\left[1\right]}}\\ & =& \frac{\partial J}{\partial a_i^{\left[1\right]}}{1}_{\{z_i^{\left[1\right]}\ge 0\}}\end{array}$$

$$\frac{\partial J}{\partial z^{\left[1\right]}}=\frac{\partial J}{\partial a^{\left[1\right]}}\odot {\sigma }^{{}^{\prime }}\left(z\right)$$ where ${\sigma }^{{}^{\prime }}(\cdot )$ is the element-wise derivative of the activation function $\sigma$ (here $ReLU$ function}) and $\odot$ denotes the element-wise product of two vectors of the same dimensionality.

Step 4:

$$\begin{array}{ccc}\frac{\partial J}{\partial a_i^{\left[1\right]}}& =& \frac{\partial J}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial a_i^{\left[1\right]}}\\ & =& (\hat{y}-y)w_i^{\left[2\right]}\end{array}$$

where $\hat{y}={\sum }_{i=1}^{m_1}{W}_i^{\left[2\right]}{a}_i^{\left[1\right]}+b^{\left[2\right]}$

$$\frac{\partial J}{\partial a^{\left[1\right]}}=(\hat{y}-y)W^{\left[2\right]T}$$

These steps enable us to define the Back-propagation algorithm for this simple Neural Network.

4.4.2 Back-propagation Algorithm

Copy link

Algorithm : Back-propagation for two-layer neural netwoks

1. Compute the values of $z^{\left[1\right]}$, $a^{\left[1\right]}$ and $\hat{y}$ using forward pass
2. Compute

$$\begin{array}{ccc}{\delta }^{\left[2\right]}& =& \frac{\partial J}{\partial \hat{y}}=(\hat{y}-y)\\ {\delta }^{\left[1\right]}& =& \frac{\partial J}{\partial Z^{\left[1\right]}}=\left(W^{\left[2\right]T}\right(\hat{y}-y\left)\right)\odot 1_{\{z^{\left[1\right]}\ge 0\}}\end{array}$$

3. Compute

$$\begin{array}{ccc}\frac{\partial J}{\partial W^{\left[2\right]}}& =& {\delta }^{\left[2\right]}{a}^{\left[1\right]T}\\ \frac{\partial J}{\partial b^{\left[2\right]}}& =& {\delta }^{\left[2\right]}\\ \frac{\partial J}{\partial W^{\left[1\right]}}& =& {\delta }^{\left[1\right]}{x}^T\\ \frac{\partial J}{\partial b^{\left[1\right]}}& =& {\delta }^{\left[1\right]}\end{array}$$

4.4.3 General Case with r layers

Copy link

Consider the general case of a fully-connected Multi-layer networks defining by the following equations:

$$\{\begin{array}{ccc}{a}^{\left[0\right]}& =& x\\ z^{\left[1\right]}& =& W^{\left[1\right]}{a}^{\left[0\right]}+b^{\left[1\right]}\\ a^{\left[1\right]}& =& ReLu\left(Z^{\left[1\right]}\right)\\ z^{\left[2\right]}& =& W^{\left[2\right]}{a}^{\left[1\right]}+b^{\left[2\right]}\\ a^{\left[2\right]}& =& ReLu\left(Z^{\left[2\right]}\right)\\ \dots & =& \dots \\ z^{[r-1]}& =& W^{[r-1]}{a}^{[r-2]}+b^{[r-1]}\\ a^{[r-1]}& =& ReLu\left(Z^{[r-1]}\right)\\ z^{\left[r\right]}& =& W^{\left[r\right]}{a}^{[r-1]}+b^{\left[r\right]}\\ \hat{y}=a^{\left[r\right]}& =& z^{\left[r\right]}\\ J& =& \frac{1}{2}(y-\hat{y}{)}^2\end{array}$$

4.4.4 Back-propagation multi-layer

Copy link

First, note that the updated parameter of interests (the weights and bias) depend of intermediate following intermediate variables: $$z^{\left[k\right]}=W^{\left[k\right]}{a}^{[k-1]}+b^{\left[k\right]},k\in \{1,\dots ,r\}$$

Then the cost function depends of weights and bias via the intermediate variables $z^{\left[k\right]}$. Using chain rule we get

$$\{\begin{array}{ccc}\frac{\partial J}{\partial W^{\left[k\right]}}& =& \frac{\partial J}{\partial z^{\left[k\right]}}{a}^{[k-1]T}\\ \frac{\partial J}{\partial b^{\left[k\right]}}& =& \frac{\partial J}{\partial z^{\left[k\right]}}\end{array}$$

Using similar notation as our simple NN, we define ${\delta }^{\left[k\right]}=\frac{\partial J}{\partial z^{\left[k\right]}}$ and compute it in a backward manner from $k=r$ to 1.

k=r:

$${\delta }^{\left[r\right]}=\frac{\partial J}{\partial z^{\left[r\right]}}=(z^{\left[r\right]}-y)$$

k<r:

$${\delta }^{\left[k\right]}=\frac{\partial J}{\partial z^{\left[k\right]}}=\frac{\partial J}{\partial a^{\left[k\right]}}\odot {\text{ReLU}}^{{}^{\prime }}\left(z^{\left[k\right]}\right)$$ By noting that $z^{[k+1]}=W^{[k+1]}{a}^{\left[k\right]}+b^{[k+1]}$ and assuming we have computed ${\delta }^{[k+1]}$ then we try to compute ${\delta }^{\left[k\right]}$. First note that