2 Logistic Regression Type Neural Networks

2.1.1 Sigmoid function

Copy link

The sigmoid function σ(⋅)$\sigma (\cdot )$, also known as the logistic function, is defined as follows:

∀z∈R,σ(z)=11+e−z∈]0,1[$$\forall z\in R,\sigma \left(z\right)=\frac{1}{1+e^{-z}}\in ]0,1[$$

Figure 2.1: Sigmoid function

z <- seq(-5, 5, 0.01)
sigma = 1 / (1 + exp(-z))

plot(sigma~z,type="l",ylab=expression(sigma(z)))

2.1.2 Logistic regression

Copy link

The logistic regression is a probabilistic model that aims to predict the probability that the outcome variable y$y$ is 1. It is defined by assuming that y|x;θ∼Bernoulli(ϕ)$y|x;\theta \sim \text{Bernoulli}\left(\varphi \right)$. Then, the logistic regression is defined by applying the soft sigmoid function to the linear predictor θTx${\theta }^{T}x$:

ϕ=hθ(x)=p(y=1|x;θ)=11+exp(−θTx)=σ(θTx)$$\varphi =h_{\theta }\left(x\right)=p(y=1|x;\theta )=\frac{1}{1+\exp (-{\theta }^{T}x)}=\sigma \left({\theta }^{T}x\right)$$

The logistic regression is also presented:

Logit[hθ(x)]=logit[p(y=1|x;θ)]=θTx$$\text{Logit}\left[h_{\theta }\right(x\left)\right]=logit\left[p\right(y=1|x;\theta \left)\right]={\theta }^{T}x$$ where Logit(p)=log(p1−p)$\text{Logit}\left(p\right)=log\left(\frac{p}{1-p}\right)$.

Remark about notation

• x=(x0,…,xd)T$x=(x_0,\dots ,x_d{)}^T$ represent a vector of d+1$d+1$ features/predictors and by convention x0=1$x_0=1$
• θ=(θ0,…,θd)T$\theta =({\theta }_0,\dots ,{\theta }_d{)}^T$ is the vector of parameter related to the features x$x$
• θ0${\theta }_0$ is called the intercept by the statistician and named bias by the computer scientist (noted generally b$b$)

2.1.3 Logistic regression for classification

Copy link

Let’s play with a simple example with 10 points, and two classes (red and blue)

Figure 2.2: Classify red and blue points

clr1 <- c(rgb(1,0,0,1),rgb(0,0,1,1))
clr2 <- c(rgb(1,0,0,.2),rgb(0,0,1,.2))
x <- c(.4,.55,.65,.9,.1,.35,.5,.15,.2,.85)
y <- c(.85,.95,.8,.87,.5,.55,.5,.2,.1,.3)
z <- c(1,1,1,1,1,0,0,1,0,0)
df <- data.frame(x,y,z)
plot(x,y,pch=19,cex=2,col=clr1[z+1])

In order to classify the points, we run a logistic regression to get predictions

model <- glm(z~x+y,data=df,family=binomial)
#summary(model)

Then, we use the fitted model to define our classifier which is defined as attributed the class that is the most likely.

pred_model <- function(x,y){
 predict(model,newdata=data.frame(x=x,
 y=y),type="response")>.5
}

Using our decision rule, we can visualise the produced partition of the space.

Figure 2.3: Partition using the logistic model

x_grid<-seq(0,1,length=101)
y_grid<-seq(0,1,length=101)
z_grid <- outer(x_grid,y_grid,pred_model)
image(x_grid,y_grid,z_grid,col=clr2)
points(x,y,pch=19,cex=2,col=clr1[z+1])

2.1.4 Likelihood of the logistic model

Copy link

The maximum likelihood estimation procedure is generally used to estimate the parameters of the models θ0,…,θd${\theta }_0,\dots ,{\theta }_d$.

p(y|x;θ)={hθ(x)if y=1, and1−hθ(x)otherwise.$$p(y|x;\theta )=\{\begin{array}{cc}{h}_{\theta }\left(x\right)& \text{if}y=1,\text{and}\\ 1-h_{\theta }\left(x\right)& \text{otherwise}.\end{array}$$ which could be written as

p(y|x;θ)=hθ(x)y(1−hθ(x))1−y,$$p(y|x;\theta )=h_{\theta }\left(x{)}^y\right(1-h_{\theta }\left(x\right){)}^{1-y},$$

Consider now the observation of m$m$ training samples denoted by {(x(1),y(1)),…,(x(m),y(m))}$\left\{(x^{\left(1\right)},y^{\left(1\right)}),\dots ,(x^{\left(m\right)},y^{\left(m\right)})\right\}$ as i.i.d. observations from the logistic model. The likelihood is

L(θ)=m∏i=1p(y(i)|x(i);θ)=m∏i=1hθ(x(i))y(1−hθ(x(i)))1−y$$\begin{array}{ccc}L\left(\theta \right)& =& \prod _{i=1}^{m}p(y^{\left(i\right)}|x^{\left(i\right)};\theta )\\ & =& \prod _{i=1}^m{h}_{\theta }\left(x^{\left(i\right)}{)}^y\right(1-h_{\theta }\left(x^{\left(i\right)}\right){)}^{1-y}\end{array}$$

Then, the following log likelihood is maximized to the estimates of θ$\theta$:

ℓ(θ)=log L(θ)=m∑i=1[y(i)loghθ(x(i))+(1−y(i))log(1−hθ(x(i)))]$$\ell \left(\theta \right)=\text{log}L\left(\theta \right)=\sum _{i=1}^m\left[y^{\left(i\right)}\log h_{\theta }\left(x^{\left(i\right)}\right)+(1-y^{\left(i\right)})\log (1-h_{\theta }(x^{\left(i\right)}\left)\right)\right]$$

2.1.5 Shallow Neural Network

Copy link

The logistic model can ve viewed as a shallow Neural Network.

This figure used here the same notation as the regression logistic model presented by the statistical point of view. However, in the following we will adopt the notation used the most frequently in deep learning framework.

Figure 2.4: Shallow Neural Network

In this figure, z=wTz+b=w1x1+…+wdxd+b$z=w^{T}z+b=w_1{x}_1+\dots +w_d{x}_d+b$ is the linear combination of the d$d$ features/predictors and a=σ(Z)$a=\sigma \left(Z\right)$ is called the activation function which is the non-linear part of the Neural Network to get a close prediction ˆy≈y$\hat{y}\approx y$.

Remark: In the sequel, we will adopt the following notation:

• x=(x1,…,xd)T∈ℜd$x=(x_1,\dots ,x_d{)}^T\in R^d$ representz a vector of d$d$ features/predictors
• w=(w1,…,wd)T$w=(w_1,\dots ,w_d{)}^T$ is the vector of weight related to the features x$x$
• b$b$ is called the biais
• We consider the observations of m$m$ training samples denoted by {(x(1),y(1)),…,(x(m),y(m))}$\left\{(x^{\left(1\right)},y^{\left(1\right)}),\dots ,(x^{\left(m\right)},y^{\left(m\right)})\right\}$

2.1.6 Entropy, Cross-entropy and Kullback-Leibler

Copy link

Let’s first talk about the cross-entropy which is widely used as loss function for classification purpose.

Cross Entropy (CE) is related to the entropy and the kullback-Leibler risk.

The entropy of a discrete probability distribution p=(p1,…,pn)$p=(p_1,\dots ,p_n)$ is defined as

H(p)=H(p1,…,pn)=−n∑i=1pilogpi$$H\left(p\right)=H(p_1,\dots ,p_n)=-\sum _{i=1}^n{p}_i\log p_i$$ which is a ``measurement of the disorder or randomness of a system’’.

Kullback and Leibler known also as KL divergence quantifies how similar a probability distribution p$p$ is to a candidate distribution q$q$.

KL(p;q)=−n∑i=1pilogpiqi$$KL(p;q)=-\sum _{i=1}^n{p}_i\log \frac{p_i}{q_i}$$ Note that the KL$KL$ divergence is not a distance measure as KL(p;q)≠KL(q;p)$KL(p;q)\ne KL(q;p)$. KL$KL$ is non-negative and zero if and only if pi=qi$p_i=q_i$ for all i$i$.

One can easily show that

KL(p;q)=n∑i=1pilog1qi⏟cross entropy−H(p)$$KL(p;q)=\underset{\text{cross entropy}}{\underset{}{\sum _{i=1}^n{p}_i\log \frac{1}{q_i}}}-H\left(p\right)$$

where the first term of the right part is the cross entropy:

CE(p,q)=n∑i=1pilog1qi=−n∑i=1pilogqi$$CE(p,q)=\sum _{i=1}^n{p}_i\log \frac{1}{q_i}=-\sum _{i=1}^n{p}_i\log q_i$$ And we have the relation CE(p,q)=H(p)+KL(p;q)$$CE(p,q)=H\left(p\right)+KL(p;q)$$

Thus, the cross entropy can be interpreted as the uncertainty implicit in H(p)$H\left(p\right)$ plus the likelihood that the distribution p$p$ could have be generated by the distribution q$q$.

2.1.7 Mathematical expression of the Neural Network:

Copy link

For one example x(i)$x^{\left(i\right)}$, the ouput of this Neural Network is given by:

ˆy(i)=σ(wTx(i)+b)⏟a(i)⏟activation function,$${\hat{y}}^{\left(i\right)}=\underset{\underset{\text{activation function}}{\underset{}{a^{\left(i\right)}}}}{\underset{}{\sigma (w^T{x}^{\left(i\right)}+b)}},$$

where σ(⋅)$\sigma (\cdot )$ is the sigmoid function.

We aim to get the weight w$w$ and the biais b$b$ such that ˆy(i)≈y(i)${\hat{y}}^{\left(i\right)}\approx y^{\left(i\right)}$.

The loss function used for this network is the cross-entropy which is defined for one sample (x,y)$(x,y)$:

L(ˆy,y)=−(ylogˆy+(1−y)log(1−ˆy))$$L(\hat{y},y)=-(y\log \hat{y}+(1-y\left)\log \right(1-\hat{y}\left)\right)$$

Then the Cost function for the entire training data set is:

J(w,b)=1mm∑i=1L(ˆy(i),y(i))$$J(w,b)=\frac{1}{m}\sum _{i=1}^{m}L({\hat{y}}^{\left(i\right)},y^{\left(i\right)})$$

Further, it is easy to see the connection with the log-likelihood function of the logistic model:

J(w,b)=1mm∑i=1L(ˆy(i),y(i))=−1mm∑i=1[y(i)loga(i))+(1−y(i))log(1−a(i))]=−1mm∑i=1[y(i)loghθ(x(i))+(1−y(i))log(1−hθ(x(i)))]≡−1mℓ(θ)$$\begin{array}{ccc}J(w,b)& =& \frac{1}{m}\sum _{i=1}^{m}L({\hat{y}}^{\left(i\right)},y^{\left(i\right)})\\ & =& -\frac{1}{m}\sum _{i=1}^m\left[y^{\left(i\right)}\log a^{\left(i\right)})+(1-y^{\left(i\right)})\log (1-a^{\left(i\right)})\right]\\ & =& -\frac{1}{m}\sum _{i=1}^m\left[y^{\left(i\right)}\log h_{\theta }\left(x^{\left(i\right)}\right)+(1-y^{\left(i\right)})\log (1-h_{\theta }(x^{\left(i\right)}\left)\right)\right]\\ & \equiv & -\frac{1}{m}\ell \left(\theta \right)\end{array}$$

where b≡θ0$b\equiv {\theta }_0$ and w≡(θ1,…,θd)$w\equiv ({\theta }_1,\dots ,{\theta }_d)$.

The optimization step will be carried using Gradient Descent procedures and extension which will be briefly presented in the sub-section Optimization

2.2.1 Multinomial regression for classification

Copy link

We illustrate the Multinomial model by considering three classes: red, yellow and blue.

Figure 2.5: Classify for three color points

clr1 <- c(rgb(1,0,0,1),rgb(1,1,0,1),rgb(0,0,1,1))
clr2 <- c(rgb(1,0,0,.2),rgb(1,1,0,.2),rgb(0,0,1,.2))
x <- c(.4,.55,.65,.9,.1,.35,.5,.15,.2,.85)
y <- c(.85,.95,.8,.87,.5,.55,.5,.2,.1,.3)
z <- c(1,2,2,2,1,0,0,1,0,0)
df <- data.frame(x,y,z)
plot(x,y,pch=19,cex=2,col=clr1[z+1])

One can use the R package to run a mutinomial regression model

library(nnet)
model.mult <- multinom(z~x+y,data=df)

# weights:  12 (6 variable)
initial  value 10.986123 
iter  10 value 0.794930
iter  20 value 0.065712
iter  30 value 0.064409
iter  40 value 0.061612
iter  50 value 0.058756
iter  60 value 0.056225
iter  70 value 0.055332
iter  80 value 0.052887
iter  90 value 0.050644
iter 100 value 0.048117
final  value 0.048117 
stopped after 100 iterations

Then, the output gives a predicted probability to the three colours and we attribute the color that is the most likely.

pred_mult <- function(x,y){
res <- predict(model.mult,
newdata=data.frame(x=x,y=y),type="probs")
apply(res,MARGIN=1,which.max)
}

x_grid<-seq(0,1,length=101)
y_grid<-seq(0,1,length=101)
z_grid <- outer(x_grid,y_grid,FUN=pred_mult)

We can now visualize the three regions, the frontier being linear, and the intersection being the equiprobable case.

Figure 2.6: Classifier using multinomial model

image(x_grid,y_grid,z_grid,col=clr2)
points(x,y,pch=19,cex=2,col=clr1[z+1])

2.2.2 Likelihood of the softmax model

Copy link

The maximum likelihood estimation procedure consists of maximizing the log-likelihood:

ℓ(θ)=m∑i=1logp(y(i)|x(i);θ)=m∑i=1logK∏l=1(eθTlx(i)∑Kj=1eθTjx(i))1{y(i)=l}$$\begin{array}{ccc}\ell \left(\theta \right)& =& \sum _{i=1}^m\log p\left(y^{\left(i\right)|x^{\left(i\right)};\theta }\right)\\ & =& \sum _{i=1}^m\log \prod _{l=1}^K{\left(\frac{e^{{\theta }_l^T{x}^{\left(i\right)}}}{{\sum }_{j=1}^K{e}^{{\theta }_j^T{x}^{\left(i\right)}}}\right)}^{1_{\{y^{\left(i\right)}=l\}}}\end{array}$$

2.2.3 Softmax regression as shallow Neural Network

Copy link

The Softmax regression model can be viewed as a shallow Neural Network.

In this representation, there are K$K$ neurons where each neuron is defined by his own set of weights wi∈ℜd$w_i\in R^d$ and a bais term bi$b_i$. The linear part is denoted by zi=wTix+bi$z_i=w_i^{T}x+b_i$ and the non linear part (activation part) is σi=ai=exp(zi)K∑j=1exp(zj)${\sigma }_i=a_i=\frac{\exp \left(z_i\right)}{\sum _{j=1}^K\exp \left(z_j\right)}$. Note that the denominator of the activation part is defined using the weights from the other neurons. The output is a vector of probabilities (a1,…,aK)$(a_1,\dots ,a_K)$ and the function is used for classification purpose:

ˆy=argmax  aii∈{1,…,K}

2.2.4 Loss function: cross-entropy for categorical variable

Copy link

Let consider first one training sample (x,y). The cross entropy loss for categorical response variable, also called Softmax Loss is defined as:

CE=−K∑i=1˜yilogp(y=i)=−K∑i=1˜yilogai=−K∑i=1˜yilog(exp(zi)K∑j=1exp(zj)) where ˜yi=1{y=i} is a binary variable indicating if y is in the class i.

This expression can be rewritten as

CE=−logK∏i=1(exp(zi)K∑j=1exp(zj))1{y=i}

Then, the cost function for the m training samples is defined as

J(w,b)=−1mm∑i=1logK∏k=1(exp(z(i)k)K∑j=1exp(z(i)j))1{y(i)=k}≡−1mℓ(θ)

2.3.1 Gradient Descent

Copy link

Consider unconstrained, smooth convex optimization

minθ∈ℜd f(θ),

Algorithm : Gradient Descent

1. Choose initial point θ(0)∈Rd
2. Repeat θ(k+1)=θ(k)−α.∇f(θ(k)),  k=1,2,3,…
3. Stop at some point: ||θ(k+1)−θ(k)||22<ϵ

Here, α is called the learning rate.

Suppose that we want to find x that minimizes: f(x)=1.2(x−2)2+3.2

Figure 2.7: Closed from solution (red)

f.x <- function(x) 1.2*(x-2)**2+3.2
curve(1.2*(x-2)**2+3.2,0,4,ylab="fx)")
abline(v=2,col="red")

In general, we cannot find a closed form solution, but can compute ∇f(x)

simple.grad.des <- function(x0,alpha,epsilon=0.00001,max.iter=300){
  tol <- 1; xold <- x0; res <- x0; iter <- 1
  while (tol>epsilon & iter < max.iter){
    xnew <- xold - alpha*2.4*(xold-2)
    tol <- abs(xnew-xold)
    xold <- xnew
  res <- c(res,xnew)
  iter <- iter +1
  }
 return(res)
}
result <- simple.grad.des(0,0.01,max.iter=200)
#result[length(result)]

Convergence with a learning rate=0.01

Figure 2.8: alpha=0.01

curve(1.2*(x-2)**2+3.2,0,4,ylab="fx)")
abline(v=2,col="red")
points(result,f.x(result),col="blue")

Convergence with a learning rate=0.83

Figure 2.9: alpha=0.83

result2 <- simple.grad.des(0,0.83,max.iter=200)
#result2[length(result2)]
curve(1.2*(x-2)**2+3.2,0,4,ylab="fx)")
abline(v=2,col="red")
points(result2,f.x(result2),col="blue",type="o")

2.3.2 Gradient Descent for logistic regression

Copy link

Given (x(i),y(i))∈ℜ×{0,1} for i=1,…,m, consider the cross-entropy loss function for this data set:

f(w)=1mm∑i=1(−y(i)wTx(i)+log(1+exp(wTx(i))))=m∑i=1fi(w)

The gradient is

∇f(w)=1mm∑i=1(p(i)(w)−y(i))x(i)

where p(i)(w))=p(Y=1|x(i),w)=exp(wTx(i))/(1+exp(wTx(i))),   i=1,…,m

Algorithm : Batch Gradient Descent

1. Initialize w=(0,…,0)
2. Repeat until convergence
   • Let g=(0,…,0) be the gradient vector
   • for i=1:m do p(i)=exp(wTx(i))/(1+exp(wTx(i))) error(i)=p(i)−yi g=g+error(i).w(i)
   • end
   w=w−α.g/m
3. End repeat until convergence

Note that algorithm uses all samples to compute the gradient. This approach is called batch gradient descent.

2.3.3 Stochastic gradient descent

Copy link

Algorithm : Stochastic Gradient Descent

1. Initialize w=(0,…,0)

2. Repeat until convergence

   Pick sample i

   p(i)=exp(wTx(i))/(1+exp(wTx(i)))

   error(i)=p(i)−yi

   w=w−α(error(i)×x(i))

3. End repeat until convergence

Coefficient w is updated after each sample.

Remark: The gradient computation ∇f(w)=∑mi=1(p(i)(w)−y(i))x(i) is doable when m is moderate, but not when m∼500million.

• One batch update costs O(md)

• One stochastic update costs O(d)

2.3.4 Mini-Batches

Copy link

In practice, mini-batch is often used to:

1. Compute gradient based on a small subset of samples
2. Make update to coefficient vector

2.3.5 Example with logistic regression d=2

Copy link

• Simulate some m samples from true model:

p(Y=1|x(i),w)=11+exp(−w1x(i)1−w2x(i))

set.seed(10)
m <- 5000 ;d <- 2 ;w <- c(0.5,-1.5)
x <- matrix(rnorm(m*2),ncol=2,nrow=m)
ptrue <- 1/(1+exp(-x%*%matrix(w,ncol=1)))
y <- rbinom(m,size=1,prob = ptrue)
(w.est <- coef(glm(y~x[,1]+x[,2]-1,family=binomial)))

##    x[, 1]    x[, 2] 
##  0.557587 -1.569509

• The cross-entropy loss for this dataset

Cost.fct <- function(w1,w2) {
  w <- c(w1,w2)
  cost <- sum(-y*x%*%matrix(w,ncol=1)+log(1+exp(x%*%matrix(w,ncol=1))))
  return(cost)
}

Figure 2.10: Contour plot of the Cost function

w1 <- seq(0, 1, 0.05)
w2 <- seq(-2, -1, 0.05)
cost <- outer(w1, w2, function(x,y) mapply(Cost.fct,x,y))
contour(x = w1, y = w2, z = cost)
points(x=w.est[1],y=w.est[2],col="black",lwd=2,lty=2,pch=8)

• Implementation of Batch Gradient Descent

sigmoid <- function(x) 1/(1+exp(-x))

batch.GD <- function(theta,alpha,epsilon,iter.max=500){
  tol <- 1
  iter <-1
  res.cost <- Cost.fct(theta[1],theta[2])
  res.theta <- theta
  while (tol > epsilon & iter<iter.max) {
      error <- sigmoid(x%*%matrix(theta,ncol=1))-y
      theta.up <- theta-as.vector(alpha*matrix(error,nrow=1)%*%x)
      res.theta <- cbind(res.theta,theta.up)
      tol <- sum((theta-theta.up)**2)^0.5
      theta <- theta.up
      cost <- Cost.fct(theta[1],theta[2])
      res.cost <- c(res.cost,cost)
      iter <- iter +1
    }
  result <- list(theta=theta,res.theta=res.theta,res.cost=res.cost,iter=iter,tol.theta=tol)
  
  return(result)
}

dim(x);length(y)

## [1] 5000    2

## [1] 5000

Figure 2.11: Convergence Batch Gradient Descent

theta0 <- c(0,-1); alpha=0.001
test <- batch.GD(theta=theta0,alpha,epsilon = 0.0000001)
plot(test$res.cost,ylab="cost function",xlab="iteration",main="alpha=0.01",type="l")
abline(h=Cost.fct(w.est[1],w.est[2]),col="red")

Figure 2.12: Convergence of BGD

contour(x = w1, y = w2, z = cost)
points(x=w.est[1],y=w.est[2],col="black",lwd=2,lty=2,pch=8)
record <- as.data.frame(t(test$res.theta))
points(record,col="red",type="o")

• Implementation of Stochastic Gradient Descent

Stochastic.GD <- function(theta,alpha,epsilon=0.0001,epoch=50){
  epoch.max <- epoch
  tol <- 1
  epoch <-1
  res.cost <- Cost.fct(theta[1],theta[2])
  res.cost.outer <- res.cost
  res.theta <- theta
  while (tol > epsilon & epoch<epoch.max) {
    for (i in 1:nrow(x)){
      errori <- sigmoid(sum(x[i,]*theta))-y[i]
      xi <- x[i,]
      theta.up <- theta-alpha*errori*xi
      res.theta <- cbind(res.theta,theta.up)
      tol <- sum((theta-theta.up)**2)^0.5
      theta <- theta.up
      cost <- Cost.fct(theta[1],theta[2])
      res.cost <- c(res.cost,cost)
    }
    epoch <- epoch +1
    cost.outer <- Cost.fct(theta[1],theta[2])
    res.cost.outer <- c(res.cost.outer,cost.outer)
    }
  result <- list(theta=theta,res.theta=res.theta,res.cost=res.cost,epoch=epoch,tol.theta=tol)
}

test.SGD <- Stochastic.GD(theta=theta0,alpha,epsilon = 0.0001,epoch=10)

Figure 2.13: Convergence Stochastic Gradient Descent

plot(test.SGD$res.cost,ylab="cost function",xlab="iteration",main="alpha=0.01",type="l")
abline(h=Cost.fct(w.est[1],w.est[2]),col="red")

Figure 2.14: Convergence of Stochastic Gradient Descent

contour(x = w1, y = w2, z = cost)
points(x=w.est[1],y=w.est[2],col="black",lwd=2,lty=2,pch=8)
record2 <- as.data.frame(t(test.SGD$res.theta))
points(record2,col="red",lwd=0.5)

• Implementation of mini batch Gradient Descent

Mini.Batch <- function (theta,dataTrain, alpha = 0.1, maxIter = 10, nBatch = 2, seed = NULL,intercept=NULL) 
{
    batchRate <- 1/nBatch
    dataTrain <- matrix(unlist(dataTrain), ncol = ncol(dataTrain), byrow = FALSE)
    set.seed(seed)
    dataTrain <- dataTrain[sample(nrow(dataTrain)), ]
    set.seed(NULL)
    
    res.cost <- Cost.fct(theta[1],theta[2])
    res.cost.outer <- res.cost
    res.theta <- theta
    
    if(!is.null(intercept)) dataTrain <- cbind(1, dataTrain)
    
    temporaryTheta <- matrix(ncol = length(theta), nrow = 1)
    theta <- matrix(theta,ncol = length(theta), nrow = 1)
    
    for (iteration in 1:maxIter ) {
        if (iteration%%nBatch == 1 | nBatch == 1) {
            temp <- 1
            x <- nrow(dataTrain) * batchRate
            temp2 <- x
        }
        batch <- dataTrain[temp:temp2, ]
        inputData <- batch[, 1:ncol(batch) - 1]
        outputData <- batch[, ncol(batch)]
        rowLength <- nrow(batch)
        temp <- temp + x
        temp2 <- temp2 + x
        error <- matrix(sigmoid(inputData %*% t(theta)),ncol=1) - outputData
        for (column in 1:length(theta)) {
            term <- error * inputData[, column]
            gradient <- sum(term)#/rowLength
            temporaryTheta[1, column] = theta[1, column] - (alpha * 
                gradient)
        }
        
        theta <- temporaryTheta
        res.theta <- cbind(res.theta,as.vector(theta))
        cost.outer <- Cost.fct(theta[1,1],theta[1,2])
        res.cost.outer <- c(res.cost.outer,cost.outer)
        
    }
    result <- list(theta=theta,res.theta=res.theta,res.cost.outer=res.cost.outer)
    return(result)
}

theta0 <- c(0,-1); alpha=0.001
data.Train <- cbind(x,y)
test.miniBatch <- Mini.Batch(theta=theta0,dataTrain=data.Train, alpha = 0.001, maxIter = 100, nBatch = 10, seed = NULL,intercept=NULL)
##Result frm Mini-Batch
test.miniBatch$theta

##           [,1]      [,2]
## [1,] 0.5515469 -1.561252

Figure 2.15: Convergence Mini Batch

plot(test.miniBatch$res.cost.outer,ylab="cost function",xlab="iteration",main="alpha=0.001",type="l")
abline(h=Cost.fct(w.est[1],w.est[2]),col="red")

Figure 2.16: Convergence of Stochastic Gradient Descent

contour(x = w1, y = w2, z = cost)
points(x=w.est[1],y=w.est[2],col="black",lwd=2,lty=2,pch=8)
record3 <- as.data.frame(t(test.miniBatch$res.theta))
points(record3,col="red",lwd=0.5,type="o")

2.4.1 Univariate Chain rule

Copy link

• Univariate chain rule

∂f(g(w))∂w=∂f(g(w))∂g(w).∂g(w)∂w

2.4.2 Multivariate Chain Rule

Copy link

• Part I: Let z=f(x,y), x=g(t) and y=h(t), where f,g and h are differentiable functions. Then z=f(x,y)=f(g(t),h(t))) is a function of t, and

dzdt=dfdt=fx(x,y)dxdt+fy(x,y)dydt=∂f∂xdxdt+∂f∂ydydt.

• Part II:

1. Let z=f(x,y), x=g(s,t) and y=h(s,t), where f,g and h are differentiable functions. Then z is a function of s and t, and

∂z∂s=∂f∂x∂x∂s+∂f∂y∂y∂s

and

∂z∂t=∂f∂x∂x∂t+∂f∂y∂y∂t

2. Let z=f(x1,x2,…,xm) be a differentiable function of m variables, where each of the xi is a differentiable function of the variables t1,t2,…,tn. Then z is a function of the ti, and

∂z∂ti=∂f∂x1∂x1∂ti+∂f∂x2∂x2∂ti+⋯+∂f∂xm∂xm∂ti.