Logistic Regression

Logistic Regression is used for classification problems which is used to predict the dependent variable which is categorical in nature.

It is considered as a part of supervised learning algorithms where we are making use of the actual values of dependent variable to calculate the coefficients for the model.

In logistic regression, instead of modeling the outcome variable y, we predict the probabilities that y belong to a particular category.

To learn about how to implement Logistic Regression in Python you can read this article by us: Logistic Regression in Python

Sigmoid Function

Logistic Regression makes use of the sigmoid function p(x) given by:

which leads to an S-shaped sigmoid curve as shown in the figure below:

Here p(x) takes the values between 0 and 1. If x is highly negative, then p(x) is 0, while it is 1in case of large values of x. However, if x = 0, then p(x) = 0.5.

In other words, p(x) is 0 for large negative x and then it keeps on increasing as x keeps on increasing, passing through 1/2 when x = 0, becoming close to 1 as x becomes large.

Logistic Regression Equation

In case of a single variable X, the logistic regression equation is:

Where β0 and β1 are logistic regression coefficients, X is the independent variable and p(x) is the probability given by the equation.

Likelihood Function

To solve the logistic regression equation, method of maximum likelihood is used, where we try to find βˆ 0 and βˆ 1 so that the probabilities are close to the actual outcome. These estimates are obtained by maximising the likelihood function

By taking log, we get the log-likelihood function:

The log-likelihood, defined in equation 2.4 can be considered as the sum of the log-probabilities. If the output for an instance is 1, we need to add log(p(xi)) to this sum. When an output is 0, we will add log(1 − p(xi)).

Making predictions with Logistic Regression Equation

Let us say we wanted to predict whether a person should be on the basis of his income. Thus, our dependent variable would be: 1 means grant the loan, 0 means do not grant the loan.

Our logistic regression equation would look like:

y = β0 + β1 X Income

Suppose β0 = -20, β1 = 0.6 and income for a person is 50

then p(x) = exp( -20 + 0.6 * 50) / ( 1 + exp( -20 + 0.6 * 50) )

p(x) = 10 / (1+10) = 0.9091

Thus, the probability the person should be given a loan is 0.91

Now, if p(x) > 0.5 then we should predict as 1, otherwise if p(x) < 0.5 then prediction is 0.

Odds Ratio

To understand odds ratio, let us understand what are odds?

Odds is defined as the ratio of the probability of occurence of an event to the probability of non-occurence of an event

i.e., Odds =

Let us say ,

probability of a person having cancer = 0.2 when he does not smoke, thus odds of having cancer for a non-smoker = 0.2 / (1-0.2) = 0.25

Probability of a person having cancer when he smokes = 0.7, thus odds of having cancer for a smoker = 0.7 / (1-0.7) = 2.33

Thus, odds ratio of smoking = Odds of having cancer when a person smokes / Odds of having cancer when a person does not smoke

i.e., extra chances of having cancer, when a person smokes is given by : 2.33 / 0.25 = 9.33

In logistic regression, odds ratio is defined as the extra chances of P(Y = 1) when one variable (say X1) increases by one unit. provided all other variables in the model are held constant.

Interpretation of Odds Ratio:

Odds ratio lies between 0 and 1. If odds ratio is < 1 then it depicts the chances of occurence of Y=1 are low as X increased. However, if odds ratio is > 1 then it depicts the extra chances of occurence of Y=1 as X increased by 1 unit.

Let us say odds ratio > 1 (say 1.15) then there are 15% more chances of occurence of Y=1 if X increases.

Similarly if odds ratio < 1 (say 0.85) then there are (100-85)% = 15% less chances of occurence of Y = 1 if X increases.

How Odds Ratio is calculated for a logistic regression model:

Earlier we learnt that odds are defined as the ratio of the probability of occurence of an event to the probability of non-occurence of an event i.e., Odds =