top of page
Writer's picturerayindaputrirp

LOGISTIC Regression vs. LINEAR Regression

In the realm of statistics and data analysis, regression analysis stands as a fundamental tool for making sense of complex relationships between variables. Two prominent members of this analytical family are linear regression and logistic regression. While they may share a common surname, their roles, abilities, and underlying principles couldn't be more distinct. Linear regression, like a reliable compass, guides us along a continuous path, predicting numerical outcomes. Logistic regression, on the other hand, serves as a binary navigator, steering us through the choppy waters of classification problems. In this article, we embark on a journey to unravel the stark differences between these two vital techniques, clarifying when to employ one over the other and ensuring you're well-equipped to navigate the twists and turns of data analysis.


The distinctive essence of logistic regression becomes evident when we delve into the realm of binary outcomes. While ordinary regression techniques are suitable for estimating continuous variables, logistic regression steps in when we need to gauge the impact of a variable on a binary dependent variable. This binary world, comprising outcomes like true or false, male or female, formal or informal sector, introduces a unique challenge.


Consider this scenario: we aim to unravel the relationship between the weekly number of hours of sleep and the probability of a child advancing to the next grade. The number of hours of sleep represents our explanatory variable, a continuous spectrum stretching from 0 to 168—imagine values like 15, 26.5, 47.3, 68, or 100. Conversely, our dependent variable exists in a binary universe, strictly adhering to the values of 0 or 1. According to this, our regression equation becomes:



Let me explain why this makes sense.


Suppose we are using a linear regression equation,

This situation involves the regression of a binary variable, which solely takes values of 0 or 1, against a continuous variable. However, the nature of this equation's right-hand side presents a considerable challenge. In the realm of continuous variables, values can span a virtually limitless range, extending beyond 1, dipping below 0, and occupying positions throughout the spectrum in between. The incongruity between these two domains highlights a fundamental issue—the conventional approach employed here fails to yield coherent results.


Then, if we use the following regression equation,

Another issue arises from this equation due to the nature of the natural logarithm function. When applied to the binary variable values, ln(0) results in negative infinity (-∞), while ln(1) equals 0. In contrast, the predictor variable remains unrestricted in its range and can potentially reach positive infinity (+∞). Therefore, this equation proves unviable for meaningful analysis and interpretation.


The correct equation is presented below,

Solving this equation, assuming zero error, leads to the following result:


Suppose β1 is a positive number. In this case, as hi approaches ∞, the probability p approaches 1, and as hi approaches -∞, p approaches 0. However, for different values of hi, p can take on values between 0 and 1. Thus, p is not restricted to only 0 or 1, but rather ranges from 0 to 1. This variation in p creates an S-shaped curve when plotted against hi, known as the logistic function curve.




The curve above illustrates a significant difference between logistic regression and linear regression. In linear regression, we determine the best-fit line using the least squares method, aiming to minimize the sum of the squares of the residuals and subsequently calculating R-squared. In contrast, logistic regression does not employ R-squared but relies on the "maximum likelihood" approach. Additionally, it reveals that as students accumulate more hours of sleep, their likelihood of promotion increases, demonstrating the utility of logistic regression for analyzing binary outcomes.



18 views0 comments

Recent Posts

See All

Comments


bottom of page