Week 4 – Multiple Regression and Understanding Relationships in Data with Python

In Week 3, we introduced OLS regression and learned how to run our first model.

We looked at how one variable can help explain another.

Now we move beyond simple regression.

In real-world analysis, outcomes are rarely explained by only one factor.

For example:

Does salary depend only on experience?

Or does it also depend on:

• education program
• location
• industry
• other factors?

This is where multiple regression becomes useful.

📑 In This Article

• What is multiple regression?
• Simple vs multiple regression
• Adding multiple predictors
• Understanding categorical variables
• Creating dummy variables
• Interpreting regression output
• Avoiding common mistakes
• Key takeaways

What is Multiple Regression?

Multiple regression allows us to analyze the relationship between:

• one dependent variable
• multiple independent variables

Instead of asking:

Does experience affect salary?

We can ask:

Does experience, education and other factors help explain salary?

Example:

Salary = Experience + Program + Location

Each variable contributes information to the model.

Simple Regression vs Multiple Regression

A simple regression uses one predictor.

Example:

Salary = Experience

Question:

Does experience help explain salary?

A multiple regression uses several predictors.

Example:

Salary = Experience + Education + Age

Question:

Do several factors together help explain salary?

Multiple regression gives us a more realistic view because real-world outcomes are usually influenced by multiple factors.

However, both simple and multiple regression use statistical tests to determine whether relationships are meaningful.

A regression model commonly uses:

• t-tests to evaluate individual variables
• F-tests to evaluate the model overall

This allows us to answer two different questions:

1. Does this specific variable matter?
2. Does the model overall explain the outcome?

Running a Multiple Regression Model

Before running regression, we need:

• pandas to load and prepare data
• statsmodels to build the model

import pandas as pd
import statsmodels.api as sm

df = pd.read_csv('sample_data.csv')

X = df[['Experience', 'Age']]
y = df['Salary']

X = sm.add_constant(X)

model = sm.OLS(y, X).fit()

print(model.summary())

Understanding Multiple Coefficients

When you run the model, each variable gets its own coefficient.

Example:

Variable	Coefficient
Experience	5000
Age	1000

The interpretation:

Experience:

For every additional unit of experience, salary increases by approximately 5000 units, holding other variables constant.

Age:

For every additional year of age, salary increases by approximately 1000 units, holding other variables constant.

The phrase:

“holding other variables constant”

is very important.

It means we are looking at one relationship while controlling for other factors.

Working with Categorical Variables

Many datasets contain text categories.

Examples:

Program

Computer Science
Business
Engineering

Regression models cannot directly use text.

A model understands numbers, not words.

So we need to convert categories into numeric values.

This is called creating:

👉 dummy variables

Creating Dummy Variables with pandas

We can use pandas to convert categories into numbers.

import pandas as pd

df = pd.read_csv('sample_data.csv')

df_encoded = pd.get_dummies(
    df,
    columns=['Program'],
    drop_first=True
)

print(df_encoded.head())

Before:

Program
Computer Science
Business
Engineering

After:

Program_Business	Program_Engineering
0	0
1	0
0	1

Now the regression model can use these values.

Why Use drop_first=True?

When creating dummy variables, one category is removed.

Example:

Three programs:

• Computer Science
• Business
• Engineering

We only need two columns:

• Business
• Engineering

The remaining category becomes the reference group.

This avoids creating redundant information.

Running Regression with Categorical Data

Now we can include our encoded variables.

import pandas as pd
import statsmodels.api as sm

df = pd.read_csv('sample_data.csv')

df_encoded = pd.get_dummies(
    df,
    columns=['Program'],
    drop_first=True
)

X = df_encoded[
    [
        'Experience',
        'Program_Business',
        'Program_Engineering'
    ]
]

y = df_encoded['Salary']

X = sm.add_constant(X)

model = sm.OLS(y, X).fit()

print(model.summary())

Interpreting Dummy Variable Results

Suppose:

Program_Engineering coefficient = 8000

This means:

Engineering students have an average salary 8000 higher than the reference program, while controlling for other variables.

The reference group is the category that was removed.

Understanding Model Quality

Regression output contains several useful measurements.

R-squared

Measures how much variation in the outcome is explained by the model.

Example:

R-squared = 0.70

Meaning:

The model explains about 70% of the variation in salary.

Adjusted R-squared

Adjusted R-squared considers the number of variables in the model.

Useful when comparing models with different numbers of predictors.

F-statistic

Tests whether the model as a whole provides useful information.

A significant result suggests:

At least one predictor contributes to explaining the outcome.

Common Regression Mistakes

Mistake 1: Using unclean data

Regression depends on good input data.

Always:

• clean values
• handle missing data
• Check data types

Mistake 2: Treating categories as numbers

Incorrect:

Computer Science = 1
Business = 2
Engineering = 3

The numbers do not have meaningful order.

Use dummy variables instead.

Mistake 3: Assuming correlation means causation

Regression finds relationships.

It does not automatically prove:

X causes Y

Other factors may influence the result.

Key Takeaways

• Multiple regression analyzes several predictors at once
• Categorical variables need dummy encoding
• Coefficients explain relationships between variables
• Always interpret results in context
• Good models start with good data

Coming Next (Week 5 Preview)

Next week we will explore:

• improving regression models
• checking assumptions
• residual analysis
• detecting problems in models
• making better statistical decisions