In Week 1, we introduced Python and pandas.
In Week 2, we focused on cleaning and preparing data for analysis.
Now in Week 3, we take the next step:
👉 Filtering data, transforming variables, and running our first regression model
This is where data starts turning into insights.
Why Filtering and Transformation Matter
Before running statistical models, we often need to:
- Focus on a subset of data
- create new variables
- reshape data for analysis
Raw data is rarely ready for modelling. This step bridges clean data → usable features for regression
Filtering Data in pandas
Filtering allows us to isolate specific parts of a dataset.
Example: Filter by numeric condition
import pandas as pd
df = pd.read_csv('sample_data.csv')
df_filtered = df[df['Salary'] > 50000]
print(df_filtered.head())This keeps only rows where the salary is greater than 50,000.
Example: Filter by category
import pandas as pd
df = pd.read_csv('sample_data.csv')
df_filtered = df[df['Program'] == 'Computer Science']
print(df_filtered.head())This isolates one group for analysis.
Why filtering matters
Filtering helps you:
- Focus on relevant groups
- reduce noise in data
- improve model clarity
Transforming Data (Feature Engineering Basics)
Transformation means creating new variables from existing ones.
Example: Convert salary into thousands
import pandas as pd
df = pd.read_csv('sample_data.csv')
df['Salary_k'] = df['Salary'] / 1000
print(df.head())This makes values easier to interpret.
Example: Create a binary variable
import pandas as pd
df = pd.read_csv('sample_data.csv')
df['High_Salary'] = df['Salary'] > 50000
print(df.head())This creates True/False categories for analysis.
Why transformation matters
It helps you:
- simplify interpretation
- Prepare variables for regression
- uncover patterns more clearly
Introduction to OLS Regression
OLS (Ordinary Least Squares) is a method used to find relationships between variables.
We want to understand:
👉 Does one variable help explain another?
Example:
- Does work experience affect salary?
- Does program type affect salary?
Running Your First Regression
import pandas as pd
import statsmodels.api as sm
df = pd.read_csv('sample_data.csv')
X = df[['Term']]
y = df['Salary']
X = sm.add_constant(X)
model = sm.OLS(y, X).fit()
print(model.summary())Understanding regression output
When you run:
model.summary()You will see a table containing key results.
R-squared
How well the model explains the data.
- 0 → no explanation
- 1 → perfect explanation
Coefficient
Shows direction and strength of the relationship.
- positive → increases
- negative → decreases
P-value
Shows statistical significance.
- < 0.05 → significant
- 0.05 → weak evidence
Simple interpretation
If Term has a positive coefficient:
👉 Higher term value is associated with higher salary (on average)
What this actually means
Regression does NOT prove causation.
It only shows:
👉 “There is a relationship between variables”
not
👉 “One causes the other”
Key Takeaways
- Filtering helps isolate relevant data
- Transformation creates meaningful variables
- OLS regression identifies relationships
- Interpretation matters more than execution
Coming Next (Week 4 Preview)
Next week, we will go deeper into:
- multiple regression models
- dummy variables
- interpreting complex relationships
- improving model accuracy
Thank you for reading
