Machine Learning Part 2: Prediction and Classification
Loading comments...
Loading comments...
Part 2 of the ML learning notes. This time: actually building models instead of just setting up the toolchain. Linear regression → logistic regression → cross-validation → a mini house price prediction project.
I started this part with linear regression because it made coefficient changes easy to inspect. It finds linear relationships between independent and dependent variables to predict future values.
1import numpy as np
2import pandas as pd
3from sklearn.linear_model import LinearRegression
4from sklearn.model_selection import train_test_split
5from sklearn.metrics import mean_squared_error, r2_score
6
7# Data preparation
8X = np.random.rand(100, 1) * 10
9y = 2 * X + 1 + np.random.randn(100, 1) * 0.5
10
11# Model training
12model = LinearRegression()
13model.fit(X, y)
14
15# Prediction
16y_pred = model.predict(X)
17print(f"Coefficient: {model.coef_[0][0]:.2f}")
18print(f"Intercept: {model.intercept_[0]:.2f}")In real problems, multiple variables act in combination. Let's find complex patterns with multiple linear regression.
1# Multi-variable data
2X_multi = np.random.rand(100, 3) * 10
3y_multi = 2*X_multi[:, 0] + 1.5*X_multi[:, 1] - 0.5*X_multi[:, 2] + np.random.randn(100) * 0.5
4
5# Multiple linear regression
6model_multi = LinearRegression()
7model_multi.fit(X_multi, y_multi)
8
9# Check variable importance
10for i, coef in enumerate(model_multi.coef_):
11 print(f"Variable {i+1} coefficient: {coef:.2f}")Logistic regression is a powerful tool for converting continuous probabilities into binary classification.
1from sklearn.linear_model import LogisticRegression
2from sklearn.metrics import classification_report, confusion_matrix
3
4# Generate binary classification data
5X_binary = np.random.randn(200, 2)
6y_binary = (X_binary[:, 0] + X_binary[:, 1] > 0).astype(int)
7
8# Logistic regression model
9logistic_model = LogisticRegression()
10logistic_model.fit(X_binary, y_binary)
11
12# Prediction and evaluation
13y_pred_binary = logistic_model.predict(X_binary)
14print(classification_report(y_binary, y_pred_binary))For classifying three or more classes, we use multi-class logistic regression.
1from sklearn.datasets import make_classification
2from sklearn.multiclass import OneVsRestClassifier
3
4# Multi-class data
5X_multi_class, y_multi_class = make_classification(
6 n_samples=300, n_features=4, n_classes=3, n_clusters_per_class=1
7)
8
9# Multi-class logistic regression
10multi_logistic = LogisticRegression(multi_class='ovr')
11multi_logistic.fit(X_multi_class, y_multi_class)
12
13# Prediction
14y_pred_multi = multi_logistic.predict(X_multi_class)
15print(classification_report(y_multi_class, y_pred_multi))Let's practice cross-validation to prevent overfitting and evaluate model generalization performance.
1from sklearn.model_selection import cross_val_score, KFold
2
3# K-Fold cross-validation
4kfold = KFold(n_splits=5, shuffle=True, random_state=42)
5scores = cross_val_score(logistic_model, X_binary, y_binary, cv=kfold)
6
7print(f"Cross-validation score: {scores.mean():.3f} (+/- {scores.std() * 2:.3f})")I used GridSearchCV next to check how much C/penalty choices changed accuracy.
1from sklearn.model_selection import GridSearchCV
2
3# Hyperparameter grid
4param_grid = {
5 'C': [0.1, 1, 10, 100],
6 'penalty': ['l1', 'l2'],
7 'solver': ['liblinear', 'saga']
8}
9
10# Grid search
11grid_search = GridSearchCV(
12 LogisticRegression(), param_grid, cv=5, scoring='accuracy'
13)
14grid_search.fit(X_binary, y_binary)
15
16print(f"Best parameters: {grid_search.best_params_}")
17print(f"Best score: {grid_search.best_score_:.3f}")Let's proceed with a project to predict house prices using real data.
1import pandas as pd
2from sklearn.preprocessing import StandardScaler
3from sklearn.ensemble import RandomForestRegressor
4
5# House price data (example)
6data = pd.DataFrame({
7 'area': np.random.rand(1000) * 200 + 50,
8 'bedrooms': np.random.randint(1, 6, 1000),
9 'bathrooms': np.random.randint(1, 4, 1000),
10 'age': np.random.randint(0, 50, 1000)
11})
12
13# Price generation (in reality, more complex relationships)
14data['price'] = (
15 data['area'] * 100 +
16 data['bedrooms'] * 50000 +
17 data['bathrooms'] * 30000 -
18 data['age'] * 2000 +
19 np.random.randn(1000) * 10000
20)
21
22# Separate features and target
23X_house = data[['area', 'bedrooms', 'bathrooms', 'age']]
24y_house = data['price']
25
26# Data splitting
27X_train, X_test, y_train, y_test = train_test_split(
28 X_house, y_house, test_size=0.2, random_state=42
29)
30
31# Scaling
32scaler = StandardScaler()
33X_train_scaled = scaler.fit_transform(X_train)
34X_test_scaled = scaler.transform(X_test)
35
36# Random Forest model
37rf_model = RandomForestRegressor(n_estimators=100, random_state=42)
38rf_model.fit(X_train_scaled, y_train)
39
40# Prediction and evaluation
41y_pred_house = rf_model.predict(X_test_scaled)
42mse = mean_squared_error(y_test, y_pred_house)
43r2 = r2_score(y_test, y_pred_house)
44
45print(f"Mean Squared Error: {mse:.2f}")
46print(f"R² Score: {r2:.3f}")Running GridSearchCV and watching how much the results swing based on whether I scaled the data or not was more instructive than any article I've read about it. The house price project also made feature importance feel concrete rather than abstract, seeing area dominate the chart made the Random Forest output feel explainable.
Ensemble methods (Random Forest, XGBoost), then neural networks.