Regression with Deep Learning#
In this workshop we see
what regression is and some metrics
how we set up a model to make predictions.
Setup#
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from IPython.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
import requests
url = "https://raw.githubusercontent.com/AstroStat-Academy/assets-public/main/colab/clone_and_cd_colab.py"
colab = requests.get(url).text
exec(colab)
url = "https://raw.githubusercontent.com/AstroStat-Academy/assets-public/main/styles/plot_style.py"
style = requests.get(url).text
exec(style)
import warnings
warnings.filterwarnings(action='ignore')
import time
import keras
from keras.layers import Activation, Dropout, Flatten, Dense, Input, BatchNormalization,Conv3D, MaxPooling3D, Dense, Add, Activation
from keras.regularizers import l2
from keras.models import Model, Sequential
from keras.optimizers import Adam, SGD, Adagrad, RMSprop
from astropy.io import fits
from astropy.table import Table
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from IPython.display import clear_output
What is regression ?#
Modeling problems where the output is a continuous numeric value.
Linear regression and least squares#
A linear regression model will try to predict the single value of the dependent variable \(y\) given the independent variable \(x\), with the most generic form being:
where \(x\) corresponds to the input variable(s) and \(β\) is an array of parameters.
The simplest linear model we can think of is a line:
where \(β_0\) and \(β_1\) have the usual meaning of intercept and slope of a line.
Generalizing a bit, for an input vector \(x^T = (x_1, x_2, ..., x_N)\), where \(N\) is the total number of observations (or samples), the linear model to predict the real-valued output \(y\) is:
To find these parameters we use the Ordinary Least Squares approach, i.e. we try to minimize the residual sum of the squares between the observations and the predictions by the model (i.e. the loss or cost function):
The more general linear regression#
Linear regression refers to modeling functions that are linear with respect to the parameters (coefficients) and not with respect to the variables! For example, the function:
describes a linear problem as long as the sub-functions \(g_i(x)\) do not depend on any of the parameters \(β_i\). This is not the most generic formulation of the linear regression but we are going to use this form in the following applications.
On metrics … or how well can we do#
Accuracy is a measure for classification not regression
We cannot calculate accuracy for a regression modelby Jason Brownlee Regression Metrics for Machine Learning
In regression we are dealing with continuous values. Therefore, it is actually impossible to predict the exact same values. The idea is to get an estimate of how close the predictions are to the expected values.
We are going to refer only a few of the available metrics in sklearn. In the following \(y\) refers to the dependent values while \(\hat{y}\) to the predicted values.
–> Mean Squared Error (MSE)
It is actually the cost function of the Ordinary Least Squares (check \(RSS(β)\) above). The units returned in this case are squared. Best score is 0.
–> Root Mean Squared Error (RMSE)
It returns the square root of MSE so that the units match the units of the target value (so better interpretation). Best score is 0.
–> Mean Absolute Error (MAE)
It is less sensitive to large errors when compared to (R)MSE. The score is in units of the target value.
Comment: Although arithetically the best scores for (R)MSE and MAE is 0 this cannot be the case in real-life problems. Instead a baseline model has to be determined and calculate its score. Then, any model that can achieve a score better that the baselie model is accepted as a skilful model.
–> R2 (coefficient of determination)
where \(\bar{y} = \frac{1}{n}\sum_{i=1}^N y_i\).
It represents the proportion of variance (of y) that has been explained by the independent variables in the model. It provides an indication of goodness of fit and therefore a measure of how well unseen samples are likely to be predicted by the model, through the proportion of explained variance. Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse).
Application 1: Estimate SFR#
We will use data derived from the Heraklion Extragalactic Catalogue (HECATE; Kovlakas et al, 2021) which is an all-sky galaxy catalogue, containing about 200k galaxies (up to z=0.047, D≲200Mpc), and it offers positions, sizes, distances, morphological classifications, star formation rates, stellar masses, metallicities, and nuclear activity classifications.
In particular, we are going to use a dataset from the work of Kouroumpatzakis et al. 2023 where they estimate the Star Formation Rates (SFR) and stellar mass using model Spectral Enerny Distributions with a variety of parameters (IR photometry, extinction, and stellar populations).
Exercise 1:
Objective: Build a regression model to predict SFR
Task: For this follow these steps:
load and investigate the data
select features
pre-process the data
split data (train/validation/test)
build model (architecture with dense layers)
train the model
evaluate its performance
compare with baseline model and check results.
Loading and checking data#
class PlotLosses(keras.callbacks.Callback):
# NOTE: the current version can print only the first metric from the list provided
def on_train_begin(self, logs={}):
self.i = 0
self.x = []
self.losses = []
self.val_losses = []
self.losses2 = []
self.val_losses2 = []
self.fig = plt.figure()
self.logs = []
def on_epoch_end(self, epoch, logs={}):
self.logs.append(logs)
self.x.append(self.i)
self.losses.append(logs.get('loss'))
self.val_losses.append(logs.get('val_loss'))
self.losses2.append(logs.get( test_metrics[0] ) )
self.val_losses2.append(logs.get( f'val_{test_metrics[0]}' ))
# print(test_metrics[0] , logs.get( test_metrics[0] ), logs.get( f'val_{test_metrics[0]}' ))
# print(self.losses2)
self.i += 1
clear_output(wait=True)
plt.subplot(1,2,1)
plt.plot(self.x, self.losses2, label="Train",linestyle='-')
plt.plot(self.x, self.val_losses2, label="Validate",linestyle='--')
# plt.ylim(0,1)
plt.legend()
plt.xlabel('Epoch')
plt.ylabel(test_metrics[0])
plt.subplot(1,2,2)
plt.plot(self.x, self.losses, label="Train",linestyle='-')
plt.plot(self.x, self.val_losses, label="Validate",linestyle='--')
plt.legend()
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.tight_layout()
plt.show();
plot_losses = PlotLosses()
def sub_ploots(plots_cols, plots_array):
""" Automatic adjustment of subplots
Given the
plots_cols : plots per row (set manually)
plots_array : array from which the number of
individual subplots is derived
the output is given as
plots_cols : plots per row
plots_rows : number of necessary rows to create
"""
plots_unique = len((list(set(plots_array))))
plots_rows = int(np.ceil(plots_unique/plots_cols))
return plots_rows, plots_cols
# following this tutorial :https://learn.astropy.org/tutorials/FITS-tables.html
data_fits = fits.open('data/CIGALE_SFR.fits', memmap=True)
# printing some information
data_fits.info()
print('-----')
print(data_fits[1].columns)
Note on columns:
'sfr' : Star Formation Rate
'mstar' : Stellar Mass
'IRX' : IR excess ratio
'EBVs' : extinction
'LYoungOld' : ratin of luminosity of young to old populations
'NIR200' : JWST NIR-F200W
'MIRI2100' : JWST MIRI-F2100W
'W1' : WISE 3.4 μm
'W2' : WISE 4.6 μm
'W3' : WISE 12 μm
'W4' : WISE 22 μm
Printing the data table.
data_tab = Table(data_fits[1].data)
data_tab
Selecting features#
In this part we can select which features we want to keep and use to train our model. Feel free to choose any number of feature - keep in mind though NOT to select sfr and mstar as these are needed for the output (i.e. are the values we want to predict).
Task: select the features to work with
sel_features = [ ... ]
fig = plt.figure(figsize=(8,8))
rws, cls = sub_ploots(2, sel_features) # for subplots
for f in range(len(sel_features)):
f_name = sel_features[f]
ax = fig.add_subplot(rws, cls, f+1)
ax.set_title(f'Feature {f_name}')
ax.hist(data_tab[f_name], bins='auto', align='mid', density=True)
ax.set_xlabel('feature value')
ax.set_ylabel('counts or probability density')
plt.tight_layout()
plt.show()
len(data_tab)
and the data table with the selected features looks like …
data_tab[sel_features]
We convert table objects into numpy arrays:
values = data_tab[sel_features].as_array()
values = values.view((float, len(values.dtype.names)))
target = np.array(data_tab['sfr']).reshape(-1,1)
print('The values for selected features:')
print(values)
print()
print('The target quantity:')
print(target)
Split the sample and … ?#
Tasks:
split the sample into train/validation/test
what should you do before training the model?
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
# keep aside the test set first (coming from the future!)
X_train_full, X_test_un, y_train_full, y_test = train_test_split( ..., ... ,
test_size= ... ) #, random_state=42)
# split the remaining sample intro train and validation
X_train_un, X_valid_un, y_train, y_valid = train_test_split( ... , ... ,
test_size= ... ) #, random_state=42)
scaler = ...
X_train = scaler.fit_transform(...)
X_valid = scaler.transform( ... )
X_test = scaler.transform( ... )
print(f'> From {len(target)} sources:')
print(f' {len(X_train)} (train)')
print(f' {len(X_valid)} (validation)')
print(f' {len(X_test)} (test)')
print('\n> Statistics per feature:')
print(f'Mean: {scaler.mean_}')
print(f'Variance: {scaler.var_}')
print()
print(f' Target shape: {y_train.shape}') # print the shape of the output values
Time to build our model!#
# if you want you can add more output metrics!
# but keep in mind that the plot_losses will use the first one.
test_metrics = [ 'mean_squared_error', 'mean_absolute_error' ]
Tasks:
build a DL network with a number of dense layers - you can add as many layers as you want (probably you may need to return to correct this).
how many nodes the last layer should have?
model = Sequential()
model.add( Dense( ... , input_shape=X_train_full.shape[1:], activation='relu') )
model.add( Dense( ..., activation='relu') )
...
model.add( Dense(...) )
optzr = Adam(learning_rate=1e-4)
model.compile(loss='mean_squared_error', optimizer=optzr,
metrics = test_metrics)
model.summary()
Finally … train the model!#
Task: Fill the missing code to train the model
start_time = time.time()
history=model.fit( ... , ... ,
batch_size= ... ,
epochs= ... ,
validation_data=[ ... , ... ],
callbacks=[plot_losses],shuffle=True)
# to print history contents
# history.history
elapsed_time = time.time() - start_time
time.strftime("%H:%M:%S", time.gmtime(elapsed_time))
… and the moment of truth - evaluation#
Task: evaluate the model (take care on which dataset!)
evaluation = model.evaluate( ... , ... )
print(f"Loss value: {evaluation[0]:.2f}")
for m in range(len(test_metrics)):
print(f"{test_metrics[m]}: {evaluation[m+1]:0.2f}")
Comparing with a baseline model…#
Task: fill in the code to train and evaluate the baseline model.
lr_model = LinearRegression()
lr_model.fit( ... , ... )
y_hat_lr = lr_model.predict( ... )
mse = mean_squared_error( ... , ... )
mae = mean_absolute_error( ... , ... )
r2 = r2_score( ... , ...)
print(f"MSE: {mse:.2f}")
print(f"MAE: {mae:.2f}")
print(f"R2 : {r2:.2f}" )
Checking the predicitons#
Task: fill in the code to check the results.
y_hat = model.predict( ... )
r2 = r2_score( ... , ... )
min_ax, max_ax = np.min(y_test), np.max(y_test)
plt.plot(y_test, y_hat, '.', alpha=0.4)
plt.title( f'R2 = {r2:0.2f}', fontsize=20)
plt.xlabel('true', fontsize=16)
plt.ylabel('prediction', fontsize=16)
plt.xlim([min_ax, max_ax])
# 1-to-1 line:
xx = np.linspace(min_ax, max_ax, 100)
plt.plot(xx, xx, c="grey", linestyle='--')
plt.show()
Application 2: Estimate SFR and stellar mass#
Exercise 2:
Objective: Build a regression model to predict both SFR (‘sfr’) and stellar mass (‘mstar’).
Task: Follow the same steps with the previous exercise. You need to adjust the model to train and predict both values.
In this case we want to use all/part of the available features to estimate both the SFR (sfr) and the stellar mass (mstar).
Reminder of the available features:
print(data_fits[1].columns)
Task: Select the features to work with. The features we want to predict now are two, the SFR and the stellar mass.
sel_features = [ ... ]
pre_features = [ ... ]
values = data_tab[sel_features].as_array()
values = values.view((float, len(values.dtype.names)))
target = data_tab[pre_features].as_array()
target = target.view((float, len(target.dtype.names)))
print('The values for selected features:')
print(values)
print()
print('The target quantity:')
print(target)
Tasks:
split the sample into train/validation/test
what should you do before training the model?
# keep aside the test set first (coming from the future!)
X_train_full, X_test_un, y_train_full, y_test = train_test_split(... ,...,
test_size= ...) #, random_state=42)
# split the remaining sample intro train and validation
X_train_un, X_valid_un, y_train, y_valid = train_test_split( ... , ...,
test_size= ...) #, random_state=42)
scaler = ...
X_train = scaler.fit_transform( ... )
X_valid = scaler.transform( ... )
X_test = scaler.transform( ... )
print(f'> From {len(target)} sources:')
print(f' {len(X_train)} (train)')
print(f' {len(X_valid)} (validation)')
print(f' {len(X_test)} (test)')
print('\n> Statistics per feature:')
print(f'Mean: {scaler.mean_}')
print(f'Variance: {scaler.var_}')
print()
print(f' Target shape: {y_train.shape}') # print the shape of the output values
We define the metrics again here:
# if you want to add more output metrics
test_metrics = ['mean_squared_error', 'mean_absolute_error', ]
Task: Building our new model. Watch out the number of nodes at the very last step!
model = Sequential()
model.add( Dense(,,,, input_shape=X_train.shape[1:], activation='relu') )
model.add( Dense(..., activation='relu') )
...
model.add( Dense(...) )
optzr = Adam(learning_rate=1e-4)
model.compile(loss= 'mean_squared_error', # custom_loss,
optimizer=optzr,
metrics = test_metrics)
model.summary()
Task: Train the model
start_time = time.time()
history=model.fit( ... , ... ,
batch_size= ...,
epochs= ...,
validation_data=[ ... , ...],
callbacks=[plot_losses],shuffle=True)
# to print history contents
# history.history
elapsed_time = time.time() - start_time
time.strftime("%H:%M:%S", time.gmtime(elapsed_time))
Task: Compare with baseline model.
lr_model = LinearRegression()
lr_model.fit( ... , ...)
y_hat_lr = lr_model.predict( ... )
mse_lr = mean_squared_error( ... , ...)
mae_lr = mean_absolute_error( ... , ...)
r2_lr = r2_score(y_test, y_hat_lr, multioutput='raw_values')
print(f"Mean Squared Error (MSE): {mse_lr:.2f}")
print(f"Mean Absolute Error (MAE): {mae_lr:.2f}")
print(f"R2 (sfr): {r2_lr[0]:0.2f} ")
print(f"R2 (mstar): {r2_lr[1]:0.2f} ")
Task: check the results.
evaluation = model.evaluate( ... , ...)
print(f"Loss value: {evaluation[0]:.2f}")
for m in range(len(test_metrics)):
print(f"{test_metrics[m]}: {evaluation[m+1]:0.2f}")
print('------')
y_pred = model.predict( ... )
fig, axes = plt.subplots(1,2, figsize=(12,6))
axes = axes.flatten()
for i, ax in enumerate(axes):
r2 = r2_score( y_test[:,i] , y_pred[:,i])
print(i, r2)
min_ax, max_ax = np.min(y_test[:,i]), np.max(y_test[:,i])
ax.plot(y_test[:,i], y_pred[:,i], '.', alpha=0.4)
ax.set_title( f'R2 = {r2:0.2f} for {pre_features[i]}', fontsize=20)
ax.set_xlabel('true', fontsize=16)
ax.set_ylabel('prediction', fontsize=16)
ax.set_xlim([min_ax, max_ax])
min_ax, max_ax = np.min(y_test[:,i]), np.max(y_test[:,i])
# 1-to-1 line:
xx = np.linspace(min_ax, max_ax, 100)
ax.plot(xx, xx, c="grey", linestyle='--')
plt.show()
Solutions#
Application 1: Estimate SFR#
[Solution below]
sel_features = ['LYoungOld', 'IRX', 'EBVs', 'W1']
fig = plt.figure(figsize=(8,8))
rws, cls = sub_ploots(2, sel_features) # for subplots
for f in range(len(sel_features)):
f_name = sel_features[f]
ax = fig.add_subplot(rws, cls, f+1)
ax.set_title(f'Feature {f_name}')
ax.hist(data_tab[f_name], bins='auto', align='mid', density=True)
ax.set_xlabel('feature value')
ax.set_ylabel('counts or probability density')
plt.tight_layout()
plt.show()
# keep aside the test set first (coming from the future!)
X_train_full, X_test_un, y_train_full, y_test = train_test_split( values, target,
test_size=0.3) #, random_state=42)
# split the remaining sample intro train and validation
X_train_un, X_valid_un, y_train, y_valid = train_test_split( X_train_full, y_train_full,
test_size=0.2) #, random_state=42)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train_un)
X_valid = scaler.transform(X_valid_un)
X_test = scaler.transform(X_test_un)
print(f'> From {len(target)} sources:')
print(f' {len(X_train)} (train)')
print(f' {len(X_valid)} (validation)')
print(f' {len(X_test)} (test)')
print('\n> Statistics per feature:')
print(f'Mean: {scaler.mean_}')
print(f'Variance: {scaler.var_}')
print()
print(f' Target shape: {y_train.shape}') # print the shape of the output values
model = Sequential()
model.add( Dense(16, input_shape=X_train_full.shape[1:], activation='relu') )
# model.add( Dropout(0.3) ) # Dropout layer with 50% dropout rate
# model.add( Dense(64, activation='relu', kernel_regularizer=l2(0.01)),
model.add( Dense(64, activation='relu') )
model.add( Dense(32, activation='relu') )
model.add( Dense(1) )
start_time = time.time()
history=model.fit(X_train, y_train,
batch_size=512,
epochs=15,
validation_data=[ X_valid, y_valid],
callbacks=[plot_losses],shuffle=True)
# to print history contents
# history.history
elapsed_time = time.time() - start_time
time.strftime("%H:%M:%S", time.gmtime(elapsed_time))
evaluation = model.evaluate(X_test, y_test)
print(f"Loss value: {evaluation[0]:.2f}")
for m in range(len(test_metrics)):
print(f"{test_metrics[m]}: {evaluation[m+1]:0.2f}")
lr_model = LinearRegression()
lr_model.fit(X_train, y_train)
y_hat_lr = lr_model.predict(X_test)
mse = mean_squared_error(y_test, y_hat_lr)
mae = mean_absolute_error(y_test, y_hat_lr)
r2 = r2_score( y_test , y_hat_lr)
print(f"MSE: {mse:.2f}")
print(f"MAE: {mae:.2f}")
print(f"R2 : {r2:.2f}" )
y_hat = model.predict( X_test)
r2 = r2_score( y_test , y_hat)
min_ax, max_ax = np.min(y_test), np.max(y_test)
plt.plot(y_test, y_hat, '.', alpha=0.4)
plt.title( f'R2 = {r2:0.2f}', fontsize=20)
plt.xlabel('true', fontsize=16)
plt.ylabel('prediction', fontsize=16)
plt.xlim([min_ax, max_ax])
# 1-to-1 line:
xx = np.linspace(min_ax, max_ax, 100)
plt.plot(xx, xx, c="grey", linestyle='--')
plt.show()
[End of solution]
Application 2: Estimate SFR and stellar mass#
[Solution below]
sel_features = ['LYoungOld', 'IRX', 'W1']
pre_features = ['sfr', 'mstar']
values = data_tab[sel_features].as_array()
values = values.view((float, len(values.dtype.names)))
target = data_tab[pre_features].as_array()
target = target.view((float, len(target.dtype.names)))
print('The values for selected features:')
print(values)
print()
print('The target quantity:')
print(target)
# keep aside the test set first (coming from the future!)
X_train_full, X_test_un, y_train_full, y_test = train_test_split( values, target,
test_size=0.3) #, random_state=42)
# split the remaining sample intro train and validation
X_train_un, X_valid_un, y_train, y_valid = train_test_split( X_train_full, y_train_full,
test_size=0.2) #, random_state=42)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train_un)
X_valid = scaler.transform(X_valid_un)
X_test = scaler.transform(X_test_un)
print(f'> From {len(target)} sources:')
print(f' {len(X_train)} (train)')
print(f' {len(X_valid)} (validation)')
print(f' {len(X_test)} (test)')
print('\n> Statistics per feature:')
print(f'Mean: {scaler.mean_}')
print(f'Variance: {scaler.var_}')
print()
print(f' Target shape: {y_train.shape}') # print the shape of the output values
model = Sequential()
model.add( Dense(128, input_shape=X_train.shape[1:], activation='relu') )
model.add( Dense(64, activation='relu') )
model.add( Dense(32, activation='relu') )
model.add( Dense(2) )
start_time = time.time()
history=model.fit(X_train, y_train,
batch_size=512,
epochs=40,
validation_data=[ X_valid, y_valid],
callbacks=[plot_losses],shuffle=True)
# to print history contents
# history.history
elapsed_time = time.time() - start_time
time.strftime("%H:%M:%S", time.gmtime(elapsed_time))
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error
lr_model = LinearRegression()
lr_model.fit(X_train, y_train)
y_hat_lr = lr_model.predict(X_test)
mse_lr = mean_squared_error(y_test, y_hat_lr)
mae_lr = mean_absolute_error(y_test, y_hat_lr)
r2_lr = r2_score(y_test, y_hat_lr, multioutput='raw_values')
print(f"Mean Squared Error (MSE): {mse_lr:.2f}")
print(f"Mean Absolute Error (MAE): {mae_lr:.2f}")
print(f"R2 (sfr): {r2_lr[0]:0.2f} ")
print(f"R2 (mstar): {r2_lr[1]:0.2f} ")
Task: Check the results (with the linear modeling)
from sklearn.metrics import r2_score
evaluation = model.evaluate(X_test, y_test)
print(f"Loss value: {evaluation[0]:.2f}")
for m in range(len(test_metrics)):
print(f"{test_metrics[m]}: {evaluation[m+1]:0.2f}")
print('------')
y_pred = model.predict( X_test)
fig, axes = plt.subplots(1,2, figsize=(12,6))
axes = axes.flatten()
for i, ax in enumerate(axes):
r2 = r2_score( y_test[:,i] , y_pred[:,i])
print(i, r2)
min_ax, max_ax = np.min(y_test[:,i]), np.max(y_test[:,i])
ax.plot(y_test[:,i], y_pred[:,i], '.', alpha=0.4)
ax.set_title( f'R2 = {r2:0.2f} for {pre_features[i]}', fontsize=20)
ax.set_xlabel('true', fontsize=16)
ax.set_ylabel('prediction', fontsize=16)
ax.set_xlim([min_ax, max_ax])
min_ax, max_ax = np.min(y_test[:,i]), np.max(y_test[:,i])
# 1-to-1 line:
xx = np.linspace(min_ax, max_ax, 100)
ax.plot(xx, xx, c="grey", linestyle='--')
plt.show()
[End of solution]
###EOF