Machine Learning: Classification#
In this session we focus on another important application for machine learning, the Classification.
We will provide a simple problem (with multiple intermediate tasks):
to demonstrate basic visualization of the data
to show the basic steps in an application of an algorith (or two!)
to present the necessary performance metrics
to investigate differences in binary and multiclass approaches
\(\rightarrow\) The target is to have an practical application example of an algorith to understand the approach and (hopefully!) use it for your own research.
Setup#
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
import warnings
warnings.filterwarnings(action='ignore')
import requests
url = "https://raw.githubusercontent.com/AstroStat-Academy/assets-public/main/colab/clone_and_cd_colab.py"
colab = requests.get(url).text
exec(colab)
url = "https://raw.githubusercontent.com/AstroStat-Academy/assets-public/main/styles/plot_style.py"
style = requests.get(url).text
exec(style)
Types of Classification#
Flavors#
Classification can come in two major flavors, based on the type of intervention by the user:
Unsupervised: The classification is defined “unsupervised” when the user does not provide labels during the training process and the machine learns the definition of each class from the data. This is exactly the case of “Clustering” that we saw in the previous session.
In practice, the machine learns to cluster objects with similar properties.
Supervised: The classification is defined “supervised” when the user provides a label for each object in the training set. In this case, the idea is that we can train the model to associate the label with some given characteristics of the training data.
In practice, the machine learns to find similarities between objects with the same label.
Terminology#
Keep in mind that:
class or label are used interchangeably
samples (i.e. the items) can be of any arbitrary type (e.g. a string, an image, or numbers) - however, in astronomy we mainly refer to objects or sources, since samples correspond to their collections
(so… samples in ml terminology and objects/sources in astronomy!)the objects/samples are categorized based on their properties or features (the actual information that we pass to the machine, e.g. the pixel intensities in the case of an image).
The sample for our classification example#
To properly classify a star we need a spectrum. However, this is time consuming and limited to one (single slit spectroscopy) or a few tens of sources (multi-object spectroscopy). On the contrary, imaging in various different filters can be done easily and for thousands of sources per image. Photometry at different wavelengths result in a very low-resolution “spectrum”.
The filter and detector transmission curves for a number of different systems, along with indicative spectra of Vega, the Sun, and a M5 giant. (Fig 3. from Girardi et al. (2002))
In this example we are using a set of photometric measurements (from optical to mid-IR bands) for a sample of massive evolved stars in the Large Magellanic Cloud (based on these works: Bonanos et al. (2009) AJ, 138, 1003, Neugent et al. (2012), ApJ, 749, 177, and Davies, Crowther & Beasor (2018), MNRAS, 478, 313).

A slightly modified version of Fig. 1 from Massey et al. (2013))
Our aim is to use a method that will help us distinguish different classes of objects. For our purposes we will use OBA stars (main-sequense objects), OBAe (a subcategory of OBA stars with circumstellar disks and emission lines), Wolf-Rayet stars (hot evolved stars with strong stellar winds that actually strip their envelopes), Yellow and Red supergiants (evolved massive stars). For convenience we will use OBA, OBAe, WR, YSG, and RSG, respectively, as labels.
★ A similar, but more elaborated, implementation is performed in Maravelias et al. (2022), A&A, 666, 122.
Load and examine data#
dfile = "data/LMC_phot_data.csv"
data = pd.read_csv(dfile)
#examine data
print("Let us see what we have:\n")
print(data)
print("-"*25)
print("The column names:")
print(data.columns)
print("-"*25)
print("Let's print the spectral types only:")
print(data['SpT'])
Let’s now explore the statistics per group (class) of sources.
unique_cls = data['SpT'].unique()
print("> Unique classes:", unique_cls)
print("="*25)
print("> SUMMARY of loaded data:")
print("="*25)
for sptype in unique_cls:
number = data[data['SpT']==sptype].shape[0]
print(f"{sptype:-<6s}--> {number:>3} stars")
bands = [b for b in data.columns if 'e_' not in b][3:-1]
def reminder():
"""
A simple function to print all bands
and classes available.
"""
print('Available bands to use: ')
print(','.join(bands))
print('-'*25)
print('Available classes to use:')
print(','.join(unique_cls))
results = []
for spt in unique_cls:
data_spt = data[data['SpT']==spt]
number_spt = data_spt.shape[0]
# print(spt, number_spt)
row = {
'Class': spt,
'Total': number_spt
}
for b in bands:
data_spt_bnd = data_spt[b]
number_nn = data_spt_bnd.notna().sum()
fraction = number_nn/number_spt
row[b] = fraction * 100
# print(b, fraction)
# print(row)
results.append(row)
phot_data_per = pd.DataFrame(results)
print(f'Available photometry for: {", ".join(bands)}')
print("\nNumber of stars per band (in %)\n")
phot_data_per.round(2)
Question
Check the table, what do you notice here ?
[Spoiler] (click here to expand)
Many sources come without measurements across all filters. This is a particular problem for algortihms that need values to work with.
This problem of ”missing values” is actually the norm than the exception. There are ways to tackle this, e.g. by taking the mean values of the corresponding filters or more elaborated techinques. Check sklearn’s imputation of missing values for more.
Visualize data - select features#
reminder()
Let’s plot the data.
def selmags( band1, band2, cls):
"""
Function to select sources of a specific
spectral class (cls) and return the magnitudes
that correspond to bands 1 and 2.
"""
# select all rows of the "cls" type
data_cls = data[data['SpT']==cls]
# print(band1, band2)
mag1 = data_cls[band1].values
mag2 = data_cls[band2].values
return mag1, mag2
def plotmags( selected_bnd, selected_spt):
""" Function to plot magnitudes and colors
for sources corresponding to the selected classes.
NOTEs:
- selected_bnd: a list of four magnitudes
- selected_spt: a list of selected spectral types
Output: two sets of plots.
"""
fig, ax = plt.subplots(2,2, figsize=(10, 10))
legend_fs = 16
print('- plot1:')
for s in selected_spt:
plt1 = selmags( selected_bnd[0], selected_bnd[1], s)
# calculate the number of sources to be plotted
# both arrays have nan values, and plot is smart enough to
# not plot those points.
# but we need to know what we are missing!
mask1 = np.isfinite(plt1[0]) & np.isfinite(plt1[1])
n_points_1_kept = np.count_nonzero(mask1)
print(f'-- {s}: excluding {len(plt1[0])-n_points_1_kept} out of {len(plt1[0])} sources ({(len(plt1[0])-n_points_1_kept)/len(plt1[0])*100:.1f}%)')
ax[0,0].plot(plt1[0], plt1[1], 'o', label=f'{s}: {n_points_1_kept}')
ax[0,0].set_xlabel(f'{selected_bnd[0]}') #'-{band1_2}')
ax[0,0].set_ylabel(selected_bnd[1])
ax[0,0].invert_yaxis()
ax[0,0].invert_xaxis()
ax[0,0].legend(fontsize=legend_fs)
ax[0,1].plot(plt1[0]-plt1[1], plt1[1], 'o', label=f'{s}: {n_points_1_kept}')
ax[0,1].set_xlabel(f'{selected_bnd[0]}-{selected_bnd[1]}')
ax[0,1].set_ylabel(selected_bnd[1])
ax[0,1].invert_yaxis()
ax[0,1].legend(fontsize=legend_fs)
print()
print('- plot2:')
for s in selected_spt:
plt2 = selmags( selected_bnd[2], selected_bnd[3], s)
mask2 = np.isfinite(plt2[0]) & np.isfinite(plt2[1])
n_points_2_kept = np.count_nonzero(mask2)
print(f'-- {s}: excluding {len(plt2[0])-n_points_2_kept} out of {len(plt2[0])} sources ({(len(plt2[0])-n_points_2_kept)/len(plt2[0])*100:.1f}%)')
ax[1,0].plot(plt2[0], plt2[1], 'o', label=f'{s}: {n_points_2_kept}')
ax[1,0].set_xlabel(f'{selected_bnd[2]}') #'-{band2_2}')
ax[1,0].set_ylabel(selected_bnd[3])
ax[1,0].invert_yaxis()
ax[1,0].invert_xaxis()
ax[1,0].legend(fontsize=legend_fs)
ax[1,1].plot(plt2[0]-plt2[1], plt2[1], 'o', label=f'{s}: {len(plt2[0])}')
ax[1,1].set_xlabel(f'{selected_bnd[2]}-{selected_bnd[3]}')
ax[1,1].set_ylabel(selected_bnd[3])
ax[1,1].invert_yaxis()
ax[1,1].legend(fontsize=legend_fs)
plt.show()
Exercise 1:
Objective: Data visualization and exploration - which features to pick ?
Task: Change the bands you use to plot the CMDs as well as the number of classes to consider, and try to answer:
What happens if you start increasing the number of classes to consider ?
How the selection of bands (e.g. adding more bands) influence the numbr of objects to plot (or get rejected) ?
Would you prefer to use combinations with few or more objects ?
plotmags(['B', 'V', 'H','K'], ['WR', 'OBA'])
[Spoiler] (click here to expand)
The task becomes more difficult as classes start to overlap. Some combinations may prove better to separate some classes but there is not a single optimal one.
In order to make the plots we need sources with measuremets in both bands. By enforcing this we actually exclude objects with missing data. That results in decreasing number of sources.
In principle we would like to take advantage of all available sources. That give us a better sampling of their parameters’ space. If only few sources remain then we actually lose information.
Hence, we would like to maximize the number of sources used. In problems with already small numbers (like in this example) this can be crucial.
A working combination:
plotmags(['V', 'J', 'K','[4.5]'], ['RSG', 'OBA'])
What have we achieved so far?
To examine our data and select which features to use.
Now is the time to select an algorithm.
The Support Vector Machine (SVM) algorithm#
The boundary (in 2D)#
Support vector machine (SVM) is a way of choosing a decision boundary between different classes.
The classification boundary is provided by the line (in 2D) that is found in the middle of the two lines defined by the closest points of each class, which are called support vectors. The margin is the distance between these two parallel lines. By maximizing this margin, the SVM achieves the best possible separation with the largest safety zone between classes.
|
|
The left panel of the figure above shows two different classes distributing in a scatter plot according to variable \(x_1\) and \(x_2\). The right panel explains the origin of the name support vectors: the closest points support the (solid) lines equally distant from the decision (dashed) line.
Infinite possible boundaries can separate the two classes. SVM algorithms find the one that maximizes the distance between the lines defined by the support vectors
Minimizing the loss function - again !#
The supported (solid) lines can be defined as:
where x is the coordinate on the (x1, x2) plane, w is a 2\(\times\)1 matrix and b a scalar. It turns out that these lines are separated by a distance 2 / ||w||. Finding the ideal classification boundary, i.e. the one maximizing the distance, is therefore a problem of minimizing ||w||
where \(\gamma\) is a hyperparameter which shall be learned (in our example we use an arbitrary value but in principle we should use cross-validation methods).
Final remarks on SVM#
Pros
Good at dealing with high dimensional data
Works well on small data sets
Cons
Picking the right kernel and parameters can be computationally intensive
It suffers from contamination
★ For further information on SVM, consult Support Vector Machine - Classification, by Saed Sayad.
SVM in practice#
The binary problem#
In this case we examine a binary classification problem where we select one class (or more that will group into a single one) and the rest classes as contaminants. The purpose is to check if we can separate efficiently these two classes.
def process_data( bands2use, binary_classes2use ):
"""
Process input data to return arrays
of magnitudes and (consecutive) colors
based on the input bands (band2use).
Option to prepare data for binary classification
if binary_classes2use contains classes or not.
Selected objects are assigned 1, while the contaminants 0.
"""
data_diff = -data[bands2use].diff(axis=1).iloc[:, 1:]
data_diff.columns = [f"{c1}-{c2}" for c1, c2 in zip(bands2use[:-1], bands2use[1:])]
diff_columns = data_diff.columns.values
# print(data_diff)
# print(data_diff.columns.values)
working = pd.concat([data[['Name']+bands2use+['SpT']], data_diff], axis=1)
# print(working)
if len(binary_classes2use)!=0:
working['keep'] = working['SpT'].isin(binary_classes2use).astype(int)
else:
working['keep'] = working['SpT']
# remove all nan rows
working_clean = working.dropna()
print(f'# stars with mags in:\n',
f'{",".join([bb for bb in bands2use])}'
)
print("=========================")
print("Type initial final ")
print("-------------------------")
for s in unique_cls:
print(
f"{s:<4} {(working['SpT']==s).sum():>9}",
f"{(working_clean['SpT']==s).sum():>8}"
)
print('-'*24)
print(f'TOTAL: {working.shape[0]:>7} {working_clean.shape[0]:>7}')
if len(binary_classes2use)!=0:
print('='*24)
print(f'classifying: {(working_clean['keep']==1).sum():>10}')
print(f'contaminants: {(working_clean['keep']==0).sum():>10}')
data_labels = working_clean['keep'].to_numpy()
else:
data_labels = working_clean['SpT'].to_numpy()
# print(working_clean[diff_columns])
data_mags = working_clean[bands2use].to_numpy()
data_clrs = working_clean[diff_columns].to_numpy()
data_objects = working_clean['Name'].to_numpy()
return data_mags, data_clrs, data_objects, data_labels
reminder()
Select here the class(es) you would like to distinguish from the rest, along with the bands to use.
class2keep = ['RSG']
# Select the bands you want to use here:
bands_selected = ['V', 'J', '[4.5]']
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected, class2keep)
NOTE: the process_data() function examines and keeps only the sources with values across all bands. In other words, we remove sources with missing values (according to the bands selected).
Let’s print the labels to see how they look like.
print(ml_labels)
NOTE: If you have more than 2 bands selected the following plot will use the first two. Modify accordingly to plot other combinations.
fig = plt.figure(figsize=(8,8))
conts = np.where( ml_labels==0 )[0]
clasf = np.where( ml_labels!=0 )[0]
plt.plot( ml_data_mags[conts][:,0], ml_data_mags[conts][:,1], 'o',
label='Contaminants')
plt.plot( ml_data_mags[clasf][:,0], ml_data_mags[clasf][:,1], '*',
label=f'Selected ({"+".join(class2keep)})')
plt.gca().invert_yaxis()
plt.xlabel(f'{bands_selected[0]}') #'-{bands_selected[1]}')
plt.ylabel(bands_selected[1])
plt.legend(fontsize=14)
plt.show()
Introducing data splitting and the Golden rule#
In supervised approaches we want to “teach” the algorithms what they need to learn, before start the predictions. In this case we want the SVM to identify the common properties of the two sub groups (1 including all possible classes used, and 0 as contaminants). However, if we provide all data the algorithm will learn this “by heart” and fit them perfectly (called overfitting), and when new data appear will probably misclassify.
So, what can we do ? We have to split the data and follow the Golden rule.
Indicative recipe for splitting a dataset into the analysis sets. For very large datasets (above 10$^5$ entries), the percentages of validation and test can be substantially smaller. |
What are the different sets for?
Train set \(~~~~~~\rightarrow\) Learn the model parameters
Validation set \(\rightarrow\) Check that the learnt model is not overfitting/underfitting the train set
Test set \(~~~~~~~\rightarrow\) Assess the model performance
The Golden Rule of Machine Learning
Treat the test data as if they come from the future
So let’s go on and split our data now.
NOTE: We will only use the train/test split as our purpose now is not to optimize the model (you will see more on that in ML Basic Practices).
NOTE: You can either use magnitudes (ml_data_mags) or color indeces (ml_data_clrs).
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, ml_labels,
test_size=0.3) #, random_state=42)
print(f'- From {len(ml_objects)} sources:')
print(f' {len(X_train)} (training)')
print(f' {len(X_test)} (test)')
print()
print(f'Test labels: {y_test}')
Questions
What happens when you repeat the process?
How can we solve this?
Where does 42 comes from?
HINTS:
- Use Ctrl+Enter to run a cell without continuing to the next one
- Check the online documentation (train-test_split)
[Spoiler] (click here to expand)
The process is random so every time different objects are selected as train to test.
We can use
random_stateto have the same split all the time (this is mainly for developing and debugging pusposes, as it helps to reproduce the results).Whaat? You do not know?
- shame on you!
- look it up (many ways to do it nowadays!)
Now, let’s use the classifier to fit our training set (X_train) and predict the classes of the test samples (X_test). We will print some metrics to check the performance.
from sklearn import metrics
clf = SVC(kernel='linear', C=1.0) # 1.0 is the default value, must be positive
clf.fit(X_train, y_train)
yhat = clf.predict(X_test)
#print(yhat)
print(f"Classification report:\n\n {metrics.classification_report(y_test, yhat)}")
print(f"Confusion matrix: \n\n {metrics.confusion_matrix(y_test, yhat)}")
but… aaarg, what are all these metrics about ?
Model evaluation metrics - a mini discussion#
There is a number of assessment metrics for the performance of a classifier. We start by introducing the idea of the confusion matrix.
Confusion matrix for classification.(Credit: from EvidentlyAI.com)
Defining some metrics:
Precision, the number of correct predictions: $\( \rm{Precision} = \frac{\rm{True~Positives}}{\rm{Actual~Results}} = \frac{\rm{True~ Positives}}{\rm{True~Positives\,+\,False~Positives}} \)$
Recall, the number of real samples that are correctly predicted: $\( \rm{Recall} = \frac{\rm{True~Positives}}{\rm{Predicted~Results}} = \frac{\rm{True~ Positives}}{\rm{True~Positives + False~Negatives}} \)$
F1-score, the harmonic mean of precisions and recall (for balance): $\( \rm{F1-score} = 2 \times \frac{\rm{Precision}\times\rm{Recall}}{\rm{Precision}+\rm{Recall}}\)$
Accuracy, the fraction of the total correct predictions over the total sample: $\( \rm{Accuracy} = \frac{\rm{True~Positives}\,+\,\rm{True~Negatives}}{\rm{Total}} \)$
Macro averaging, the mean for any metric and category, providing the same weight for all categories (indepedant of their sample).
Weighted averaging: the mean of all metrics by considering the number of samples per class (i.e. it gives higher weights to the classes with larger samples).
Support: the number of samples per class on the test set.
Note 1: You may also encounter the terms sensitivity and specificity which corresponds to the recall of the positive and the negative class, respectively, in binary problems.
Note 2: In astrophysics we use the terms completeness and contamination (see Classification, by Andy Connolly):
Question
What metric of performance shall we maximize, to optimize our model parameters?
It depends on the objective of the model.
Example 1: We want to identify best follow-up candidates of GW sources, without wasting precious time on wrong targets
TP: identified as GW, and it is GW
FP: identified as GW, but not GW
TN: not identified as GW, and not GW
FN: not identified as GW, but is GW
[Spoiler] (click here to expand)
Observing GW candidates means that no time should be spent on FP, so we need to minimize this and maximize Precision.
Example 2: We want to create a catalog of OB stars, so we would like to retrieve the whole population
TP: identified as OB, and it is OB
FP: identified as OB, but not OB
TN: not identified as OB, and not ON
FN: not identified as OB, but is OB
[Spoiler] (click here to expand)
We would like to be complete on the catalog, so we should not miss any of the OB population (even at the expense of some FP). Therefore, we need to minimize FN, and maximize Recall.
An interesting presentation of this topic can be found here.
And now let’s go on with our example… plotting a prettier presentation of the previous confusion matrix.
def plot_confusion_matrix(cm,
target_names,
title='Confusion matrix',
cmap=None,
normalize=True):
"""
given a sklearn confusion matrix (cm), make a nice plot
Arguments
---------
cm: confusion matrix from sklearn.metrics.confusion_matrix
target_names: given classification classes such as [0, 1, 2]
the class names, for example: ['high', 'medium', 'low']
title: the text to display at the top of the matrix
cmap: the gradient of the values displayed from matplotlib.pyplot.cm
see http://matplotlib.org/examples/color/colormaps_reference.html
plt.get_cmap('jet') or plt.cm.Blues
normalize: If False, plot the raw numbers
If True, plot the proportions
Usage
-----
plot_confusion_matrix(cm = cm, # confusion matrix created by
# sklearn.metrics.confusion_matrix
normalize = True, # show proportions
target_names = y_labels_vals, # list of names of the classes
title = best_estimator_name) # title of graph
Citation
---------
http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html
"""
import matplotlib.pyplot as plt
import numpy as np
import itertools
accuracy = np.trace(cm) / float(np.sum(cm))
misclass = 1 - accuracy
if cmap is None:
cmap = plt.get_cmap('Blues')
plt.figure(figsize=(6,6))
plt.imshow(cm, interpolation='nearest', cmap=cmap, alpha=0.5)
# plt.title(title)
cbar = plt.colorbar()
cbar.set_label('# sources', fontsize=16)
cbar.ax.tick_params(labelsize=16) # (fontsize=15)
if target_names is not None:
tick_marks = np.arange(len(target_names))
plt.xticks(tick_marks, target_names, rotation=45, fontsize=15)
plt.yticks(tick_marks, target_names, fontsize=15)
if normalize:
cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
thresh = cm.max() / 1.5 if normalize else cm.max() / 2
for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
if normalize:
plt.text(j, i, "{:0.3f}".format(cm[i, j]),
horizontalalignment="center", color="black", fontsize=14 )
#color="white" if cm[i, j] > thresh else "black")
else:
plt.text(j, i, "{:,}".format(cm[i, j]),
horizontalalignment="center", color="black")
# color="white" if cm[i, j] > thresh else "black")
plt.tight_layout()
plt.ylabel('True label', fontsize=16)
plt.xlabel('Predicted label (accuracy={:0.2f})'.format(accuracy, misclass), fontsize=16)
plt.show()
plot_confusion_matrix( metrics.confusion_matrix( y_test, yhat),
[1, 0],
title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
normalize=False # True returns fraction, False raw numbers
) # YlOrBr
Exercise 2:
Objective: Explore the effect of classification results by changing some input parameters.
Task: Go back to the previous cells (or if you want make new ones and copy the code from above) so that you experiment with the code. What happens if we start adding more classes into the selected one ?
Optional task: What happens when we start changing the \(C\) parameter?
HINT: the default value is 1, so start increasing/decreasing from this value.
reminder()
# enter code here if you want
[Solution below]
# select classes and bands
class2keep = ['RSG', 'WR']
bands_selected = ['V', 'J', '[4.5]']
# process data, pick either mag or clrs
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected, class2keep)
print()
# split data
X_train, X_test, y_train, y_test = train_test_split(ml_data_mags,
ml_labels,
test_size=0.3) #, random_state=42)
print(f'- From {len(ml_objects)} sources:')
print(f' {len(X_train)} (training)')
print(f' {len(X_test)} (test)')
print()
#print(f'Test labels: {y_test}')
# initiate the model, change parameter C
clf = SVC(kernel='linear')
# fit and predict
clf.fit(X_train, y_train)
yhat = clf.predict(X_test)
# plot the confusion matrix
plot_confusion_matrix( metrics.confusion_matrix( y_test, yhat),
[1, 0],
title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
normalize=False # True returns fraction, False raw numbers
) # YlOrBr
[End of solution]
[Spoiler] (click here to expand)
The accuracy drops as more classes results in a more complex problem and the classes may overlap in the feature space (confusing the classifier even more!).
A lower value off C allows for more misclassifications, i.e. more sources are found away from the diagonal in the confusion matrix. As we increase C, the sources start gathering around the diagonal, but we will lose in predictive power in new data.
The multi-class problem#
Now we approach the same problem as a multi-class one, i.e. we are using the SVM as a classifier that can handle all classes simultaneously.
reminder()
class2keep = [] # that means all classes
bands_selected = ['V', 'J', '[4.5]']
NOTE: in this case the list for the classes should be empty ([]) to include all classes.
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected, class2keep)
Now if we print the labels again, we notice that the array is totally different and the multiclass output is evident.
print(ml_labels)
Splitting into train and test sets for multi-class#
We are using again the train-test split approach but considering all classes now.
indices = np.arange(len(ml_labels))
X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, ml_labels,
# .... , .... ,
test_size=0.1) #, random_state=42)
print(f'> From {len(ml_objects)} sources we use {len(X_train)} for training and {len(X_test)} for testing.')
print('\nStatistics per class:')
k = 0
for c in unique_cls:
items_train = np.where( y_train==c )[0]
items_test = np.where( y_test==c )[0]
items_total = np.where( ml_labels==c )[0]
print(f'> For {c} there are {len(items_total)} sources split in {len(items_train)} (train) and {len(items_test)} (test) samples')
Questions
Run train-test split a few times as it is - do you notice anything strange?
HINT: try to reduce the test_size, to become more evident.
How can we correct for the effect in the previous question?
HINT: Check the documentation (train-test_split) and find out which parameters help.
[Spoiler] (click here to expand)
It is possible to get 0 test objects for some classes. This is due to the randomness of the selection and the low number of sources in some of the classes.
Add shuffle=True and stratify ensures that the test sample will keep the demographics of the original one.
indices = np.arange(len(ml_labels))
X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, ml_labels,
shuffle=True, stratify=ml_labels,
test_size=0.1) #, random_state=42)
print(f'> From {len(ml_objects)} sources we use {len(X_train)} for training and {len(X_test)} for testing.')
print('\nStatistics per class:')
k = 0
for c in unique_cls:
items_train = np.where( y_train==c )[0]
items_test = np.where( y_test==c )[0]
items_total = np.where( ml_labels==c )[0]
print(f'> For {c} there are {len(items_total)} sources split in {len(items_train)} (train) and {len(items_test)} (test) samples')
Important take-away message #1
sklearn documentation is your friend.
Let’s produce some results now…
clf2 = SVC(kernel='linear')
clf2.fit(X_train, y_train)
yhat = clf2.predict(X_test)
#print(yhat)
#confmatrix = metrics.confusion_matrix( y_test, yhat)
print(f"Classification report:\n\n {metrics.classification_report(y_test, yhat)}")
print(f"Confusion matrix: \n\n {metrics.confusion_matrix(y_test, yhat)}")
plot_confusion_matrix( metrics.confusion_matrix( y_test, yhat),
unique_cls,
title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
normalize=False # True returns precent, False raw numbers
) # YlOrBr
Questions
How does the result changes with respect to the binary case?
How does the result change with kernel ?
HINT: check sklearn.svm.SVC
[Spoiler] (click here to expand)
We see a drop in the accuracy, because we try to classify more classes which are actually not easilly separated in the feature space. In other words, some classes may have similar magnitudes and colors, which makes it difficult to separate them.
Adding more classes (and bands) increases the dimensionality of the problem, therefore, there is a more complex representation of the input data. Kernels help to fit them more efficiently since they are able to “see” in more dimensions.
Important take-away message #2
Accuracy is not the best metric to use when we have imbalanced datasets.
The Random Forest algorithm#
Decision Tree#
A Decision Tree (DT) is simply a top-to-bottom tree-like structure where each node corresponds to a question (or a set of features more generally) that distinguishes objects to two groups, left and right from the node. A decision tree presents the drawback of learning extremely well the training set. That means that DTs overfit the data and they cannot predict very accurately new data.
Quick introduction to Decision Trees - how to fulfill an everyday need.(Credit: G. Maravelias)
Minimizing what?#
To find the optimal way to partition objects, the algorithm needs a metric to quantify this process. One such metric is the Gini impurity, which calculates the probability of an object being incorrectly classified (or assigned to the wrong class) when chosen randomly. It is defined by the formula
where \(n\) is the number of classes, and \(p_i\) is the probability of an object belonging to a specific class for a given value of the variable (feature) under consideration. These probabilities are summed across all values of the feature and all corresponding classes.
The Gini impurity for a node is determined by the average of these values across all classes, weighted by their frequency (for a detailed presentation).
If all objects in a set belong to the same class, the Gini impurity is 0. If it is 1, the class assignments of the objects are random.
Random Forest#
Random Forest (RF) or Random Decision Trees (Breiman (2001), Machine Learning, 45, 5) is a generalization of the DTs, as it utilises a multitude of decision trees. When RF are used as a classification method, for each input datum the final output is the class (/value) given by the mode of the classes of the individual trees.
Schematic description of the Random Forest classifier.(Credit: G. Maravelias)
RF creates a large number of DTs through random selection of a subset of the training set as well as a random selection of features. This randomness reduces the correlation between the different DTs. Since the DTs have different conditions in their nodes and different overall structures, this diversity yield overall robust predictions.
Once the RF has been trained, the data of an unlabeled source (to be classified) are fed into all DTs of the forest. According to its properties and the nodes in each DT, it follows a specific path which leads to a given class. The final output of the RF (the prediction) is an aggregation of all DTs by means of a majority vote.
The fact that RF combines the prediction for a number of individual trees makes it an ensemble method.
★ Reis, Baron, & Shahaf (2019), AJ, 157, 16 provide an excellent description of RFs (for a two-class problem) and present a probabilistic RF method which takes into account the uncertainties on the data and their labels.
Final remarks on Random Forest#
In machine learning most of the effort is actually spent on the sample selection and, most importantly, on the selection of the features used for the classification (feature engineering).
RF partially overcome the latter problem by training each DT on a different sub-set of features, hence training the algorithm to recognize the features which mostly differentiate the objects.
PROS
No need of scaling or transformation of the initial data.
Implicit feature selection.
Suitable for large datasets with many features.
CONS
Not easily interpretable.
Hyperparameter needs good tuning for high accuracy.
Random Forest in practice#
For the implementation, we will use the sklearn.RandomForestClassifier, whose basic parameter (among others) is n_estimators, i.e. the number of trees (default 100).
Using the same dataset we are now approaching the same problem using the Random Forest (both as a binary and a multiclass classifier).
Exercise 3:
Objective: Apply the Random Forest classifier to our dataset.
Task: Follow the actions below to get to some results:
Complete the train-test split and select magnitudes to work with.
Find the proper function for RF and make predictions.
reminder()
Note: Add any class if you want to use RF as binary classifier, or keep the list class2keep_RF empty for all classes. Add any number of bands to use.
class2keep_RF = []
bands_selected_RF = [...]
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected_RF, class2keep_RF)
# splitting data on magnitudes or colors
#X_train, X_test, y_train, y_test = train_test_split(ml_data_mags,
X_train, X_test, y_train, y_test = train_test_split(...,
...,
shuffle=True, stratify=...,
test_size=...) #, random_state=42)
print(f'> From {len(ml_objects)} sources we use {len(X_train)} for training and {len(X_test)} for testing.')
# check if in binary or multi-label mode
if 1 in ml_labels: # binary
confmat_classes = [1,0]
else:
confmat_classes = unique_cls
print('\nStatistics per class:')
k = 0
for c in unique_cls:
items_train = np.where( y_train==c )[0]
items_test = np.where( y_test==c )[0]
items_total = np.where( ml_labels==c )[0]
print(f'> For {c} there are {len(items_total)} sources split in {len(items_train)} (train) and {len(items_test)} (test) samples')
from sklearn.ensemble import RandomForestClassifier
clfrf = ...
# code to fit and predict here (follow SVM structure)
...
...
print(f"Classification report:\n\n {metrics.classification_report(..., ...)}")
plot_confusion_matrix( metrics.confusion_matrix( ..., ...),
confmat_classes,
title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
normalize=False # True returns percent, False raw numbers
) # YlOrBr
[Solution below]
reminder()
class2keep_RF = []
bands_selected_RF = ['U','V', 'I', 'J', '[3.6]']
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected_RF, class2keep_RF)
# splitting data on magnitudes or colors
#X_train, X_test, y_train, y_test = train_test_split(ml_data_mags,
X_train, X_test, y_train, y_test = train_test_split(ml_data_clrs,
ml_labels,
shuffle=True, stratify=ml_labels,
test_size=0.3) #, random_state=42)
print(f'> From {len(ml_objects)} sources we use {len(X_train)} for training and {len(X_test)} for testing.')
# check if in binary or multi-label mode
if 1 in ml_labels: # binary
confmat_classes = [1,0]
else:
confmat_classes = unique_cls
print('\nStatistics per class:')
k = 0
for c in unique_cls:
items_train = np.where( y_train==c )[0]
items_test = np.where( y_test==c )[0]
items_total = np.where( ml_labels==c )[0]
print(f'> For {c} there are {len(items_total)} sources split in {len(items_train)} (train) and {len(items_test)} (test) samples')
from sklearn.ensemble import RandomForestClassifier
clfrf = RandomForestClassifier()
# code to fit and predict here (follow SVM structure)
clfrf.fit(X_train, y_train)
yhat = clfrf.predict(X_test)
print(f"Classification report:\n\n {metrics.classification_report(y_test, yhat)}")
print(f"Confusion matrix: \n\n {metrics.confusion_matrix(y_test, yhat)}")
plot_confusion_matrix( metrics.confusion_matrix( y_test, yhat),
confmat_classes,
title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
normalize=False # True returns percent, False raw numbers
) # YlOrBr
[End of solution]
Questions
Why the results change by re-running it ?
What is the difference on the accuracy of the algorithm if we use the colors instead of the magnitudes? Why do it?
[Spoiler] (click here to expand)
RF as the name suggest create always a different set of trees with random eselection of features and data. Consequently, the RF from one run to the other is not the same and there are small variations. However, the results are robust because they are averaging the results from all trees.
Depending on the classes and bands selected the accuracy will (slightly) change. RF are rather insensitive to the input data (scaled or not).
However, colors are distance-independent features, which means that a model build with these features could be applicable to other galaxies also. Colors are distance-independent features, which means that the model could be applicable to other galaxies also. On the contrary, if we use only magnitudes this would mess up the prediction as with distance the sources become fainter.
Classification algorithms overview#
This is a presentation of the set of sklearn algorithms with toy datasets, and then describe some of them.
This serves as a showcase of the available methods and how they compare. You can easily adapt any of these methods to the following examples or your own problems.
Overview of the clustering algorithms in sklearn
Question
Explore the results above - what differences do you notice?
[Spoiler] (click here to expand)
An indicative list of noticable differences:
- There is not a single best algorithm.
- Some algorithms are more able to separate classes.
- Some algorithms have limited ability (e.g. the linear SVM with non-linear distributed data).
Recap: steps to follow when building a classification model#
1. Examine / visualize / preprocessing data
2. Select features
3. Select algorithm
4. Split data and follow Golden rule
5. Select metrics
6. Examine results and re-iterate
###EOF