Credits
AstroStat Academy logo This notebook contains original work by the authors unless stated otherwise. Any external material is properly credited to its sources.
References to papers, datasets, and software are acknowledged. Original content is licensed under the GNU General Public License v3.0 (GNU GPLv3).

Machine Learning: Classification#

In this session we focus on another important application for machine learning, the Classification.
We will provide a simple problem (with multiple intermediate tasks):

  • to demonstrate basic visualization of the data

  • to show the basic steps in an application of an algorith (or two!)

  • to present the necessary performance metrics

  • to investigate differences in binary and multiclass approaches

\(\rightarrow\) The target is to have an practical application example of an algorith to understand the approach and (hopefully!) use it for your own research.

Setup#

import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns

from IPython.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))

import warnings
warnings.filterwarnings(action='ignore')

import requests

url = "https://raw.githubusercontent.com/AstroStat-Academy/assets-public/main/colab/clone_and_cd_colab.py"
colab = requests.get(url).text
exec(colab)

url = "https://raw.githubusercontent.com/AstroStat-Academy/assets-public/main/styles/plot_style.py"
style = requests.get(url).text
exec(style)

Types of Classification#

Flavors#

Classification can come in two major flavors, based on the type of intervention by the user:

  • Unsupervised: The classification is defined “unsupervised” when the user does not provide labels during the training process and the machine learns the definition of each class from the data. This is exactly the case of “Clustering” that we saw in the previous session.

In practice, the machine learns to cluster objects with similar properties.

  • Supervised: The classification is defined “supervised” when the user provides a label for each object in the training set. In this case, the idea is that we can train the model to associate the label with some given characteristics of the training data.

In practice, the machine learns to find similarities between objects with the same label.

Terminology#

Keep in mind that:

  • class or label are used interchangeably

  • samples (i.e. the items) can be of any arbitrary type (e.g. a string, an image, or numbers) - however, in astronomy we mainly refer to objects or sources, since samples correspond to their collections
    (so… samples in ml terminology and objects/sources in astronomy!)

  • the objects/samples are categorized based on their properties or features (the actual information that we pass to the machine, e.g. the pixel intensities in the case of an image).

The sample for our classification example#

To properly classify a star we need a spectrum. However, this is time consuming and limited to one (single slit spectroscopy) or a few tens of sources (multi-object spectroscopy). On the contrary, imaging in various different filters can be done easily and for thousands of sources per image. Photometry at different wavelengths result in a very low-resolution “spectrum”.

The filter and detector transmission curves for a number of different systems, along with indicative spectra of Vega, the Sun, and a M5 giant.
(Fig 3. from Girardi et al. (2002))

In this example we are using a set of photometric measurements (from optical to mid-IR bands) for a sample of massive evolved stars in the Large Magellanic Cloud (based on these works: Bonanos et al. (2009) AJ, 138, 1003, Neugent et al. (2012), ApJ, 749, 177, and Davies, Crowther & Beasor (2018), MNRAS, 478, 313).


A slightly modified version of Fig. 1 from Massey et al. (2013))

Our aim is to use a method that will help us distinguish different classes of objects. For our purposes we will use OBA stars (main-sequense objects), OBAe (a subcategory of OBA stars with circumstellar disks and emission lines), Wolf-Rayet stars (hot evolved stars with strong stellar winds that actually strip their envelopes), Yellow and Red supergiants (evolved massive stars). For convenience we will use OBA, OBAe, WR, YSG, and RSG, respectively, as labels.

★ A similar, but more elaborated, implementation is performed in Maravelias et al. (2022), A&A, 666, 122.

Load and examine data#

dfile = "data/LMC_phot_data.csv"

data = pd.read_csv(dfile)

#examine data
print("Let us see what we have:\n")
print(data)
print("-"*25)
print("The column names:")
print(data.columns)
print("-"*25)
print("Let's print the spectral types only:")
print(data['SpT'])

Let’s now explore the statistics per group (class) of sources.

unique_cls = data['SpT'].unique()

print("> Unique classes:", unique_cls)
print("="*25)
print("> SUMMARY of loaded data:")
print("="*25)

for sptype in unique_cls:
    number = data[data['SpT']==sptype].shape[0]
    print(f"{sptype:-<6s}--> {number:>3} stars")
bands = [b for b in data.columns if 'e_' not in b][3:-1]

def reminder():
    """ 
    A simple function to print all bands
    and classes available.
    """
    print('Available bands to use: ')
    print(','.join(bands))
    print('-'*25)
    print('Available classes to use:')
    print(','.join(unique_cls))
results = []

for spt in unique_cls:
    data_spt = data[data['SpT']==spt]
    number_spt = data_spt.shape[0]
#    print(spt, number_spt)

    row = {
        'Class': spt,
        'Total': number_spt
    }
    
    for b in bands:
        data_spt_bnd = data_spt[b]
        number_nn = data_spt_bnd.notna().sum()
        fraction = number_nn/number_spt
        
        row[b] = fraction * 100
            
#        print(b, fraction)
        
#    print(row)
    results.append(row)
    
phot_data_per = pd.DataFrame(results)

print(f'Available photometry for: {", ".join(bands)}')
print("\nNumber of stars per band (in %)\n")

phot_data_per.round(2)

Question

Check the table, what do you notice here ?

[Spoiler] (click here to expand)

Many sources come without measurements across all filters. This is a particular problem for algortihms that need values to work with.

This problem of ”missing values” is actually the norm than the exception. There are ways to tackle this, e.g. by taking the mean values of the corresponding filters or more elaborated techinques. Check sklearn’s imputation of missing values for more.

Visualize data - select features#

reminder()

Let’s plot the data.

def selmags( band1, band2, cls):
    """
    Function to select sources of a specific
    spectral class (cls) and return the magnitudes
    that correspond to bands 1 and 2.
        """
    # select all rows of the "cls" type
    data_cls = data[data['SpT']==cls]
#    print(band1, band2)
    mag1 = data_cls[band1].values
    mag2 = data_cls[band2].values
    
    return mag1, mag2


def plotmags( selected_bnd, selected_spt):
    """ Function to plot magnitudes and colors 
    for sources corresponding to the selected classes.
    NOTEs: 
        - selected_bnd: a list of four magnitudes
        - selected_spt: a list of selected spectral types
    
    Output: two sets of plots.
    """
    fig, ax = plt.subplots(2,2, figsize=(10, 10))
    legend_fs = 16

    print('- plot1:')
    for s in selected_spt: 
        plt1 = selmags( selected_bnd[0], selected_bnd[1], s)
        
        # calculate the number of sources to be plotted
        # both arrays have nan values, and plot is smart enough to 
        # not plot those points.
        # but we need to know what we are missing!
        mask1 = np.isfinite(plt1[0]) & np.isfinite(plt1[1])
        n_points_1_kept = np.count_nonzero(mask1)  
        print(f'-- {s}: excluding {len(plt1[0])-n_points_1_kept} out of {len(plt1[0])} sources ({(len(plt1[0])-n_points_1_kept)/len(plt1[0])*100:.1f}%)')


        ax[0,0].plot(plt1[0], plt1[1], 'o', label=f'{s}: {n_points_1_kept}')
        ax[0,0].set_xlabel(f'{selected_bnd[0]}') #'-{band1_2}')
        ax[0,0].set_ylabel(selected_bnd[1])
        ax[0,0].invert_yaxis()
        ax[0,0].invert_xaxis()
        ax[0,0].legend(fontsize=legend_fs)

        ax[0,1].plot(plt1[0]-plt1[1], plt1[1], 'o', label=f'{s}: {n_points_1_kept}')
        ax[0,1].set_xlabel(f'{selected_bnd[0]}-{selected_bnd[1]}')
        ax[0,1].set_ylabel(selected_bnd[1])
        ax[0,1].invert_yaxis()
        ax[0,1].legend(fontsize=legend_fs)


    print()
    print('- plot2:')        
    for s in selected_spt:
        plt2 = selmags( selected_bnd[2], selected_bnd[3], s)

        mask2 = np.isfinite(plt2[0]) & np.isfinite(plt2[1])
        n_points_2_kept = np.count_nonzero(mask2)
        print(f'-- {s}: excluding {len(plt2[0])-n_points_2_kept} out of {len(plt2[0])} sources ({(len(plt2[0])-n_points_2_kept)/len(plt2[0])*100:.1f}%)')

        ax[1,0].plot(plt2[0], plt2[1], 'o', label=f'{s}: {n_points_2_kept}')
        ax[1,0].set_xlabel(f'{selected_bnd[2]}') #'-{band2_2}')
        ax[1,0].set_ylabel(selected_bnd[3])
        ax[1,0].invert_yaxis()
        ax[1,0].invert_xaxis()    
        ax[1,0].legend(fontsize=legend_fs)

        ax[1,1].plot(plt2[0]-plt2[1], plt2[1], 'o', label=f'{s}: {len(plt2[0])}')
        ax[1,1].set_xlabel(f'{selected_bnd[2]}-{selected_bnd[3]}')
        ax[1,1].set_ylabel(selected_bnd[3])
        ax[1,1].invert_yaxis()
        ax[1,1].legend(fontsize=legend_fs)


    plt.show()

Exercise 1:

Objective: Data visualization and exploration - which features to pick ?

Task: Change the bands you use to plot the CMDs as well as the number of classes to consider, and try to answer:

  1. What happens if you start increasing the number of classes to consider ?

  2. How the selection of bands (e.g. adding more bands) influence the numbr of objects to plot (or get rejected) ?

  3. Would you prefer to use combinations with few or more objects ?

plotmags(['B', 'V', 'H','K'], ['WR', 'OBA'])

[Spoiler] (click here to expand)

  1. The task becomes more difficult as classes start to overlap. Some combinations may prove better to separate some classes but there is not a single optimal one.

  2. In order to make the plots we need sources with measuremets in both bands. By enforcing this we actually exclude objects with missing data. That results in decreasing number of sources.

  3. In principle we would like to take advantage of all available sources. That give us a better sampling of their parameters’ space. If only few sources remain then we actually lose information.

Hence, we would like to maximize the number of sources used. In problems with already small numbers (like in this example) this can be crucial.

A working combination:

plotmags(['V', 'J', 'K','[4.5]'], ['RSG', 'OBA'])

What have we achieved so far?

To examine our data and select which features to use.

Now is the time to select an algorithm.

The Support Vector Machine (SVM) algorithm#

The boundary (in 2D)#

Support vector machine (SVM) is a way of choosing a decision boundary between different classes.

The classification boundary is provided by the line (in 2D) that is found in the middle of the two lines defined by the closest points of each class, which are called support vectors. The margin is the distance between these two parallel lines. By maximizing this margin, the SVM achieves the best possible separation with the largest safety zone between classes.

Left: The dashed line separating two classes (red and green). Right: The closest points to the line from each class constitute the "tip" of the support vectors.

The left panel of the figure above shows two different classes distributing in a scatter plot according to variable \(x_1\) and \(x_2\). The right panel explains the origin of the name support vectors: the closest points support the (solid) lines equally distant from the decision (dashed) line.

Infinite possible boundaries can separate the two classes. SVM algorithms find the one that maximizes the distance between the lines defined by the support vectors

Minimizing the loss function - again !#

The supported (solid) lines can be defined as:

\[ w \cdot x + b = +1 \]
\[ w \cdot x + b = -1 \]

where x is the coordinate on the (x1, x2) plane, w is a 2\(\times\)1 matrix and b a scalar. It turns out that these lines are separated by a distance 2 / ||w||. Finding the ideal classification boundary, i.e. the one maximizing the distance, is therefore a problem of minimizing ||w||

\[ \begin{align}\begin{aligned} min(Loss) = min(||w||) $$.\\And... this is what SVM algorithms do!\\&#9733; For a complete mathematical formulation, consult the [Idiot’s guide to Support vector machines, by Robert Berwick]( http://web.mit.edu/6.034/wwwbob/svm-notes-long-08.pdf).\\## Separatable classes (or not)\\We cannot always assume that 2 classes are separable without "contamination". That is why SVM algorithms includes a tunable parameter ($C$) which penalizes misclassifications.\\ <div style="text-align: center;"> <img src="images/svm-parameter-c-example.png" width=800> The effect of <i>C</i> parameter in the misclassifications.<br> (Credit: <a href="https://learnopencv.com/svm-using-scikit-learn-in-python/" target="_blank" rel="noopener noreferrer">SVM: What makes it superior to the Maximal-Margin and Support Vector Classifiers?, by Shivam Sharma</a>) </img> </div> - Small $C$ &#8594; wide margin &#8594; allows more misclassification <br> - Large $C$ &#8594; narrow margin &#8594; allows less misclassification \\The SVM finds the line that maximizes the margin, and indirectly minimizes the misclassifications. However, SVM is not designed to minimize the contamination _per se_.\\ ## Multiple classes\\The SVM method can be applied for multiple classes as well.\\<center><img src="images/svm_many_classes.png" width=400> SVM applied to 3 different classes.<br> </center>\\## Multiple dimensions\\In the case of a sample characterized by three parameters (X, Y, Z), the scatter plot vecomes a 3D plot. Then, the boundary between the classes is not a line but a plane. Because of the fact that the method can be extrapolated at N-dimensions, the boundary can be generilized as a *hyperplane*.\\<img src="images/svm_3d.png" width=400> <center> Support vector machine applied for 3D features and three classes. </center>\\## Non-linear boundaries\\Sometimes, linear boundaries may not be optimal and a non-linear SVM should be used instead. The left panel of the figure above shows an 2D scatter plot of two different classes (e.g. red and green stars with different radii and temperatures) which cannot be linearly separated.\\In order to find non-linear boundaries we can tackle the problem in an higher dimensional space. We use a process called **kernelization**, which consists in using a kernel function to attribute to our data a value in the additional dimension. Then, we draw the decision hyperplane into this higher dimensional space.\\The central panel of the figure shows that once the 2D data are mapped to a 3D space by attributing a $z$ value through a Gaussian-like function, the classes are easily separable by a 3D hyperplane. Projecting back the plane in 2D, we obtain the non-linear boundary (right panel).\\<img src="images/kernel.png" width=800> <center> When no linear boundaries can be used the SVM method can be applied by using kernel. </center>\\## Choosing the kernel function\\Useful kernel functions [shall satisfy specific conditions](https://en.wikipedia.org/wiki/Kernel_method), so that in practice only a few are used. In the previous example, the Gaussian Radial Basis Function is used:\\$$K(x,y) = e^{-\gamma(x-y)^2}\end{aligned}\end{align} \]

where \(\gamma\) is a hyperparameter which shall be learned (in our example we use an arbitrary value but in principle we should use cross-validation methods).

Final remarks on SVM#

Pros

  • Good at dealing with high dimensional data

  • Works well on small data sets

Cons

  • Picking the right kernel and parameters can be computationally intensive

  • It suffers from contamination

★ For further information on SVM, consult Support Vector Machine - Classification, by Saed Sayad.

SVM in practice#

The binary problem#

In this case we examine a binary classification problem where we select one class (or more that will group into a single one) and the rest classes as contaminants. The purpose is to check if we can separate efficiently these two classes.

def process_data( bands2use, binary_classes2use  ):
    """
    Process input data to return arrays 
    of magnitudes and (consecutive) colors
    based on the input bands (band2use).

    Option to prepare data for binary classification
    if binary_classes2use contains classes or not.
    
    Selected objects are assigned 1, while the contaminants 0.
    
    """

    data_diff = -data[bands2use].diff(axis=1).iloc[:, 1:]   
    data_diff.columns = [f"{c1}-{c2}" for c1, c2 in zip(bands2use[:-1], bands2use[1:])]
    diff_columns = data_diff.columns.values
#    print(data_diff)
#    print(data_diff.columns.values)

    working = pd.concat([data[['Name']+bands2use+['SpT']], data_diff], axis=1)
#    print(working)
    
    
    if len(binary_classes2use)!=0:
        working['keep'] = working['SpT'].isin(binary_classes2use).astype(int)
    else:
        working['keep'] = working['SpT']
        
    # remove all nan rows
    working_clean = working.dropna()
    

    print(f'# stars with mags in:\n',
          f'{",".join([bb for bb in bands2use])}'
         )
    print("=========================")
    print("Type    initial    final ")
    print("-------------------------")
       
    for s in unique_cls:
        print(
            f"{s:<4}  {(working['SpT']==s).sum():>9}",
            f"{(working_clean['SpT']==s).sum():>8}"
        )
    print('-'*24)
    print(f'TOTAL:  {working.shape[0]:>7}  {working_clean.shape[0]:>7}') 

    if len(binary_classes2use)!=0:
        print('='*24)
        print(f'classifying:  {(working_clean['keep']==1).sum():>10}') 
        print(f'contaminants: {(working_clean['keep']==0).sum():>10}') 
        
        data_labels = working_clean['keep'].to_numpy()
    
    else:
        data_labels = working_clean['SpT'].to_numpy()
    
#    print(working_clean[diff_columns])
    
    data_mags = working_clean[bands2use].to_numpy()
    data_clrs = working_clean[diff_columns].to_numpy() 
    data_objects = working_clean['Name'].to_numpy()
          
    return data_mags, data_clrs, data_objects, data_labels  
reminder()

Select here the class(es) you would like to distinguish from the rest, along with the bands to use.

class2keep = ['RSG']
# Select the bands you want to use here:
bands_selected = ['V', 'J', '[4.5]'] 
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected, class2keep)  

NOTE: the process_data() function examines and keeps only the sources with values across all bands. In other words, we remove sources with missing values (according to the bands selected).

Let’s print the labels to see how they look like.

print(ml_labels)

NOTE: If you have more than 2 bands selected the following plot will use the first two. Modify accordingly to plot other combinations.

fig = plt.figure(figsize=(8,8))

conts = np.where( ml_labels==0 )[0]
clasf = np.where( ml_labels!=0 )[0]

plt.plot( ml_data_mags[conts][:,0], ml_data_mags[conts][:,1], 'o', 
             label='Contaminants')
plt.plot( ml_data_mags[clasf][:,0], ml_data_mags[clasf][:,1], '*', 
             label=f'Selected ({"+".join(class2keep)})')
plt.gca().invert_yaxis()

plt.xlabel(f'{bands_selected[0]}') #'-{bands_selected[1]}')
plt.ylabel(bands_selected[1])
plt.legend(fontsize=14)
plt.show()

Introducing data splitting and the Golden rule#

In supervised approaches we want to “teach” the algorithms what they need to learn, before start the predictions. In this case we want the SVM to identify the common properties of the two sub groups (1 including all possible classes used, and 0 as contaminants). However, if we provide all data the algorithm will learn this “by heart” and fit them perfectly (called overfitting), and when new data appear will probably misclassify.

So, what can we do ? We have to split the data and follow the Golden rule.


Indicative recipe for splitting a dataset into the analysis sets.
For very large datasets (above 10$^5$ entries), the percentages of validation and test can be substantially smaller.

What are the different sets for?

  • Train set \(~~~~~~\rightarrow\) Learn the model parameters

  • Validation set \(\rightarrow\) Check that the learnt model is not overfitting/underfitting the train set

  • Test set \(~~~~~~~\rightarrow\) Assess the model performance

The Golden Rule of Machine Learning

Treat the test data as if they come from the future

So let’s go on and split our data now.

NOTE: We will only use the train/test split as our purpose now is not to optimize the model (you will see more on that in ML Basic Practices).

NOTE: You can either use magnitudes (ml_data_mags) or color indeces (ml_data_clrs).

from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, ml_labels, 
                        test_size=0.3) #, random_state=42) 

print(f'- From {len(ml_objects)} sources:')
print(f'   {len(X_train)} (training)')
print(f'   {len(X_test)} (test)') 
print()
print(f'Test labels: {y_test}')

Questions

  1. What happens when you repeat the process?

  2. How can we solve this?

  3. Where does 42 comes from?


HINTS:
  • Use Ctrl+Enter to run a cell without continuing to the next one
  • Check the online documentation (train-test_split)

[Spoiler] (click here to expand)

  1. The process is random so every time different objects are selected as train to test.

  2. We can use random_state to have the same split all the time (this is mainly for developing and debugging pusposes, as it helps to reproduce the results).

  3. Whaat? You do not know?

  • shame on you!
  • look it up (many ways to do it nowadays!)

Now, let’s use the classifier to fit our training set (X_train) and predict the classes of the test samples (X_test). We will print some metrics to check the performance.

from sklearn import metrics

clf = SVC(kernel='linear', C=1.0) # 1.0 is the default value, must be positive 
clf.fit(X_train, y_train)
yhat = clf.predict(X_test)
#print(yhat)

print(f"Classification report:\n\n {metrics.classification_report(y_test, yhat)}") 
print(f"Confusion matrix: \n\n {metrics.confusion_matrix(y_test, yhat)}")

but… aaarg, what are all these metrics about ?

Model evaluation metrics - a mini discussion#

There is a number of assessment metrics for the performance of a classifier. We start by introducing the idea of the confusion matrix.

Confusion matrix for classification.
(Credit: from EvidentlyAI.com)

Defining some metrics:

  • Precision, the number of correct predictions: $\( \rm{Precision} = \frac{\rm{True~Positives}}{\rm{Actual~Results}} = \frac{\rm{True~ Positives}}{\rm{True~Positives\,+\,False~Positives}} \)$

  • Recall, the number of real samples that are correctly predicted: $\( \rm{Recall} = \frac{\rm{True~Positives}}{\rm{Predicted~Results}} = \frac{\rm{True~ Positives}}{\rm{True~Positives + False~Negatives}} \)$

  • F1-score, the harmonic mean of precisions and recall (for balance): $\( \rm{F1-score} = 2 \times \frac{\rm{Precision}\times\rm{Recall}}{\rm{Precision}+\rm{Recall}}\)$

  • Accuracy, the fraction of the total correct predictions over the total sample: $\( \rm{Accuracy} = \frac{\rm{True~Positives}\,+\,\rm{True~Negatives}}{\rm{Total}} \)$

  • Macro averaging, the mean for any metric and category, providing the same weight for all categories (indepedant of their sample).

  • Weighted averaging: the mean of all metrics by considering the number of samples per class (i.e. it gives higher weights to the classes with larger samples).

  • Support: the number of samples per class on the test set.


Note 1: You may also encounter the terms sensitivity and specificity which corresponds to the recall of the positive and the negative class, respectively, in binary problems.

Note 2: In astrophysics we use the terms completeness and contamination (see Classification, by Andy Connolly):

\[ \rm{completeness} = \frac{\rm{True~Positives}}{\rm{All~real~Positives}} = \frac{True~Positives}{True~Positives + False~Negatives} = recall\]
\[ \rm{contamination} = \frac{False~Positives}{All~detected~Positives} = \frac{False~Positives}{True~Positives + False~Positives}\]

Question

What metric of performance shall we maximize, to optimize our model parameters?
It depends on the objective of the model.

Example 1: We want to identify best follow-up candidates of GW sources, without wasting precious time on wrong targets

TP: identified as GW, and it is GW
FP: identified as GW, but not GW
TN: not identified as GW, and not GW
FN: not identified as GW, but is GW

[Spoiler] (click here to expand)

Observing GW candidates means that no time should be spent on FP, so we need to minimize this and maximize Precision.

Example 2: We want to create a catalog of OB stars, so we would like to retrieve the whole population

TP: identified as OB, and it is OB
FP: identified as OB, but not OB
TN: not identified as OB, and not ON
FN: not identified as OB, but is OB

[Spoiler] (click here to expand)

We would like to be complete on the catalog, so we should not miss any of the OB population (even at the expense of some FP). Therefore, we need to minimize FN, and maximize Recall.

An interesting presentation of this topic can be found here.


And now let’s go on with our example… plotting a prettier presentation of the previous confusion matrix.

def plot_confusion_matrix(cm,
                          target_names,
                          title='Confusion matrix',
                          cmap=None,
                          normalize=True):
    """
    given a sklearn confusion matrix (cm), make a nice plot

    Arguments
    ---------
    cm:           confusion matrix from sklearn.metrics.confusion_matrix

    target_names: given classification classes such as [0, 1, 2]
                  the class names, for example: ['high', 'medium', 'low']

    title:        the text to display at the top of the matrix

    cmap:         the gradient of the values displayed from matplotlib.pyplot.cm
                  see http://matplotlib.org/examples/color/colormaps_reference.html
                  plt.get_cmap('jet') or plt.cm.Blues

    normalize:    If False, plot the raw numbers
                  If True, plot the proportions
                  
                  
    Usage
    -----
    plot_confusion_matrix(cm           = cm,                  # confusion matrix created by
                                                              # sklearn.metrics.confusion_matrix
                          normalize    = True,                # show proportions
                          target_names = y_labels_vals,       # list of names of the classes
                          title        = best_estimator_name) # title of graph

    Citation
    ---------
    http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html

    """
    import matplotlib.pyplot as plt
    import numpy as np
    import itertools

    accuracy = np.trace(cm) / float(np.sum(cm))
    misclass = 1 - accuracy

    if cmap is None:
        cmap = plt.get_cmap('Blues')

    plt.figure(figsize=(6,6))
    plt.imshow(cm, interpolation='nearest', cmap=cmap, alpha=0.5)
#    plt.title(title)
    cbar = plt.colorbar()
    cbar.set_label('# sources', fontsize=16)
    cbar.ax.tick_params(labelsize=16) # (fontsize=15)
   

    if target_names is not None:
        tick_marks = np.arange(len(target_names))
        plt.xticks(tick_marks, target_names, rotation=45, fontsize=15)
        plt.yticks(tick_marks, target_names, fontsize=15)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]


    thresh = cm.max() / 1.5 if normalize else cm.max() / 2
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        if normalize:
            plt.text(j, i, "{:0.3f}".format(cm[i, j]),
                     horizontalalignment="center", color="black", fontsize=14 )
                     #color="white" if cm[i, j] > thresh else "black")
        else:
            plt.text(j, i, "{:,}".format(cm[i, j]),
                     horizontalalignment="center", color="black") 
#                     color="white" if cm[i, j] > thresh else "black")


    plt.tight_layout()
    plt.ylabel('True label', fontsize=16)
    plt.xlabel('Predicted label (accuracy={:0.2f})'.format(accuracy, misclass), fontsize=16)
    plt.show()
plot_confusion_matrix( metrics.confusion_matrix( y_test, yhat),
                      [1, 0],
                      title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
                      normalize=False  # True returns fraction, False raw numbers
                      ) # YlOrBr

Exercise 2:

Objective: Explore the effect of classification results by changing some input parameters.

Task: Go back to the previous cells (or if you want make new ones and copy the code from above) so that you experiment with the code. What happens if we start adding more classes into the selected one ?

Optional task: What happens when we start changing the \(C\) parameter?

HINT: the default value is 1, so start increasing/decreasing from this value.

reminder()
# enter code here if you want
# select classes and bands
class2keep = ['RSG', 'WR']
bands_selected = ['V', 'J', '[4.5]'] 

# process data, pick either mag or clrs
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected, class2keep)  
print() 

# split data
X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, 
                                                    ml_labels, 
                        test_size=0.3) #, random_state=42) 

print(f'- From {len(ml_objects)} sources:')
print(f'   {len(X_train)} (training)')
print(f'   {len(X_test)} (test)') 
print()
#print(f'Test labels: {y_test}')

# initiate the model, change parameter C
clf = SVC(kernel='linear') 
# fit and predict
clf.fit(X_train, y_train)
yhat = clf.predict(X_test)

# plot the confusion matrix
plot_confusion_matrix( metrics.confusion_matrix( y_test, yhat),
                      [1, 0],
                      title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
                      normalize=False  # True returns fraction, False raw numbers
                      ) # YlOrBr

[Spoiler] (click here to expand)

  1. The accuracy drops as more classes results in a more complex problem and the classes may overlap in the feature space (confusing the classifier even more!).

  2. A lower value off C allows for more misclassifications, i.e. more sources are found away from the diagonal in the confusion matrix. As we increase C, the sources start gathering around the diagonal, but we will lose in predictive power in new data.

The multi-class problem#

Now we approach the same problem as a multi-class one, i.e. we are using the SVM as a classifier that can handle all classes simultaneously.

reminder()
class2keep = [] # that means all classes
bands_selected = ['V', 'J', '[4.5]'] 

NOTE: in this case the list for the classes should be empty ([]) to include all classes.

ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected, class2keep)  

Now if we print the labels again, we notice that the array is totally different and the multiclass output is evident.

print(ml_labels)

Splitting into train and test sets for multi-class#

We are using again the train-test split approach but considering all classes now.

indices = np.arange(len(ml_labels))
X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, ml_labels,
                                # .... , .... , 
                                test_size=0.1) #, random_state=42) 

print(f'> From {len(ml_objects)} sources we use {len(X_train)} for training and {len(X_test)} for testing.') 

print('\nStatistics per class:')
k = 0
for c in unique_cls:
    items_train = np.where( y_train==c )[0]
    items_test  = np.where( y_test==c )[0]
    items_total = np.where( ml_labels==c )[0]
    print(f'> For {c} there are {len(items_total)} sources split in {len(items_train)} (train) and {len(items_test)} (test) samples')

Questions

  1. Run train-test split a few times as it is - do you notice anything strange?

HINT: try to reduce the test_size, to become more evident.

  1. How can we correct for the effect in the previous question?

HINT: Check the documentation (train-test_split) and find out which parameters help.

[Spoiler] (click here to expand)

  1. It is possible to get 0 test objects for some classes. This is due to the randomness of the selection and the low number of sources in some of the classes.

  2. Add shuffle=True and stratify ensures that the test sample will keep the demographics of the original one.

indices = np.arange(len(ml_labels))
X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, ml_labels,
                                shuffle=True, stratify=ml_labels, 
                                test_size=0.1) #, random_state=42) 

print(f'> From {len(ml_objects)} sources we use {len(X_train)} for training and {len(X_test)} for testing.') 

print('\nStatistics per class:')
k = 0
for c in unique_cls:
    items_train = np.where( y_train==c )[0]
    items_test  = np.where( y_test==c )[0]
    items_total = np.where( ml_labels==c )[0]
    print(f'> For {c} there are {len(items_total)} sources split in {len(items_train)} (train) and {len(items_test)} (test) samples')

Important take-away message #1

sklearn documentation is your friend.

Let’s produce some results now…

clf2 = SVC(kernel='linear') 
clf2.fit(X_train, y_train)
yhat = clf2.predict(X_test)
#print(yhat)

#confmatrix = metrics.confusion_matrix( y_test, yhat)
print(f"Classification report:\n\n {metrics.classification_report(y_test, yhat)}") 
print(f"Confusion matrix: \n\n {metrics.confusion_matrix(y_test, yhat)}")

plot_confusion_matrix( metrics.confusion_matrix( y_test, yhat),
                      unique_cls,
                      title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
                      normalize=False  # True returns precent, False raw numbers
                      ) # YlOrBr

Questions

  1. How does the result changes with respect to the binary case?

  2. How does the result change with kernel ?

HINT: check sklearn.svm.SVC

[Spoiler] (click here to expand)

  1. We see a drop in the accuracy, because we try to classify more classes which are actually not easilly separated in the feature space. In other words, some classes may have similar magnitudes and colors, which makes it difficult to separate them.

  2. Adding more classes (and bands) increases the dimensionality of the problem, therefore, there is a more complex representation of the input data. Kernels help to fit them more efficiently since they are able to “see” in more dimensions.

Important take-away message #2

Accuracy is not the best metric to use when we have imbalanced datasets.

The Random Forest algorithm#

Decision Tree#

A Decision Tree (DT) is simply a top-to-bottom tree-like structure where each node corresponds to a question (or a set of features more generally) that distinguishes objects to two groups, left and right from the node. A decision tree presents the drawback of learning extremely well the training set. That means that DTs overfit the data and they cannot predict very accurately new data.

Quick introduction to Decision Trees - how to fulfill an everyday need.
(Credit: G. Maravelias)

Minimizing what?#

To find the optimal way to partition objects, the algorithm needs a metric to quantify this process. One such metric is the Gini impurity, which calculates the probability of an object being incorrectly classified (or assigned to the wrong class) when chosen randomly. It is defined by the formula

\[ Gini = 1 - \sum\limits_{i=1}^{n} (p_i)^2 \]

where \(n\) is the number of classes, and \(p_i\) is the probability of an object belonging to a specific class for a given value of the variable (feature) under consideration. These probabilities are summed across all values of the feature and all corresponding classes.

The Gini impurity for a node is determined by the average of these values across all classes, weighted by their frequency (for a detailed presentation).

If all objects in a set belong to the same class, the Gini impurity is 0. If it is 1, the class assignments of the objects are random.

Random Forest#

Random Forest (RF) or Random Decision Trees (Breiman (2001), Machine Learning, 45, 5) is a generalization of the DTs, as it utilises a multitude of decision trees. When RF are used as a classification method, for each input datum the final output is the class (/value) given by the mode of the classes of the individual trees.

Schematic description of the Random Forest classifier.
(Credit: G. Maravelias)

RF creates a large number of DTs through random selection of a subset of the training set as well as a random selection of features. This randomness reduces the correlation between the different DTs. Since the DTs have different conditions in their nodes and different overall structures, this diversity yield overall robust predictions.

Once the RF has been trained, the data of an unlabeled source (to be classified) are fed into all DTs of the forest. According to its properties and the nodes in each DT, it follows a specific path which leads to a given class. The final output of the RF (the prediction) is an aggregation of all DTs by means of a majority vote.

The fact that RF combines the prediction for a number of individual trees makes it an ensemble method.

★ Reis, Baron, & Shahaf (2019), AJ, 157, 16 provide an excellent description of RFs (for a two-class problem) and present a probabilistic RF method which takes into account the uncertainties on the data and their labels.

Final remarks on Random Forest#

In machine learning most of the effort is actually spent on the sample selection and, most importantly, on the selection of the features used for the classification (feature engineering).

RF partially overcome the latter problem by training each DT on a different sub-set of features, hence training the algorithm to recognize the features which mostly differentiate the objects.

PROS

  • No need of scaling or transformation of the initial data.

  • Implicit feature selection.

  • Suitable for large datasets with many features.

CONS

  • Not easily interpretable.

  • Hyperparameter needs good tuning for high accuracy.

Random Forest in practice#

For the implementation, we will use the sklearn.RandomForestClassifier, whose basic parameter (among others) is n_estimators, i.e. the number of trees (default 100).

Using the same dataset we are now approaching the same problem using the Random Forest (both as a binary and a multiclass classifier).


Exercise 3:

Objective: Apply the Random Forest classifier to our dataset.

Task: Follow the actions below to get to some results:

  1. Complete the train-test split and select magnitudes to work with.

  2. Find the proper function for RF and make predictions.

reminder()

Note: Add any class if you want to use RF as binary classifier, or keep the list class2keep_RF empty for all classes. Add any number of bands to use.

class2keep_RF = []
bands_selected_RF = [...]  
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected_RF, class2keep_RF)  
# splitting data on magnitudes or colors

#X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, 
X_train, X_test, y_train, y_test = train_test_split(..., 
                                                    ...,
                                shuffle=True, stratify=..., 
                                test_size=...) #,  random_state=42) 

print(f'> From {len(ml_objects)} sources we use {len(X_train)} for training and {len(X_test)} for testing.') 

# check if in binary or multi-label mode
if 1 in ml_labels: # binary
    confmat_classes = [1,0]
else:
    confmat_classes = unique_cls

    print('\nStatistics per class:')
    k = 0
    for c in unique_cls:
        items_train = np.where( y_train==c )[0]
        items_test  = np.where( y_test==c )[0]
        items_total = np.where( ml_labels==c )[0]
        print(f'> For {c} there are {len(items_total)} sources split in {len(items_train)} (train) and {len(items_test)} (test) samples')
from sklearn.ensemble import RandomForestClassifier

clfrf = ...
# code to fit and predict here (follow SVM structure)
...
...

print(f"Classification report:\n\n {metrics.classification_report(..., ...)}") 

plot_confusion_matrix( metrics.confusion_matrix( ..., ...),
                      confmat_classes,
                      title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
                      normalize=False  # True returns percent, False raw numbers
                      ) # YlOrBr
reminder()
class2keep_RF = []
bands_selected_RF = ['U','V', 'I', 'J', '[3.6]']  
ml_data_mags, ml_data_clrs, ml_objects, ml_labels = process_data( bands_selected_RF, class2keep_RF)  
# splitting data on magnitudes or colors

#X_train, X_test, y_train, y_test = train_test_split(ml_data_mags, 
X_train, X_test, y_train, y_test = train_test_split(ml_data_clrs, 
                                                    ml_labels,
                                shuffle=True, stratify=ml_labels, 
                                test_size=0.3) #,  random_state=42) 

print(f'> From {len(ml_objects)} sources we use {len(X_train)} for training and {len(X_test)} for testing.') 

# check if in binary or multi-label mode
if 1 in ml_labels: # binary
    confmat_classes = [1,0]
else:
    confmat_classes = unique_cls

    print('\nStatistics per class:')
    k = 0
    for c in unique_cls:
        items_train = np.where( y_train==c )[0]
        items_test  = np.where( y_test==c )[0]
        items_total = np.where( ml_labels==c )[0]
        print(f'> For {c} there are {len(items_total)} sources split in {len(items_train)} (train) and {len(items_test)} (test) samples')
from sklearn.ensemble import RandomForestClassifier

clfrf = RandomForestClassifier()
# code to fit and predict here (follow SVM structure)
clfrf.fit(X_train, y_train)
yhat = clfrf.predict(X_test)

print(f"Classification report:\n\n {metrics.classification_report(y_test, yhat)}") 
print(f"Confusion matrix: \n\n {metrics.confusion_matrix(y_test, yhat)}")

plot_confusion_matrix( metrics.confusion_matrix( y_test, yhat),
                      confmat_classes,
                      title='Confusion matrix', cmap='BuPu', # for more options see: https://matplotlib.org/stable/tutorials/colors/colormaps.html
                      normalize=False  # True returns percent, False raw numbers
                      ) # YlOrBr

Questions

  1. Why the results change by re-running it ?

  2. What is the difference on the accuracy of the algorithm if we use the colors instead of the magnitudes? Why do it?

[Spoiler] (click here to expand)

  1. RF as the name suggest create always a different set of trees with random eselection of features and data. Consequently, the RF from one run to the other is not the same and there are small variations. However, the results are robust because they are averaging the results from all trees.

  2. Depending on the classes and bands selected the accuracy will (slightly) change. RF are rather insensitive to the input data (scaled or not).
    However, colors are distance-independent features, which means that a model build with these features could be applicable to other galaxies also. Colors are distance-independent features, which means that the model could be applicable to other galaxies also. On the contrary, if we use only magnitudes this would mess up the prediction as with distance the sources become fainter.

Classification algorithms overview#

This is a presentation of the set of sklearn algorithms with toy datasets, and then describe some of them.

This serves as a showcase of the available methods and how they compare. You can easily adapt any of these methods to the following examples or your own problems.

Overview of the clustering algorithms in sklearn

Question

Explore the results above - what differences do you notice?

[Spoiler] (click here to expand)

An indicative list of noticable differences:

  • There is not a single best algorithm.
  • Some algorithms are more able to separate classes.
  • Some algorithms have limited ability (e.g. the linear SVM with non-linear distributed data).

Recap: steps to follow when building a classification model#

1. Examine / visualize / preprocessing data

2. Select features

3. Select algorithm

4. Split data and follow Golden rule

5. Select metrics

6. Examine results and re-iterate

###EOF