Online Feature Selection¶
In this experiment, we use float to compare the effect of different online feature selection techniques. We use a prequential evaluation strategy.
# Import modules
import copy
import matplotlib.pyplot as plt
from sklearn.metrics import zero_one_loss, accuracy_score, precision_score
from skmultiflow.neural_networks import PerceptronMask
# Import float modules
from float.data import DataLoader
from float.feature_selection.evaluation import FeatureSelectionEvaluator
from float.feature_selection.evaluation.measures import nogueira_stability
from float.feature_selection import FIRES, OFS
from float.pipeline import PrequentialPipeline
from float.prediction.evaluation import PredictionEvaluator
from float.prediction.skmultiflow import SkmultiflowClassifier
import float.visualization as fvis
Setup and Run Pipeline¶
Create a DataLoader object¶
We load the TüEyeQ data set (76 features) from https://www.nature.com/articles/s41597-021-00938-3
data_loader = DataLoader(path='./datasets/iq.csv', # This path might have to be adjusted!
target_col=-1)
Create a Predictor object¶
We set up a single predictor in the default configuration of scikit-multiflow. Note that you may also run multiple predictors at once (see the remaining examples for reference).
predictor = SkmultiflowClassifier(model=PerceptronMask(),
classes=data_loader.stream.target_values)
Create a PredictionEvaluator object¶
The evaluator will compute and store the performance measures when running the pipeline.
pred_evaluator = PredictionEvaluator(measure_funcs=[zero_one_loss, accuracy_score, precision_score],
zero_division = 0) # Specify the zero_division parameter of the precision_score
Create FeatureSelector objects¶
We may use feature selection models from river or any of the pre-built feature selection methods. For illustration, we compare the OFS and FIRES feature selection models.
In online feature selection, the selected feature set may change over time. As most online predictive models cannot deal with arbitrary patterns of missing features, we need to replace non-selected features by a reference value (which ideally has no effect on the training). Float offers various baseline strategies. Here, we use a gaussian baseline to replace non-selected features at each time step. Note that this baseline requires a reference sample.
Both OFS and FIRES require us to specify the size of the selected feature set, which we set to 10 for illustration.
# We load a reference sample to compute the Gaussian baseline and reset the stream afterwards
ref_sample, _ = data_loader.get_data(50)
data_loader.stream.reset()
feature_selectors = [FIRES(n_total_features=data_loader.stream.n_features,
n_selected_features=10,
classes=data_loader.stream.target_values,
baseline='gaussian',
ref_sample=ref_sample),
OFS(n_total_features=data_loader.stream.n_features,
n_selected_features=10,
baseline='gaussian',
ref_sample=ref_sample)]
Create a FeatureSelectionEvaulator object for the FeatureSelectors¶
Aside from monitoring their effect on the overall predictive performance of the classifier, feature selection models can be evaluated w.r.t to the stability of the obtained feature sets. Float provides the feature set stability measure due to Nogueira et al., which we use in this experiment.
Similar to the PredictionEvaluator, the FeatureSelectionEvaluator will compute and store the performance measures w.r.t a feature selection object when running the pipeline. Float automatically clones the provided evaluator for each specified feature selector.
fs_evaluator = FeatureSelectionEvaluator([nogueira_stability])
Create and run a PrequentialPipeline¶
We use a batch-incremental scheme, processing the data in batches of size 100. Moreover, we pretrain the classifiers on 200 observations and set a random state for reproducibility.
We run the pipeline twice, using the two different feature selectors FIRES and OFS in turn.
pipelines = []
for feature_selector in feature_selectors:
pipeline = PrequentialPipeline(data_loader=data_loader, # The data loader will be automatically reset after finishing the pipeline.
feature_selector=feature_selector,
# We clone the evaluator, predictor and prediction_evaluator since we want separate objects for each run.
# This is more concise than specifying separate objects for both pipeline runs.
feature_selection_evaluator=copy.deepcopy(fs_evaluator),
predictor=copy.deepcopy(predictor),
prediction_evaluator=copy.deepcopy(pred_evaluator),
batch_size=100,
n_pretrain=200,
n_max=data_loader.stream.n_samples, # We use all observations
random_state=0)
pipeline.run()
pipelines.append(pipeline)
Pretrain the predictor with 200 observation(s). [====================] 100% ################################## SUMMARY ################################## Evaluation has finished after 0.5322020053863525s Data Set: float/data/datasets/iq.csv The PrequentialPipeline has processed 15762 instances, using batches of size 100. ------------------------------------------------------------------------- *** Prediction *** Model: SkmultiflowClassifier.PerceptronMask | Performance Measure | Value | |-----------------------|-------------| | Avg. Test Comp. Time | 5.23237e-05 | | Avg. Train Comp. Time | 0.000270608 | | Avg. zero_one_loss | 0.223005 | | Avg. accuracy_score | 0.776995 | | Avg. precision_score | 0.810469 | ------------------------------------------------------------------------- *** Online Feature Selection *** Model: FIRES Selected Features: 10/76 | Performance Measure | Value | |-------------------------|-------------| | Avg. Comp. Time | 0.000618074 | | Avg. nogueira_stability | 0.981779 | ############################################################################# Pretrain the predictor with 200 observation(s). [========= ] 46%
/Users/johannes/Documents/float/float/feature_selection/base_feature_selector.py:107: UserWarning: The weight vector contains negative values. The absolute weights will be used for feature selection.
warnings.warn("The weight vector contains negative values. The absolute weights will be used for "
[====================] 100% ################################## SUMMARY ################################## Evaluation has finished after 0.4634978771209717s Data Set: float/data/datasets/iq.csv The PrequentialPipeline has processed 15762 instances, using batches of size 100. ------------------------------------------------------------------------- *** Prediction *** Model: SkmultiflowClassifier.PerceptronMask | Performance Measure | Value | |-----------------------|-------------| | Avg. Test Comp. Time | 5.63294e-05 | | Avg. Train Comp. Time | 0.000270614 | | Avg. zero_one_loss | 0.230529 | | Avg. accuracy_score | 0.769471 | | Avg. precision_score | 0.822894 | ------------------------------------------------------------------------- *** Online Feature Selection *** Model: OFS Selected Features: 10/76 | Performance Measure | Value | |-------------------------|-------------| | Avg. Comp. Time | 0.000367463 | | Avg. nogueira_stability | 0.638539 | #############################################################################
The summary shows that the Perceptron classifier achieves very similar predictive performance for both feature selection models. However, the FIRES model obtains an average feature set stability close to one. We may gain further insight into both feature selection strategies by using the visualization module of float.
Visualize Results¶
We begin by plotting the accuracy of the Perceptron for both feature selection methods over time.
ax = fvis.plot(measures=[pipeline.prediction_evaluators[0].result['accuracy_score']['measures'] for pipeline in pipelines],
legend_labels=['FIRES', 'OFS'],
fig_size=(13, 5),
y_label='Accuary')
As before, we observe that the predictor has almost equal performance when using the different feature selection models FIRES and OFS. Based on this plot, it is difficult to decide between the two feature selectors.
Aside from standardized plot types like line, bar or scatter, float provides dedicated plot types for change detection and online feature selection. In the following, we demonstrate the feature_selection_bar, feature_selection_scatter and feature_weight_box plots.
ax = fvis.feature_selection_bar(selected_features=[pipeline.feature_selector.selected_features_history for pipeline in pipelines],
model_names=['FIRES', 'OFS'],
feature_names=data_loader.stream.feature_names,
fig_size=(25, 5))
plt.show()
ax = fvis.feature_selection_bar(selected_features=[pipeline.feature_selector.selected_features_history for pipeline in pipelines],
model_names=['FIRES', 'OFS'],
feature_names=data_loader.stream.feature_names,
top_n_features=10)
plt.show()
In the first bar plot, we show the number of times each feature has been selected. In the second plot, we show the top 10 selected features. Note that the first provided model (i.e. FIRES in our case) is used as a reference to select the top n=10 features.
Based on these plots, we observe that grades seem to play an important role in the prediction (in particular the math grad). In general, however, we observe considerable differences between the FIRES and OFS feature selectors, i.e., FIRES uses a significanlty smaller subset of features over time.
The feature_selection_scatter plot of float allows us to illustrate the selected features over time.
ax = fvis.feature_selection_scatter(selected_features=pipelines[0].feature_selector.selected_features_history,
fig_size=(13, 5))
plt.title('FIRES Selected Features', fontsize=16)
plt.show()
ax = fvis.feature_selection_scatter(selected_features=pipelines[1].feature_selector.selected_features_history,
fig_size=(13, 5))
plt.title('OFS Selected Features', fontsize=16)
plt.show()
This plot type is useful if we want to investigate time periods in which the selected feature set has changed a lot. That is, this plot type offers an intuition about the temporal stability of the selected feature set. In this particular example, we observe much more variation in the selected features of OFS compared to FIRES. This is matched by the different Nogueira stability measures that we calculated, which are ~0.98 for FIRES and ~0.65 for OFS.
Finally, most feature selection models compute weights that are then used to rank the features. The feature_weight_box plot of float allows us to plot and compare these weights.
ax = fvis.feature_weight_box(feature_weights=[pipeline.feature_selector.weights_history for pipeline in pipelines],
model_names=['FIRES', 'OFS'],
feature_names=data_loader.stream.feature_names,
top_n_features=10)
