ensemble_learner.model_comb_cross_val_bagging

A bagging of many base hyperbox-based models trained on the full set of features and a subset of samples. Each base learner is trained by k-fold cross validation and random search based hyper-parameter tuning. After formulation of base learners models, their hyperboxes are combined into a single model. The predicted class is computed based on the final single model.

class hbbrain.numerical_data.ensemble_learner.model_comb_cross_val_bagging.ModelCombinationCrossValBagging(base_estimator=None, base_estimator_params={}, model_level_estimator=None, n_estimators=10, max_samples=0.5, bootstrap=False, class_balanced=False, n_iter=10, scoring='accuracy', k_fold=5, n_jobs=1, random_state=None)

Bases: ClassifierMixin, BaseCrossValBagging

A Bagging classifier of base hyperbox-based models trained on a full set of features and a subset of samples. Each base learner is trained by k-fold cross validation and random search based hyper-parameter tuning. Then, the hyperboxes from all base learners are combined to a single model.

A model-level combination cross validation Bagging classifier of hyperbox-based models is an ensemble meta-estimator that fits base hyperbox-based classifiers each on random subsets of the original samples using k-fold cross validation and random search of hyper-paramters, and then aggregate their hyperboxes into a single model and use this model for prediction. Such a meta-estimator can typically be used as a way to reduce the variance of a single estimator, by introducing randomization into its construction procedure and then making an ensemble out of it. This algorithm encompasses several works from the literature. When random subsets of the dataset are drawn as random subsets of the samples, then this algorithm is known as Pasting [1]. If samples are drawn with replacement, then the method is known as Bagging [2]. See [3] for more detailed information regarding the combination of hyperboxes from all base hyperbox-based models.

Parameters:

base_estimatorobject, default=None

The base estimator to fit on random subsets of the dataset. If None, then the base estimator is a OnlineGFMM.

base_estimator_paramsdict or list of dicts, default={}

Dictionary with parameters names (str) as keys and distributions or lists of parameters to try. If a list is given, it is sampled uniformly. If a list of dicts is given, first a dict is sampled uniformly, and then a parameter is sampled using that dict as above.

model_level_estimatorobject, default=None

The estimator is used to aggregate all resulting hyperboxes from all base learners into a single model. If None, then the base estimator is a AccelAgglomerativeLearningGFMM.

n_estimatorsint, default=10

The number of base estimators in the ensemble.

max_samplesint or float, default=0.5

The number of samples to draw from X to train each base estimator (with no replacement by default, see bootstrap for more details). - If int, then draw max_samples samples. - If float, then draw max_samples * X.shape[0] samples.

bootstrapbool, default=False

Whether samples are drawn with replacement. If False, sampling without replacement is performed.

class_balancedbool, default=False

Whether samples are drawn without replacement to build a final subset with the equal number of samples among classes.

n_iterint, default=10

Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution.

scoringstr or callable default=’accuracy’

Strategy to evaluate the performance of the cross-validated model on the test set. If scoring represents a single score, one can use: - a single string (see The scoring parameter: defining model evaluation rules in sklearn). - a callable (see Defining your scoring strategy from metric functions) that returns a single value.

k_foldint, default=5

Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 5-fold cross validation, - integer, to specify the number of folds in a (Stratified)KFold, For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, Stratified K-Fold is used.

n_jobsint, default=1

The number of jobs to run in parallel for both fit() and predict(). None means 1 unless in a joblib.parallel_backend context. -1 means using all processors.

random_stateint, RandomState instance or None, default=None

Controls the random resampling of the original dataset (sample wise and feature wise). If the base estimator accepts a random_state attribute, a different seed is generated for each instance in the ensemble. Pass an int for reproducible output across multiple function calls.

References

[1]

L. Breiman, “Pasting small votes for classification in large databases and on-line”, Machine Learning, vol. 36, no. 1, pp. 85-103, 1999.

[2]

L. Breiman, “Bagging predictors”, Machine Learning, vol. 24, no. 2, pp. 123-140, 1996.

[3]

B. Gabrys,”Combining neuro-fuzzy classifiers for improved generalisation and reliability”, in Proceedings of the 2002 International Joint Conference on Neural Networks, vol. 3, pp. 2410-2415, 2002.

Examples

>>> from hbbrain.numerical_data.incremental_learner.onln_gfmm import OnlineGFMM
>>> from hbbrain.numerical_data.ensemble_learner.model_comb_cross_val_bagging import ModelCombinationCrossValBagging
>>> from hbbrain.numerical_data.batch_learner.accel_agglo_gfmm import AccelAgglomerativeLearningGFMM
>>> from sklearn.datasets import make_classification
>>> X, y = make_classification(n_samples=100, n_features=4,
...                            n_informative=2, n_redundant=0,
...                            random_state=0, shuffle=False)
>>> from sklearn.preprocessing import MinMaxScaler
>>> scaler = MinMaxScaler()
>>> scaler.fit(X)
MinMaxScaler()
>>> X = scaler.transform(X)
>>> clf = ModelCombinationCrossValBagging(base_estimator=OnlineGFMM(0.1),
...                            base_estimator_params = {'theta': np.arange(0.05, 1.01, 0.05), 'theta_min':[1], 'gamma':[0.5, 1, 2, 4, 8, 16]},
...                            model_level_estimator=AccelAgglomerativeLearningGFMM(0.1),
...                            n_estimators=10, random_state=0).fit(X, y)
>>> clf.predict([[1, 0.6, 0.5, 0.2]])
array([1])

Attributes:

base_estimator_estimator

The base estimator from which the ensemble is grown.

model_level_estimator_estimator

The final estimator is the combination of hyperboxes from all base learners.

estimators_list of estimators

The collection of fitted base estimators.

estimators_samples_list of arrays

The subset of drawn samples for each base estimator.

classes_ndarray of shape (n_classes,)

The classes labels.

n_classes_int or list

The number of classes.

Methods

fit(X, y[, is_pruning_base_learners, X_val, ...])	Build a Bagging ensemble of estimators from the training set (X, y).
get_n_hyperboxes()	Get total number of hyperboxes in all base learners.
get_n_hyperboxes_comb_model()	Get number of hyperboxes in the final combined model from all hyperboxes of base learners
get_params([deep])	Get parameters for this estimator.
predict(X[, type_boundary_handling])	Predict class for X.
predict_proba(X)	Predict class probabilities of the input samples X.
predict_proba_all_base_learners(X)	Predict mean class probabilities for X from all base learners.
predict_voting(X)	Predict class for X.
predict_with_membership(X)	Predict class membership values of the input samples X.
predict_with_membership_all_base_learners(X)	Predict mean class memberships for X from all base learners.
score(X, y[, sample_weight])	Return the mean accuracy on the given test data and labels.
set_params(**params)	Set the parameters of this estimator.
simple_pruning(X_val, y_val[, ...])	Simply prune low qualitied hyperboxes based on a pre-defined accuracy threshold for each hyperbox.
simple_pruning_base_estimators(X_val, y_val)	Simply prune low qualitied hyperboxes based on a pre-defined accuracy threshold for each hyperbox.

fit(X, y, is_pruning_base_learners=False, X_val=None, y_val=None, acc_threshold=0.5, keep_empty_boxes=False)

Build a Bagging ensemble of estimators from the training set (X, y).

Parameters:

Xarray-like of shape (n_samples, n_features)

The training input samples.

yarray-like of shape (n_samples,)

The real class labels

is_pruning_base_learnersboolean, optional, default=False

Whether the pruning procedure can be applied for base learners or not

X_valarray-like of shape (n_samples, n_features)

The data matrix contains validation patterns.

y_valndarray of shape (n_samples,)

A vector contains the true class label corresponding to each validation pattern.

acc_thresholdfloat, optional, default=0.5

The minimum accuracy for each hyperbox to be kept unchanged.

keep_empty_boxesboolean, optional, default=False

Whether to keep the hyperboxes which do not join the prediction process on the validation set. If True, keep them, else the decision for keeping or removing based on the classification accuracy on the validation dataset

Returns:

selfobject

Fitted estimator.

get_n_hyperboxes_comb_model()

Get number of hyperboxes in the final combined model from all hyperboxes of base learners

Returns:

int

Total number of hyperboxes in the final combined model from all resulting hyperboxes of all base learners.

predict(X, type_boundary_handling=-1)

Predict class for X.

The predicted class of an input sample is computed as the class with the highest mean predicted probability using voting.

Parameters:

Xarray-like of shape (n_samples, n_features)

The testing input samples.

type_boundary_handlingint, optional, default=-1

The way of handling many winner hyperboxes. This parameter is only used in the case of model_level_estimator being improved online learing algorithm or aggolomerative learning algorithms.

Returns:

y_predndarray of shape (n_samples,)

The predicted classes.

predict_proba(X)

Predict class probabilities of the input samples X.

The predicted class probability is the fraction of the membership value of the representative hyperbox of that class and the sum of all membership values of all representative hyperboxes of all classes in the prediction procedure using the final combined model.

Parameters:

Xarray-like of shape (n_samples, n_features)

The input samples.

Returns:

probandarray of shape (n_samples, n_classes)

The class probabilities of the input samples. The order of the classes corresponds to that in ascending integers of class labels.

predict_proba_all_base_learners(X)

Predict mean class probabilities for X from all base learners.

The predicted class probabilities of an input sample are computed as the mean predicted class probabilities of all hyperbox-based learners in the ensemble model. The class probability of a single hyperbox-based learner is the fraction of the membership value of the representative hyperbox of that class and the sum of all membership values of all representative hyperboxes of all classes.

Parameters:

Xarray-like of shape (n_samples, n_features)

The input samples for prediction.

Returns:

all_probasndarray of shape (n_samples, n_classes)

The class probabilities of the input samples. The order of the classes corresponds to that in ascending integers of class labels.

predict_voting(X)

Predict class for X.

The predicted class of an input sample is computed as the class with the highest mean predicted probability using voting from all base learners.

Parameters:

Xarray-like of shape (n_samples, n_features)

The testing input samples.

Returns:

yndarray of shape (n_samples,)

The predicted classes.

predict_with_membership(X)

Predict class membership values of the input samples X.

The predicted class membership value is the membership value of the representative hyperbox of that class in the prediction procedure using the final combined model.

Parameters:

Xarray-like of shape (n_samples, n_features)

The input samples.

Returns:

mem_valsndarray of shape (n_samples, n_classes)

The class membership values of the input samples. The order of the classes corresponds to that in ascending integers of class labels.

predict_with_membership_all_base_learners(X)

Predict mean class memberships for X from all base learners.

The predicted class memberships of an input sample are computed as the mean predicted class memberships of the hyperbox-based learners in the ensemble model. The class membership of a single hyperbox-based learner is the membership from the input X to the representative hyperbox of that class to join the prediction procedure.

Parameters:

Xarray-like of shape (n_samples, n_features)

The input samples for prediction.

Returns:

mem_valsndarray of shape (n_samples, n_classes)

The class memberships of the input samples. The order of the classes corresponds to that in ascending integers of class labels.

simple_pruning(X_val, y_val, acc_threshold=0.5, keep_empty_boxes=False)

Simply prune low qualitied hyperboxes based on a pre-defined accuracy threshold for each hyperbox. This operation is applied for the final combined model.

Parameters:

X_valarray-like of shape (n_samples, n_features)

The data matrix contains validation patterns.

y_valndarray of shape (n_samples,)

A vector contains the true class label corresponding to each validation pattern.

acc_thresholdfloat, optional, default=0.5

The minimum accuracy for each hyperbox to be kept unchanged.

keep_empty_boxesboolean, optional, default=False

Whether to keep the hyperboxes which do not join the prediction process on the validation set. If True, keep them, else the decision for keeping or removing based on the classification accuracy on the validation dataset

Returns:

self

A final hyperbox-based model is prunned of low-quality hyperboxes.