Alibi Detect is a source-available Python library focused on outlier, adversarial and drift detection. The package aims to cover both online and offline detectors for tabular data, text, images and time series. Both TensorFlow and PyTorch backends are supported for drift detection.
For more background on the importance of monitoring outliers and distributions in a production setting, check out this talk from the Challenges in Deploying and Monitoring Machine Learning Systems ICML 2020 workshop, based on the paper Monitoring and explainability of models in production and referencing Alibi Detect.
For a thorough introduction to drift detection, check out Protecting Your Machine Learning Against Drift: An Introduction. The talk covers what drift is and why it pays to detect it, the different types of drift, how it can be detected in a principled manner and also describes the anatomy of a drift detector.
The package, alibi-detect can be installed from:
pip)conda/mamba)alibi-detect can be installed from PyPI:
Alternatively, the development version can be installed:
pip install git+https://github.com/SeldonIO/alibi-detect.git
To install with the TensorFlow backend:
pip install alibi-detect[tensorflow]
To install with the PyTorch backend:
pip install alibi-detect[torch]
To install with the KeOps backend:
pip install alibi-detect[keops]
To use the Prophet time series outlier detector:
pip install alibi-detect[prophet]
To install from conda-forge it is recommended to use mamba, which can be installed to the base conda enviroment with:
conda install mamba -n base -c conda-forge
To install alibi-detect:
mamba install -c conda-forge alibi-detect
We will use the VAE outlier detector to illustrate the API.
from alibi_detect.od import OutlierVAE from alibi_detect.saving import save_detector, load_detector # initialize and fit detector od = OutlierVAE(threshold=0.1, encoder_net=encoder_net, decoder_net=decoder_net, latent_dim=1024) od.fit(x_train) # make predictions preds = od.predict(x_test) # save and load detectors filepath = './my_detector/' save_detector(od, filepath) od = load_detector(filepath)
The predictions are returned in a dictionary with as keys meta and data. meta contains the detector's metadata while data is in itself a dictionary with the actual predictions. It contains the outlier, adversarial or drift scores and thresholds as well as the predictions whether instances are e.g. outliers or not. The exact details can vary slightly from method to method, so we encourage the reader to become familiar with the types of algorithms supported.
The following tables show the advised use cases for each algorithm. The column Feature Level indicates whether the detection can be done at the feature level, e.g. per pixel for an image. Check the algorithm reference list for more information with links to the documentation and original papers as well as examples for each of the detectors.
Detector Tabular Image Time Series Text Categorical Features Online Feature Level Isolation Forest ✔ ✔ Mahalanobis Distance ✔ ✔ ✔ AE ✔ ✔ ✔ VAE ✔ ✔ ✔ AEGMM ✔ ✔ VAEGMM ✔ ✔ Likelihood Ratios ✔ ✔ ✔ ✔ ✔ Prophet ✔ Spectral Residual ✔ ✔ ✔ Seq2Seq ✔ ✔ Detector Tabular Image Time Series Text Categorical Features Online Feature Level Adversarial AE ✔ ✔ Model distillation ✔ ✔ ✔ ✔ ✔ Detector Tabular Image Time Series Text Categorical Features Online Feature Level Kolmogorov-Smirnov ✔ ✔ ✔ ✔ ✔ Cramér-von Mises ✔ ✔ ✔ ✔ Fisher's Exact Test ✔ ✔ ✔ ✔ Maximum Mean Discrepancy (MMD) ✔ ✔ ✔ ✔ ✔ Learned Kernel MMD ✔ ✔ ✔ ✔ Context-aware MMD ✔ ✔ ✔ ✔ ✔ Least-Squares Density Difference ✔ ✔ ✔ ✔ ✔ Chi-Squared ✔ ✔ ✔ Mixed-type tabular data ✔ ✔ ✔ Classifier ✔ ✔ ✔ ✔ ✔ Spot-the-diff ✔ ✔ ✔ ✔ ✔ ✔ Classifier Uncertainty ✔ ✔ ✔ ✔ ✔ Regressor Uncertainty ✔ ✔ ✔ ✔ ✔ TensorFlow and PyTorch supportThe drift detectors support TensorFlow, PyTorch and (where applicable) KeOps backends. However, Alibi Detect does not install these by default. See the installation options for more details.
from alibi_detect.cd import MMDDrift cd = MMDDrift(x_ref, backend='tensorflow', p_val=.05) preds = cd.predict(x)
The same detector in PyTorch:
cd = MMDDrift(x_ref, backend='pytorch', p_val=.05) preds = cd.predict(x)
Or in KeOps:
cd = MMDDrift(x_ref, backend='keops', p_val=.05) preds = cd.predict(x)Built-in preprocessing steps
Alibi Detect also comes with various preprocessing steps such as randomly initialized encoders, pretrained text embeddings to detect drift on using the transformers library and extraction of hidden layers from machine learning models. This allows to detect different types of drift such as covariate and predicted distribution shift. The preprocessing steps are again supported in TensorFlow and PyTorch.
from alibi_detect.cd.tensorflow import HiddenOutput, preprocess_drift model = # TensorFlow model; tf.keras.Model or tf.keras.Sequential preprocess_fn = partial(preprocess_drift, model=HiddenOutput(model, layer=-1), batch_size=128) cd = MMDDrift(x_ref, backend='tensorflow', p_val=.05, preprocess_fn=preprocess_fn) preds = cd.predict(x)
Check the example notebooks (e.g. CIFAR10, movie reviews) for more details.
Isolation Forest (FT Liu et al., 2008)
Mahalanobis Distance (Mahalanobis, 1936)
Variational Auto-Encoder (VAE) (Kingma et al., 2013)
Auto-Encoding Gaussian Mixture Model (AEGMM) (Zong et al., 2018)
Variational Auto-Encoding Gaussian Mixture Model (VAEGMM)
Likelihood Ratios (Ren et al., 2019)
Prophet Time Series Outlier Detector (Taylor et al., 2018)
Spectral Residual Time Series Outlier Detector (Ren et al., 2019)
Sequence-to-Sequence (Seq2Seq) Outlier Detector (Sutskever et al., 2014; Park et al., 2017)
Adversarial Auto-Encoder (Vacanti and Van Looveren, 2020)
Maximum Mean Discrepancy (Gretton et al, 2012)
Learned Kernel MMD (Liu et al, 2020)
Context-aware MMD (Cobb and Van Looveren, 2022)
Classifier (Lopez-Paz and Oquab, 2017)
Spot-the-diff (adaptation of Jitkrittum et al, 2016)
Classifier and Regressor Uncertainty
Online Maximum Mean Discrepancy
Online Least-Squares Density Difference (Bu et al, 2017)
The package also contains functionality in alibi_detect.datasets to easily fetch a number of datasets for different modalities. For each dataset either the data and labels or a Bunch object with the data, labels and optional metadata are returned. Example:
from alibi_detect.datasets import fetch_ecg (X_train, y_train), (X_test, y_test) = fetch_ecg(return_X_y=True)Sequential Data and Time Series
Genome Dataset: fetch_genome
from alibi_detect.datasets import fetch_genome (X_train, y_train), (X_val, y_val), (X_test, y_test) = fetch_genome(return_X_y=True)
ECG 5000: fetch_ecg
NAB: fetch_nab
alibi_detect.datasets.get_list_nab().CIFAR-10-C: fetch_cifar10c
fetch_cifar10c allows you to pick any severity level or corruption type. The list with available corruption types can be retrieved with alibi_detect.datasets.corruption_types_cifar10c(). The dataset can be used in research on robustness and drift. The original data can be found here. Example:from alibi_detect.datasets import fetch_cifar10c corruption = ['gaussian_noise', 'motion_blur', 'brightness', 'pixelate'] X, y = fetch_cifar10c(corruption=corruption, severity=5, return_X_y=True)
Adversarial CIFAR-10: fetch_attack
from alibi_detect.datasets import fetch_attack
(X_train, y_train), (X_test, y_test) = fetch_attack('cifar10', 'resnet56', 'cw', return_X_y=True)fetch_kdd
fetch_kdd allows you to select a subset of network intrusions as targets or pick only specified features. The original data can be found here.Models and/or building blocks that can be useful outside of outlier, adversarial or drift detection can be found under alibi_detect.models. Main implementations:
PixelCNN++: alibi_detect.models.pixelcnn.PixelCNN
Variational Autoencoder: alibi_detect.models.autoencoder.VAE
Sequence-to-sequence model: alibi_detect.models.autoencoder.Seq2Seq
ResNet: alibi_detect.models.resnet
from alibi_detect.utils.fetching import fetch_tf_model
model = fetch_tf_model('cifar10', 'resnet32')Alibi-detect is integrated in the machine learning model deployment platform Seldon Core and model serving framework KFServing.
If you use alibi-detect in your research, please consider citing it.
BibTeX entry:
@software{alibi-detect,
title = {Alibi Detect: Algorithms for outlier, adversarial and drift detection},
author = {Van Looveren, Arnaud and Klaise, Janis and Vacanti, Giovanni and Cobb, Oliver and Scillitoe, Ashley and Samoilescu, Robert and Athorne, Alex},
url = {https://github.com/SeldonIO/alibi-detect},
version = {0.12.1.dev0},
date = {2024-04-17},
year = {2019}
}
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4