quapy package

Submodules

quapy.error

Implementation of error measures used for quantification

quapy.error.absolute_error(prevs, prevs_hat)
Computes the absolute error between the two prevalence vectors.

Absolute error between two prevalence vectors \(p\) and \(\hat{p}\) is computed as \(AE(p,\hat{p})=\frac{1}{|\mathcal{Y}|}\sum_{y\in \mathcal{Y}}|\hat{p}(y)-p(y)|\), where \(\mathcal{Y}\) are the classes of interest.

Parameters:

  • prevs – array-like of shape (n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_classes,) with the predicted prevalence values

Returns:

absolute error

quapy.error.acc_error(y_true, y_pred)

Computes the error in terms of 1-accuracy. The accuracy is computed as \(\frac{tp+tn}{tp+fp+fn+tn}\), with tp, fp, fn, and tn standing for true positives, false positives, false negatives, and true negatives, respectively

Parameters:

  • y_true – array-like of true labels

  • y_pred – array-like of predicted labels

Returns:

1-accuracy

quapy.error.acce(y_true, y_pred)

Computes the error in terms of 1-accuracy. The accuracy is computed as \(\frac{tp+tn}{tp+fp+fn+tn}\), with tp, fp, fn, and tn standing for true positives, false positives, false negatives, and true negatives, respectively

Parameters:

  • y_true – array-like of true labels

  • y_pred – array-like of predicted labels

Returns:

1-accuracy

quapy.error.ae(prevs, prevs_hat)
Computes the absolute error between the two prevalence vectors.

Absolute error between two prevalence vectors \(p\) and \(\hat{p}\) is computed as \(AE(p,\hat{p})=\frac{1}{|\mathcal{Y}|}\sum_{y\in \mathcal{Y}}|\hat{p}(y)-p(y)|\), where \(\mathcal{Y}\) are the classes of interest.

Parameters:

  • prevs – array-like of shape (n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_classes,) with the predicted prevalence values

Returns:

absolute error

quapy.error.f1_error(y_true, y_pred)

F1 error: simply computes the error in terms of macro \(F_1\), i.e., \(1-F_1^M\), where \(F_1\) is the harmonic mean of precision and recall, defined as \(\frac{2tp}{2tp+fp+fn}\), with tp, fp, and fn standing for true positives, false positives, and false negatives, respectively. Macro averaging means the \(F_1\) is computed for each category independently, and then averaged.

Parameters:

  • y_true – array-like of true labels

  • y_pred – array-like of predicted labels

Returns:

\(1-F_1^M\)

quapy.error.f1e(y_true, y_pred)

F1 error: simply computes the error in terms of macro \(F_1\), i.e., \(1-F_1^M\), where \(F_1\) is the harmonic mean of precision and recall, defined as \(\frac{2tp}{2tp+fp+fn}\), with tp, fp, and fn standing for true positives, false positives, and false negatives, respectively. Macro averaging means the \(F_1\) is computed for each category independently, and then averaged.

Parameters:

  • y_true – array-like of true labels

  • y_pred – array-like of predicted labels

Returns:

\(1-F_1^M\)

quapy.error.from_name(err_name)

Gets an error function from its name. E.g., from_name(“mae”) will return function quapy.error.mae()

Parameters:

err_name – string, the error name

Returns:

a callable implementing the requested error

quapy.error.kld(prevs, prevs_hat, eps=None)
Computes the Kullback-Leibler divergence between the two prevalence distributions.

Kullback-Leibler divergence between two prevalence distributions \(p\) and \(\hat{p}\) is computed as \(KLD(p,\hat{p})=D_{KL}(p||\hat{p})= \sum_{y\in \mathcal{Y}} p(y)\log\frac{p(y)}{\hat{p}(y)}\), where \(\mathcal{Y}\) are the classes of interest. The distributions are smoothed using the eps factor (see quapy.error.smooth()).

Parameters:

  • prevs – array-like of shape (n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_classes,) with the predicted prevalence values

  • eps – smoothing factor. KLD is not defined in cases in which the distributions contain zeros; eps is typically set to be \(\frac{1}{2T}\), with \(T\) the sample size. If eps=None, the sample size will be taken from the environment variable SAMPLE_SIZE (which has thus to be set beforehand).

Returns:

Kullback-Leibler divergence between the two distributions

quapy.error.mae(prevs, prevs_hat)

Computes the mean absolute error (see quapy.error.ae()) across the sample pairs.

Parameters:

  • prevs – array-like of shape (n_samples, n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_samples, n_classes,) with the predicted prevalence values

Returns:

mean absolute error

quapy.error.mean_absolute_error(prevs, prevs_hat)

Computes the mean absolute error (see quapy.error.ae()) across the sample pairs.

Parameters:

  • prevs – array-like of shape (n_samples, n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_samples, n_classes,) with the predicted prevalence values

Returns:

mean absolute error

quapy.error.mean_relative_absolute_error(prevs, prevs_hat, eps=None)

Computes the mean relative absolute error (see quapy.error.rae()) across the sample pairs. The distributions are smoothed using the eps factor (see quapy.error.smooth()).

Parameters:

  • prevs – array-like of shape (n_samples, n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_samples, n_classes,) with the predicted prevalence values

  • eps – smoothing factor. mrae is not defined in cases in which the true distribution contains zeros; eps is typically set to be \(\frac{1}{2T}\), with \(T\) the sample size. If eps=None, the sample size will be taken from the environment variable SAMPLE_SIZE (which has thus to be set beforehand).

Returns:

mean relative absolute error

quapy.error.mkld(prevs, prevs_hat, eps=None)

Computes the mean Kullback-Leibler divergence (see quapy.error.kld()) across the sample pairs. The distributions are smoothed using the eps factor (see quapy.error.smooth()).

Parameters:

  • prevs – array-like of shape (n_samples, n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_samples, n_classes,) with the predicted prevalence values

  • eps – smoothing factor. KLD is not defined in cases in which the distributions contain zeros; eps is typically set to be \(\frac{1}{2T}\), with \(T\) the sample size. If eps=None, the sample size will be taken from the environment variable SAMPLE_SIZE (which has thus to be set beforehand).

Returns:

mean Kullback-Leibler distribution

quapy.error.mnkld(prevs, prevs_hat, eps=None)

Computes the mean Normalized Kullback-Leibler divergence (see quapy.error.nkld()) across the sample pairs. The distributions are smoothed using the eps factor (see quapy.error.smooth()).

Parameters:

  • prevs – array-like of shape (n_samples, n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_samples, n_classes,) with the predicted prevalence values

  • eps – smoothing factor. NKLD is not defined in cases in which the distributions contain zeros; eps is typically set to be \(\frac{1}{2T}\), with \(T\) the sample size. If eps=None, the sample size will be taken from the environment variable SAMPLE_SIZE (which has thus to be set beforehand).

Returns:

mean Normalized Kullback-Leibler distribution

quapy.error.mrae(prevs, prevs_hat, eps=None)

Computes the mean relative absolute error (see quapy.error.rae()) across the sample pairs. The distributions are smoothed using the eps factor (see quapy.error.smooth()).

Parameters:

  • prevs – array-like of shape (n_samples, n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_samples, n_classes,) with the predicted prevalence values

  • eps – smoothing factor. mrae is not defined in cases in which the true distribution contains zeros; eps is typically set to be \(\frac{1}{2T}\), with \(T\) the sample size. If eps=None, the sample size will be taken from the environment variable SAMPLE_SIZE (which has thus to be set beforehand).

Returns:

mean relative absolute error

quapy.error.mse(prevs, prevs_hat)

Computes the mean squared error (see quapy.error.se()) across the sample pairs.

Parameters:

  • prevs – array-like of shape (n_samples, n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_samples, n_classes,) with the predicted prevalence values

Returns:

mean squared error

quapy.error.nkld(prevs, prevs_hat, eps=None)
Computes the Normalized Kullback-Leibler divergence between the two prevalence distributions.

Normalized Kullback-Leibler divergence between two prevalence distributions \(p\) and \(\hat{p}\) is computed as math:NKLD(p,hat{p}) = 2frac{e^{KLD(p,hat{p})}}{e^{KLD(p,hat{p})}+1}-1, where \(\mathcal{Y}\) are the classes of interest. The distributions are smoothed using the eps factor (see quapy.error.smooth()).

Parameters:

  • prevs – array-like of shape (n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_classes,) with the predicted prevalence values

  • eps – smoothing factor. NKLD is not defined in cases in which the distributions contain zeros; eps is typically set to be \(\frac{1}{2T}\), with \(T\) the sample size. If eps=None, the sample size will be taken from the environment variable SAMPLE_SIZE (which has thus to be set beforehand).

Returns:

Normalized Kullback-Leibler divergence between the two distributions

quapy.error.rae(prevs, prevs_hat, eps=None)
Computes the absolute relative error between the two prevalence vectors.

Relative absolute error between two prevalence vectors \(p\) and \(\hat{p}\) is computed as \(RAE(p,\hat{p})= \frac{1}{|\mathcal{Y}|}\sum_{y\in \mathcal{Y}}\frac{|\hat{p}(y)-p(y)|}{p(y)}\), where \(\mathcal{Y}\) are the classes of interest. The distributions are smoothed using the eps factor (see quapy.error.smooth()).

Parameters:

  • prevs – array-like of shape (n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_classes,) with the predicted prevalence values

  • eps – smoothing factor. rae is not defined in cases in which the true distribution contains zeros; eps is typically set to be \(\frac{1}{2T}\), with \(T\) the sample size. If eps=None, the sample size will be taken from the environment variable SAMPLE_SIZE (which has thus to be set beforehand).

Returns:

relative absolute error

quapy.error.relative_absolute_error(prevs, prevs_hat, eps=None)
Computes the absolute relative error between the two prevalence vectors.

Relative absolute error between two prevalence vectors \(p\) and \(\hat{p}\) is computed as \(RAE(p,\hat{p})= \frac{1}{|\mathcal{Y}|}\sum_{y\in \mathcal{Y}}\frac{|\hat{p}(y)-p(y)|}{p(y)}\), where \(\mathcal{Y}\) are the classes of interest. The distributions are smoothed using the eps factor (see quapy.error.smooth()).

Parameters:

  • prevs – array-like of shape (n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_classes,) with the predicted prevalence values

  • eps – smoothing factor. rae is not defined in cases in which the true distribution contains zeros; eps is typically set to be \(\frac{1}{2T}\), with \(T\) the sample size. If eps=None, the sample size will be taken from the environment variable SAMPLE_SIZE (which has thus to be set beforehand).

Returns:

relative absolute error

quapy.error.se(prevs, prevs_hat)
Computes the squared error between the two prevalence vectors.

Squared error between two prevalence vectors \(p\) and \(\hat{p}\) is computed as \(SE(p,\hat{p})=\frac{1}{|\mathcal{Y}|}\sum_{y\in \mathcal{Y}}(\hat{p}(y)-p(y))^2\), where \(\mathcal{Y}\) are the classes of interest.

Parameters:

  • prevs – array-like of shape (n_classes,) with the true prevalence values

  • prevs_hat – array-like of shape (n_classes,) with the predicted prevalence values

Returns:

absolute error

quapy.error.smooth(prevs, eps)

Smooths a prevalence distribution with \(\epsilon\) (eps) as: \(\underline{p}(y)=\frac{\epsilon+p(y)}{\epsilon|\mathcal{Y}|+ \displaystyle\sum_{y\in \mathcal{Y}}p(y)}\)

Parameters:

  • prevs – array-like of shape (n_classes,) with the true prevalence values

  • eps – smoothing factor

Returns:

array-like of shape (n_classes,) with the smoothed distribution

quapy.evaluation

quapy.evaluation.evaluate(model: BaseQuantifier, protocol: AbstractProtocol, error_metric: Union[str, Callable], aggr_speedup: Union[str, bool] = 'auto', verbose=False)

Evaluates a quantification model according to a specific sample generation protocol and in terms of one evaluation metric (error).

Parameters:

  • model – a quantifier, instance of quapy.method.base.BaseQuantifier

  • protocol – quapy.protocol.AbstractProtocol; if this object is also instance of quapy.protocol.OnLabelledCollectionProtocol, then the aggregation speed-up can be run. This is the protocol in charge of generating the samples in which the model is evaluated.

  • error_metric – a string representing the name(s) of an error function in qp.error (e.g., ‘mae’), or a callable function implementing the error function itself.

  • aggr_speedup – whether or not to apply the speed-up. Set to “force” for applying it even if the number of instances in the original collection on which the protocol acts is larger than the number of instances in the samples to be generated. Set to True or “auto” (default) for letting QuaPy decide whether it is convenient or not. Set to False to deactivate.

  • verbose – boolean, show or not information in stdout

Returns:

if the error metric is not averaged (e.g., ‘ae’, ‘rae’), returns an array of shape (n_samples,) with the error scores for each sample; if the error metric is averaged (e.g., ‘mae’, ‘mrae’) then returns a single float

quapy.evaluation.evaluate_on_samples(model: BaseQuantifier, samples: Iterable[LabelledCollection], error_metric: Union[str, Callable], verbose=False)

Evaluates a quantification model on a given set of samples and in terms of one evaluation metric (error).

Parameters:

  • model – a quantifier, instance of quapy.method.base.BaseQuantifier

  • samples – a list of samples on which the quantifier is to be evaluated

  • error_metric – a string representing the name(s) of an error function in qp.error (e.g., ‘mae’), or a callable function implementing the error function itself.

  • verbose – boolean, show or not information in stdout

Returns:

if the error metric is not averaged (e.g., ‘ae’, ‘rae’), returns an array of shape (n_samples,) with the error scores for each sample; if the error metric is averaged (e.g., ‘mae’, ‘mrae’) then returns a single float

quapy.evaluation.evaluation_report(model: BaseQuantifier, protocol: AbstractProtocol, error_metrics: Iterable[Union[str, Callable]] = 'mae', aggr_speedup: Union[str, bool] = 'auto', verbose=False)

Generates a report (a pandas’ DataFrame) containing information of the evaluation of the model as according to a specific protocol and in terms of one or more evaluation metrics (errors).

Parameters:

  • model – a quantifier, instance of quapy.method.base.BaseQuantifier

  • protocol – quapy.protocol.AbstractProtocol; if this object is also instance of quapy.protocol.OnLabelledCollectionProtocol, then the aggregation speed-up can be run. This is the protocol in charge of generating the samples in which the model is evaluated.

  • error_metrics – a string, or list of strings, representing the name(s) of an error function in qp.error (e.g., ‘mae’, the default value), or a callable function, or a list of callable functions, implementing the error function itself.

  • aggr_speedup – whether or not to apply the speed-up. Set to “force” for applying it even if the number of instances in the original collection on which the protocol acts is larger than the number of instances in the samples to be generated. Set to True or “auto” (default) for letting QuaPy decide whether it is convenient or not. Set to False to deactivate.

  • verbose – boolean, show or not information in stdout

Returns:

a pandas’ DataFrame containing the columns ‘true-prev’ (the true prevalence of each sample), ‘estim-prev’ (the prevalence estimated by the model for each sample), and as many columns as error metrics have been indicated, each displaying the score in terms of that metric for every sample.

quapy.evaluation.prediction(model: BaseQuantifier, protocol: AbstractProtocol, aggr_speedup: Union[str, bool] = 'auto', verbose=False)

Uses a quantification model to generate predictions for the samples generated via a specific protocol. This function is central to all evaluation processes, and is endowed with an optimization to speed-up the prediction of protocols that generate samples from a large collection. The optimization applies to aggregative quantifiers only, and to OnLabelledCollectionProtocol protocols, and comes down to generating the classification predictions once and for all, and then generating samples over the classification predictions (instead of over the raw instances), so that the classifier prediction is never called again. This behaviour is obtained by setting aggr_speedup to ‘auto’ or True, and is only carried out if the overall process is convenient in terms of computations (e.g., if the number of classification predictions needed for the original collection exceed the number of classification predictions needed for all samples, then the optimization is not undertaken).

Parameters:

  • model – a quantifier, instance of quapy.method.base.BaseQuantifier

  • protocol – quapy.protocol.AbstractProtocol; if this object is also instance of quapy.protocol.OnLabelledCollectionProtocol, then the aggregation speed-up can be run. This is the protocol in charge of generating the samples for which the model has to issue class prevalence predictions.

  • aggr_speedup – whether or not to apply the speed-up. Set to “force” for applying it even if the number of instances in the original collection on which the protocol acts is larger than the number of instances in the samples to be generated. Set to True or “auto” (default) for letting QuaPy decide whether it is convenient or not. Set to False to deactivate.

  • verbose – boolean, show or not information in stdout

Returns:

a tuple (true_prevs, estim_prevs) in which each element in the tuple is an array of shape (n_samples, n_classes) containing the true, or predicted, prevalence values for each sample

quapy.protocol

New in version 0.1.7.

class quapy.protocol.APP(data: LabelledCollection, sample_size=None, n_prevalences=21, repeats=10, smooth_limits_epsilon=0, random_state=0, sanity_check=10000, return_type='sample_prev')

Bases: AbstractStochasticSeededProtocol, OnLabelledCollectionProtocol

Implementation of the artificial prevalence protocol (APP). The APP consists of exploring a grid of prevalence values containing n_prevalences points (e.g., [0, 0.05, 0.1, 0.15, …, 1], if n_prevalences=21), and generating all valid combinations of prevalence values for all classes (e.g., for 3 classes, samples with [0, 0, 1], [0, 0.05, 0.95], …, [1, 0, 0] prevalence values of size sample_size will be yielded). The number of samples for each valid combination of prevalence values is indicated by repeats.

Parameters:

  • data – a LabelledCollection from which the samples will be drawn

  • sample_size – integer, number of instances in each sample; if None (default) then it is taken from qp.environ[“SAMPLE_SIZE”]. If this is not set, a ValueError exception is raised.

  • n_prevalences – the number of equidistant prevalence points to extract from the [0,1] interval for the grid (default is 21)

  • repeats – number of copies for each valid prevalence vector (default is 10)

  • smooth_limits_epsilon – the quantity to add and subtract to the limits 0 and 1

  • random_state – allows replicating samples across runs (default 0, meaning that the sequence of samples will be the same every time the protocol is called)

  • sanity_check – int, raises an exception warning the user that the number of examples to be generated exceed this number; set to None for skipping this check

  • return_type – set to “sample_prev” (default) to get the pairs of (sample, prevalence) at each iteration, or to “labelled_collection” to get instead instances of LabelledCollection

prevalence_grid()

Generates vectors of prevalence values from an exhaustive grid of prevalence values. The number of prevalence values explored for each dimension depends on n_prevalences, so that, if, for example, n_prevalences=11 then the prevalence values of the grid are taken from [0, 0.1, 0.2, …, 0.9, 1]. Only valid prevalence distributions are returned, i.e., vectors of prevalence values that sum up to 1. For each valid vector of prevalence values, repeat copies are returned. The vector of prevalence values can be implicit (by setting return_constrained_dim=False), meaning that the last dimension (which is constrained to 1 - sum of the rest) is not returned (note that, quite obviously, in this case the vector does not sum up to 1). Note that this method is deterministic, i.e., there is no random sampling anywhere.

Returns:

a np.ndarray of shape (n, dimensions) if return_constrained_dim=True or of shape (n, dimensions-1) if return_constrained_dim=False, where n is the number of valid combinations found in the grid multiplied by repeat

sample(index)

Realizes the sample given the index of the instances.

Parameters:

index – indexes of the instances to select

Returns:

an instance of qp.data.LabelledCollection

samples_parameters()

Return all the necessary parameters to replicate the samples as according to the APP protocol.

Returns:

a list of indexes that realize the APP sampling

total()

Returns the number of samples that will be generated

Returns:

int

class quapy.protocol.AbstractProtocol

Bases: object

Abstract parent class for sample generation protocols.

total()

Indicates the total number of samples that the protocol generates.

Returns:

The number of samples to generate if known, or None otherwise.

class quapy.protocol.AbstractStochasticSeededProtocol(random_state=0)

Bases: AbstractProtocol

An AbstractStochasticSeededProtocol is a protocol that generates, via any random procedure (e.g., via random sampling), sequences of quapy.data.base.LabelledCollection samples. The protocol abstraction enforces the object to be instantiated using a seed, so that the sequence can be fully replicated. In order to make this functionality possible, the classes extending this abstraction need to implement only two functions, samples_parameters() which generates all the parameters needed for extracting the samples, and sample() that, given some parameters as input, deterministically generates a sample.

Parameters:

random_state – the seed for allowing to replicate any sequence of samples. Default is 0, meaning that the sequence will be consistent every time the protocol is called.

collator(sample, *args)

The collator prepares the sample to accommodate the desired output format before returning the output. This collator simply returns the sample as it is. Classes inheriting from this abstract class can implement their custom collators.

Parameters:

  • sample – the sample to be returned

  • args – additional arguments

Returns:

the sample adhering to a desired output format (in this case, the sample is returned as it is)

property random_state
abstract sample(params)

Extract one sample determined by the given parameters

Parameters:

params – all the necessary parameters to generate a sample

Returns:

one sample (the same sample has to be generated for the same parameters)

abstract samples_parameters()

This function has to return all the necessary parameters to replicate the samples

Returns:

a list of parameters, each of which serves to deterministically generate a sample

quapy.protocol.ArtificialPrevalenceProtocol

alias of APP

class quapy.protocol.DomainMixer(domainA: LabelledCollection, domainB: LabelledCollection, sample_size, repeats=1, prevalence=None, mixture_points=11, random_state=0, return_type='sample_prev')

Bases: AbstractStochasticSeededProtocol

Generates mixtures of two domains (A and B) at controlled rates, but preserving the original class prevalence.

Parameters:

  • domainA – one domain, an object of qp.data.LabelledCollection

  • domainB – another domain, an object of qp.data.LabelledCollection

  • sample_size – integer, the number of instances in each sample; if None (default) then it is taken from qp.environ[“SAMPLE_SIZE”]. If this is not set, a ValueError exception is raised.

  • repeats – int, number of samples to draw for every mixture rate

  • prevalence – the prevalence to preserv along the mixtures. If specified, should be an array containing one prevalence value (positive float) for each class and summing up to one. If not specified, the prevalence will be taken from the domain A (default).

  • mixture_points – an integer indicating the number of points to take from a linear scale (e.g., 21 will generate the mixture points [1, 0.95, 0.9, …, 0]), or the array of mixture values itself. the specific points

  • random_state – allows replicating samples across runs (default 0, meaning that the sequence of samples will be the same every time the protocol is called)

sample(indexes)

Realizes the sample given a pair of indexes of the instances from A and B.

Parameters:

indexes – indexes of the instances to select from A and B

Returns:

an instance of qp.data.LabelledCollection

samples_parameters()

Return all the necessary parameters to replicate the samples as according to the this protocol.

Returns:

a list of zipped indexes (from A and B) that realize the sampling

total()

Returns the number of samples that will be generated (equals to “repeats * mixture_points”)

Returns:

int

class quapy.protocol.IterateProtocol(samples: [<class 'quapy.data.base.LabelledCollection'>])

Bases: AbstractProtocol

A very simple protocol which simply iterates over a list of previously generated samples

Parameters:

samples – a list of quapy.data.base.LabelledCollection

total()

Returns the number of samples in this protocol

Returns:

int

class quapy.protocol.NPP(data: LabelledCollection, sample_size=None, repeats=100, random_state=0, return_type='sample_prev')

Bases: AbstractStochasticSeededProtocol, OnLabelledCollectionProtocol

A generator of samples that implements the natural prevalence protocol (NPP). The NPP consists of drawing samples uniformly at random, therefore approximately preserving the natural prevalence of the collection.

Parameters:

  • data – a LabelledCollection from which the samples will be drawn

  • sample_size – integer, the number of instances in each sample; if None (default) then it is taken from qp.environ[“SAMPLE_SIZE”]. If this is not set, a ValueError exception is raised.

  • repeats – the number of samples to generate. Default is 100.

  • random_state – allows replicating samples across runs (default 0, meaning that the sequence of samples will be the same every time the protocol is called)

  • return_type – set to “sample_prev” (default) to get the pairs of (sample, prevalence) at each iteration, or to “labelled_collection” to get instead instances of LabelledCollection

sample(index)

Realizes the sample given the index of the instances.

Parameters:

index – indexes of the instances to select

Returns:

an instance of qp.data.LabelledCollection

samples_parameters()

Return all the necessary parameters to replicate the samples as according to the NPP protocol.

Returns:

a list of indexes that realize the NPP sampling

total()

Returns the number of samples that will be generated (equals to “repeats”)

Returns:

int

quapy.protocol.NaturalPrevalenceProtocol

alias of NPP

class quapy.protocol.OnLabelledCollectionProtocol

Bases: object

Protocols that generate samples from a qp.data.LabelledCollection object.

RETURN_TYPES = ['sample_prev', 'labelled_collection', 'index']
classmethod get_collator(return_type='sample_prev')

Returns a collator function, i.e., a function that prepares the yielded data

Parameters:

return_type – either ‘sample_prev’ (default) if the collator is requested to yield tuples of (sample, prevalence), or ‘labelled_collection’ when it is requested to yield instances of qp.data.LabelledCollection

Returns:

the collator function (a callable function that takes as input an instance of qp.data.LabelledCollection)

get_labelled_collection()

Returns the labelled collection on which this protocol acts.

Returns:

an object of type qp.data.LabelledCollection

on_preclassified_instances(pre_classifications, in_place=False)

Returns a copy of this protocol that acts on a modified version of the original qp.data.LabelledCollection in which the original instances have been replaced with the outputs of a classifier for each instance. (This is convenient for speeding-up the evaluation procedures for many samples, by pre-classifying the instances in advance.)

Parameters:

  • pre_classifications – the predictions issued by a classifier, typically an array-like with shape (n_instances,) when the classifier is a hard one, or with shape (n_instances, n_classes) when the classifier is a probabilistic one.

  • in_place – whether or not to apply the modification in-place or in a new copy (default).

Returns:

a copy of this protocol

class quapy.protocol.UPP(data: LabelledCollection, sample_size=None, repeats=100, random_state=0, return_type='sample_prev')

Bases: AbstractStochasticSeededProtocol, OnLabelledCollectionProtocol

A variant of APP that, instead of using a grid of equidistant prevalence values, relies on the Kraemer algorithm for sampling unit (k-1)-simplex uniformly at random, with k the number of classes. This protocol covers the entire range of prevalence values in a statistical sense, i.e., unlike APP there is no guarantee that it is covered precisely equally for all classes, but it is preferred in cases in which the number of possible combinations of the grid values of APP makes this endeavour intractable.

Parameters:

  • data – a LabelledCollection from which the samples will be drawn

  • sample_size – integer, the number of instances in each sample; if None (default) then it is taken from qp.environ[“SAMPLE_SIZE”]. If this is not set, a ValueError exception is raised.

  • repeats – the number of samples to generate. Default is 100.

  • random_state – allows replicating samples across runs (default 0, meaning that the sequence of samples will be the same every time the protocol is called)

  • return_type – set to “sample_prev” (default) to get the pairs of (sample, prevalence) at each iteration, or to “labelled_collection” to get instead instances of LabelledCollection

sample(index)

Realizes the sample given the index of the instances.

Parameters:

index – indexes of the instances to select

Returns:

an instance of qp.data.LabelledCollection

samples_parameters()

Return all the necessary parameters to replicate the samples as according to the UPP protocol.

Returns:

a list of indexes that realize the UPP sampling

total()

Returns the number of samples that will be generated (equals to “repeats”)

Returns:

int

quapy.protocol.UniformPrevalenceProtocol

alias of UPP

quapy.functional

quapy.functional.HellingerDistance(P, Q)

Computes the Hellingher Distance (HD) between (discretized) distributions P and Q. The HD for two discrete distributions of k bins is defined as:

\[HD(P,Q) = \frac{ 1 }{ \sqrt{ 2 } } \sqrt{ \sum_{i=1}^k ( \sqrt{p_i} - \sqrt{q_i} )^2 }\]
Parameters:

  • P – real-valued array-like of shape (k,) representing a discrete distribution

  • Q – real-valued array-like of shape (k,) representing a discrete distribution

Returns:

float

quapy.functional.TopsoeDistance(P, Q, epsilon=1e-20)

Topsoe distance between two (discretized) distributions P and Q. The Topsoe distance for two discrete distributions of k bins is defined as:

\[Topsoe(P,Q) = \sum_{i=1}^k \left( p_i \log\left(\frac{ 2 p_i + \epsilon }{ p_i+q_i+\epsilon }\right) + q_i \log\left(\frac{ 2 q_i + \epsilon }{ p_i+q_i+\epsilon }\right) \right)\]
Parameters:

  • P – real-valued array-like of shape (k,) representing a discrete distribution

  • Q – real-valued array-like of shape (k,) representing a discrete distribution

Returns:

float

quapy.functional.adjusted_quantification(prevalence_estim, tpr, fpr, clip=True)

Implements the adjustment of ACC and PACC for the binary case. The adjustment for a prevalence estimate of the positive class p comes down to computing:

\[ACC(p) = \frac{ p - fpr }{ tpr - fpr }\]
Parameters:

  • prevalence_estim – float, the estimated value for the positive class

  • tpr – float, the true positive rate of the classifier

  • fpr – float, the false positive rate of the classifier

  • clip – set to True (default) to clip values that might exceed the range [0,1]

Returns:

float, the adjusted count

quapy.functional.check_prevalence_vector(p, raise_exception=False, toleranze=1e-08)

Checks that p is a valid prevalence vector, i.e., that it contains values in [0,1] and that the values sum up to 1.

Parameters:

p – the prevalence vector to check

Returns:

True if p is valid, False otherwise

quapy.functional.get_nprevpoints_approximation(combinations_budget: int, n_classes: int, n_repeats: int = 1)

Searches for the largest number of (equidistant) prevalence points to define for each of the n_classes classes so that the number of valid prevalence values generated as combinations of prevalence points (points in a n_classes-dimensional simplex) do not exceed combinations_budget.

Parameters:

  • combinations_budget – integer, maximum number of combinations allowed

  • n_classes – integer, number of classes

  • n_repeats – integer, number of repetitions for each prevalence combination

Returns:

the largest number of prevalence points that generate less than combinations_budget valid prevalences

quapy.functional.normalize_prevalence(prevalences)

Normalize a vector or matrix of prevalence values. The normalization consists of applying a L1 normalization in cases in which the prevalence values are not all-zeros, and to convert the prevalence values into 1/n_classes in cases in which all values are zero.

Parameters:

prevalences – array-like of shape (n_classes,) or of shape (n_samples, n_classes,) with prevalence values

Returns:

a normalized vector or matrix of prevalence values

quapy.functional.num_prevalence_combinations(n_prevpoints: int, n_classes: int, n_repeats: int = 1)

Computes the number of valid prevalence combinations in the n_classes-dimensional simplex if n_prevpoints equally distant prevalence values are generated and n_repeats repetitions are requested. The computation comes down to calculating:

\[\binom{N+C-1}{C-1} \times r\]

where N is n_prevpoints-1, i.e., the number of probability mass blocks to allocate, C is the number of classes, and r is n_repeats. This solution comes from the Stars and Bars problem.

Parameters:

  • n_classes – integer, number of classes

  • n_prevpoints – integer, number of prevalence points.

  • n_repeats – integer, number of repetitions for each prevalence combination

Returns:

The number of possible combinations. For example, if n_classes=2, n_prevpoints=5, n_repeats=1, then the number of possible combinations are 5, i.e.: [0,1], [0.25,0.75], [0.50,0.50], [0.75,0.25], and [1.0,0.0]

quapy.functional.prevalence_from_labels(labels, classes)

Computed the prevalence values from a vector of labels.

Parameters:

  • labels – array-like of shape (n_instances) with the label for each instance

  • classes – the class labels. This is needed in order to correctly compute the prevalence vector even when some classes have no examples.

Returns:

an ndarray of shape (len(classes)) with the class prevalence values

quapy.functional.prevalence_from_probabilities(posteriors, binarize: bool = False)

Returns a vector of prevalence values from a matrix of posterior probabilities.

Parameters:

  • posteriors – array-like of shape (n_instances, n_classes,) with posterior probabilities for each class

  • binarize – set to True (default is False) for computing the prevalence values on crisp decisions (i.e., converting the vectors of posterior probabilities into class indices, by taking the argmax).

Returns:

array of shape (n_classes,) containing the prevalence values

quapy.functional.prevalence_linspace(n_prevalences=21, repeats=1, smooth_limits_epsilon=0.01)

Produces an array of uniformly separated values of prevalence. By default, produces an array of 21 prevalence values, with step 0.05 and with the limits smoothed, i.e.: [0.01, 0.05, 0.10, 0.15, …, 0.90, 0.95, 0.99]

Parameters:

  • n_prevalences – the number of prevalence values to sample from the [0,1] interval (default 21)

  • repeats – number of times each prevalence is to be repeated (defaults to 1)

  • smooth_limits_epsilon – the quantity to add and subtract to the limits 0 and 1

Returns:

an array of uniformly separated prevalence values

quapy.functional.strprev(prevalences, prec=3)

Returns a string representation for a prevalence vector. E.g.,

>>> strprev([1/3, 2/3], prec=2)
>>> '[0.33, 0.67]'
Parameters:

  • prevalences – a vector of prevalence values

  • prec – float precision

Returns:

string

quapy.functional.uniform_prevalence_sampling(n_classes, size=1)

Implements the Kraemer algorithm for sampling uniformly at random from the unit simplex. This implementation is adapted from this post <https://cs.stackexchange.com/questions/3227/uniform-sampling-from-a-simplex>_.

Parameters:

  • n_classes – integer, number of classes (dimensionality of the simplex)

  • size – number of samples to return

Returns:

np.ndarray of shape (size, n_classes,) if size>1, or of shape (n_classes,) otherwise

quapy.functional.uniform_simplex_sampling(n_classes, size=1)

Implements the Kraemer algorithm for sampling uniformly at random from the unit simplex. This implementation is adapted from this post <https://cs.stackexchange.com/questions/3227/uniform-sampling-from-a-simplex>_.

Parameters:

  • n_classes – integer, number of classes (dimensionality of the simplex)

  • size – number of samples to return

Returns:

np.ndarray of shape (size, n_classes,) if size>1, or of shape (n_classes,) otherwise

quapy.model_selection

class quapy.model_selection.GridSearchQ(model: ~quapy.method.base.BaseQuantifier, param_grid: dict, protocol: ~quapy.protocol.AbstractProtocol, error: ~typing.Union[~typing.Callable, str] = <function mae>, refit=True, timeout=-1, n_jobs=None, verbose=False)

Bases: BaseQuantifier

Grid Search optimization targeting a quantification-oriented metric.

Optimizes the hyperparameters of a quantification method, based on an evaluation method and on an evaluation protocol for quantification.

Parameters:

  • model (BaseQuantifier) – the quantifier to optimize

  • param_grid – a dictionary with keys the parameter names and values the list of values to explore

  • protocol – a sample generation protocol, an instance of quapy.protocol.AbstractProtocol

  • error – an error function (callable) or a string indicating the name of an error function (valid ones are those in quapy.error.QUANTIFICATION_ERROR

  • refit – whether or not to refit the model on the whole labelled collection (training+validation) with the best chosen hyperparameter combination. Ignored if protocol=’gen’

  • timeout – establishes a timer (in seconds) for each of the hyperparameters configurations being tested. Whenever a run takes longer than this timer, that configuration will be ignored. If all configurations end up being ignored, a TimeoutError exception is raised. If -1 (default) then no time bound is set.

  • verbose – set to True to get information through the stdout

best_model()

Returns the best model found after calling the fit() method, i.e., the one trained on the combination of hyper-parameters that minimized the error function.

Returns:

a trained quantifier

fit(training: LabelledCollection)
Learning routine. Fits methods with all combinations of hyperparameters and selects the one minimizing

the error metric.

Parameters:

training – the training set on which to optimize the hyperparameters

Returns:

self

get_params(deep=True)

Returns the dictionary of hyper-parameters to explore (param_grid)

Parameters:

deep – Unused

Returns:

the dictionary param_grid

quantify(instances)

Estimate class prevalence values using the best model found after calling the fit() method.

Parameters:

instances – sample contanining the instances

Returns:

a ndarray of shape (n_classes) with class prevalence estimates as according to the best model found by the model selection process.

set_params(**parameters)

Sets the hyper-parameters to explore.

Parameters:

parameters – a dictionary with keys the parameter names and values the list of values to explore

quapy.model_selection.cross_val_predict(quantifier: BaseQuantifier, data: LabelledCollection, nfolds=3, random_state=0)

Akin to scikit-learn’s cross_val_predict but for quantification.

Parameters:

  • quantifier – a quantifier issuing class prevalence values

  • data – a labelled collection

  • nfolds – number of folds for k-fold cross validation generation

  • random_state – random seed for reproducibility

Returns:

a vector of class prevalence values

quapy.plot

quapy.plot.binary_bias_bins(method_names, true_prevs, estim_prevs, pos_class=1, title=None, nbins=5, colormap=<matplotlib.colors.ListedColormap object>, vertical_xticks=False, legend=True, savepath=None)

Box-plots displaying the local bias (i.e., signed error computed as the estimated value minus the true value) for different bins of (true) prevalence of the positive classs, for each quantification method.

Parameters:

  • method_names – array-like with the method names for each experiment

  • true_prevs – array-like with the true prevalence values (each being a ndarray with n_classes components) for each experiment

  • estim_prevs – array-like with the estimated prevalence values (each being a ndarray with n_classes components) for each experiment

  • pos_class – index of the positive class

  • title – the title to be displayed in the plot

  • nbins – number of bins

  • colormap – the matplotlib colormap to use (default cm.tab10)

  • vertical_xticks – whether or not to add secondary grid (default is False)

  • legend – whether or not to display the legend (default is True)

  • savepath – path where to save the plot. If not indicated (as default), the plot is shown.

quapy.plot.binary_bias_global(method_names, true_prevs, estim_prevs, pos_class=1, title=None, savepath=None)

Box-plots displaying the global bias (i.e., signed error computed as the estimated value minus the true value) for each quantification method with respect to a given positive class.

Parameters:

  • method_names – array-like with the method names for each experiment

  • true_prevs – array-like with the true prevalence values (each being a ndarray with n_classes components) for each experiment

  • estim_prevs – array-like with the estimated prevalence values (each being a ndarray with n_classes components) for each experiment

  • pos_class – index of the positive class

  • title – the title to be displayed in the plot

  • savepath – path where to save the plot. If not indicated (as default), the plot is shown.

quapy.plot.binary_diagonal(method_names, true_prevs, estim_prevs, pos_class=1, title=None, show_std=True, legend=True, train_prev=None, savepath=None, method_order=None)

The diagonal plot displays the predicted prevalence values (along the y-axis) as a function of the true prevalence values (along the x-axis). The optimal quantifier is described by the diagonal (0,0)-(1,1) of the plot (hence the name). It is convenient for binary quantification problems, though it can be used for multiclass problems by indicating which class is to be taken as the positive class. (For multiclass quantification problems, other plots like the error_by_drift() might be preferable though).

Parameters:

  • method_names – array-like with the method names for each experiment

  • true_prevs – array-like with the true prevalence values (each being a ndarray with n_classes components) for each experiment

  • estim_prevs – array-like with the estimated prevalence values (each being a ndarray with n_classes components) for each experiment

  • pos_class – index of the positive class

  • title – the title to be displayed in the plot

  • show_std – whether or not to show standard deviations (represented by color bands). This might be inconvenient for cases in which many methods are compared, or when the standard deviations are high – default True)

  • legend – whether or not to display the leyend (default True)

  • train_prev – if indicated (default is None), the training prevalence (for the positive class) is hightlighted in the plot. This is convenient when all the experiments have been conducted in the same dataset.

  • savepath – path where to save the plot. If not indicated (as default), the plot is shown.

  • method_order – if indicated (default is None), imposes the order in which the methods are processed (i.e., listed in the legend and associated with matplotlib colors).

quapy.plot.brokenbar_supremacy_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, n_bins=20, binning='isomerous', x_error='ae', y_error='ae', ttest_alpha=0.005, tail_density_threshold=0.005, method_order=None, savepath=None)

Displays (only) the top performing methods for different regions of the train-test shift in form of a broken bar chart, in which each method has bars only for those regions in which either one of the following conditions hold: (i) it is the best method (in average) for the bin, or (ii) it is not statistically significantly different (in average) as according to a two-sided t-test on independent samples at confidence ttest_alpha. The binning can be made “isometric” (same size), or “isomerous” (same number of experiments – default). A second plot is displayed on top, that displays the distribution of experiments for each bin (when binning=”isometric”) or the percentiles points of the distribution (when binning=”isomerous”).

Parameters:

  • method_names – array-like with the method names for each experiment

  • true_prevs – array-like with the true prevalence values (each being a ndarray with n_classes components) for each experiment

  • estim_prevs – array-like with the estimated prevalence values (each being a ndarray with n_classes components) for each experiment

  • tr_prevs – training prevalence of each experiment

  • n_bins – number of bins in which the y-axis is to be divided (default is 20)

  • binning – type of binning, either “isomerous” (default) or “isometric”

  • x_error – a string representing the name of an error function (as defined in quapy.error) to be used for measuring the amount of train-test shift (default is “ae”)

  • y_error – a string representing the name of an error function (as defined in quapy.error) to be used for measuring the amount of error in the prevalence estimations (default is “ae”)

  • ttest_alpha – the confidence interval above which a p-value (two-sided t-test on independent samples) is to be considered as an indicator that the two means are not statistically significantly different. Default is 0.005, meaning that a p-value > 0.005 indicates the two methods involved are to be considered similar

  • tail_density_threshold – sets a threshold on the density of experiments (over the total number of experiments) below which a bin in the tail (i.e., the right-most ones) will be discarded. This is in order to avoid some bins to be shown for train-test outliers.

  • method_order – if indicated (default is None), imposes the order in which the methods are processed (i.e., listed in the legend and associated with matplotlib colors).

  • savepath – path where to save the plot. If not indicated (as default), the plot is shown.

Returns:

quapy.plot.error_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, n_bins=20, error_name='ae', show_std=False, show_density=True, show_legend=True, logscale=False, title='Quantification error as a function of distribution shift', vlines=None, method_order=None, savepath=None)

Plots the error (along the x-axis, as measured in terms of error_name) as a function of the train-test shift (along the y-axis, as measured in terms of quapy.error.ae()). This plot is useful especially for multiclass problems, in which “diagonal plots” may be cumbersone, and in order to gain understanding about how methods fare in different regions of the prior probability shift spectrum (e.g., in the low-shift regime vs. in the high-shift regime).

Parameters:

  • method_names – array-like with the method names for each experiment

  • true_prevs – array-like with the true prevalence values (each being a ndarray with n_classes components) for each experiment

  • estim_prevs – array-like with the estimated prevalence values (each being a ndarray with n_classes components) for each experiment

  • tr_prevs – training prevalence of each experiment

  • n_bins – number of bins in which the y-axis is to be divided (default is 20)

  • error_name – a string representing the name of an error function (as defined in quapy.error, default is “ae”)

  • show_std – whether or not to show standard deviations as color bands (default is False)

  • show_density – whether or not to display the distribution of experiments for each bin (default is True)

  • show_density – whether or not to display the legend of the chart (default is True)

  • logscale – whether or not to log-scale the y-error measure (default is False)

  • title – title of the plot (default is “Quantification error as a function of distribution shift”)

  • vlines – array-like list of values (default is None). If indicated, highlights some regions of the space using vertical dotted lines.

  • method_order – if indicated (default is None), imposes the order in which the methods are processed (i.e., listed in the legend and associated with matplotlib colors).

  • savepath – path where to save the plot. If not indicated (as default), the plot is shown.

quapy.util

class quapy.util.EarlyStop(patience, lower_is_better=True)

Bases: object

A class implementing the early-stopping condition typically used for training neural networks.

>>> earlystop = EarlyStop(patience=2, lower_is_better=True)
>>> earlystop(0.9, epoch=0)
>>> earlystop(0.7, epoch=1)
>>> earlystop.IMPROVED  # is True
>>> earlystop(1.0, epoch=2)
>>> earlystop.STOP  # is False (patience=1)
>>> earlystop(1.0, epoch=3)
>>> earlystop.STOP  # is True (patience=0)
>>> earlystop.best_epoch  # is 1
>>> earlystop.best_score  # is 0.7
Parameters:

  • patience – the number of (consecutive) times that a monitored evaluation metric (typically obtaind in a held-out validation split) can be found to be worse than the best one obtained so far, before flagging the stopping condition. An instance of this class is callable, and is to be used as follows:

  • lower_is_better – if True (default) the metric is to be minimized.

Variables:

  • best_score – keeps track of the best value seen so far

  • best_epoch – keeps track of the epoch in which the best score was set

  • STOP – flag (boolean) indicating the stopping condition

  • IMPROVED – flag (boolean) indicating whether there was an improvement in the last call

quapy.util.create_if_not_exist(path)

An alias to os.makedirs(path, exist_ok=True) that also returns the path. This is useful in cases like, e.g.:

>>> path = create_if_not_exist(os.path.join(dir, subdir, anotherdir))
Parameters:

path – path to create

Returns:

the path itself

quapy.util.create_parent_dir(path)

Creates the parent dir (if any) of a given path, if not exists. E.g., for ./path/to/file.txt, the path ./path/to is created.

Parameters:

path – the path

quapy.util.download_file(url, archive_filename)

Downloads a file from a url

Parameters:

  • url – the url

  • archive_filename – destination filename

quapy.util.download_file_if_not_exists(url, archive_filename)

Dowloads a function (using download_file()) if the file does not exist.

Parameters:

  • url – the url

  • archive_filename – destination filename

quapy.util.get_quapy_home()

Gets the home directory of QuaPy, i.e., the directory where QuaPy saves permanent data, such as dowloaded datasets. This directory is ~/quapy_data

Returns:

a string representing the path

quapy.util.map_parallel(func, args, n_jobs)

Applies func to n_jobs slices of args. E.g., if args is an array of 99 items and `n_jobs`=2, then func is applied in two parallel processes to args[0:50] and to args[50:99]. func is a function that already works with a list of arguments.

Parameters:

  • func – function to be parallelized

  • args – array-like of arguments to be passed to the function in different parallel calls

  • n_jobs – the number of workers

quapy.util.parallel(func, args, n_jobs, seed=None)

A wrapper of multiprocessing:

>>> Parallel(n_jobs=n_jobs)(
>>>      delayed(func)(args_i) for args_i in args
>>> )

that takes the quapy.environ variable as input silently. Seeds the child processes to ensure reproducibility when n_jobs>1

quapy.util.pickled_resource(pickle_path: str, generation_func: callable, *args)

Allows for fast reuse of resources that are generated only once by calling generation_func(*args). The next times this function is invoked, it loads the pickled resource. Example:

>>> def some_array(n):  # a mock resource created with one parameter (`n`)
>>>     return np.random.rand(n)
>>> pickled_resource('./my_array.pkl', some_array, 10)  # the resource does not exist: it is created by calling some_array(10)
>>> pickled_resource('./my_array.pkl', some_array, 10)  # the resource exists; it is loaded from './my_array.pkl'
Parameters:

  • pickle_path – the path where to save (first time) and load (next times) the resource

  • generation_func – the function that generates the resource, in case it does not exist in pickle_path

  • args – any arg that generation_func uses for generating the resources

Returns:

the resource

quapy.util.save_text_file(path, text)

Saves a text file to disk, given its full path, and creates the parent directory if missing.

Parameters:

  • path – path where to save the path.

  • text – text to save.

quapy.util.temp_seed(random_state)

Can be used in a “with” context to set a temporal seed without modifying the outer numpy’s current state. E.g.:

>>> with temp_seed(random_seed):
>>>  pass # do any computation depending on np.random functionality
Parameters:

random_state – the seed to set within the “with” context

Subpackages

Module contents

QuaPy module for quantification