diff --git a/README.md b/README.md index 10c769f..404fa71 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ for facilitating the analysis and interpretation of the experimental results. ### Last updates: +* Version 0.1.7 is released! major changes can be consulted [here](quapy/CHANGE_LOG.txt). * A detailed documentation is now available [here](https://hlt-isti.github.io/QuaPy/) * The developer API documentation is available [here](https://hlt-isti.github.io/QuaPy/build/html/modules.html) @@ -22,6 +23,20 @@ for facilitating the analysis and interpretation of the experimental results. pip install quapy ``` +### Cite QuaPy + +If you find QuaPy useful (and we hope you will), plese consider citing the original paper in your research: + +``` +@inproceedings{moreo2021quapy, + title={QuaPy: a python-based framework for quantification}, + author={Moreo, Alejandro and Esuli, Andrea and Sebastiani, Fabrizio}, + booktitle={Proceedings of the 30th ACM International Conference on Information \& Knowledge Management}, + pages={4534--4543}, + year={2021} +} +``` + ## A quick example: The following script fetches a dataset of tweets, trains, applies, and evaluates a quantifier based on the @@ -59,13 +74,14 @@ See the [Wiki](https://github.com/HLT-ISTI/QuaPy/wiki) for detailed examples. ## Features * Implementation of many popular quantification methods (Classify-&-Count and its variants, Expectation Maximization, -quantification methods based on structured output learning, HDy, QuaNet, and quantification ensembles). -* Versatile functionality for performing evaluation based on artificial sampling protocols. +quantification methods based on structured output learning, HDy, QuaNet, quantification ensembles, among others). +* Versatile functionality for performing evaluation based on sampling generation protocols (e.g., APP, NPP, etc.). * Implementation of most commonly used evaluation metrics (e.g., AE, RAE, SE, KLD, NKLD, etc.). * Datasets frequently used in quantification (textual and numeric), including: * 32 UCI Machine Learning datasets. * 11 Twitter quantification-by-sentiment datasets. * 3 product reviews quantification-by-sentiment datasets. + * 4 tasks from LeQua competition (_new in v0.1.7!_) * Native support for binary and single-label multiclass quantification scenarios. * Model selection functionality that minimizes quantification-oriented loss functions. * Visualization tools for analysing the experimental results. @@ -80,29 +96,6 @@ quantification methods based on structured output learning, HDy, QuaNet, and qua * pandas, xlrd * matplotlib -## SVM-perf with quantification-oriented losses -In order to run experiments involving SVM(Q), SVM(KLD), SVM(NKLD), -SVM(AE), or SVM(RAE), you have to first download the -[svmperf](http://www.cs.cornell.edu/people/tj/svm_light/svm_perf.html) -package, apply the patch -[svm-perf-quantification-ext.patch](./svm-perf-quantification-ext.patch), and compile the sources. -The script [prepare_svmperf.sh](prepare_svmperf.sh) does all the job. Simply run: - -``` -./prepare_svmperf.sh -``` - -The resulting directory [svm_perf_quantification](./svm_perf_quantification) contains the -patched version of _svmperf_ with quantification-oriented losses. - -The [svm-perf-quantification-ext.patch](./svm-perf-quantification-ext.patch) is an extension of the patch made available by -[Esuli et al. 2015](https://dl.acm.org/doi/abs/10.1145/2700406?casa_token=8D2fHsGCVn0AAAAA:ZfThYOvrzWxMGfZYlQW_y8Cagg-o_l6X_PcF09mdETQ4Tu7jK98mxFbGSXp9ZSO14JkUIYuDGFG0) -that allows SVMperf to optimize for -the _Q_ measure as proposed by [Barranquero et al. 2015](https://www.sciencedirect.com/science/article/abs/pii/S003132031400291X) -and for the _KLD_ and _NKLD_ measures as proposed by [Esuli et al. 2015](https://dl.acm.org/doi/abs/10.1145/2700406?casa_token=8D2fHsGCVn0AAAAA:ZfThYOvrzWxMGfZYlQW_y8Cagg-o_l6X_PcF09mdETQ4Tu7jK98mxFbGSXp9ZSO14JkUIYuDGFG0). -This patch extends the above one by also allowing SVMperf to optimize for -_AE_ and _RAE_. - ## Documentation @@ -113,6 +106,8 @@ are provided: * [Datasets](https://github.com/HLT-ISTI/QuaPy/wiki/Datasets) * [Evaluation](https://github.com/HLT-ISTI/QuaPy/wiki/Evaluation) +* [Protocols](https://github.com/HLT-ISTI/QuaPy/wiki/Protocols) * [Methods](https://github.com/HLT-ISTI/QuaPy/wiki/Methods) +* [SVMperf](https://github.com/HLT-ISTI/QuaPy/wiki/ExplicitLossMinimization) * [Model Selection](https://github.com/HLT-ISTI/QuaPy/wiki/Model-Selection) * [Plotting](https://github.com/HLT-ISTI/QuaPy/wiki/Plotting) diff --git a/TODO.txt b/TODO.txt index 8a674a7..36b7e95 100644 --- a/TODO.txt +++ b/TODO.txt @@ -1,7 +1,20 @@ +sample_size should not be mandatory when qp.environ['SAMPLE_SIZE'] has been specified +clean all the cumbersome methods that have to be implemented for new quantifiers (e.g., n_classes_ prop, etc.) +make truly parallel the GridSearchQ +make more examples in the "examples" directory +merge with master, because I had to fix some problems with QuaNet due to an issue notified via GitHub! +added cross_val_predict in qp.model_selection (i.e., a cross_val_predict for quantification) --would be nice to have + it parallelized + +check the OneVsAll module(s) + +check the set_params de neural.py, because the separation of estimator__ is not implemented; see also + __check_params_colision + +HDy can be customized so that the number of bins is specified, instead of explored within the fit method Packaging: ========================================== -Documentation with sphinx Document methods with paper references unit-tests clean wiki_examples! diff --git a/docs/build/html/Datasets.html b/docs/build/html/Datasets.html index 6af836e..775690d 100644 --- a/docs/build/html/Datasets.html +++ b/docs/build/html/Datasets.html @@ -2,23 +2,26 @@ - + - - Datasets — QuaPy 0.1.6 documentation + + + Datasets — QuaPy 0.1.7 documentation + + - - + + - - - -
-
-
-
- -
-

quapy.tests package

-
-

Submodules

-
-
-

quapy.tests.test_base module

-
-
-

quapy.tests.test_datasets module

-
-
-

quapy.tests.test_methods module

-
-
-

Module contents

-
-
- - -
-
-
-
- -
-
- - - - \ No newline at end of file diff --git a/docs/build/html/readme.html b/docs/build/html/readme.html deleted file mode 100644 index c223f24..0000000 --- a/docs/build/html/readme.html +++ /dev/null @@ -1,129 +0,0 @@ - - - - - - - - - Getting Started — QuaPy 0.1.6 documentation - - - - - - - - - - - - - - - - - -
-
-
-
- -
-

Getting Started

-

QuaPy is an open source framework for Quantification (a.k.a. Supervised Prevalence Estimation) written in Python.

-
-

Installation

-
>>> pip install quapy
-
-
-
-
- - -
-
-
-
- -
-
- - - - \ No newline at end of file diff --git a/docs/build/html/readme2.html b/docs/build/html/readme2.html deleted file mode 100644 index e5ff4a6..0000000 --- a/docs/build/html/readme2.html +++ /dev/null @@ -1,92 +0,0 @@ - - - - - - - - - <no title> — QuaPy 0.1.6 documentation - - - - - - - - - - - - - - - -
-
-
-
- -

.. include:: ../../README.md

- - -
-
-
-
- -
-
- - - - \ No newline at end of file diff --git a/docs/build/html/search.html b/docs/build/html/search.html index 2090979..480e246 100644 --- a/docs/build/html/search.html +++ b/docs/build/html/search.html @@ -2,11 +2,11 @@ - + - Search — QuaPy 0.1.6 documentation + Search — QuaPy 0.1.7 documentation @@ -14,7 +14,9 @@ + + @@ -37,7 +39,7 @@
  • modules |
    • -
  • QuaPy 0.1.6 documentation »
    • +
  • QuaPy 0.1.7 documentation »
  • Search
    • @@ -97,13 +99,13 @@
  • modules |
    • -
  • QuaPy 0.1.6 documentation »
    • +
  • QuaPy 0.1.7 documentation »
  • Search
    • \ No newline at end of file diff --git a/docs/build/html/searchindex.js b/docs/build/html/searchindex.js index 2c03e3a..bc79dbe 100644 --- a/docs/build/html/searchindex.js +++ b/docs/build/html/searchindex.js @@ -1 +1 @@ -Search.setIndex({docnames:["Datasets","Evaluation","Installation","Methods","Model-Selection","Plotting","index","modules","quapy","quapy.classification","quapy.data","quapy.method"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":4,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":3,"sphinx.domains.rst":2,"sphinx.domains.std":2,sphinx:56},filenames:["Datasets.md","Evaluation.md","Installation.rst","Methods.md","Model-Selection.md","Plotting.md","index.rst","modules.rst","quapy.rst","quapy.classification.rst","quapy.data.rst","quapy.method.rst"],objects:{"":{quapy:[8,0,0,"-"]},"quapy.classification":{methods:[9,0,0,"-"],neural:[9,0,0,"-"],svmperf:[9,0,0,"-"]},"quapy.classification.methods":{LowRankLogisticRegression:[9,1,1,""]},"quapy.classification.methods.LowRankLogisticRegression":{fit:[9,2,1,""],get_params:[9,2,1,""],predict:[9,2,1,""],predict_proba:[9,2,1,""],set_params:[9,2,1,""],transform:[9,2,1,""]},"quapy.classification.neural":{CNNnet:[9,1,1,""],LSTMnet:[9,1,1,""],NeuralClassifierTrainer:[9,1,1,""],TextClassifierNet:[9,1,1,""],TorchDataset:[9,1,1,""]},"quapy.classification.neural.CNNnet":{document_embedding:[9,2,1,""],get_params:[9,2,1,""],vocabulary_size:[9,3,1,""]},"quapy.classification.neural.LSTMnet":{document_embedding:[9,2,1,""],get_params:[9,2,1,""],vocabulary_size:[9,3,1,""]},"quapy.classification.neural.NeuralClassifierTrainer":{device:[9,3,1,""],fit:[9,2,1,""],get_params:[9,2,1,""],predict:[9,2,1,""],predict_proba:[9,2,1,""],reset_net_params:[9,2,1,""],set_params:[9,2,1,""],transform:[9,2,1,""]},"quapy.classification.neural.TextClassifierNet":{dimensions:[9,2,1,""],document_embedding:[9,2,1,""],forward:[9,2,1,""],get_params:[9,2,1,""],predict_proba:[9,2,1,""],vocabulary_size:[9,3,1,""],xavier_uniform:[9,2,1,""]},"quapy.classification.neural.TorchDataset":{asDataloader:[9,2,1,""]},"quapy.classification.svmperf":{SVMperf:[9,1,1,""]},"quapy.classification.svmperf.SVMperf":{decision_function:[9,2,1,""],fit:[9,2,1,""],predict:[9,2,1,""],set_params:[9,2,1,""],valid_losses:[9,4,1,""]},"quapy.data":{base:[10,0,0,"-"],datasets:[10,0,0,"-"],preprocessing:[10,0,0,"-"],reader:[10,0,0,"-"]},"quapy.data.base":{Dataset:[10,1,1,""],LabelledCollection:[10,1,1,""],isbinary:[10,5,1,""]},"quapy.data.base.Dataset":{SplitStratified:[10,2,1,""],binary:[10,3,1,""],classes_:[10,3,1,""],kFCV:[10,2,1,""],load:[10,2,1,""],n_classes:[10,3,1,""],stats:[10,2,1,""],vocabulary_size:[10,3,1,""]},"quapy.data.base.LabelledCollection":{Xy:[10,3,1,""],artificial_sampling_generator:[10,2,1,""],artificial_sampling_index_generator:[10,2,1,""],binary:[10,3,1,""],counts:[10,2,1,""],kFCV:[10,2,1,""],load:[10,2,1,""],n_classes:[10,3,1,""],natural_sampling_generator:[10,2,1,""],natural_sampling_index_generator:[10,2,1,""],prevalence:[10,2,1,""],sampling:[10,2,1,""],sampling_from_index:[10,2,1,""],sampling_index:[10,2,1,""],split_stratified:[10,2,1,""],stats:[10,2,1,""],uniform_sampling:[10,2,1,""],uniform_sampling_index:[10,2,1,""]},"quapy.data.datasets":{fetch_UCIDataset:[10,5,1,""],fetch_UCILabelledCollection:[10,5,1,""],fetch_reviews:[10,5,1,""],fetch_twitter:[10,5,1,""],warn:[10,5,1,""]},"quapy.data.preprocessing":{IndexTransformer:[10,1,1,""],index:[10,5,1,""],reduce_columns:[10,5,1,""],standardize:[10,5,1,""],text2tfidf:[10,5,1,""]},"quapy.data.preprocessing.IndexTransformer":{add_word:[10,2,1,""],fit:[10,2,1,""],fit_transform:[10,2,1,""],transform:[10,2,1,""],vocabulary_size:[10,2,1,""]},"quapy.data.reader":{binarize:[10,5,1,""],from_csv:[10,5,1,""],from_sparse:[10,5,1,""],from_text:[10,5,1,""],reindex_labels:[10,5,1,""]},"quapy.error":{absolute_error:[8,5,1,""],acc_error:[8,5,1,""],acce:[8,5,1,""],ae:[8,5,1,""],f1_error:[8,5,1,""],f1e:[8,5,1,""],from_name:[8,5,1,""],kld:[8,5,1,""],mae:[8,5,1,""],mean_absolute_error:[8,5,1,""],mean_relative_absolute_error:[8,5,1,""],mkld:[8,5,1,""],mnkld:[8,5,1,""],mrae:[8,5,1,""],mse:[8,5,1,""],nkld:[8,5,1,""],rae:[8,5,1,""],relative_absolute_error:[8,5,1,""],se:[8,5,1,""],smooth:[8,5,1,""]},"quapy.evaluation":{artificial_prevalence_prediction:[8,5,1,""],artificial_prevalence_protocol:[8,5,1,""],artificial_prevalence_report:[8,5,1,""],evaluate:[8,5,1,""],gen_prevalence_prediction:[8,5,1,""],gen_prevalence_report:[8,5,1,""],natural_prevalence_prediction:[8,5,1,""],natural_prevalence_protocol:[8,5,1,""],natural_prevalence_report:[8,5,1,""]},"quapy.functional":{HellingerDistance:[8,5,1,""],adjusted_quantification:[8,5,1,""],artificial_prevalence_sampling:[8,5,1,""],get_nprevpoints_approximation:[8,5,1,""],normalize_prevalence:[8,5,1,""],num_prevalence_combinations:[8,5,1,""],prevalence_from_labels:[8,5,1,""],prevalence_from_probabilities:[8,5,1,""],prevalence_linspace:[8,5,1,""],strprev:[8,5,1,""],uniform_prevalence_sampling:[8,5,1,""],uniform_simplex_sampling:[8,5,1,""]},"quapy.method":{aggregative:[11,0,0,"-"],base:[11,0,0,"-"],meta:[11,0,0,"-"],neural:[11,0,0,"-"],non_aggregative:[11,0,0,"-"]},"quapy.method.aggregative":{ACC:[11,1,1,""],AdjustedClassifyAndCount:[11,4,1,""],AggregativeProbabilisticQuantifier:[11,1,1,""],AggregativeQuantifier:[11,1,1,""],CC:[11,1,1,""],ClassifyAndCount:[11,4,1,""],ELM:[11,1,1,""],EMQ:[11,1,1,""],ExpectationMaximizationQuantifier:[11,4,1,""],ExplicitLossMinimisation:[11,4,1,""],HDy:[11,1,1,""],HellingerDistanceY:[11,4,1,""],MAX:[11,1,1,""],MS2:[11,1,1,""],MS:[11,1,1,""],MedianSweep2:[11,4,1,""],MedianSweep:[11,4,1,""],OneVsAll:[11,1,1,""],PACC:[11,1,1,""],PCC:[11,1,1,""],ProbabilisticAdjustedClassifyAndCount:[11,4,1,""],ProbabilisticClassifyAndCount:[11,4,1,""],SLD:[11,4,1,""],SVMAE:[11,1,1,""],SVMKLD:[11,1,1,""],SVMNKLD:[11,1,1,""],SVMQ:[11,1,1,""],SVMRAE:[11,1,1,""],T50:[11,1,1,""],ThresholdOptimization:[11,1,1,""],X:[11,1,1,""]},"quapy.method.aggregative.ACC":{aggregate:[11,2,1,""],classify:[11,2,1,""],fit:[11,2,1,""],solve_adjustment:[11,2,1,""]},"quapy.method.aggregative.AggregativeProbabilisticQuantifier":{posterior_probabilities:[11,2,1,""],predict_proba:[11,2,1,""],probabilistic:[11,3,1,""],quantify:[11,2,1,""],set_params:[11,2,1,""]},"quapy.method.aggregative.AggregativeQuantifier":{aggregate:[11,2,1,""],aggregative:[11,3,1,""],classes_:[11,3,1,""],classify:[11,2,1,""],fit:[11,2,1,""],get_params:[11,2,1,""],learner:[11,3,1,""],quantify:[11,2,1,""],set_params:[11,2,1,""]},"quapy.method.aggregative.CC":{aggregate:[11,2,1,""],fit:[11,2,1,""]},"quapy.method.aggregative.ELM":{aggregate:[11,2,1,""],classify:[11,2,1,""],fit:[11,2,1,""]},"quapy.method.aggregative.EMQ":{EM:[11,2,1,""],EPSILON:[11,4,1,""],MAX_ITER:[11,4,1,""],aggregate:[11,2,1,""],fit:[11,2,1,""],predict_proba:[11,2,1,""]},"quapy.method.aggregative.HDy":{aggregate:[11,2,1,""],fit:[11,2,1,""]},"quapy.method.aggregative.OneVsAll":{aggregate:[11,2,1,""],binary:[11,3,1,""],classes_:[11,3,1,""],classify:[11,2,1,""],fit:[11,2,1,""],get_params:[11,2,1,""],posterior_probabilities:[11,2,1,""],probabilistic:[11,3,1,""],quantify:[11,2,1,""],set_params:[11,2,1,""]},"quapy.method.aggregative.PACC":{aggregate:[11,2,1,""],classify:[11,2,1,""],fit:[11,2,1,""]},"quapy.method.aggregative.PCC":{aggregate:[11,2,1,""],fit:[11,2,1,""]},"quapy.method.aggregative.ThresholdOptimization":{aggregate:[11,2,1,""],fit:[11,2,1,""]},"quapy.method.base":{BaseQuantifier:[11,1,1,""],BinaryQuantifier:[11,1,1,""],isaggregative:[11,5,1,""],isbinary:[11,5,1,""],isprobabilistic:[11,5,1,""]},"quapy.method.base.BaseQuantifier":{aggregative:[11,3,1,""],binary:[11,3,1,""],classes_:[11,3,1,""],fit:[11,2,1,""],get_params:[11,2,1,""],n_classes:[11,3,1,""],probabilistic:[11,3,1,""],quantify:[11,2,1,""],set_params:[11,2,1,""]},"quapy.method.base.BinaryQuantifier":{binary:[11,3,1,""]},"quapy.method.meta":{EACC:[11,5,1,""],ECC:[11,5,1,""],EEMQ:[11,5,1,""],EHDy:[11,5,1,""],EPACC:[11,5,1,""],Ensemble:[11,1,1,""],ensembleFactory:[11,5,1,""],get_probability_distribution:[11,5,1,""]},"quapy.method.meta.Ensemble":{VALID_POLICIES:[11,4,1,""],aggregative:[11,3,1,""],binary:[11,3,1,""],classes_:[11,3,1,""],fit:[11,2,1,""],get_params:[11,2,1,""],probabilistic:[11,3,1,""],quantify:[11,2,1,""],set_params:[11,2,1,""]},"quapy.method.neural":{QuaNetModule:[11,1,1,""],QuaNetTrainer:[11,1,1,""],mae_loss:[11,5,1,""]},"quapy.method.neural.QuaNetModule":{device:[11,3,1,""],forward:[11,2,1,""]},"quapy.method.neural.QuaNetTrainer":{classes_:[11,3,1,""],clean_checkpoint:[11,2,1,""],clean_checkpoint_dir:[11,2,1,""],fit:[11,2,1,""],get_params:[11,2,1,""],quantify:[11,2,1,""],set_params:[11,2,1,""]},"quapy.method.non_aggregative":{MaximumLikelihoodPrevalenceEstimation:[11,1,1,""]},"quapy.method.non_aggregative.MaximumLikelihoodPrevalenceEstimation":{classes_:[11,3,1,""],fit:[11,2,1,""],get_params:[11,2,1,""],quantify:[11,2,1,""],set_params:[11,2,1,""]},"quapy.model_selection":{GridSearchQ:[8,1,1,""]},"quapy.model_selection.GridSearchQ":{best_model:[8,2,1,""],classes_:[8,3,1,""],fit:[8,2,1,""],get_params:[8,2,1,""],quantify:[8,2,1,""],set_params:[8,2,1,""]},"quapy.plot":{binary_bias_bins:[8,5,1,""],binary_bias_global:[8,5,1,""],binary_diagonal:[8,5,1,""],brokenbar_supremacy_by_drift:[8,5,1,""],error_by_drift:[8,5,1,""]},"quapy.util":{EarlyStop:[8,1,1,""],create_if_not_exist:[8,5,1,""],create_parent_dir:[8,5,1,""],download_file:[8,5,1,""],download_file_if_not_exists:[8,5,1,""],get_quapy_home:[8,5,1,""],map_parallel:[8,5,1,""],parallel:[8,5,1,""],pickled_resource:[8,5,1,""],save_text_file:[8,5,1,""],temp_seed:[8,5,1,""]},quapy:{classification:[9,0,0,"-"],data:[10,0,0,"-"],error:[8,0,0,"-"],evaluation:[8,0,0,"-"],functional:[8,0,0,"-"],isbinary:[8,5,1,""],method:[11,0,0,"-"],model_selection:[8,0,0,"-"],plot:[8,0,0,"-"],util:[8,0,0,"-"]}},objnames:{"0":["py","module","Python module"],"1":["py","class","Python class"],"2":["py","method","Python method"],"3":["py","property","Python property"],"4":["py","attribute","Python attribute"],"5":["py","function","Python function"]},objtypes:{"0":"py:module","1":"py:class","2":"py:method","3":"py:property","4":"py:attribute","5":"py:function"},terms:{"0":[0,1,3,4,5,8,9,10,11],"00":[0,1,4,8],"000":1,"0001":[4,11],"000e":1,"001":[4,9,11],"005":8,"008":[],"009":1,"0097":[],"01":[8,9,11],"017":1,"018":0,"02":1,"021":0,"02552":4,"03":1,"034":1,"035":1,"037":1,"04":1,"041":1,"042":1,"046":1,"048":1,"05":[5,8,10],"055":1,"063":[0,10],"065":0,"070":1,"073":1,"075":1,"078":0,"081":[0,10],"082":[0,1],"083":0,"086":0,"091":1,"099":0,"1":[0,1,3,4,5,8,9,10,11],"10":[0,1,4,5,8,9,11],"100":[0,1,3,4,5,9,10,11],"1000":[0,4,11],"10000":4,"100000":4,"1007":[],"101":[4,8,10],"1010":4,"1024":11,"104":0,"108":1,"109":0,"11":[0,1,6,8,10],"11338":0,"114":1,"1145":[],"12":9,"120":0,"1215742":0,"1271":0,"13":[0,9],"139":0,"14":3,"142":1,"146":3,"1473":0,"148":0,"1484":0,"15":[3,8,10],"150":0,"153":0,"157":0,"158":0,"159":0,"1593":0,"1594":0,"1599":0,"161":0,"163":[0,1],"164":[0,3],"167":0,"17":0,"1771":1,"1775":[0,3],"1778":[0,3],"178":0,"1823":0,"1839":0,"18399":0,"1853":0,"19":[3,10],"193":0,"199151":0,"19982":4,"1e":9,"1st":0,"2":[0,1,3,5,8,10,11],"20":[5,8,11],"200":[1,9],"2000":0,"2002":3,"2006":11,"2008":11,"2011":4,"2013":3,"2015":[0,2,3,9,11],"2016":[3,10,11],"2017":[0,3,10,11],"2018":[0,3,10],"2019":[3,10,11],"2020":4,"2021":11,"20342":4,"206":0,"207":0,"208":0,"21":[1,3,5,8,10],"210":[],"211":0,"2126":0,"2155":0,"21591":[0,10],"218":3,"2184":0,"219e":1,"22":[0,3,9,10],"222":0,"222046":0,"226":0,"229":1,"229399":0,"23":9,"235":1,"238":0,"2390":0,"24":[0,9],"243":0,"248563":0,"24866":4,"24987":4,"25":[0,5,8,9,11],"25000":0,"256":[0,9],"26":9,"261":0,"265":0,"266":0,"267":0,"27":[1,3,9],"270":0,"2700406":[],"271":0,"272":0,"274":0,"275":1,"27th":[0,3,10],"28":3,"280":0,"281":0,"282":0,"283":[0,1],"288":0,"289":0,"2971":0,"2nd":0,"2t":[1,8],"2tp":8,"2x5fcv":0,"3":[0,1,3,5,6,8,9,10,11],"30":[0,1,3,11],"300":[0,1,9],"305":0,"306":0,"312":0,"32":[0,6],"3227":8,"3269206":[],"3269287":[],"33":[0,5,8],"331":0,"333":0,"335":0,"337":0,"34":[0,3,10,11],"341":0,"346":1,"347":0,"350":0,"351":0,"357":1,"359":0,"361":0,"366":1,"372":0,"373":0,"376132":0,"3765":0,"3813":0,"3821":[0,10],"383e":1,"387e":1,"392":0,"394":0,"399":0,"3f":[1,6],"3rd":0,"4":[0,1,3,4,5,8,11],"40":[0,3,4,11],"404333":0,"407":0,"41":3,"412":0,"412e":1,"413":0,"414":0,"417":0,"41734":4,"42":[1,8],"421":0,"4259":0,"426e":1,"427":0,"430":0,"434":0,"435":1,"43676":4,"437":0,"44":0,"4403":10,"446":0,"45":[3,5,10],"452":0,"459":1,"4601":0,"461":0,"463":0,"465":0,"466":0,"470":0,"48":3,"481":0,"48135":4,"486":0,"4898":0,"492":0,"496":0,"4960":1,"497":0,"5":[0,1,3,4,5,8,9,10,11],"50":[0,5,8,11],"500":[0,1,4,5,11],"5000":[1,5],"5005":4,"507":0,"508":0,"512":[9,11],"514":0,"515e":1,"530":0,"534":0,"535":0,"535e":1,"5379":4,"539":0,"541":1,"546":0,"5473":0,"54it":4,"55":5,"55it":4,"565":1,"569":0,"57":0,"573":0,"578":1,"583":0,"591":3,"5f":4,"5fcv":[],"5fcvx2":10,"6":[0,1,3,5,8,10],"60":0,"600":1,"601":0,"604":3,"606":0,"625":0,"627":0,"633e":1,"634":1,"64":[9,11],"640":0,"641":0,"650":0,"653":0,"654":1,"66":[1,11],"665":0,"667":0,"669":0,"67":[5,8],"683":0,"688":0,"691":0,"694582":0,"7":[1,5,8,9,11],"70":0,"700":0,"701e":1,"711":0,"717":1,"725":1,"730":0,"735":0,"740e":1,"748":0,"75":[0,5,8],"762":0,"774":0,"778":0,"787":0,"794":0,"798":0,"8":[0,1,5,10,11],"8000":0,"830":0,"837":1,"858":1,"861":0,"87":[0,3,10],"8788":0,"889504":0,"8d2fhsgcvn0aaaaa":[],"9":[0,1,3,5,8],"90":[5,8],"901":0,"909":1,"914":1,"917":0,"919":[0,10],"922":0,"923":0,"935":0,"936":0,"937":[0,10],"945":1,"95":[8,10],"9533":0,"958":0,"97":0,"979":0,"982":0,"99":8,"abstract":[3,9,10,11],"boolean":[8,10,11],"case":[0,1,3,4,5,8,10,11],"class":[0,1,3,4,5,6,8,9,10,11],"d\u00edez":3,"default":[1,3,8,9,10,11],"do":[0,1,3,4,8,9,10,11],"final":[1,3,5,11],"float":[0,3,8,9,10,11],"function":[0,1,3,4,5,6,7,9,10,11],"g\u00e1llego":[0,3,10,11],"gonz\u00e1lez":3,"import":[0,1,3,4,5,6,10,11],"int":[0,5,8,10,11],"long":[4,9],"new":[0,3,10],"p\u00e9rez":[0,3,10,11],"return":[0,1,3,4,5,8,9,10,11],"rodr\u0131":3,"short":9,"static":[3,11],"true":[0,1,3,4,5,6,8,9,10,11],"try":4,"while":[3,5,8,9,10,11],A:[0,3,8,9,10,11],As:[3,4],By:[1,3,8],For:[0,1,5,6,8,10],If:[3,5,8,10,11],In:[0,1,2,3,4,5,6,9],It:[3,4,5,8],One:[0,1,3,11],That:[1,4],The:[0,1,2,4,5,6,8,9,10,11],Then:3,These:0,To:[5,10],_:[5,8,10],__:[],__class__:5,__name__:5,_adjust:[],_ae_:[],_classify_:[],_error_name_:[],_fit_learner_:[],_kld_:[],_labelledcollection_:[],_learner_:[],_mean:[],_min_df_:[],_my:[],_nkld_:[],_posterior_probabilities_:11,_q_:[],_rae_:[],_svmperf_:[],ab:[],aboud:3,about:[0,5,8,10],abov:[0,3,5,8],absolut:[1,3,5,6,8,11],absolute_error:8,abstractmethod:3,acc:[1,3,5,6,8,11],acc_error:8,accept:3,access:[0,3,10,11],accommod:0,accord:[1,3,4,8,9,10,11],accordingli:5,accuraci:[1,5,8,11],accuracy_polici:[],achiev:[1,3,4,5],acm:[0,3,10],across:[0,1,4,5,6,8],action:0,actual:[10,11],acut:0,ad:6,adapt:8,add:[3,4,8,10],add_word:10,addit:3,addition:0,adjust:[3,6,8,11],adjusted_quantif:8,adjustedclassifyandcount:11,adopt:[3,4,10],advanc:[0,6],advantag:[3,11],ae:[1,2,5,8,11],ae_:1,affect:8,after:[8,11],afterward:11,again:5,against:5,aggreg:[1,4,5,6,7,8],aggregativeprobabilisticquantifi:[3,11],aggregativequantifi:[3,11],aggregg:[],aim:[4,5],aka:[10,11],al:[0,2,9,10,11],alaiz:3,alegr:3,alejandro:4,algorithm:[8,11],alia:[3,8,11],all:[0,1,2,3,5,8,10,11],allia:3,alloc:[8,9],allow:[0,1,2,3,5,8,9,10,11],almost:3,along:[0,3,8,11],alreadi:[3,11],also:[0,1,2,3,5,6,8,9],altern:4,although:[3,4,5,11],alwai:[3,4,5,11],among:3,amount:8,an:[0,1,2,3,4,5,6,8,9,10,11],analys:[5,6],analysi:[0,3,6,10],analyz:5,ani:[0,1,3,4,5,6,8,9,10,11],anoth:[0,1,3,5],anotherdir:8,anyon:0,anyth:11,api:6,app:[8,10,11],appeal:1,appear:5,append:5,appli:[2,3,4,5,8,9,10,11],appropri:4,approxim:[1,5,8,9,10],ar:[0,1,3,4,5,8,9,10,11],archive_filenam:8,archive_path:[],arg:[8,10],argmax:8,args_i:8,argu:4,argument:[0,1,3,5,8,10,11],arifici:[],aris:1,around:[1,10],arrai:[1,3,5,8,9,10,11],articl:[3,4],artifici:[0,1,3,4,5,6,8,10],artificial_prevalence_predict:8,artificial_prevalence_protocol:8,artificial_prevalence_report:8,artificial_prevalence_sampl:8,artificial_sampling_ev:[1,4],artificial_sampling_gener:[0,10],artificial_sampling_index_gener:10,artificial_sampling_predict:[1,5],artificial_sampling_report:1,arxiv:4,asarrai:1,asdataload:9,asonam:0,assert:10,assess:4,assign:[3,8,10],associ:[8,10],assum:[1,6,11],assumpion:11,assumpt:[1,5,6],astyp:[],attempt:[3,11],attribut:11,august:0,autom:[0,3,6],automat:[0,1],av:[3,11],avail:[0,1,2,3,5,6,9,11],averag:[1,3,8,10,11],avoid:[1,8],ax:11,axi:[5,8],b:[0,10,11],balanc:[0,4,11],band:[5,8],bar:8,barranquero:[2,3,9,11],base:[0,3,6,7,8,9],base_classifi:5,base_estim:3,base_quantifier_class:11,baseestim:[9,11],baselin:6,basequantifi:[3,8,11],basic:[5,11],batch:9,batch_siz:9,batch_size_test:9,beat:11,been:[0,3,4,5,8,10,11],befor:[3,8,9,10,11],beforehand:8,behav:[3,5],being:[4,8,11],belief:1,belong:[3,11],below:[0,2,3,5,8,10],best:[4,8,9],best_epoch:8,best_model:8,best_model_:4,best_params_:4,best_scor:8,better:4,between:[4,5,6,8,9,11],beyond:5,bia:[6,8],bias:5,bidirect:11,bin:[5,8,11],bin_bia:5,bin_diag:5,binar:[8,10],binari:[3,5,6,8,9,10,11],binary_bias_bin:[5,8],binary_bias_glob:[5,8],binary_diagon:[5,8],binary_quantifi:11,binaryquantifi:11,binom:8,block:[0,8],bool:8,both:5,bound:[8,11],box:[5,8],breast:0,brief:1,bring:11,broken:[5,8],brokenbar_supremacy_by_drift:8,budg:1,budget:[1,4],build:[],bypass:11,c:[3,4,8,9,10,11],calcul:8,calibr:3,calibratedclassifi:3,calibratedclassifiercv:3,calibratedcv:[],call:[0,1,5,8,10,11],callabl:[0,8,10],can:[0,1,2,3,4,5,8,10,11],cancer:0,cannot:[],cardiotocographi:0,care:11,carri:[3,10,11],casa_token:[],castano:[3,10],castro:3,categor:[3,10],categori:[1,8],cc:[3,5,11],ceil:8,cell:11,center:5,chang:[0,1,3,10],character:[3,6],characteriz:[0,3,10],charg:[0,8,10],chart:8,check:[3,4],checkpoint:[9,11],checkpointdir:11,checkpointnam:11,checkpointpath:9,choic:4,choos:11,chosen:[4,8],cl:0,cla:[],class2int:10,class_weight:[4,11],classes_:[8,10,11],classif:[0,1,3,7,8,10,11],classif_posterior:[3,11],classif_predict:[3,11],classif_predictions_bin:11,classifi:[1,4,5,6,8,9,11],classifier_net:9,classifiermixin:9,classifyandcount:[3,11],classmethod:[0,10,11],classnam:10,classs:8,clean_checkpoint:11,clean_checkpoint_dir:11,clear:5,clearer:1,clearli:5,clip:8,close:[1,10],closer:1,closest:11,cm:8,cmc:0,cnn:[3,11],cnnnet:[3,9,11],code:[0,3,4,5,9],codifi:10,coincid:[0,6],col:[0,10],collect:[0,8,9,10],collet:10,color:[5,8],colormap:8,column:[0,8,10],com:8,combin:[0,1,4,8,10,11],combinatio:8,combinations_budget:8,come:[0,8,10,11],commandlin:[],common:11,commonli:6,compar:[5,8],comparison:5,compat:11,compil:[2,3],complement:11,complet:[3,5,11],compon:[8,9],compress:0,comput:[1,3,5,8,11],computation:4,compute_fpr:[],compute_t:[],compute_tpr:[],concept:6,concur:[],condit:[8,11],conduct:[0,8],confer:[0,3,10],confid:8,configur:[4,8],conform:10,connect:11,consecut:[8,9,11],consid:[3,5,8,9,10,11],consist:[0,4,5,8,9,10,11],constrain:[1,5,8,10],constructor:3,consult:[0,1],contain:[1,2,3,5,8,9,10,11],contanin:8,content:7,context:8,contrast:1,control:[1,4,10],conv_block:[],conv_lay:[],conveni:8,converg:11,convert:[1,3,8,9,10,11],convolut:9,copi:[8,10],cornel:[],correct:11,correctli:8,correspond:[5,8,10],cosest:11,cost:1,costli:4,could:[0,1,3,4,5,6],count:[4,5,6,8,10,11],count_:[],counter:10,countvector:10,covari:10,cover:[1,4,9],coz:[0,3,10],cpu:[1,9,11],creat:[0,6,8],create_if_not_exist:8,create_parent_dir:8,crisp:[3,8],criteria:4,cross:[3,10,11],cs:8,csr:10,csr_matrix:10,csv:10,ctg:0,cuda:[3,9,11],cumbersom:1,cumberson:8,cumul:11,curios:5,current:[3,8,9,10,11],custom:[3,6,8,10],customarili:[3,4],cv:[3,4],cyan:5,d:11,d_:8,dat:[0,9],data:[1,3,4,5,6,7,8,9,11],data_hom:10,datafram:[1,8],dataload:9,dataset:[1,3,4,5,6,7,8,9,11],dataset_nam:10,deal:0,decaesteck:[3,11],decai:9,decid:10,decim:1,decis:[3,8,9,11],decision_funct:9,decomposit:9,dedic:[1,10],deep:[3,8,11],def:[0,1,3,5,8],defin:[0,3,8,9,10,11],degre:4,del:[0,3,10],delai:8,deliv:[3,11],denomin:11,dens:[0,11],densiti:8,depend:[0,1,4,5,8,11],describ:[3,8,11],descript:0,design:4,desir:[0,1,10],despit:1,destin:8,detail:[0,1,3,6,9,10,11],determin:[1,4,5],detriment:5,devel:10,develop:[4,6],deviat:[0,1,5,8,10],devic:[0,3,5,9,11],df:1,df_replac:[],diabet:0,diagon:[6,8],dict:[8,10,11],dictionari:[8,9,10,11],differ:[0,1,3,4,5,6,8,10,11],difficult:5,digit:0,dimens:[8,9,10,11],dimension:[8,9,10,11],dir:8,directli:[0,1,3],directori:[2,8,9,10,11],discard:8,discoveri:3,discret:8,discuss:5,disjoint:9,disk:8,displai:[1,5,8],displaystyl:8,distanc:[8,11],distant:[1,8],distribut:[0,3,5,8,10,11],diverg:[1,3,8,11],divid:8,dl:[],doabl:0,doc_embed:11,doc_embedding_s:11,doc_posterior:11,document:[0,1,3,5,9,10,11],document_embed:9,doe:[0,2,3,8,11],doi:[],done:3,dot:[5,8],dowload:8,down:[5,8,10],download:[0,2,3,8],download_fil:8,download_file_if_not_exist:8,draw:[8,10],drawn:[0,1,4,8,10],drift:6,drop:9,drop_p:9,dropout:[9,11],ds:[3,11],ds_polici:[],ds_policy_get_posterior:[],dtype:[1,10],dump:10,dure:[1,5,11],dynam:[3,9,10,11],e:[0,1,3,4,5,6,8,9,10,11],eacc:11,each:[0,1,3,4,5,8,9,10,11],earli:[8,9,11],early_stop:[],earlystop:8,easili:[0,2,5,9],ecc:11,edu:[],eemq:11,effect:3,effici:3,ehdi:11,either:[1,3,8,10,11],element:[3,10,11],elm:[3,11],els:11,em:11,emb:9,embed:[3,9,11],embed_s:9,embedding_s:9,empti:10,emq:[5,11],enabl:9,encod:10,end:[4,8,11],endeavour:6,enough:5,ensembl:[0,6,10,11],ensemblefactori:11,ensure_probabilist:[],entir:[0,3,4,5,8],entri:11,environ:[1,3,4,5,8,11],ep:[1,8],epacc:11,epoch:[8,9,11],epsilon:[1,8,11],equal:[1,8],equidist:[0,8],equip:[3,5],equival:11,err:[],err_drift:5,err_nam:8,error:[3,4,6,7,9,11],error_:[],error_by_drift:[5,8],error_funct:1,error_metr:[1,4,8],error_nam:[5,8],especi:8,establish:8,estim:[1,3,5,6,8,9,10,11],estim_prev:[1,5,8],estim_preval:[3,6,11],estimant:11,esuli:[0,2,3,9,10,11],et:[0,2,9,10,11],etc:6,eval_budget:[4,8],evalu:[0,3,4,5,6,7,9,10,11],even:8,eventu:[9,10],everi:[3,11],everyth:3,evinc:5,ex:[],exact:[0,10],exactli:0,exampl:[0,1,3,4,5,8,9,10,11],exce:8,excel:0,except:[3,8,11],exemplifi:0,exhaust:8,exhibit:[4,5],exist:8,exist_ok:8,expand_frame_repr:1,expect:[6,11],expectationmaximizationquantifi:[3,11],experi:[1,2,3,4,5,8],explain:[1,5],explicit:11,explicitlossminim:[],explicitlossminimis:11,explor:[4,8,10],express:10,ext:2,extend:[2,3,11],extens:[0,2,5],extern:3,extract:[1,8,10],f1:[1,8,9],f1_error:8,f1e:[1,8],f:[0,1,3,4,5,6,10],f_1:8,fabrizio:4,facilit:6,fact:[3,5],factor:8,factori:11,fals:[1,3,5,8,9,10,11],famili:[3,11],familiar:3,far:[8,9,10],fare:8,fast:8,faster:[0,10],feat1:10,feat2:10,featn:10,featur:[0,10],feature_extract:10,fetch:[0,6],fetch_review:[0,1,3,4,5,10,11],fetch_twitt:[0,3,6,10],fetch_ucidataset:[0,3,10],fetch_ucilabelledcollect:[0,10],ff:11,ff_layer:11,fhe:0,file:[0,5,8,9,10,11],filenam:8,fin:0,find:[0,4],finish:4,first:[0,1,2,3,5,8,10,11],fit:[1,3,4,5,6,8,9,10,11],fit_learn:[3,11],fit_transform:10,fix:[1,4],flag:8,float64:1,fn:8,fold:[3,10,11],folder:[0,11],follow:[0,1,3,4,5,6,8,11],fomart:10,for_model_select:[0,10],form:[0,8,10],forman:11,format:[0,5,10],former:[2,11],forward:[9,11],found:[0,3,4,8,9,10],four:3,fp:8,fpr:[8,11],frac:8,framework:6,frequenc:[0,10,11],from:[0,1,3,4,5,6,8,10,11],from_csv:10,from_nam:[1,8],from_spars:10,from_text:10,full:[1,8],fulli:0,func:8,further:[0,1,3,9,10,11],fusion:[0,3,10],futur:3,g:[0,1,3,4,6,8,10,11],gain:8,gao:[0,3,10,11],gap:10,gasp:[0,10],gen:8,gen_data:5,gen_fn:8,gen_prevalence_predict:8,gen_prevalence_report:8,gener:[0,1,3,4,5,8,9,10,11],generation_func:8,german:0,get:[0,1,5,8,9,10,11],get_aggregative_estim:[],get_nprevpoints_approxim:[1,8],get_param:[3,8,9,11],get_probability_distribut:11,get_quapy_hom:8,ggener:8,github:[],give:11,given:[1,3,4,8,9,10,11],global:8,goal:11,goe:4,good:[4,5],got:4,govern:1,gpu:[9,11],grant:[],greater:10,grid:[4,8,10,11],gridsearchcv:[4,11],gridsearchq:[4,8,11],ground:11,group:3,guarante:10,guez:3,gzip:0,ha:[3,4,5,8,9,10,11],haberman:[0,3],had:10,handl:0,happen:[4,5],hard:3,harder:5,harmon:8,harri:0,hat:8,have:[0,1,2,3,4,5,8,10,11],hcr:[0,3,10],hd:8,hdy:[6,11],held:[3,4,8,9,11],helling:11,hellingerdist:8,hellingerdistancei:[3,11],hellingh:8,help:5,henc:[8,10],here:[1,11],heurist:11,hidden:[5,9,11],hidden_s:9,hide:5,high:[5,8],higher:[1,5],highlight:8,hightlight:8,histogram:11,hlt:[],hold:[6,8,11],home:[8,10],hook:11,how:[0,1,3,4,5,8,10,11],howev:[0,4,5],hp:[0,3,4,10],html:10,http:[8,10],hyper:[4,8,9],hyperparam:4,hyperparamet:[3,8],i:[0,1,3,4,5,8,9,10,11],id:[0,3,10],identifi:8,idf:0,ieee:0,ignor:[8,10,11],ii:8,iid:[1,5,6],illustr:[3,4,5],imdb:[0,5,10],implement:[0,1,3,4,5,6,8,9,10,11],implicit:8,impos:[4,8],improv:[3,8,9,11],includ:[0,1,3,5,6,10,11],inconveni:8,inde:[3,4],independ:[8,11],index:[0,3,6,8,9,10,11],indextransform:10,indic:[0,1,3,4,5,8,10,11],individu:[1,3],infer:[0,10],inform:[0,1,3,4,8,10,11],infrequ:10,inherit:3,init:3,init_hidden:[],initi:[0,9],inplac:[1,3,10,11],input:[3,5,8,9,11],insight:5,inspir:3,instal:[0,3,6,9,11],instanc:[0,3,4,5,6,8,9,10,11],instanti:[0,1,3,4,9,11],instead:[1,3,4,11],integ:[3,8,9,10,11],integr:6,interest:[1,5,6,8,10],interestingli:5,interfac:[0,1,11],intern:[0,3,10],interpret:[5,6,11],interv:[1,5,8,10],introduc:1,invok:[0,1,3,8,10],involv:[2,5,8],io:[],ionospher:0,iri:0,irrespect:[5,11],isaggreg:11,isbinari:[8,10,11],isomer:8,isometr:[5,8],isprobabilist:11,isti:[],item:8,iter:[0,8,11],its:[3,4,8,9,11],itself:[3,8,11],j:[0,3,10,11],joachim:[3,9,11],job:[2,8],joblib:2,join:8,just:[1,3],k:[3,6,8,10,11],keep:8,kei:[8,10],kept:10,kernel:9,kernel_height:9,keyword:[10,11],kfcv:[0,10,11],kindl:[0,1,3,5,10,11],kl:8,kld:[1,2,8,9,11],know:3,knowledg:[0,3,10],known:[0,3,4,11],kraemer:8,kullback:[1,3,8,11],kwarg:[9,10,11],l1:[8,11],l:11,label:[0,3,4,5,6,8,9,10,11],labelledcollect:[0,3,4,8,10,11],larg:4,larger:[10,11],largest:8,last:[1,3,5,8,9,10],lastli:3,latex:5,latinn:[3,11],latter:11,layer:[3,9,11],lazi:11,lead:[1,10],learn:[1,2,3,4,6,8,9,10,11],learner:[3,4,9,11],least:[0,10],leav:10,left:10,legend:8,leibler:[1,3,8,11],len:8,length:[9,10],less:[8,10],let:[1,3],level:[],leverag:3,leyend:8,like:[0,1,3,5,8,9,10,11],likelihood:11,limit:[5,8,10,11],line:[1,3,8],linear:[5,11],linear_model:[1,3,4,6,9],linearsvc:[3,5,10],link:[],linspac:5,list:[0,5,8,9,10,11],listedcolormap:8,literatur:[0,1,4,6],load:[0,3,8,10,11],loader:[0,10],loader_func:[0,10],loader_kwarg:10,local:8,log:[8,10],logist:[1,3,9,11],logisticregress:[1,3,4,6,9,11],logscal:8,logspac:[4,11],longer:8,longest:9,look:[0,1,3,5,11],loop:11,loss:[6,9,11],low:[5,8,9],lower:[5,8,11],lower_is_bett:8,lowest:5,lowranklogisticregress:9,lr:[1,3,9,11],lstm:[3,9,11],lstm_class_nlay:9,lstm_hidden_s:11,lstm_nlayer:11,lstmnet:9,m:[3,8,11],machin:[1,4,6],macro:8,made:[0,2,8,10,11],mae:[1,4,6,8,9,11],mae_loss:11,mai:8,main:5,maintain:[3,11],make:[0,1,3,11],makedir:8,mammograph:0,manag:[0,3,10],mani:[1,3,4,5,6,8,10,11],manner:0,manual:0,map:[1,9],map_parallel:8,margin:9,mass:8,math:[],mathcal:8,matplotlib:[2,8],matric:[0,5,10],matrix:[5,8,11],max:11,max_it:11,max_sample_s:11,maxim:[6,11],maximum:[1,8,9,11],maximumlikelihoodprevalenceestim:11,md:[],mean:[0,1,3,4,5,6,8,9,10,11],mean_absolute_error:8,mean_relative_absolute_error:8,measur:[2,3,4,5,6,8,11],median:11,mediansweep2:11,mediansweep:11,member:[3,11],memori:9,mention:3,merg:5,met:10,meta:[6,7,8],meth:[],method:[0,1,4,5,6,7,8],method_data:5,method_nam:[5,8],method_ord:8,metric:[1,3,4,6,8,11],might:[1,8,10],min_df:[1,3,4,5,10,11],min_po:11,mine:[0,3],minim:[8,11],minimum:[10,11],minimun:10,mining6:10,minu:8,misclassif:11,miss:8,mixtur:[3,11],mkld:[1,8,11],ml:10,mlpe:11,mnkld:[1,8,11],mock:[8,9],modal:4,model:[0,1,5,6,8,9,11],model_select:[4,7,11],modifi:[3,8],modul:[0,1,3,5,6,7],moment:[0,3],monitor:8,more:[3,5,8,11],moreo:[0,3,4,10,11],most:[0,3,5,6,8,10,11],movi:0,mrae:[1,6,8,9,11],ms2:11,ms:11,mse:[1,3,6,8,11],msg:[],multiclass:8,multipli:8,multiprocess:8,multivari:[3,9],must:[3,10,11],mutual:11,my:[],my_arrai:8,my_collect:10,my_custom_load:0,my_data:0,mycustomloss:3,n:[0,1,8,9,11],n_bin:[5,8],n_class:[1,3,8,9,10,11],n_classes_:11,n_compon:9,n_dimens:9,n_epoch:11,n_featur:9,n_instanc:[8,9,11],n_job:[1,3,4,8,10,11],n_preval:[0,8,10],n_prevpoint:[1,4,5,8],n_repeat:[1,8],n_repetit:[1,4,5,8],n_sampl:[8,9],name:[5,8,9,10,11],nativ:6,natur:[1,8,10,11],natural_prevalence_predict:8,natural_prevalence_protocol:8,natural_prevalence_report:8,natural_sampling_gener:10,natural_sampling_index_gener:10,nbin:[5,8],ndarrai:[1,3,8,10,11],necessarili:[],need:[0,3,8,10,11],neg:[0,5,8,11],nest:[],net:9,network:[0,8,9,10,11],neural:[0,7,8,10],neuralclassifiertrain:[3,9,11],neutral:0,next:[4,8,9,10],nfold:[0,10],nkld:[1,2,6,8,9,11],nn:[9,11],nogap:10,non:3,non_aggreg:[7,8],none:[1,4,8,9,10,11],nonetheless:4,nor:3,normal:[0,1,3,8,10,11],normalize_preval:8,note:[1,3,4,5,8,10],noth:11,now:5,nowadai:3,np:[1,3,4,5,8,10,11],npp:[8,10],nprevpoint:[],nrepeat:[0,10],num_prevalence_combin:[1,8],number:[0,1,3,5,8,9,10,11],numer:[0,1,3,6,10,11],numpi:[2,4,8,9,11],o_l6x_pcf09mdetq4tu7jk98mxfbgsxp9zso14jkuiyudgfg0:[],object:[0,8,9,10,11],observ:1,obtain:[1,4,8,11],obtaind:8,obvious:8,occur:[5,10],occurr:10,octob:[0,3],off:9,offer:[3,6],older:2,omd:[0,10],ommit:[1,8],onc:[1,3,5,8],one:[0,1,3,4,5,8,10,11],ones:[1,3,5,8,10],onevsal:[3,11],onli:[0,3,5,8,9,10,11],open:[0,6,10],oper:3,opt:4,optim:[2,3,4,8,9,11],optimize_threshold:[],option:[0,1,3,5,8,10,11],order:[0,2,3,5,8,10,11],order_bi:11,org:10,orient:[3,6,8,11],origin:[0,3,10],os:[0,8],other:[1,3,5,6,8,10,11],otherwis:[0,3,8,10,11],our:[],out:[3,4,5,8,9,10,11],outcom:5,outer:8,outlier:8,output:[0,1,3,4,8,9,10,11],outsid:11,over:[3,4,8],overal:1,overestim:5,overrid:3,overridden:[3,11],own:4,p:[0,3,8,10,11],p_hat:8,p_i:8,pacc:[1,3,5,8,11],packag:[0,2,3,6,7],pad:[9,10],pad_length:9,padding_length:9,page:[0,2,6],pageblock:0,pair:[0,8,11],panda:[1,2,8],paper:[0,3],parallel:[1,3,8,10,11],param:[4,9,11],param_grid:[4,8,11],param_mod_sel:11,param_model_sel:11,paramet:[1,3,4,8,9,10,11],parent:8,part:[3,10],particular:[0,1,3],particularli:1,pass:[0,1,5,8,9,11],past:1,patch:[2,3,9,11],path:[0,3,5,8,9,10,11],patienc:[8,9,11],pattern:3,pca:[],pcalr:[],pcc:[3,4,5,11],pd:1,pdf:5,peopl:[],percentil:8,perf:[6,9,11],perform:[1,3,4,5,6,8,9,11],perman:8,phase:11,phonem:0,pick:4,pickl:[3,8,10,11],pickle_path:8,pickled_resourc:8,pii:[],pip:2,pipelin:[],pkl:8,plai:0,plan:3,pleas:3,plot:[6,7],png:5,point:[0,1,3,8,10],polici:[3,11],popular:6,portion:4,pos_class:[8,10],posit:[0,3,5,8,10,11],possibl:[1,3,8],post:8,posterior:[3,8,9,11],posterior_prob:[3,11],postpon:3,potter:0,pp:[0,3],pprox:[],practic:[0,4],pre:[0,3],prec:[0,8],preced:10,precis:[0,1,8],preclassifi:3,predefin:10,predict:[3,4,5,8,9,11],predict_proba:[3,9,11],predictor:1,prefer:8,preliminari:11,prepare_svmperf:[2,3],preprint:4,preprocess:[0,1,3,7,8,11],present:[0,3,10],preserv:[1,5,8,10],pretti:5,prev:[0,1,8,10],prevail:3,preval:[0,1,3,4,5,6,8,10,11],prevalence_estim:8,prevalence_from_label:8,prevalence_from_prob:8,prevalence_linspac:8,prevel:11,previou:3,previous:[],prevs_estim:11,prevs_hat:[1,8],princip:9,print:[0,1,3,4,6,9,10],prior:[1,3,4,5,6,8,11],priori:3,probabilist:[3,11],probabilisticadjustedclassifyandcount:11,probabilisticclassifyandcount:11,probabl:[1,3,4,5,6,8,9,11],problem:[0,3,5,8,10,11],procedur:[3,6],proceed:[0,3,10],process:[3,4,8],processor:3,procol:1,produc:[0,1,5,8],product:3,progress:[8,10],properli:0,properti:[3,8,9,10,11],proport:[3,4,8,9,10,11],propos:[2,3,11],protocl:8,protocol:[0,3,4,5,6,8,10,11],provid:[0,3,5,6,11],ptecondestim:11,ptr:[3,11],ptr_polici:[],purpos:[0,11],put:11,python:[0,6],pytorch:[2,11],q:[0,2,3,8,9,11],q_i:8,qacc:9,qdrop_p:11,qf1:9,qgm:9,qp:[0,1,3,4,5,6,8,10,11],quanet:[2,6,9,11],quanetmodul:11,quanettrain:11,quantif:[0,1,6,8,9,10,11],quantifi:[3,4,5,6,8,11],quantification_error:8,quantiti:8,quapi:[0,1,2,3,4,5],quapy_data:0,quay_data:10,question:8,quevedo:[0,3,10],quick:[],quit:8,r:[0,3,8,10],rac:[],rae:[1,2,8,11],rais:[3,8,11],rand:8,random:[1,3,4,5,8,10],random_se:[1,8],random_st:10,randomli:0,rang:[0,5,8,11],rank:[3,9],rare:10,rate:[3,8,9,11],rather:[1,4],raw:10,rb:0,re:[3,4,10],reach:11,read:10,reader:[7,8],readm:[],real:[8,9,10,11],reason:[3,5,6],recal:8,receiv:[0,3,5],recip:11,recognit:3,recommend:[1,5,11],recomput:11,recurr:[0,3,10],recurs:11,red:0,red_siz:[3,11],reduc:[0,10],reduce_column:[0,10],refer:[9,10],refit:[4,8],regard:4,regardless:10,regim:8,region:8,regist:11,regress:9,regressor:[1,3],reindex_label:10,reiniti:9,rel:[1,3,8,10,11],relative_absolute_error:8,reli:[1,3,11],reliabl:3,rememb:5,remov:[10,11],repeat:[8,10],repetit:8,repl:[],replac:[0,3,10],replic:[1,4,8],report:[1,8],repositori:[0,10],repr_siz:9,repres:[1,3,5,8,10,11],represent:[0,3,8,9,11],reproduc:10,request:[0,8,10],requir:[0,1,3,6,9],reset_net_param:9,resourc:8,resp:11,respect:[0,1,5,8,11],respond:3,rest:[8,10,11],result:[1,2,3,4,5,6,8,11],retain:[0,3,9,11],retrain:4,return_constrained_dim:8,reus:[0,3,8],review:[5,6,10],reviews_sentiment_dataset:[0,10],rewrit:5,right:[4,8,10],role:0,root:6,roughli:0,round:10,routin:[8,10,11],row:[8,10],run:[0,1,2,3,4,5,8,10,11],s003132031400291x:[],s10618:[],s:[0,1,3,4,5,8,9,10,11],saeren:[3,11],sai:[],said:3,same:[0,3,5,8,10,11],sampl:[0,1,3,4,5,6,8,9,10,11],sample_s:[0,1,3,4,5,8,10,11],sampling_from_index:[0,10],sampling_index:[0,10],sander:[0,10],save:[5,8],save_or_show:[],save_text_fil:8,savepath:[5,8],scale:8,scall:10,scenario:[1,3,4,5,6],scienc:3,sciencedirect:[],scikit:[2,3,4,10],scipi:[2,10],score:[0,1,4,8,9,10],script:[1,2,3,6,11],se:[1,8],search:[3,4,6,8],sebastiani:[0,3,4,10,11],second:[0,1,3,5,8,10],secondari:8,section:4,see:[0,1,2,3,4,5,6,8,9,10,11],seed:[1,4,8],seem:3,seemingli:5,seen:[5,8,11],select:[0,3,6,8,10,11],selector:3,self:[3,8,9,10,11],semeion:0,semev:0,semeval13:[0,10],semeval14:[0,10],semeval15:[0,10],semeval16:[0,6,10],sentenc:10,sentiment:[3,6,10],separ:[8,10],sequenc:8,seri:0,serv:3,set:[0,1,3,4,5,6,8,9,10,11],set_opt:1,set_param:[3,8,9,11],set_siz:[],sever:0,sh:[2,3],shape:[5,8,9,10,11],share:[0,10],shift:[1,4,6,8,11],shorter:9,shoud:3,should:[0,1,3,4,5,6,9,10,11],show:[0,1,3,4,5,8,9,10,11],show_dens:8,show_std:[5,8],showcas:5,shown:[1,5,8],shuffl:[9,10],side:8,sign:8,signific:1,significantli:8,silent:[8,11],simeq:[],similar:[8,11],simpl:[0,3,5,11],simplest:3,simplex:[0,8],simpli:[1,2,3,4,5,6,8,11],sinc:[0,1,3,5,8,10,11],singl:[1,3,6,11],size:[0,1,3,8,9,10,11],sklearn:[1,3,4,5,6,9,10,11],sld:[3,11],slice:8,smooth:[1,8],smooth_limits_epsilon:8,so:[0,1,3,5,8,9,10,11],social:[0,3,10],soft:3,softwar:0,solid:5,solut:8,solv:[4,11],solve_adjust:11,some:[0,1,3,5,8,10,11],some_arrai:8,sometim:1,sonar:0,sort:11,sourc:[2,3,6,9],sout:[],space:[0,4,8,9],spambas:0,spars:[0,10],special:[0,5,10],specif:[3,4],specifi:[0,1,3,5,8,9,10],spectf:0,spectrum:[0,1,4,5,8],speed:[3,11],split:[0,3,4,5,8,9,10,11],split_stratifi:10,splitstratifi:10,spmatrix:10,springer:[],sqrt:8,squar:[1,3,8],sst:[0,10],stabil:[1,11],stabl:10,stackexchang:8,stand:[8,11],standard:[0,1,5,8,10,11],star:8,start:4,stat:10,state:8,statist:[0,1,8,11],stats_siz:11,std:9,stdout:8,step:[5,8],stop:[8,9,11],store:[0,9,10,11],str:[0,8,10],strategi:[3,4],stratif:10,stratifi:[0,3,10,11],stride:9,string:[1,8,10,11],strongli:[4,5],strprev:[0,1,8],structur:[3,11],studi:[0,3,10],style:10,subclass:11,subdir:8,subinterv:5,sublinear_tf:10,submit:0,submodul:7,subobject:[],suboptim:4,subpackag:7,subsequ:10,subtract:[0,8,10],subtyp:10,suffic:5,suffici:[],sum:[8,11],sum_:8,summar:0,supervis:[4,6],support:[3,6,9,10],surfac:10,surpass:1,svm:[3,5,6,9,10,11],svm_light:[],svm_perf:[],svm_perf_classifi:9,svm_perf_learn:9,svm_perf_quantif:[2,3],svmae:[3,11],svmkld:[3,11],svmnkld:[3,11],svmperf:[2,3,7,8,11],svmperf_bas:[9,11],svmperf_hom:3,svmq:[3,11],svmrae:[3,11],sweep:11,syntax:5,system:[4,11],t50:11,t:[0,1,3,8],tab10:8,tail:8,tail_density_threshold:8,take:[0,3,5,8,10,11],taken:[3,8,9,10],target:[3,5,6,8,9,11],task:[3,4,10],te:[8,10],temp_se:8,tempor:8,tend:5,tendenc:5,tensor:9,term:[0,1,3,4,5,6,8,9,10,11],test:[0,1,3,4,5,6,8,9,10,11],test_bas:[],test_dataset:[],test_method:[],test_path:[0,10],test_sampl:8,test_split:10,text2tfidf:[0,1,3,10],text:[0,3,8,9,10,11],textclassifiernet:9,textual:[0,6,10],tf:[0,10],tfidf:[0,4,5,10],tfidfvector:10,than:[1,4,5,8,9,10],thei:[0,3,11],them:[0,3,11],theoret:4,thereaft:1,therefor:[8,10],thi:[0,1,2,3,4,5,6,8,9,10,11],thing:3,third:[1,5],thorsten:9,those:[1,3,4,5,8,9,11],though:[3,8],three:[0,5],threshold:[8,11],thresholdoptim:11,through:[3,8],thu:[3,4,5,8,11],tictacto:0,time:[0,1,3,8,10],timeout:8,timeouterror:8,timer:8,titl:8,tj:[],tn:8,token:[0,9,10],tool:[1,6],top:[3,8,11],torch:[3,9,11],torchdataset:9,total:8,toward:[5,10],tp:8,tpr:[8,11],tqdm:2,tr:10,tr_iter_per_poch:11,tr_prev:[5,8,11],track:8,trade:9,tradition:1,train:[0,1,3,4,5,6,8,9,10,11],train_path:[0,10],train_prev:[5,8],train_prop:10,train_siz:10,train_val_split:[],trainer:9,training_help:[],training_preval:5,training_s:5,transact:3,transform:[0,9,10,11],transfus:0,trivial:3,true_prev:[1,5,8],true_preval:6,truncatedsvd:9,truth:11,ttest_alpha:8,tupl:[8,10,11],turn:4,tweet:[0,3,10],twitter:[6,10],twitter_sentiment_datasets_test:[0,10],twitter_sentiment_datasets_train:[0,10],two:[0,1,3,4,5,8,10,11],txt:8,type:[0,3,8,10,11],typic:[1,4,5,8,9,10,11],u1:10,uci:[6,10],uci_dataset:10,unabl:0,unadjust:5,unalt:9,unbias:5,uncompress:0,under:1,underestim:5,underlin:8,understand:8,unfortun:5,unifi:[0,11],uniform:[8,10],uniform_prevalence_sampl:8,uniform_sampl:10,uniform_sampling_index:10,uniform_simplex_sampl:8,uniformli:[8,10],union:[8,11],uniqu:10,unit:[0,8],unix:0,unk:10,unknown:10,unlabel:11,unless:11,unlik:[1,4],until:11,unus:[8,9],up:[3,4,8,9,11],updat:11,url:8,us:[0,1,3,4,5,6,8,9,10,11],user:[0,1,5],utf:10,util:[7,9],v:3,va_iter_per_poch:11,val:[0,10],val_split:[3,4,8,9,11],valid:[0,1,3,4,5,8,9,10,11],valid_loss:[3,9,11],valid_polici:11,valu:[0,1,3,8,9,10,11],variabl:[1,3,5,8,10],varianc:[0,5],variant:[5,6,11],varieti:4,variou:[1,5],vector:[0,8,9,10],verbos:[0,1,4,8,9,10,11],veri:[3,5],versatil:6,version:[2,9,11],vertic:8,vertical_xtick:8,via:[0,2,3,11],view:5,visual:[5,6],vline:8,vocab_s:9,vocabulari:[9,10],vocabulary_s:[3,9,10,11],vs:[3,8],w:[0,3,10],wa:[0,3,5,8,10,11],wai:[1,11],wait:9,want:[3,4],warn:10,wb:[0,10],wdbc:0,we:[0,1,3,4,5,6],weight:[9,10],weight_decai:9,well:[0,3,4,5,11],were:0,what:3,whcih:10,when:[0,1,3,4,5,8,9,10],whenev:[5,8],where:[3,5,8,9,10,11],wherebi:4,whether:[8,9,10,11],which:[0,1,3,4,5,8,9,10,11],white:0,whole:[0,1,3,4,8],whose:[10,11],why:3,wide:5,wiki:[0,3],wine:0,within:[8,11],without:[1,3,8,10],word:[1,3,6,9,10,11],work:[1,3,4,5,10],worker:[1,8,10,11],wors:[4,5,8],would:[0,1,3,5,6,8,10,11],wrapper:[8,9,10,11],written:6,www:[],x2:10,x:[5,8,9,10,11],x_error:8,xavier:9,xavier_uniform:9,xlrd:[0,2],xy:10,y:[5,8,9,10,11],y_:[],y_error:8,y_i:11,y_j:11,y_pred:8,y_true:8,ye:[],yeast:[0,10],yield:[5,8,10,11],yin:[],you:[2,3],your:3,z:[0,10],zero:[0,8],zfthyovrzwxmgfzylqw_y8cagg:[],zip:[0,5]},titles:["Datasets","Evaluation","Installation","Quantification Methods","Model Selection","Plotting","Welcome to QuaPy\u2019s documentation!","quapy","quapy package","quapy.classification package","quapy.data package","quapy.method package"],titleterms:{"function":8,A:6,The:3,ad:0,aggreg:[3,11],base:[10,11],bia:5,classif:[4,9],classifi:3,content:[6,8,9,10,11],count:3,custom:0,data:[0,10],dataset:[0,10],diagon:5,distanc:3,document:6,drift:5,emq:3,ensembl:3,error:[1,5,8],evalu:[1,8],ex:[],exampl:6,expect:3,explicit:3,featur:6,get:[],hdy:3,helling:3,indic:6,instal:2,introduct:6,issu:0,learn:0,loss:[2,3,4],machin:0,maxim:3,measur:1,meta:[3,11],method:[3,9,11],minim:3,model:[3,4],model_select:8,modul:[8,9,10,11],network:3,neural:[3,9,11],non_aggreg:11,orient:[2,4],packag:[8,9,10,11],perf:2,plot:[5,8],preprocess:10,process:0,protocol:1,quanet:3,quantif:[2,3,4,5],quapi:[6,7,8,9,10,11],quick:6,reader:10,readm:[],requir:2,review:0,s:6,select:4,sentiment:0,start:[],submodul:[8,9,10,11],subpackag:8,svm:2,svmperf:9,tabl:6,target:4,test:[],test_bas:[],test_dataset:[],test_method:[],titl:[],twitter:0,uci:0,util:8,variant:3,welcom:6,y:3}}) \ No newline at end of file +Search.setIndex({"docnames": ["Datasets", "Evaluation", "ExplicitLossMinimization", "Home", "Installation", "Methods", "Model-Selection", "Plotting", "Protocols", "index", "modules", "quapy", "quapy.classification", "quapy.data", "quapy.method"], "filenames": ["Datasets.md", "Evaluation.md", "ExplicitLossMinimization.md", "Home.md", "Installation.rst", "Methods.md", "Model-Selection.md", "Plotting.md", "Protocols.md", "index.rst", "modules.rst", "quapy.rst", "quapy.classification.rst", "quapy.data.rst", "quapy.method.rst"], "titles": ["Datasets", "Evaluation", "Explicit Loss Minimization", "<no title>", "Installation", "Quantification Methods", "Model Selection", "Plotting", "Protocols", "Welcome to QuaPy\u2019s documentation!", "quapy", "quapy package", "quapy.classification package", "quapy.data package", "quapy.method package"], "terms": {"quapi": [0, 1, 2, 3, 4, 5, 6, 7, 8], "make": [0, 2, 5, 11, 14], "avail": [0, 1, 2, 4, 5, 7, 9, 12, 14], "sever": [0, 2, 13], "have": [0, 1, 4, 5, 6, 7, 8, 11, 13, 14], "been": [0, 5, 6, 7, 8, 11, 12, 13, 14], "us": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "quantif": [0, 1, 2, 8, 9, 11, 12, 13, 14], "literatur": [0, 1, 6, 8, 9], "well": [0, 5, 7, 14], "an": [0, 1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14], "interfac": [0, 1, 5, 14], "allow": [0, 2, 4, 5, 7, 8, 11, 12, 13, 14], "anyon": 0, "import": [0, 5, 6, 7, 8, 9, 13, 14], "A": [0, 1, 5, 11, 12, 13, 14], "object": [0, 8, 11, 12, 13, 14], "i": [0, 1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14], "roughli": 0, "pair": [0, 11], "labelledcollect": [0, 5, 6, 8, 11, 13, 14], "one": [0, 1, 2, 5, 6, 7, 8, 11, 13, 14], "plai": 0, "role": 0, "train": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "set": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "anoth": [0, 1, 5, 7, 8, 11], "test": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "class": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "consist": [0, 6, 7, 8, 11, 12, 13, 14], "iter": [0, 11, 13, 14], "instanc": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "label": [0, 5, 6, 7, 8, 9, 11, 12, 13, 14], "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14], "handl": 0, "most": [0, 1, 5, 7, 8, 9, 11, 13, 14], "sampl": [0, 1, 5, 6, 7, 9, 11, 12, 13, 14], "function": [0, 1, 5, 6, 7, 8, 9, 10, 12, 13, 14], "take": [0, 5, 7, 8, 11, 13, 14], "look": [0, 5, 7, 14], "follow": [0, 1, 5, 6, 7, 8, 9, 11, 14], "code": [0, 1, 2, 5, 6, 7, 8, 12], "qp": [0, 1, 5, 6, 7, 8, 9, 11, 13, 14], "f": [0, 1, 5, 6, 7, 8, 9, 13], "1st": 0, "posit": [0, 5, 7, 11, 13, 14], "document": [0, 5, 7, 12, 13, 14], "2nd": 0, "onli": [0, 1, 5, 7, 11, 12, 13, 14], "neg": [0, 7, 11, 14], "neutral": 0, "3rd": 0, "2": [0, 1, 5, 6, 7, 8, 11, 13, 14], "0": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "1": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "print": [0, 1, 5, 6, 8, 9, 12, 13], "strprev": [0, 1, 11], "preval": [0, 1, 5, 6, 7, 9, 11, 12, 13, 14], "prec": [0, 11], "output": [0, 1, 5, 6, 8, 11, 12, 13, 14], "show": [0, 3, 5, 6, 7, 8, 11, 12, 13, 14], "digit": 0, "precis": [0, 11], "17": [0, 5], "50": [0, 7, 11, 14], "33": [0, 7, 11], "One": [0, 1, 5, 8, 14], "can": [0, 1, 4, 5, 6, 7, 8, 11, 13, 14], "easili": [0, 4, 7, 12], "produc": [0, 1, 7, 8, 11], "new": [0, 5, 8, 11, 12, 13], "desir": [0, 1, 5, 8, 11, 13], "sample_s": [0, 1, 5, 6, 7, 8, 11, 14], "10": [0, 6, 7, 8, 11, 12, 14], "prev": [0, 1, 8, 11, 13], "4": [0, 1, 5, 6, 7, 8, 9, 13, 14], "5": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "which": [0, 1, 5, 6, 7, 11, 12, 13, 14], "40": [0, 5, 14], "made": [0, 2, 4, 11, 13, 14], "across": [0, 6, 7, 9, 11, 14], "differ": [0, 5, 6, 7, 8, 9, 11, 13, 14], "run": [0, 2, 4, 5, 7, 11, 13, 14], "e": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "g": [0, 1, 5, 6, 8, 9, 11, 13, 14], "method": [0, 1, 2, 3, 6, 7, 8, 9, 11], "same": [0, 5, 6, 7, 8, 11, 13, 14], "exact": [0, 8, 13], "retain": [0, 5, 12, 14], "index": [0, 5, 9, 11, 12, 13, 14], "gener": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "sampling_index": [0, 13], "sampling_from_index": [0, 13], "also": [0, 1, 2, 4, 5, 6, 7, 8, 9, 11, 12], "implement": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "artifici": [5, 6, 7, 9, 11], "protocol": [0, 3, 5, 6, 7, 9, 10, 13, 14], "via": [4, 5, 8, 11, 12, 14], "python": [0, 9], "": [0, 1, 5, 6, 7, 11, 12, 13, 14], "seri": [0, 6, 13], "equidist": 11, "rang": [6, 7, 8, 11, 14], "entir": [0, 1, 5, 6, 7, 8, 11], "spectrum": [7, 8, 11], "simplex": [9, 11], "space": [6, 11, 12], "artificial_sampling_gener": [], "100": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "n_preval": [8, 11], "each": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "valid": [0, 5, 6, 7, 8, 11, 12, 13, 14], "combin": [6, 8, 11, 14], "origin": [0, 1, 5, 8, 11, 13], "from": [0, 1, 5, 6, 7, 9, 11, 12, 13, 14], "split": [0, 5, 6, 7, 11, 12, 13, 14], "point": [5, 8, 11, 13], "25": [7, 11, 12, 14], "75": [6, 7, 8, 11], "00": [], "see": [0, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14], "evalu": [0, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14], "wiki": [0, 1, 3, 5], "further": [0, 5, 12, 13, 14], "detail": [0, 1, 2, 5, 9, 12, 13, 14], "how": [0, 5, 6, 7, 11, 13, 14], "properli": [5, 14], "three": [0, 7], "about": [0, 7, 11, 13], "kindl": [0, 5, 7, 13, 14], "devic": [0, 5, 7, 12, 14], "harri": 0, "potter": 0, "known": [0, 5, 6, 11, 14], "imdb": [0, 6, 7, 8, 13], "movi": 0, "fetch": [0, 9], "unifi": [0, 14], "For": [0, 1, 6, 7, 8, 9, 11, 13], "exampl": [0, 2, 3, 5, 6, 7, 8, 11, 12, 13, 14], "fetch_review": [0, 5, 6, 7, 8, 13, 14], "These": [0, 1, 2, 5, 8, 12], "esuli": [0, 2, 4, 5, 12, 13, 14], "moreo": [0, 5, 6, 13, 14], "sebastiani": [0, 5, 6, 13, 14], "2018": [0, 5, 13], "octob": [0, 5], "recurr": [0, 5, 13], "neural": [0, 11, 13], "network": [0, 11, 12, 13, 14], "In": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14], "proceed": [0, 5, 13], "27th": [0, 5, 13], "acm": [0, 5, 13, 14], "intern": [0, 1, 5, 12, 13], "confer": [0, 5, 12, 13], "inform": [0, 5, 6, 8, 11, 12, 13, 14], "knowledg": [0, 5, 13], "manag": [0, 5, 13], "pp": [0, 5, 6, 12], "1775": [0, 5], "1778": [0, 5], "The": [0, 1, 2, 4, 6, 7, 8, 9, 11, 12, 13, 14], "list": [0, 7, 11, 12, 13, 14], "id": [0, 5, 13], "reviews_sentiment_dataset": [0, 13], "some": [0, 1, 5, 7, 8, 11, 13, 14], "statist": [0, 8, 11, 14], "fhe": 0, "ar": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "summar": 0, "below": [0, 4, 5, 7, 11, 13], "size": [0, 1, 5, 11, 12, 13, 14], "type": [0, 5, 11, 13, 14], "hp": [0, 5, 13], "9533": 0, "18399": 0, "018": 0, "982": 0, "065": 0, "935": 0, "text": [0, 5, 11, 12, 13, 14], "3821": [0, 13], "21591": [0, 13], "081": [0, 13], "919": [0, 13], "063": [0, 13], "937": [0, 13], "25000": 0, "500": [0, 1, 7, 14], "11": [0, 8, 9, 11], "analysi": [0, 5, 9, 13], "access": [0, 5, 13, 14], "were": 0, "tf": [0, 13], "idf": 0, "format": [0, 7, 11, 13, 14], "present": [0, 5, 13], "two": [0, 5, 7, 8, 11, 13, 14], "val": [0, 8, 12, 13], "model": [0, 1, 3, 7, 8, 9, 11, 12, 14], "select": [0, 1, 3, 5, 8, 9, 11, 13, 14], "purpos": [0, 8, 14], "exemplifi": 0, "load": [0, 5, 8, 11, 13, 14], "fetch_twitt": [0, 5, 9, 13], "gasp": [0, 13], "for_model_select": [0, 13], "true": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "gao": [0, 5, 13, 14], "w": [0, 5, 13], "2015": [0, 2, 4, 5, 12, 14], "august": 0, "tweet": [0, 5, 13], "classif": [0, 1, 5, 9, 11, 13, 14], "ieee": 0, "advanc": [0, 6, 8, 9, 11], "social": [0, 5, 13], "mine": [0, 5], "asonam": 0, "97": 0, "104": [0, 1], "semeval13": [0, 13], "semeval14": [0, 13], "semeval15": [0, 13], "share": [0, 13], "semev": 0, "mean": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "would": [0, 1, 5, 7, 9, 13, 14], "get": [0, 7, 8, 11, 12, 13, 14], "when": [0, 1, 5, 7, 8, 11, 12, 13], "request": [0, 6, 11, 13, 14], "ani": [0, 5, 6, 7, 8, 9, 11, 12, 13, 14], "them": [0, 5, 13, 14], "consult": [0, 8], "twitter_sentiment_datasets_test": [0, 13], "9": [0, 5, 7, 11], "replac": [0, 5, 11, 13], "twitter_sentiment_datasets_train": [0, 13], "found": [0, 5, 11, 12, 13], "featur": [0, 13], "3": [0, 1, 5, 6, 7, 8, 9, 11, 12, 13, 14], "8788": 0, "3765": 0, "694582": 0, "421": 0, "496": 0, "082": 0, "407": 0, "507": 0, "086": 0, "spars": [0, 13], "hcr": [0, 5, 13], "1594": 0, "798": 0, "222046": 0, "546": 0, "211": 0, "243": 0, "640": 0, "167": 0, "193": 0, "omd": [0, 13], "1839": 0, "787": 0, "199151": 0, "463": 0, "271": 0, "266": 0, "437": 0, "283": 0, "280": 0, "sander": [0, 13], "2155": 0, "923": 0, "229399": 0, "161": 0, "691": 0, "148": 0, "164": [0, 5], "688": 0, "11338": 0, "3813": 0, "1215742": 0, "159": 0, "470": 0, "372": 0, "158": 0, "430": 0, "412": 0, "1853": 0, "109": 0, "361": 0, "530": 0, "2390": 0, "153": 0, "413": 0, "434": 0, "semeval16": [0, 9, 13], "8000": 0, "2000": 0, "889504": 0, "157": 0, "351": 0, "492": 0, "163": 0, "341": 0, "497": 0, "sst": [0, 13], "2971": 0, "1271": 0, "376132": 0, "261": 0, "452": 0, "288": 0, "207": 0, "481": 0, "312": 0, "wa": [0, 5, 7, 8, 11, 13, 14], "2184": 0, "936": 0, "248563": 0, "305": 0, "414": 0, "281": 0, "282": 0, "446": 0, "272": [0, 1], "wb": [0, 13], "4259": 0, "1823": 0, "404333": 0, "270": 0, "392": 0, "337": 0, "274": 0, "335": 0, "32": [0, 6, 9], "repositori": [0, 13], "p\u00e9rez": [0, 5, 13, 14], "g\u00e1llego": [0, 5, 13, 14], "p": [0, 5, 11, 12, 13, 14], "quevedo": [0, 5, 13], "j": [0, 5, 13, 14], "r": [0, 5, 11, 13], "del": [0, 5, 13], "coz": [0, 5, 13], "2017": [0, 5, 13, 14], "ensembl": [0, 9, 13, 14], "problem": [0, 5, 7, 11, 13, 14], "characteriz": [0, 5, 13], "chang": [0, 1, 5, 13], "distribut": [0, 1, 5, 7, 8, 11, 13, 14], "case": [0, 1, 5, 7, 8, 11, 12, 13, 14], "studi": [0, 5, 13], "fusion": [0, 5, 13], "34": [0, 5, 13, 14], "87": [0, 5, 13], "doe": [0, 2, 4, 5, 11, 14], "exactli": 0, "coincid": [0, 9], "et": [0, 2, 4, 5, 12, 13, 14], "al": [0, 2, 4, 5, 12, 13, 14], "sinc": [0, 1, 5, 6, 7, 8, 13, 14], "we": [0, 1, 3, 5, 6, 7, 8, 9, 13], "unabl": 0, "find": [0, 6, 14], "diabet": 0, "phonem": 0, "call": [0, 1, 5, 7, 8, 11, 13, 14], "fetch_ucidataset": [0, 5, 13], "yeast": [0, 13], "verbos": [0, 6, 11, 12, 13, 14], "return": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "randomli": [0, 13], "drawn": [0, 8, 11, 13], "stratifi": [0, 5, 12, 13, 14], "manner": [0, 12, 14], "whole": [0, 1, 5, 6, 11, 12], "collect": [0, 8, 11, 12, 13], "70": 0, "30": [0, 5, 6, 8, 14], "respect": [0, 1, 7, 11, 14], "option": [0, 5, 7, 13, 14], "indic": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "descript": [0, 13], "should": [0, 5, 6, 7, 9, 11, 12, 13, 14], "standard": [0, 7, 8, 11, 12, 13, 14], "paper": [0, 5, 12, 14], "submit": 0, "kfcv": [0, 12, 13, 14], "order": [0, 4, 5, 6, 7, 8, 11, 13, 14], "accommod": [0, 11], "practic": 0, "could": [0, 1, 5, 6, 7, 8, 9], "first": [0, 1, 2, 4, 5, 7, 11, 13, 14], "instanti": [0, 1, 5, 6, 8, 11, 12, 14], "creat": [0, 9, 11, 14], "time": [0, 1, 5, 8, 11, 13, 14], "fetch_ucilabelledcollect": [0, 13], "nfold": [0, 11, 13], "nrepeat": [0, 13], "abov": [0, 2, 5, 7, 11], "conduct": [0, 11], "2x5fcv": 0, "all": [0, 1, 2, 4, 5, 7, 8, 11, 12, 14], "come": [0, 1, 8, 11, 13, 14], "numer": [0, 1, 5, 9, 13, 14], "form": [0, 11, 13, 14], "dens": [0, 14], "matric": [0, 7, 13], "acut": 0, "120": 0, "6": [0, 1, 5, 7, 13], "508": 0, "b": [0, 11, 13, 14], "583": 0, "417": 0, "balanc": [0, 6, 7, 14], "625": 0, "539": 0, "461": 0, "922": 0, "078": 0, "breast": 0, "cancer": 0, "683": 0, "350": 0, "650": 0, "cmc": 0, "1473": 0, "573": 0, "427": 0, "774": 0, "226": 0, "653": 0, "347": 0, "ctg": 0, "2126": 0, "22": [0, 5, 12, 13], "222": [0, 12], "778": 0, "861": 0, "139": 0, "917": 0, "083": 0, "german": 0, "1000": [0, 6, 14], "24": [0, 1, 12], "300": [0, 1, 12], "700": 0, "haberman": [0, 5], "306": 0, "735": 0, "265": 0, "ionospher": 0, "641": 0, "359": 0, "iri": 0, "150": 0, "667": 0, "333": 0, "mammograph": 0, "830": 0, "514": 0, "486": 0, "pageblock": 0, "5473": 0, "979": 0, "021": 0, "semeion": 0, "1593": 0, "256": [0, 12], "901": 0, "099": 0, "sonar": 0, "208": 0, "60": 0, "534": 0, "466": 0, "spambas": 0, "4601": 0, "57": 0, "606": 0, "394": 0, "spectf": 0, "267": 0, "44": 0, "794": 0, "206": 0, "tictacto": 0, "958": 0, "transfus": 0, "748": 0, "762": 0, "238": 0, "wdbc": 0, "569": 0, "627": 0, "373": 0, "wine": 0, "178": 0, "13": [0, 12], "669": 0, "331": 0, "601": 0, "399": 0, "730": 0, "q": [0, 2, 4, 5, 11, 12, 14], "red": 0, "1599": 0, "465": 0, "535": 0, "white": 0, "4898": 0, "665": 0, "1484": 0, "8": [0, 6, 7, 13, 14], "711": 0, "289": 0, "download": [0, 2, 4, 5, 11, 13], "automat": [0, 12], "thei": [0, 5, 14], "store": [0, 12, 13, 14], "quapy_data": [0, 11], "folder": [0, 6, 8, 13, 14], "faster": [0, 13], "reus": [0, 5, 11, 13], "howev": [0, 6, 7], "requir": [0, 1, 2, 5, 8, 9, 12], "special": [0, 7, 13], "action": 0, "moment": [0, 1, 5], "fulli": [0, 11], "autom": [0, 9], "cardiotocographi": 0, "excel": 0, "file": [0, 7, 11, 12, 13, 14], "user": [0, 7, 8], "instal": [0, 5, 9, 12, 14], "xlrd": [0, 4], "modul": [0, 1, 5, 7, 8, 9, 10], "open": [0, 9, 13], "page": [0, 4, 9], "block": [0, 11], "need": [0, 5, 8, 11, 13, 14], "unix": 0, "compress": 0, "extens": [0, 2, 4, 7], "z": [0, 13], "directli": [0, 5], "doabl": 0, "packag": [0, 2, 4, 5, 9, 10], "like": [0, 1, 5, 7, 8, 11, 12, 13, 14], "gzip": 0, "zip": [0, 7, 11], "uncompress": 0, "o": [0, 11], "depend": [0, 6, 7, 11, 14], "softwar": 0, "manual": 0, "do": [0, 5, 6, 11, 12, 13, 14], "invok": [0, 5, 8, 11, 13], "provid": [0, 5, 7, 8, 9, 13, 14], "loader": [0, 13], "simpl": [0, 5, 7, 11, 14], "deal": 0, "t": [0, 1, 5, 11, 12, 14], "pre": [0, 5, 11], "n": [0, 8, 11, 12, 14], "second": [0, 1, 5, 7, 11, 13], "represent": [0, 5, 11, 12, 14], "col": [0, 13], "int": [0, 7, 11, 13, 14], "float": [0, 5, 11, 12, 13, 14], "charg": [0, 11, 13], "classmethod": [0, 11, 13, 14], "def": [0, 1, 5, 7, 11], "cl": 0, "path": [0, 5, 7, 11, 12, 13, 14], "str": [0, 11, 13, 14], "loader_func": [0, 13], "callabl": [0, 11, 13, 14], "defin": [0, 5, 8, 11, 12, 13, 14], "argument": [0, 1, 5, 7, 8, 11, 13, 14], "initi": [0, 12, 14], "particular": [0, 5, 14], "receiv": [0, 5, 7], "addition": 0, "number": [0, 1, 5, 6, 7, 8, 11, 12, 13, 14], "specifi": [0, 5, 7, 8, 11, 12, 13], "otherwis": [0, 5, 11, 13], "infer": [0, 13], "least": [0, 13], "pass": [0, 1, 7, 11, 12, 14], "along": [0, 1, 5, 11, 14], "train_path": [0, 13], "my_data": 0, "dat": [0, 12], "test_path": [0, 13], "my_custom_load": 0, "rb": 0, "fin": 0, "preprocess": [0, 5, 11, 14], "includ": [0, 2, 3, 5, 6, 7, 8, 9, 13, 14], "text2tfidf": [0, 5, 13], "tfidf": [0, 6, 7, 8, 13], "vector": [0, 5, 8, 11, 12, 13, 14], "reduce_column": [0, 13], "reduc": [0, 1, 13], "column": [0, 1, 11, 13], "base": [0, 5, 9, 11, 12], "term": [0, 5, 6, 7, 9, 11, 12, 13, 14], "frequenc": [0, 13, 14], "transform": [0, 12, 13, 14], "valu": [0, 1, 5, 6, 8, 11, 12, 13, 14], "score": [0, 1, 6, 11, 12, 13], "subtract": [0, 11, 13], "normal": [0, 1, 5, 11, 13, 14], "deviat": [0, 7, 8, 11, 13], "so": [0, 5, 7, 8, 11, 12, 13, 14], "zero": [0, 11], "unit": [0, 9, 11], "varianc": [0, 7], "textual": [0, 9, 13], "token": [0, 12, 13], "appeal": 1, "tool": [1, 9], "scenario": [1, 5, 6, 7, 9], "dataset": [1, 3, 5, 6, 7, 8, 9, 11, 12, 14], "shift": [1, 6, 8, 9, 11, 12, 14], "particularli": 1, "prior": [1, 5, 6, 7, 8, 9, 11, 14], "probabl": [1, 5, 6, 7, 8, 9, 11, 12, 14], "That": [1, 6], "interest": [1, 7, 8, 9, 11], "estim": [0, 1, 5, 7, 9, 11, 12, 13, 14], "aris": 1, "under": [1, 8], "belief": 1, "those": [1, 5, 6, 7, 11, 12, 14], "might": [1, 11, 13], "ones": [1, 5, 7, 11, 13, 14], "observ": [1, 14], "dure": [1, 7, 14], "other": [1, 5, 7, 9, 11, 13, 14], "word": [1, 5, 9, 12, 13, 14], "simpli": [1, 2, 4, 5, 6, 7, 9, 11, 14], "predictor": 1, "assum": [1, 9, 14], "unlik": [1, 11], "machin": [1, 6, 9, 12], "learn": [1, 4, 5, 6, 9, 11, 12, 13, 14], "govern": 1, "iid": [1, 7, 9], "assumpt": [1, 7, 9], "brief": [0, 1, 13], "dedic": [0, 1, 13], "explain": [1, 7], "here": [1, 14], "mae": [1, 6, 8, 9, 11, 12, 14], "absolut": [1, 5, 7, 9, 11, 14], "mrae": [1, 9, 11, 12, 14], "rel": [1, 5, 11, 13, 14], "mse": [1, 5, 9, 11, 14], "squar": [1, 5, 11], "mkld": [1, 11, 14], "kullback": [1, 5, 11, 14], "leibler": [1, 5, 11, 14], "diverg": [1, 5, 11, 14], "mnkld": [1, 11, 14], "ae": [1, 2, 4, 5, 7, 11], "rae": [1, 2, 4, 5, 11], "se": [1, 11], "kld": [1, 2, 4, 5, 11, 12, 14], "nkld": [1, 2, 4, 5, 9, 11, 12, 14], "individu": [1, 5], "without": [1, 5, 11, 13], "averag": [1, 5, 11, 13, 14], "acc": [1, 5, 7, 8, 9, 11, 14], "accuraci": [1, 7, 11, 14], "f1e": [1, 11], "f1": [1, 11, 12], "true_prev": [1, 7, 11], "prevs_hat": [1, 11], "ndarrai": [1, 5, 11, 13, 14], "contain": [1, 2, 4, 5, 7, 8, 11, 12, 13, 14], "smooth": [1, 11], "stabil": [1, 14], "third": [1, 7], "ep": [1, 11], "none": [1, 6, 8, 11, 12, 13, 14], "paramet": [1, 5, 6, 8, 11, 12, 13, 14], "epsilon": [1, 11, 14], "tradition": 1, "2t": [1, 11], "past": 1, "either": [1, 5, 11, 14], "environ": [1, 5, 6, 7, 8, 11, 14], "variabl": [1, 5, 7, 11, 13], "onc": [0, 1, 5, 6, 7, 8, 11, 13], "ommit": [], "thereaft": 1, "recommend": [1, 7, 14], "np": [1, 5, 6, 7, 8, 11, 13, 14], "asarrai": 1, "let": [1, 5, 6, 11, 14], "estim_prev": [1, 7, 11], "ae_": [], "3f": [1, 9], "200": [1, 12], "600": 1, "914": 1, "final": [1, 5, 7, 14], "possibl": [1, 5, 8, 11, 14], "string": [1, 11, 13, 14], "error_funct": 1, "from_nam": [1, 11], "accord": [5, 6, 11, 12, 13, 14], "fix": 8, "cover": [8, 11, 12], "full": [8, 11], "contrast": [], "natur": [9, 11], "despit": [], "introduc": [], "approxim": [5, 7, 11, 12], "preserv": [7, 11, 13], "procol": [], "equal": [8, 11, 14], "distant": [8, 11], "interv": [7, 8, 11], "n_prevpoint": [8, 11], "determin": [6, 7, 8, 11], "constrain": [7, 8, 11, 13], "obtain": [8, 11, 12, 14], "66": [8, 14], "given": [1, 5, 6, 8, 11, 12, 13, 14], "num_prevalence_combin": [8, 11], "21": [5, 7, 8, 11], "n_class": [5, 8, 11, 12, 13, 14], "n_repeat": [8, 11], "1771": 8, "note": [1, 5, 7, 8, 11, 13], "last": [5, 7, 8, 11, 12, 13], "typic": [1, 6, 7, 8, 11, 12, 13, 14], "singl": [1, 5, 8, 9, 11, 14], "higher": [7, 8], "comput": [1, 5, 7, 8, 11, 14], "perform": [1, 5, 6, 7, 8, 9, 11, 12, 14], "signific": 8, "instead": [0, 1, 5, 6, 8, 11, 13, 14], "work": [5, 7, 8, 11, 13, 14], "wai": [5, 8, 14], "around": [8, 13, 14], "maximum": [8, 11, 12, 14], "budg": 8, "close": [8, 13], "than": [1, 6, 7, 8, 11, 12, 13], "budget": 8, "achiev": [5, 6, 7, 8], "get_nprevpoints_approxim": [8, 11], "5000": [0, 1, 7, 8], "4960": 8, "cost": [], "sometim": 8, "cumbersom": 8, "control": [6, 8, 11], "overal": 11, "experi": [0, 4, 5, 6, 7, 11, 13], "rather": 6, "By": [5, 11], "avoid": 11, "lead": 13, "closer": [], "surpass": [], "script": [2, 4, 5, 9, 14], "pacc": [5, 7, 11, 14], "reli": [5, 8, 11, 14], "logist": [5, 12, 14], "regressor": 5, "classifi": [6, 7, 9, 11, 12, 14], "variou": 7, "metric": [5, 6, 9, 11, 14], "sklearn": [5, 6, 7, 8, 9, 12, 13, 14], "linear_model": [5, 6, 8, 9, 12], "logisticregress": [5, 6, 8, 9, 12, 14], "data": [5, 6, 7, 9, 11, 12, 14], "min_df": [5, 6, 7, 8, 13, 14], "inplac": [5, 13, 14], "lr": [5, 12, 14], "aggreg": [1, 6, 7, 8, 9, 11], "fit": [5, 6, 7, 8, 9, 11, 12, 13, 14], "df": [], "artificial_sampling_report": [], "mani": [1, 5, 6, 7, 8, 9, 11, 14], "extract": [11, 13], "categori": 11, "n_repetit": [], "n_job": [5, 6, 8, 11, 12, 13, 14], "parallel": [5, 6, 11, 12, 13, 14], "worker": [11, 12, 13, 14], "cpu": [12, 14], "random_se": 11, "42": [], "random": [5, 7, 8, 11, 13], "seed": [8, 11, 13], "replic": [8, 11], "error_metr": [1, 6, 8, 11], "line": [5, 11], "result": [1, 2, 4, 5, 7, 9, 14], "report": [1, 11], "panda": [1, 4, 11], "datafram": [1, 11], "displai": [1, 7, 8, 11, 12], "just": [5, 8], "clearer": [], "shown": [7, 11], "convert": [5, 11, 12, 13, 14], "repres": [5, 7, 11, 13, 14], "decim": [], "default": [5, 8, 11, 12, 13, 14], "pd": 1, "set_opt": 1, "expand_frame_repr": 1, "fals": [1, 5, 7, 11, 12, 13, 14], "map": [1, 12, 14], "000": [], "000e": [], "091": 1, "909": 1, "009": [], "048": [], "426e": [], "04": [], "837": [], "037": [], "114": [], "633e": [], "03": [], "7": [5, 6, 7, 8, 9, 11, 12, 14], "717": [], "017": [], "041": [], "383e": [], "366": [], "634": [], "034": [], "070": [], "412e": [], "459": [], "541": [], "387e": [], "565": [], "435": [], "035": 1, "073": [], "535e": [], "654": [], "346": [], "046": [], "108": [], "701e": [], "725": [], "275": [], "075": [], "235": [], "515e": [], "02": [], "858": [], "142": [], "042": [], "229": [], "740e": [], "945": [], "055": [], "27": [5, 12], "357": [], "219e": [], "578": [], "dtype": [1, 13], "float64": 1, "artificial_sampling_ev": [], "artificial_sampling_predict": [], "arrai": [5, 7, 11, 12, 13, 14], "pip": 4, "older": 4, "version": [2, 4, 11, 12], "scikit": [4, 5, 6, 11, 12, 13, 14], "numpi": [4, 6, 8, 11, 12], "scipi": [4, 13], "pytorch": [4, 14], "quanet": [4, 9, 12, 14], "svmperf": [2, 3, 4, 5, 11, 14], "patch": [2, 4, 5, 12, 14], "joblib": [4, 14], "tqdm": 4, "matplotlib": [4, 11], "involv": [4, 7, 11], "you": [4, 5], "appli": [1, 2, 4, 5, 6, 7, 11, 12, 13, 14], "ext": [2, 4], "compil": [2, 4, 5], "sourc": [2, 4, 5, 9, 12], "prepare_svmperf": [2, 4, 5], "sh": [2, 4, 5], "job": [2, 4], "directori": [2, 4, 11, 12, 13, 14], "svm_perf_quantif": [2, 4, 5], "optim": [1, 2, 4, 6, 11, 12, 14], "measur": [2, 4, 5, 6, 7, 9, 11, 14], "propos": [2, 4, 5, 8, 14], "barranquero": [2, 4, 5, 12, 14], "extend": [2, 4, 5, 11, 14], "former": [4, 14], "categor": [5, 13], "belong": [5, 6, 14], "non": [5, 14], "group": 5, "though": [5, 11], "plan": 5, "add": [5, 6, 11, 13], "more": [1, 2, 5, 7, 8, 11, 14], "futur": 5, "character": [1, 5, 9], "fact": [5, 7], "product": [0, 5, 13], "quantifi": [0, 1, 5, 6, 7, 8, 9, 11, 13, 14], "shoud": 5, "basequantifi": [5, 11, 14], "abstract": [5, 11, 12, 13, 14], "abstractmethod": 5, "self": [5, 6, 11, 12, 13, 14], "set_param": [5, 11, 12, 14], "get_param": [5, 11, 12, 14], "deep": [5, 11, 14], "familiar": 5, "structur": [5, 14], "inspir": 5, "reason": [5, 7, 8, 9], "why": 5, "ha": [1, 5, 6, 7, 8, 11, 12, 13, 14], "adopt": [5, 6, 13], "respond": 5, "predict": [1, 5, 7, 11, 12, 14], "input": [5, 7, 11, 12, 13, 14], "element": [5, 11, 13, 14], "while": [0, 5, 7, 12, 13, 14], "selector": 5, "process": [1, 6, 11], "hyperparamet": [5, 8, 11, 14], "search": [6, 9, 11, 14], "part": [5, 13], "aggregativequantifi": [1, 5, 14], "must": [5, 13, 14], "fit_learn": 5, "classif_predict": [5, 14], "mention": 5, "befor": [5, 11, 12, 13, 14], "inde": [5, 8], "alreadi": [1, 5, 11, 14], "preclassifi": [], "maintain": [5, 14], "through": [5, 11], "properti": [5, 11, 12, 13, 14], "learner": [5, 6, 12, 14], "extern": 5, "probabilist": [5, 11, 12, 14], "inherit": [5, 8, 11], "aggregativeprobabilisticquantifi": [5, 14], "posterior": [5, 11, 12, 14], "crisp": [1, 5, 11, 14], "decis": [5, 11, 12, 14], "hard": [5, 11, 12], "classif_posterior": 14, "posterior_prob": 14, "advantag": [5, 8, 14], "procedur": [1, 5, 9, 11], "veri": [5, 7, 11], "effici": 5, "everi": [0, 1, 5, 6, 8, 11, 14], "leverag": 5, "speed": [1, 5, 11, 14], "up": [1, 5, 11, 12, 14], "over": [5, 6, 11], "customarili": [5, 6], "done": [5, 6], "four": 5, "cc": [5, 7, 14], "simplest": 5, "deliv": [5, 6, 14], "adjust": [5, 9, 11, 14], "pcc": [5, 7, 14], "soft": [1, 5], "serv": [5, 11, 13], "complet": [5, 7, 14], "equip": [5, 7], "svm": [2, 5, 7, 9, 12, 13, 14], "linearsvc": [5, 7, 13], "pickl": [5, 11, 13, 14], "alia": [5, 11, 13, 14], "classifyandcount": [5, 14], "estim_preval": [5, 9, 14], "rate": [5, 11, 12, 14], "binari": [0, 5, 7, 9, 11, 12, 13, 14], "init": 5, "addit": [5, 11], "val_split": [5, 6, 12, 14], "integ": [5, 11, 12, 13, 14], "k": [5, 9, 11, 12, 13, 14], "fold": [5, 11, 13, 14], "cross": [5, 11, 12, 13, 14], "specif": [1, 5, 6, 8, 11], "held": [5, 6, 11, 12, 14], "out": [0, 1, 5, 6, 7, 11, 12, 13, 14], "postpon": [1, 5], "constructor": 5, "prevail": 5, "overrid": 5, "illustr": [3, 5, 6, 7], "seem": 5, "calibr": [5, 11], "calibratedclassifiercv": 5, "base_estim": 5, "cv": [5, 6], "predict_proba": [5, 12, 14], "As": [5, 6], "calibratedclassifi": 5, "except": [5, 11, 14], "rais": [5, 11, 14], "lastli": 5, "everyth": 5, "said": 5, "aboud": 5, "sld": [5, 14], "expectationmaximizationquantifi": [5, 14], "describ": [5, 11, 14], "saeren": [5, 14], "m": [5, 11, 14], "latinn": [5, 14], "decaesteck": [5, 14], "c": [5, 6, 11, 12, 13, 14], "2002": 5, "priori": 5, "14": 5, "41": 5, "attempt": 5, "although": [5, 6, 7, 8, 14], "improv": [5, 11, 12, 14], "rank": [5, 12], "almost": 5, "alwai": [5, 7, 14], "among": 5, "effect": 5, "carri": [0, 1, 5, 11, 13, 14], "gonz\u00e1lez": 5, "castro": 5, "v": [5, 11, 12, 14], "alaiz": 5, "rodr\u0131": 5, "guez": 5, "alegr": 5, "2013": 5, "scienc": 5, "218": 5, "146": 5, "It": [1, 5, 6, 7, 11], "allia": 5, "hellingerdistancei": [5, 14], "mixtur": [5, 11, 14], "previou": 5, "overridden": [5, 14], "proport": [5, 6, 12, 13, 14], "taken": [5, 11, 12, 13, 14], "itself": [5, 11, 14], "accept": 5, "elm": [2, 5, 14], "famili": [5, 14], "target": [5, 7, 9, 11, 12, 14], "orient": [2, 5, 9, 11, 14], "joachim": [5, 12, 14], "svmq": 5, "d\u00edez": 5, "reliabl": 5, "pattern": 5, "recognit": 5, "48": 5, "591": 5, "604": 5, "svmkld": [], "multivari": [5, 12], "transact": 5, "discoveri": 5, "articl": [5, 6], "svmnkld": [], "svmae": [], "error": [5, 6, 9, 10, 12, 14], "svmrae": [], "what": 5, "nowadai": 5, "consid": [5, 7, 8, 11, 12, 13, 14], "behav": [5, 7, 8], "If": [5, 7, 11, 13, 14], "want": [5, 6], "custom": [5, 8, 9, 11, 13], "modifi": [5, 11], "assign": [5, 13], "Then": 5, "re": [5, 6, 12, 13], "thing": [5, 8], "your": 5, "svmperf_hom": [5, 14], "valid_loss": [5, 12, 14], "mycustomloss": 5, "28": [0, 1, 5, 13], "current": [5, 11, 12, 13, 14], "support": [5, 9, 13, 14], "oper": 5, "trivial": 5, "strategi": [5, 6], "2016": [5, 13, 14], "sentiment": [5, 9, 13], "19": [5, 13], "onevsal": [5, 14], "know": [5, 6], "where": [5, 7, 11, 12, 13, 14], "top": [5, 11, 14], "thu": [1, 5, 6, 7, 11, 12, 14], "nor": 5, "castano": [5, 13], "2019": [5, 13, 14], "dynam": [5, 12, 13, 14], "task": [0, 5, 6, 9, 13], "45": [5, 7, 13], "15": [5, 11, 13], "polici": [5, 14], "processor": 5, "av": [5, 14], "ptr": [5, 14], "member": [5, 14], "d": [5, 14], "static": [5, 14], "red_siz": [5, 14], "pleas": 5, "check": [5, 11], "offer": [5, 9], "torch": [5, 12, 14], "embed": [5, 12, 14], "lstm": [5, 12, 14], "cnn": [5, 14], "its": [5, 6, 8, 11, 12, 14], "layer": [5, 12, 14], "neuralclassifiertrain": [5, 12, 14], "cnnnet": [5, 12, 14], "vocabulary_s": [5, 12, 13, 14], "cuda": [5, 12, 14], "supervis": [6, 9], "strongli": [6, 7], "good": [6, 7], "choic": [1, 6, 14], "hyper": [6, 11, 12], "wherebi": 6, "chosen": [1, 6, 11], "pick": 6, "best": [6, 11, 12, 14], "being": [1, 6, 8, 11, 14], "criteria": 6, "solv": [6, 14], "assess": 6, "own": 6, "right": [6, 11, 13], "impos": [6, 11], "aim": [6, 7], "appropri": 6, "configur": [6, 11], "design": 6, "long": [6, 12], "regard": 6, "next": [6, 11, 12, 13], "section": [6, 8], "argu": 6, "alejandro": 6, "fabrizio": 6, "count": [6, 7, 9, 11, 13, 14], "arxiv": [], "preprint": [], "2011": [], "02552": [], "2020": [5, 12], "varieti": 6, "exhibit": [6, 7, 8], "degre": 6, "model_select": [6, 8, 10, 14], "gridsearchq": [6, 8, 11, 14], "grid": [6, 8, 11, 14], "explor": [6, 11], "portion": [], "param_grid": [6, 8, 11, 14], "logspac": [6, 8, 14], "class_weight": [6, 7, 14], "eval_budget": [], "refit": [6, 11], "retrain": [6, 12], "goe": 6, "end": [6, 11, 14], "best_params_": 6, "best_model_": 6, "101": [], "5f": 6, "system": [1, 6, 14], "start": 6, "hyperparam": 6, "0001": 14, "got": [6, 14], "24987": [], "48135": [], "001": [6, 12, 14], "24866": [], "100000": [], "43676": [], "finish": [1, 6], "param": [6, 11, 12, 14], "19982": [], "develop": [6, 9], "1010": [], "5005": [], "54it": [], "20342": [], "altern": [1, 6], "computation": 6, "costli": 6, "try": 6, "theoret": 6, "suboptim": 6, "opt": 6, "gridsearchcv": [6, 14], "10000": [], "5379": [], "55it": [], "41734": [], "wors": [7, 11], "larg": 11, "between": [7, 9, 11, 12, 14], "modal": [], "turn": [], "better": [], "nonetheless": [], "happen": 7, "basic": [7, 14], "help": [1, 7, 14], "analys": [7, 9], "outcom": 7, "main": [3, 7, 8], "method_nam": [7, 11, 14], "name": [5, 7, 11, 12, 13, 14], "shape": [7, 11, 12, 13, 14], "correspond": [1, 7, 13], "matrix": [7, 11, 14], "appear": 7, "occur": [7, 13], "merg": 7, "emq": [7, 14], "55": 7, "showcas": 7, "wide": [1, 7, 8], "variant": [7, 9, 11, 14], "linear": [7, 11, 14], "review": [7, 9, 13], "step": [7, 11], "05": [7, 11, 14], "gen_data": 7, "base_classifi": 7, "yield": [7, 8, 11, 13, 14], "tr_prev": [7, 11, 14], "append": 7, "__class__": [], "__name__": [], "insight": 7, "view": 7, "y": [7, 11, 12, 13, 14], "axi": [7, 11], "against": [6, 7], "x": [1, 5, 7, 11, 12, 13, 14], "unfortun": 7, "limit": [7, 8, 11, 14], "binary_diagon": [7, 11], "train_prev": [7, 11], "savepath": [7, 11], "bin_diag": 7, "png": 7, "save": [7, 11], "pdf": [7, 14], "cyan": 7, "dot": [7, 11], "color": [7, 11], "band": [7, 11], "hidden": [7, 12, 14], "show_std": [7, 11], "unadjust": 7, "bias": 7, "toward": [7, 13], "seen": [7, 11, 14], "evinc": 7, "box": [7, 11], "binary_bias_glob": [7, 11], "bin_bia": 7, "unbias": 7, "center": 7, "tend": 7, "overestim": 7, "high": [7, 11], "lower": [7, 14], "again": [7, 11], "accordingli": 7, "20": [1, 7, 11, 14], "90": [7, 11], "rewrit": 7, "method_data": 7, "training_preval": 7, "linspac": 7, "training_s": 7, "suffic": 7, "latex": [], "syntax": [], "_": [7, 9, 11, 13], "now": [5, 7, 8], "clearli": 7, "binary_bias_bin": [7, 11], "broken": [7, 11], "down": [1, 7, 8, 11, 13], "bin": [6, 7, 11, 14], "To": [7, 13], "nbin": [6, 7, 11, 14], "isometr": [7, 11], "subinterv": 7, "interestingli": 7, "enough": 7, "seemingli": 7, "tendenc": 7, "low": [6, 7, 11, 12], "underestim": 7, "beyond": 7, "67": [7, 11], "curios": 7, "pretti": 7, "discuss": 7, "analyz": 7, "compar": [7, 11], "both": [7, 13], "irrespect": [1, 7, 14], "harder": 7, "interpret": [7, 9, 14], "error_by_drift": [7, 11], "error_nam": [7, 11], "n_bin": [7, 11, 14], "err_drift": 7, "whenev": [7, 11], "clear": 7, "lowest": 7, "difficult": 7, "rememb": 7, "solid": 7, "comparison": [7, 8], "detriment": 7, "visual": [1, 7, 9], "hide": 7, "framework": [5, 9, 14], "written": 9, "root": 9, "concept": [3, 9], "baselin": 9, "integr": 9, "commonli": [8, 9], "facilit": 9, "twitter": [9, 13], "true_preval": 9, "hold": [9, 11, 14], "endeavour": [9, 11], "popular": [8, 9], "expect": [8, 9, 14], "maxim": [9, 14], "hdy": [9, 14], "versatil": 9, "etc": [5, 9], "uci": [9, 13], "nativ": 9, "loss": [9, 12, 14], "perf": [2, 9, 12, 14], "ad": [8, 9], "meta": [9, 11], "plot": [3, 9, 10], "diagon": [9, 11], "bia": [5, 9, 11, 12, 14], "drift": 9, "api": [5, 9], "subpackag": 10, "submodul": 10, "util": [10, 12, 13], "content": 10, "bctscalibr": 12, "nbvscalibr": 12, "recalibratedprobabilisticclassifi": 12, "recalibratedprobabilisticclassifierbas": 12, "classes_": [12, 13, 14], "fit_cv": 12, "fit_tr_val": 12, "tscalibr": 12, "vscalibr": 12, "lowranklogisticregress": 12, "document_embed": 12, "lstmnet": 12, "reset_net_param": 12, "textclassifiernet": 12, "dimens": [11, 12, 13, 14], "forward": [12, 14], "xavier_uniform": 12, "torchdataset": 12, "asdataload": 12, "decision_funct": 12, "splitstratifi": 13, "stat": 13, "train_test": [5, 6, 7, 8, 13], "xp": 13, "xy": 13, "split_random": 13, "split_stratifi": [6, 8, 13], "uniform_sampl": 13, "uniform_sampling_index": 13, "fetch_lequa2022": [0, 13], "warn": 13, "indextransform": 13, "add_word": 13, "fit_transform": 13, "reader": 11, "binar": [11, 13], "from_csv": 13, "from_spars": 13, "from_text": 13, "reindex_label": 13, "getptecondestim": 14, "solve_adjust": 14, "adjustedclassifyandcount": 14, "distributionmatch": [5, 6, 14], "dy": [5, 14], "em": 14, "max_it": 14, "explicitlossminimis": [], "max": [5, 14], "ms2": [5, 14], "mediansweep": 14, "mediansweep2": 14, "probabilisticadjustedclassifyandcount": 14, "probabilisticclassifyandcount": 14, "smm": [5, 14], "t50": [5, 14], "thresholdoptim": 14, "cross_generate_predict": 14, "cross_generate_predictions_depr": 14, "binaryquantifi": 14, "onevsallgener": [5, 14], "eacc": 14, "ecc": 14, "eemq": 14, "ehdi": 14, "epacc": 14, "valid_polici": 14, "ensemblefactori": 14, "get_probability_distribut": 14, "quanetmodul": 14, "quanettrain": 14, "clean_checkpoint": 14, "clean_checkpoint_dir": 14, "mae_loss": 14, "non_aggreg": 11, "maximumlikelihoodprevalenceestim": 14, "absolute_error": 11, "hat": 11, "frac": 11, "mathcal": 11, "sum_": 11, "acc_error": 11, "y_true": 11, "y_pred": 11, "tp": 11, "tn": 11, "fp": 11, "fn": 11, "stand": [5, 11, 14], "f1_error": 11, "macro": 11, "f_1": 11, "harmon": 11, "recal": 11, "2tp": 11, "independ": [11, 14], "err_nam": 11, "p_hat": 11, "d_": 11, "kl": 11, "log": [11, 13], "factor": 11, "beforehand": 11, "n_sampl": [11, 12], "mean_absolute_error": 11, "mean_relative_absolute_error": 11, "relative_absolute_error": 11, "underlin": 11, "displaystyl": 11, "abstractprotocol": [8, 11], "union": [11, 13, 14], "aggr_speedup": [1, 11], "auto": [1, 11], "evaluation_report": [1, 11], "app": [6, 7, 8, 11, 14], "repeat": [7, 8, 11], "smooth_limits_epsilon": 11, "random_st": [7, 8, 11, 13], "return_typ": [8, 11], "sample_prev": [8, 11], "abstractstochasticseededprotocol": [8, 11], "onlabelledcollectionprotocol": [1, 8, 11], "95": 11, "copi": [11, 13], "quantiti": 11, "labelled_collect": [8, 11], "prevalence_grid": 11, "exhaust": 11, "sum": [11, 14], "implicit": 11, "return_constrained_dim": 11, "rest": [11, 12, 13], "quit": 11, "obvious": 11, "determinist": 11, "anywher": 11, "multipli": 11, "necessari": 11, "samples_paramet": 11, "total": [8, 11], "parent": 11, "sequenc": [8, 11], "enforc": 11, "collat": 11, "arg": [11, 13], "domainmix": 11, "domaina": 11, "domainb": 11, "mixture_point": 11, "domain": 11, "scale": [5, 11, 12, 14], "npp": [8, 11], "draw": 11, "uniformli": [8, 11], "therefor": 11, "get_col": 11, "get_labelled_collect": 11, "on_preclassified_inst": 11, "pre_classif": 11, "in_plac": 11, "usimplexpp": [], "kraemer": [8, 11], "algorithm": [8, 11, 14], "sens": 11, "guarante": [8, 11, 13], "prefer": [1, 8, 11], "intract": 11, "hellingerdist": 11, "hellingh": 11, "distanc": [11, 14], "hd": [5, 11, 14], "discret": [11, 14], "sqrt": 11, "p_i": 11, "q_i": 11, "real": [11, 12, 13, 14], "topsoedist": 11, "1e": [11, 12, 14], "topso": [11, 14], "adjusted_quantif": 11, "prevalence_estim": 11, "tpr": [11, 14], "fpr": [11, 14], "clip": 11, "exce": 11, "check_prevalence_vector": 11, "raise_except": 11, "toleranz": 11, "08": 11, "combinations_budget": 11, "largest": 11, "dimension": [11, 12, 13, 14], "repetit": 11, "less": [11, 13], "normalize_preval": 11, "l1": [11, 14], "calcul": 11, "binom": 11, "mass": 11, "alloc": [11, 12], "solut": 11, "star": 11, "bar": 11, "prevalence_from_label": 11, "n_instanc": [11, 12, 14], "correctli": 11, "even": 11, "len": 11, "prevalence_from_prob": 11, "bool": [11, 12, 14], "argmax": 11, "prevalence_linspac": 11, "01": [6, 11, 12, 14], "separ": [11, 13], "99": 11, "uniform_prevalence_sampl": 11, "adapt": [11, 12], "post": 11, "http": [11, 13, 14], "stackexchang": 11, "com": 11, "question": 11, "3227": 11, "uniform": [9, 11, 13], "uniform_simplex_sampl": 11, "dict": [11, 13, 14], "timeout": 11, "dictionari": [11, 12, 13, 14], "kei": [11, 13], "quantification_error": 11, "whether": [11, 12, 13, 14], "ignor": [11, 13, 14], "gen": 11, "establish": 11, "timer": 11, "longer": [11, 14], "timeouterror": 11, "bound": [11, 14], "stdout": 11, "best_model": 11, "after": [11, 14], "minim": [11, 14], "routin": [11, 13, 14], "unus": [11, 12], "contanin": 11, "cross_val_predict": 11, "akin": [11, 14], "issu": 11, "reproduc": [11, 13], "pos_class": [11, 13], "titl": 11, "colormap": 11, "listedcolormap": 11, "vertical_xtick": 11, "legend": 11, "local": 11, "sign": 11, "minu": 11, "classs": 11, "compon": [11, 12, 14], "cm": 11, "tab10": 11, "secondari": 11, "global": 11, "method_ord": 11, "henc": [11, 13], "conveni": [1, 5, 8, 11], "multiclass": [0, 5, 8, 11, 13, 14], "inconveni": 11, "leyend": 11, "hightlight": 11, "associ": 11, "brokenbar_supremacy_by_drift": 11, "isomer": 11, "x_error": 11, "y_error": 11, "ttest_alpha": 11, "005": 11, "tail_density_threshold": 11, "region": 11, "chart": 11, "condit": [8, 11, 14], "ii": 11, "significantli": 11, "side": 11, "confid": 11, "percentil": 11, "divid": 11, "amount": [1, 8, 11], "similar": [11, 14], "threshold": [11, 14], "densiti": 11, "tail": 11, "discard": 11, "outlier": 11, "show_dens": 11, "show_legend": 11, "logscal": 11, "vline": 11, "especi": 11, "mai": 11, "cumberson": 11, "gain": 11, "understand": 11, "fare": 11, "regim": 11, "highlight": 11, "vertic": 11, "earlystop": 11, "patienc": [11, 12, 14], "lower_is_bett": 11, "earli": [11, 12, 14], "stop": [11, 12, 14], "epoch": [11, 12, 14], "best_epoch": 11, "best_scor": 11, "consecut": [11, 12, 14], "monitor": 11, "obtaind": 11, "far": [11, 12, 13], "flag": 11, "keep": [11, 13], "track": 11, "boolean": [11, 13, 14], "create_if_not_exist": 11, "makedir": 11, "exist_ok": 11, "join": [11, 13], "dir": [11, 14], "subdir": 11, "anotherdir": 11, "create_parent_dir": 11, "exist": [8, 11], "txt": 11, "download_fil": 11, "url": 11, "archive_filenam": 11, "destin": 11, "filenam": 11, "download_file_if_not_exist": 11, "dowload": 11, "get_quapy_hom": 11, "home": [11, 13], "perman": 11, "map_parallel": 11, "func": 11, "slice": 11, "item": 11, "wrapper": [11, 12, 13, 14], "multiprocess": [11, 14], "delai": 11, "args_i": 11, "silent": [11, 14], "child": 11, "ensur": 11, "pickled_resourc": 11, "pickle_path": 11, "generation_func": 11, "fast": [0, 11, 13], "resourc": 11, "some_arrai": 11, "mock": [11, 12], "rand": 11, "my_arrai": 11, "pkl": 11, "save_text_fil": 11, "disk": 11, "miss": 11, "temp_se": 11, "context": 11, "tempor": [11, 12], "outer": 11, "state": 11, "within": [11, 14], "get_njob": [], "correct": [5, 12, 14], "temperatur": [5, 12, 14], "bct": [12, 14], "abstent": 12, "alexandari": [5, 12, 14], "afterward": [12, 14], "No": [12, 14], "nbv": [12, 14], "baseestim": [5, 12, 14], "calibratorfactori": 12, "n_compon": 12, "kwarg": [12, 13, 14], "decomposit": 12, "truncatedsvd": 12, "princip": 12, "regress": 12, "n_featur": 12, "length": [12, 13], "eventu": [12, 13], "unalt": 12, "emb": 12, "embedding_s": 12, "hidden_s": 12, "repr_siz": 12, "kernel_height": 12, "stride": 12, "pad": [12, 13], "drop_p": 12, "convolut": 12, "vocabulari": [12, 13], "kernel": 12, "drop": 12, "dropout": [12, 14], "batch": 12, "dataload": 12, "tensor": 12, "n_dimens": 12, "lstm_class_nlay": 12, "short": 12, "memori": 12, "net": 12, "weight_decai": 12, "batch_siz": 12, "64": [6, 12, 14], "batch_size_test": 12, "512": [12, 14], "padding_length": 12, "checkpointpath": 12, "checkpoint": [12, 14], "classifier_net": 12, "weight": [12, 13], "decai": 12, "wait": 12, "enabl": 12, "gpu": [12, 14], "vocab_s": 12, "reiniti": 12, "trainer": 12, "disjoint": 12, "embed_s": 12, "nn": 12, "pad_length": 12, "xavier": 12, "shuffl": [12, 13], "longest": 12, "shorter": 12, "svmperf_bas": [12, 14], "classifiermixin": 12, "thorsten": 12, "refer": [0, 12, 13], "svm_perf_learn": 12, "svm_perf_classifi": 12, "trade": [12, 14], "off": [12, 14], "margin": [12, 14], "std": 12, "qacc": 12, "qf1": 12, "qgm": 12, "12": 12, "26": 12, "23": [5, 12], "train_siz": 13, "conform": 13, "round": 13, "loader_kwarg": 13, "read": 13, "tupl": [8, 11, 13, 14], "tr": 13, "te": 13, "csr": 13, "csr_matrix": 13, "4403": 13, "my_collect": 13, "codefram": 13, "larger": [11, 13, 14], "actual": [13, 14], "empti": 13, "met": 13, "whose": [13, 14], "train_prop": [6, 8, 13], "left": [11, 13], "stratif": 13, "greater": 13, "dataset_nam": 13, "data_hom": 13, "test_split": 13, "predefin": 13, "uci_dataset": 13, "dump": 13, "leav": 13, "quay_data": 13, "ml": 13, "5fcvx2": 13, "x2": 13, "offici": 13, "lequa": [8, 9, 13], "competit": [0, 9, 13], "t1a": [0, 13], "t1b": [0, 13], "t2a": [0, 13], "t2b": [0, 13], "raw": [0, 11, 13], "merchandis": [0, 13], "sperduti": [0, 13], "2022": [0, 8, 13], "overview": [0, 13], "clef": [0, 13], "lequa2022_experi": [0, 13], "py": [0, 5, 8, 13], "guid": 13, "val_gen": 13, "test_gen": 13, "samplesfromdir": 13, "minimun": 13, "kept": 13, "subsequ": 13, "mining6": 13, "devel": 13, "style": 13, "countvector": 13, "keyword": [13, 14], "nogap": 13, "regardless": 13, "codifi": 13, "unknown": 13, "surfac": 13, "assert": 13, "gap": 13, "preced": 13, "decid": [8, 11, 13], "uniqu": 13, "rare": 13, "unk": 13, "minimum": [13, 14], "occurr": 13, "org": [13, 14], "stabl": 13, "feature_extract": 13, "html": 13, "subtyp": 13, "spmatrix": 13, "remov": [13, 14], "infrequ": 13, "aka": [13, 14], "sublinear_tf": 13, "scall": 13, "counter": 13, "tfidfvector": 13, "whcih": 13, "had": 13, "encod": 13, "utf": 13, "csv": 13, "feat1": 13, "feat2": 13, "featn": 13, "covari": 13, "express": 13, "row": [1, 13], "class2int": 13, "collet": 13, "fomart": 13, "progress": 13, "sentenc": 13, "classnam": 13, "u1": 13, "misclassif": 14, "n_classes_": [], "fit_classifi": 14, "bypass": 14, "y_": 14, "ptecondestim": 14, "prevs_estim": 14, "ax": 14, "entri": [0, 1, 14], "y_i": 14, "y_j": 14, "_posterior_probabilities_": 14, "attribut": 14, "subclass": 14, "give": [8, 14], "outsid": 14, "unless": 14, "noth": 14, "els": 14, "cdf": [5, 14], "match": [5, 14], "helling": 14, "sought": 14, "channel": 14, "proper": 14, "ch": 14, "di": 14, "dij": 14, "fraction": 14, "th": 14, "tol": 14, "ternari": 14, "dl": 14, "doi": 14, "1145": 14, "3219819": 14, "3220059": 14, "histogram": 14, "toler": 14, "explicit": 14, "exact_train_prev": [5, 14], "recalib": [5, 14], "updat": 14, "likelihood": [12, 14], "mutual": 14, "recurs": 14, "until": 14, "converg": 14, "suggest": [5, 14], "recalibr": 14, "reach": 14, "loop": 14, "cumul": 14, "unlabel": 14, "latter": 14, "forman": [5, 8, 14], "2006": [5, 14], "2008": [5, 14], "goal": 14, "bring": 14, "denomin": 14, "median": [5, 14], "sweep": [5, 14], "binary_quantifi": 14, "prevel": 14, "emploi": 14, "resp": 14, "subobject": 14, "nest": 14, "pipelin": 14, "__": 14, "simplif": 14, "2021": [5, 6, 14], "equival": 14, "cosest": 14, "heurist": [1, 14], "choos": [5, 14], "ground": 14, "complement": 14, "param_mod_sel": 14, "param_model_sel": 14, "min_po": 14, "max_sample_s": 14, "closest": 14, "preliminari": 14, "recomput": 14, "compat": 14, "l": 14, "base_quantifier_class": 14, "factori": 14, "common": 14, "doc_embedding_s": 14, "stats_siz": 14, "lstm_hidden_s": 14, "lstm_nlayer": 14, "ff_layer": 14, "1024": 14, "bidirect": 14, "qdrop_p": 14, "order_bi": 14, "cell": 14, "connect": 14, "ff": 14, "sort": 14, "doc_embed": 14, "doc_posterior": 14, "recip": 14, "care": 14, "regist": 14, "hook": 14, "n_epoch": 14, "tr_iter_per_poch": 14, "va_iter_per_poch": 14, "checkpointdir": 14, "checkpointnam": 14, "phase": 14, "anyth": 14, "truth": 14, "mlpe": 14, "lazi": 14, "put": 14, "assumpion": 14, "beat": [12, 14], "estimant": 14, "kundaj": 12, "shrikumar": 12, "novemb": 12, "232": 12, "pmlr": 12, "outpu": [], "partit": 12, "ight": [], "valueerror": 11, "attach": 13, "mix": [], "onevsallaggreg": [5, 14], "parallel_backend": 14, "loki": 14, "backend": 14, "cannot": 14, "temp": 14, "getonevsal": 5, "realiz": 11, "prepar": 11, "act": 11, "modif": 11, "place": [11, 13], "host_fold": 12, "tmp": 12, "delet": 12, "newelm": 14, "underli": [5, 6, 14], "newsvma": [5, 14], "newsvmkld": [5, 14], "newsvmq": [5, 14], "newsvmra": [5, 14], "newonevsal": 14, "onlabelledcollect": [], "forc": [1, 11], "deactiv": [1, 11], "evaluate_on_sampl": 11, "central": 11, "endow": 11, "never": [8, 11], "behaviour": [1, 5, 8, 11], "undertaken": 11, "artificialprevalenceprotocol": 11, "iterateprotocol": 11, "previous": 11, "naturalprevalenceprotocol": 11, "upp": [9, 11], "uniformprevalenceprotocol": 11, "n_train": 13, "n_test": 13, "quick": 13, "omit": 1, "procotol": 1, "vari": [1, 8], "u": 1, "prot": 1, "our": [1, 8], "evaluatio": 1, "4f": [1, 8], "often": 1, "account": 1, "rise": [1, 8], "straightforward": 1, "308": 1, "692": 1, "314": 1, "686": 1, "005649": 1, "013182": 1, "000074": 1, "896": 1, "013145": 1, "069323": 1, "000985": 1, "848": 1, "152": 1, "809": 1, "191": 1, "039063": 1, "149806": 1, "005175": 1, "016": 1, "984": 1, "033": 1, "967": 1, "017236": 1, "487529": 1, "005298": 1, "728": 1, "751": 1, "249": 1, "022769": 1, "057146": 1, "001350": 1, "4995": 1, "72": 1, "698": 1, "302": 1, "021752": 1, "053631": 1, "001133": 1, "4996": 1, "868": 1, "132": 1, "888": 1, "112": 1, "020490": 1, "088230": 1, "001985": 1, "4997": 1, "292": 1, "708": 1, "298": 1, "702": 1, "006149": 1, "014788": 1, "000090": 1, "4998": 1, "76": 1, "220": 1, "780": 1, "019950": 1, "054309": 1, "001127": 1, "4999": 1, "948": 1, "052": 1, "965": 1, "016941": 1, "165776": 1, "003538": 1, "023588": 1, "108779": 1, "003631": 1, "exit": 1, "smaller": 1, "1m": 1, "convers": 1, "precomput": 1, "execut": 1, "lot": 1, "welcom": 3, "behind": 3, "simplifi": 5, "remain": 5, "unchang": 5, "v0": [5, 8, 9], "construct": 5, "depart": 5, "approach": [5, 8], "firat": 5, "mutliclasshdi": 5, "maletzk": 5, "hassan": 5, "thank": 5, "pablo": 5, "contribut": 5, "newsvmnkld": 5, "experiment": 5, "plo": 5, "ONE": 5, "There": 5, "explicit_loss_minim": 5, "one_vs_al": 5, "robustli": 8, "presenc": 8, "confront": [6, 8], "stochast": 8, "fair": 8, "radom_st": 8, "technic": 8, "explan": 8, "custom_protocol": 8, "subject": 8, "2005": 8, "usag": 8, "classifier__c": [6, 8], "equial": 8, "val_app": 8, "increas": 8, "rapidli": 8, "becom": 8, "impract": 8, "legitim": 8, "drawback": 8, "elect": 8, "yet": 8, "burden": 8, "incur": 8, "idea": 8, "deprec": 8, "due": 8, "capabl": 8, "md": 2, "_new": 9, "adher": 11, "ecir": 6, "retriev": 6, "91": 6, "devot": 6, "mark": 6, "prefix": 6, "classifier__": 6, "16": 6, "mae_scor": 6, "04021": 6, "took": 6, "1356": 6, "04286": 6, "2139": 6, "04888": 6, "2491": 6, "05163": 6, "5372": 6, "02445": 6, "9056": 6, "02234": 6, "3114": 6, "03102": 6, "conceptu": 6, "flaw": 6, "hand": 6, "surrog": 6, "train_sampl": 7, "tackl": 0, "val_gener": 0, "test_gener": 0, "doc": 0, "250": 0, "20000": 0}, "objects": {"": [[11, 0, 0, "-", "quapy"]], "quapy": [[12, 0, 0, "-", "classification"], [13, 0, 0, "-", "data"], [11, 0, 0, "-", "error"], [11, 0, 0, "-", "evaluation"], [11, 0, 0, "-", "functional"], [14, 0, 0, "-", "method"], [11, 0, 0, "-", "model_selection"], [11, 0, 0, "-", "plot"], [11, 0, 0, "-", "protocol"], [11, 0, 0, "-", "util"]], "quapy.classification": [[12, 0, 0, "-", "calibration"], [12, 0, 0, "-", "methods"], [12, 0, 0, "-", "neural"], [12, 0, 0, "-", "svmperf"]], "quapy.classification.calibration": [[12, 1, 1, "", "BCTSCalibration"], [12, 1, 1, "", "NBVSCalibration"], [12, 1, 1, "", "RecalibratedProbabilisticClassifier"], [12, 1, 1, "", "RecalibratedProbabilisticClassifierBase"], [12, 1, 1, "", "TSCalibration"], [12, 1, 1, "", "VSCalibration"]], "quapy.classification.calibration.RecalibratedProbabilisticClassifierBase": [[12, 2, 1, "", "classes_"], [12, 3, 1, "", "fit"], [12, 3, 1, "", "fit_cv"], [12, 3, 1, "", "fit_tr_val"], [12, 3, 1, "", "predict"], [12, 3, 1, "", "predict_proba"]], "quapy.classification.methods": [[12, 1, 1, "", "LowRankLogisticRegression"]], "quapy.classification.methods.LowRankLogisticRegression": [[12, 3, 1, "", "fit"], [12, 3, 1, "", "get_params"], [12, 3, 1, "", "predict"], [12, 3, 1, "", "predict_proba"], [12, 3, 1, "", "set_params"], [12, 3, 1, "", "transform"]], "quapy.classification.neural": [[12, 1, 1, "", "CNNnet"], [12, 1, 1, "", "LSTMnet"], [12, 1, 1, "", "NeuralClassifierTrainer"], [12, 1, 1, "", "TextClassifierNet"], [12, 1, 1, "", "TorchDataset"]], "quapy.classification.neural.CNNnet": [[12, 3, 1, "", "document_embedding"], [12, 3, 1, "", "get_params"], [12, 4, 1, "", "training"], [12, 2, 1, "", "vocabulary_size"]], "quapy.classification.neural.LSTMnet": [[12, 3, 1, "", "document_embedding"], [12, 3, 1, "", "get_params"], [12, 4, 1, "", "training"], [12, 2, 1, "", "vocabulary_size"]], "quapy.classification.neural.NeuralClassifierTrainer": [[12, 2, 1, "", "device"], [12, 3, 1, "", "fit"], [12, 3, 1, "", "get_params"], [12, 3, 1, "", "predict"], [12, 3, 1, "", "predict_proba"], [12, 3, 1, "", "reset_net_params"], [12, 3, 1, "", "set_params"], [12, 3, 1, "", "transform"]], "quapy.classification.neural.TextClassifierNet": [[12, 3, 1, "", "dimensions"], [12, 3, 1, "", "document_embedding"], [12, 3, 1, "", "forward"], [12, 3, 1, "", "get_params"], [12, 3, 1, "", "predict_proba"], [12, 4, 1, "", "training"], [12, 2, 1, "", "vocabulary_size"], [12, 3, 1, "", "xavier_uniform"]], "quapy.classification.neural.TorchDataset": [[12, 3, 1, "", "asDataloader"]], "quapy.classification.svmperf": [[12, 1, 1, "", "SVMperf"]], "quapy.classification.svmperf.SVMperf": [[12, 3, 1, "", "decision_function"], [12, 3, 1, "", "fit"], [12, 3, 1, "", "predict"], [12, 4, 1, "", "valid_losses"]], "quapy.data": [[13, 0, 0, "-", "base"], [13, 0, 0, "-", "datasets"], [13, 0, 0, "-", "preprocessing"], [13, 0, 0, "-", "reader"]], "quapy.data.base": [[13, 1, 1, "", "Dataset"], [13, 1, 1, "", "LabelledCollection"]], "quapy.data.base.Dataset": [[13, 3, 1, "", "SplitStratified"], [13, 2, 1, "", "binary"], [13, 2, 1, "", "classes_"], [13, 3, 1, "", "kFCV"], [13, 3, 1, "", "load"], [13, 2, 1, "", "n_classes"], [13, 3, 1, "", "reduce"], [13, 3, 1, "", "stats"], [13, 2, 1, "", "train_test"], [13, 2, 1, "", "vocabulary_size"]], "quapy.data.base.LabelledCollection": [[13, 2, 1, "", "X"], [13, 2, 1, "", "Xp"], [13, 2, 1, "", "Xy"], [13, 2, 1, "", "binary"], [13, 3, 1, "", "counts"], [13, 3, 1, "", "join"], [13, 3, 1, "", "kFCV"], [13, 3, 1, "", "load"], [13, 2, 1, "", "n_classes"], [13, 2, 1, "", "p"], [13, 3, 1, "", "prevalence"], [13, 3, 1, "", "sampling"], [13, 3, 1, "", "sampling_from_index"], [13, 3, 1, "", "sampling_index"], [13, 3, 1, "", "split_random"], [13, 3, 1, "", "split_stratified"], [13, 3, 1, "", "stats"], [13, 3, 1, "", "uniform_sampling"], [13, 3, 1, "", "uniform_sampling_index"], [13, 2, 1, "", "y"]], "quapy.data.datasets": [[13, 5, 1, "", "fetch_UCIDataset"], [13, 5, 1, "", "fetch_UCILabelledCollection"], [13, 5, 1, "", "fetch_lequa2022"], [13, 5, 1, "", "fetch_reviews"], [13, 5, 1, "", "fetch_twitter"], [13, 5, 1, "", "warn"]], "quapy.data.preprocessing": [[13, 1, 1, "", "IndexTransformer"], [13, 5, 1, "", "index"], [13, 5, 1, "", "reduce_columns"], [13, 5, 1, "", "standardize"], [13, 5, 1, "", "text2tfidf"]], "quapy.data.preprocessing.IndexTransformer": [[13, 3, 1, "", "add_word"], [13, 3, 1, "", "fit"], [13, 3, 1, "", "fit_transform"], [13, 3, 1, "", "transform"], [13, 3, 1, "", "vocabulary_size"]], "quapy.data.reader": [[13, 5, 1, "", "binarize"], [13, 5, 1, "", "from_csv"], [13, 5, 1, "", "from_sparse"], [13, 5, 1, "", "from_text"], [13, 5, 1, "", "reindex_labels"]], "quapy.error": [[11, 5, 1, "", "absolute_error"], [11, 5, 1, "", "acc_error"], [11, 5, 1, "", "acce"], [11, 5, 1, "", "ae"], [11, 5, 1, "", "f1_error"], [11, 5, 1, "", "f1e"], [11, 5, 1, "", "from_name"], [11, 5, 1, "", "kld"], [11, 5, 1, "", "mae"], [11, 5, 1, "", "mean_absolute_error"], [11, 5, 1, "", "mean_relative_absolute_error"], [11, 5, 1, "", "mkld"], [11, 5, 1, "", "mnkld"], [11, 5, 1, "", "mrae"], [11, 5, 1, "", "mse"], [11, 5, 1, "", "nkld"], [11, 5, 1, "", "rae"], [11, 5, 1, "", "relative_absolute_error"], [11, 5, 1, "", "se"], [11, 5, 1, "", "smooth"]], "quapy.evaluation": [[11, 5, 1, "", "evaluate"], [11, 5, 1, "", "evaluate_on_samples"], [11, 5, 1, "", "evaluation_report"], [11, 5, 1, "", "prediction"]], "quapy.functional": [[11, 5, 1, "", "HellingerDistance"], [11, 5, 1, "", "TopsoeDistance"], [11, 5, 1, "", "adjusted_quantification"], [11, 5, 1, "", "check_prevalence_vector"], [11, 5, 1, "", "get_nprevpoints_approximation"], [11, 5, 1, "", "normalize_prevalence"], [11, 5, 1, "", "num_prevalence_combinations"], [11, 5, 1, "", "prevalence_from_labels"], [11, 5, 1, "", "prevalence_from_probabilities"], [11, 5, 1, "", "prevalence_linspace"], [11, 5, 1, "", "strprev"], [11, 5, 1, "", "uniform_prevalence_sampling"], [11, 5, 1, "", "uniform_simplex_sampling"]], "quapy.method": [[14, 0, 0, "-", "aggregative"], [14, 0, 0, "-", "base"], [14, 0, 0, "-", "meta"], [14, 0, 0, "-", "neural"], [14, 0, 0, "-", "non_aggregative"]], "quapy.method.aggregative": [[14, 1, 1, "", "ACC"], [14, 4, 1, "", "AdjustedClassifyAndCount"], [14, 1, 1, "", "AggregativeProbabilisticQuantifier"], [14, 1, 1, "", "AggregativeQuantifier"], [14, 1, 1, "", "CC"], [14, 4, 1, "", "ClassifyAndCount"], [14, 1, 1, "", "DistributionMatching"], [14, 1, 1, "", "DyS"], [14, 1, 1, "", "EMQ"], [14, 4, 1, "", "ExpectationMaximizationQuantifier"], [14, 1, 1, "", "HDy"], [14, 4, 1, "", "HellingerDistanceY"], [14, 1, 1, "", "MAX"], [14, 1, 1, "", "MS"], [14, 1, 1, "", "MS2"], [14, 4, 1, "", "MedianSweep"], [14, 4, 1, "", "MedianSweep2"], [14, 1, 1, "", "OneVsAllAggregative"], [14, 1, 1, "", "PACC"], [14, 1, 1, "", "PCC"], [14, 4, 1, "", "ProbabilisticAdjustedClassifyAndCount"], [14, 4, 1, "", "ProbabilisticClassifyAndCount"], [14, 4, 1, "", "SLD"], [14, 1, 1, "", "SMM"], [14, 1, 1, "", "T50"], [14, 1, 1, "", "ThresholdOptimization"], [14, 1, 1, "", "X"], [14, 5, 1, "", "cross_generate_predictions"], [14, 5, 1, "", "cross_generate_predictions_depr"], [14, 5, 1, "", "newELM"], [14, 5, 1, "", "newSVMAE"], [14, 5, 1, "", "newSVMKLD"], [14, 5, 1, "", "newSVMQ"], [14, 5, 1, "", "newSVMRAE"]], "quapy.method.aggregative.ACC": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "classify"], [14, 3, 1, "", "fit"], [14, 3, 1, "", "getPteCondEstim"], [14, 3, 1, "", "solve_adjustment"]], "quapy.method.aggregative.AggregativeProbabilisticQuantifier": [[14, 3, 1, "", "classify"]], "quapy.method.aggregative.AggregativeQuantifier": [[14, 3, 1, "", "aggregate"], [14, 2, 1, "", "classes_"], [14, 2, 1, "", "classifier"], [14, 3, 1, "", "classify"], [14, 3, 1, "", "fit"], [14, 3, 1, "", "quantify"]], "quapy.method.aggregative.CC": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "fit"]], "quapy.method.aggregative.DistributionMatching": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "fit"]], "quapy.method.aggregative.DyS": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "fit"]], "quapy.method.aggregative.EMQ": [[14, 3, 1, "", "EM"], [14, 4, 1, "", "EPSILON"], [14, 4, 1, "", "MAX_ITER"], [14, 3, 1, "", "aggregate"], [14, 3, 1, "", "fit"], [14, 3, 1, "", "predict_proba"]], "quapy.method.aggregative.HDy": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "fit"]], "quapy.method.aggregative.OneVsAllAggregative": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "classify"]], "quapy.method.aggregative.PACC": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "classify"], [14, 3, 1, "", "fit"], [14, 3, 1, "", "getPteCondEstim"]], "quapy.method.aggregative.PCC": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "fit"]], "quapy.method.aggregative.SMM": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "fit"]], "quapy.method.aggregative.ThresholdOptimization": [[14, 3, 1, "", "aggregate"], [14, 3, 1, "", "fit"]], "quapy.method.base": [[14, 1, 1, "", "BaseQuantifier"], [14, 1, 1, "", "BinaryQuantifier"], [14, 1, 1, "", "OneVsAll"], [14, 1, 1, "", "OneVsAllGeneric"], [14, 5, 1, "", "newOneVsAll"]], "quapy.method.base.BaseQuantifier": [[14, 3, 1, "", "fit"], [14, 3, 1, "", "quantify"]], "quapy.method.base.OneVsAllGeneric": [[14, 2, 1, "", "classes_"], [14, 3, 1, "", "fit"], [14, 3, 1, "", "quantify"]], "quapy.method.meta": [[14, 5, 1, "", "EACC"], [14, 5, 1, "", "ECC"], [14, 5, 1, "", "EEMQ"], [14, 5, 1, "", "EHDy"], [14, 5, 1, "", "EPACC"], [14, 1, 1, "", "Ensemble"], [14, 5, 1, "", "ensembleFactory"], [14, 5, 1, "", "get_probability_distribution"]], "quapy.method.meta.Ensemble": [[14, 4, 1, "", "VALID_POLICIES"], [14, 2, 1, "", "aggregative"], [14, 3, 1, "", "fit"], [14, 3, 1, "", "get_params"], [14, 2, 1, "", "probabilistic"], [14, 3, 1, "", "quantify"], [14, 3, 1, "", "set_params"]], "quapy.method.neural": [[14, 1, 1, "", "QuaNetModule"], [14, 1, 1, "", "QuaNetTrainer"], [14, 5, 1, "", "mae_loss"]], "quapy.method.neural.QuaNetModule": [[14, 2, 1, "", "device"], [14, 3, 1, "", "forward"], [14, 4, 1, "", "training"]], "quapy.method.neural.QuaNetTrainer": [[14, 2, 1, "", "classes_"], [14, 3, 1, "", "clean_checkpoint"], [14, 3, 1, "", "clean_checkpoint_dir"], [14, 3, 1, "", "fit"], [14, 3, 1, "", "get_params"], [14, 3, 1, "", "quantify"], [14, 3, 1, "", "set_params"]], "quapy.method.non_aggregative": [[14, 1, 1, "", "MaximumLikelihoodPrevalenceEstimation"]], "quapy.method.non_aggregative.MaximumLikelihoodPrevalenceEstimation": [[14, 3, 1, "", "fit"], [14, 3, 1, "", "quantify"]], "quapy.model_selection": [[11, 1, 1, "", "GridSearchQ"], [11, 5, 1, "", "cross_val_predict"]], "quapy.model_selection.GridSearchQ": [[11, 3, 1, "", "best_model"], [11, 3, 1, "", "fit"], [11, 3, 1, "", "get_params"], [11, 3, 1, "", "quantify"], [11, 3, 1, "", "set_params"]], "quapy.plot": [[11, 5, 1, "", "binary_bias_bins"], [11, 5, 1, "", "binary_bias_global"], [11, 5, 1, "", "binary_diagonal"], [11, 5, 1, "", "brokenbar_supremacy_by_drift"], [11, 5, 1, "", "error_by_drift"]], "quapy.protocol": [[11, 1, 1, "", "APP"], [11, 1, 1, "", "AbstractProtocol"], [11, 1, 1, "", "AbstractStochasticSeededProtocol"], [11, 4, 1, "", "ArtificialPrevalenceProtocol"], [11, 1, 1, "", "DomainMixer"], [11, 1, 1, "", "IterateProtocol"], [11, 1, 1, "", "NPP"], [11, 4, 1, "", "NaturalPrevalenceProtocol"], [11, 1, 1, "", "OnLabelledCollectionProtocol"], [11, 1, 1, "", "UPP"], [11, 4, 1, "", "UniformPrevalenceProtocol"]], "quapy.protocol.APP": [[11, 3, 1, "", "prevalence_grid"], [11, 3, 1, "", "sample"], [11, 3, 1, "", "samples_parameters"], [11, 3, 1, "", "total"]], "quapy.protocol.AbstractProtocol": [[11, 3, 1, "", "total"]], "quapy.protocol.AbstractStochasticSeededProtocol": [[11, 3, 1, "", "collator"], [11, 2, 1, "", "random_state"], [11, 3, 1, "", "sample"], [11, 3, 1, "", "samples_parameters"]], "quapy.protocol.DomainMixer": [[11, 3, 1, "", "sample"], [11, 3, 1, "", "samples_parameters"], [11, 3, 1, "", "total"]], "quapy.protocol.IterateProtocol": [[11, 3, 1, "", "total"]], "quapy.protocol.NPP": [[11, 3, 1, "", "sample"], [11, 3, 1, "", "samples_parameters"], [11, 3, 1, "", "total"]], "quapy.protocol.OnLabelledCollectionProtocol": [[11, 4, 1, "", "RETURN_TYPES"], [11, 3, 1, "", "get_collator"], [11, 3, 1, "", "get_labelled_collection"], [11, 3, 1, "", "on_preclassified_instances"]], "quapy.protocol.UPP": [[11, 3, 1, "", "sample"], [11, 3, 1, "", "samples_parameters"], [11, 3, 1, "", "total"]], "quapy.util": [[11, 1, 1, "", "EarlyStop"], [11, 5, 1, "", "create_if_not_exist"], [11, 5, 1, "", "create_parent_dir"], [11, 5, 1, "", "download_file"], [11, 5, 1, "", "download_file_if_not_exists"], [11, 5, 1, "", "get_quapy_home"], [11, 5, 1, "", "map_parallel"], [11, 5, 1, "", "parallel"], [11, 5, 1, "", "pickled_resource"], [11, 5, 1, "", "save_text_file"], [11, 5, 1, "", "temp_seed"]]}, "objtypes": {"0": "py:module", "1": "py:class", "2": "py:property", "3": "py:method", "4": "py:attribute", "5": "py:function"}, "objnames": {"0": ["py", "module", "Python module"], "1": ["py", "class", "Python class"], "2": ["py", "property", "Python property"], "3": ["py", "method", "Python method"], "4": ["py", "attribute", "Python attribute"], "5": ["py", "function", "Python function"]}, "titleterms": {"dataset": [0, 13], "review": 0, "twitter": 0, "sentiment": 0, "uci": 0, "machin": 0, "learn": 0, "issu": 0, "ad": 0, "custom": 0, "data": [0, 13], "process": 0, "evalu": [1, 11], "error": [1, 7, 11], "measur": 1, "protocol": [1, 8, 11], "instal": 4, "requir": 4, "svm": 4, "perf": 4, "quantif": [4, 5, 6, 7], "orient": [4, 6], "loss": [2, 4, 5, 6], "method": [5, 12, 14], "aggreg": [5, 14], "The": 5, "classifi": 5, "count": 5, "variant": 5, "expect": 5, "maxim": 5, "emq": 5, "helling": 5, "distanc": 5, "y": 5, "hdy": 5, "explicit": [2, 5], "minim": [2, 5], "meta": [5, 14], "model": [5, 6], "ensembl": 5, "quanet": 5, "neural": [5, 12, 14], "network": 5, "select": 6, "target": 6, "classif": [6, 12], "plot": [7, 11], "diagon": 7, "bia": 7, "drift": 7, "welcom": 9, "quapi": [9, 10, 11, 12, 13, 14], "": 9, "document": 9, "introduct": 9, "A": 9, "quick": 9, "exampl": 9, "featur": 9, "content": [9, 11, 12, 13, 14], "indic": 9, "tabl": 9, "packag": [11, 12, 13, 14], "subpackag": 11, "submodul": [11, 12, 13, 14], "function": 11, "model_select": 11, "util": 11, "modul": [11, 12, 13, 14], "calibr": 12, "svmperf": 12, "base": [13, 14], "preprocess": 13, "reader": 13, "non_aggreg": 14, "threshold": 5, "optim": 5, "artifici": 8, "preval": 8, "sampl": 8, "from": 8, "unit": 8, "simplex": 8, "uniform": 8, "upp": 8, "natur": 8, "other": 8, "lequa": 0}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx": 57}, "alltitles": {"Installation": [[4, "installation"]], "Requirements": [[4, "requirements"]], "SVM-perf with quantification-oriented losses": [[4, "svm-perf-with-quantification-oriented-losses"]], "quapy": [[10, "quapy"]], "Welcome to QuaPy\u2019s documentation!": [[9, "welcome-to-quapy-s-documentation"]], "Introduction": [[9, "introduction"]], "A quick example:": [[9, "a-quick-example"]], "Features": [[9, "features"]], "Contents:": [[9, null]], "Indices and tables": [[9, "indices-and-tables"]], "quapy package": [[11, "quapy-package"]], "Submodules": [[11, "submodules"], [12, "submodules"], [13, "submodules"], [14, "submodules"]], "quapy.error": [[11, "module-quapy.error"]], "quapy.evaluation": [[11, "module-quapy.evaluation"]], "quapy.protocol": [[11, "quapy-protocol"]], "quapy.functional": [[11, "module-quapy.functional"]], "quapy.model_selection": [[11, "module-quapy.model_selection"]], "quapy.plot": [[11, "module-quapy.plot"]], "quapy.util": [[11, "module-quapy.util"]], "Subpackages": [[11, "subpackages"]], "Module contents": [[11, "module-quapy"], [12, "module-quapy.classification"], [13, "module-quapy.data"], [14, "module-quapy.method"]], "quapy.classification package": [[12, "quapy-classification-package"]], "quapy.classification.calibration": [[12, "quapy-classification-calibration"]], "quapy.classification.methods": [[12, "module-quapy.classification.methods"]], "quapy.classification.neural": [[12, "module-quapy.classification.neural"]], "quapy.classification.svmperf": [[12, "module-quapy.classification.svmperf"]], "quapy.data package": [[13, "quapy-data-package"]], "quapy.data.base": [[13, "module-quapy.data.base"]], "quapy.data.datasets": [[13, "module-quapy.data.datasets"]], "quapy.data.preprocessing": [[13, "module-quapy.data.preprocessing"]], "quapy.data.reader": [[13, "module-quapy.data.reader"]], "quapy.method package": [[14, "quapy-method-package"]], "quapy.method.aggregative": [[14, "module-quapy.method.aggregative"]], "quapy.method.base": [[14, "module-quapy.method.base"]], "quapy.method.meta": [[14, "module-quapy.method.meta"]], "quapy.method.neural": [[14, "module-quapy.method.neural"]], "quapy.method.non_aggregative": [[14, "module-quapy.method.non_aggregative"]], "Datasets": [[0, "datasets"]], "Reviews Datasets": [[0, "reviews-datasets"]], "Twitter Sentiment Datasets": [[0, "twitter-sentiment-datasets"]], "UCI Machine Learning": [[0, "uci-machine-learning"]], "Issues:": [[0, "issues"]], "LeQua Datasets": [[0, "lequa-datasets"]], "Adding Custom Datasets": [[0, "adding-custom-datasets"]], "Data Processing": [[0, "data-processing"]], "Evaluation": [[1, "evaluation"]], "Error Measures": [[1, "error-measures"]], "Evaluation Protocols": [[1, "evaluation-protocols"]], "Explicit Loss Minimization": [[2, "explicit-loss-minimization"], [5, "explicit-loss-minimization"]], "Quantification Methods": [[5, "quantification-methods"]], "Aggregative Methods": [[5, "aggregative-methods"]], "The Classify & Count variants": [[5, "the-classify-count-variants"]], "Expectation Maximization (EMQ)": [[5, "expectation-maximization-emq"]], "Hellinger Distance y (HDy)": [[5, "hellinger-distance-y-hdy"]], "Threshold Optimization methods": [[5, "threshold-optimization-methods"]], "Meta Models": [[5, "meta-models"]], "Ensembles": [[5, "ensembles"]], "The QuaNet neural network": [[5, "the-quanet-neural-network"]], "Model Selection": [[6, "model-selection"]], "Targeting a Quantification-oriented loss": [[6, "targeting-a-quantification-oriented-loss"]], "Targeting a Classification-oriented loss": [[6, "targeting-a-classification-oriented-loss"]], "Plotting": [[7, "plotting"]], "Diagonal Plot": [[7, "diagonal-plot"]], "Quantification bias": [[7, "quantification-bias"]], "Error by Drift": [[7, "error-by-drift"]], "Protocols": [[8, "protocols"]], "Artificial-Prevalence Protocol": [[8, "artificial-prevalence-protocol"]], "Sampling from the unit-simplex, the Uniform-Prevalence Protocol (UPP)": [[8, "sampling-from-the-unit-simplex-the-uniform-prevalence-protocol-upp"]], "Natural-Prevalence Protocol": [[8, "natural-prevalence-protocol"]], "Other protocols": [[8, "other-protocols"]]}, "indexentries": {}}) \ No newline at end of file diff --git a/examples/custom_quantifier.py b/examples/custom_quantifier.py new file mode 100644 index 0000000..31a69cd --- /dev/null +++ b/examples/custom_quantifier.py @@ -0,0 +1,69 @@ +import quapy as qp +from quapy.data import LabelledCollection +from quapy.method.base import BinaryQuantifier +from quapy.model_selection import GridSearchQ +from quapy.method.aggregative import AggregativeProbabilisticQuantifier +from quapy.protocol import APP +import numpy as np +from sklearn.linear_model import LogisticRegression + + +# Define a custom quantifier: for this example, we will consider a new quantification algorithm that uses a +# logistic regressor for generating posterior probabilities, and then applies a custom threshold value to the +# posteriors. Since the quantifier internally uses a classifier, it is an aggregative quantifier; and since it +# relies on posterior probabilities, it is a probabilistic-aggregative quantifier. Note also it has an +# internal hyperparameter (let say, alpha) which is the decision threshold. Let's also assume the quantifier +# is binary, for simplicity. + +class MyQuantifier(AggregativeProbabilisticQuantifier, BinaryQuantifier): + def __init__(self, classifier, alpha=0.5): + self.alpha = alpha + # aggregative quantifiers have an internal self.classifier attribute + self.classifier = classifier + + def fit(self, data: LabelledCollection, fit_classifier=True): + assert fit_classifier, 'this quantifier needs to fit the classifier!' + self.classifier.fit(*data.Xy) + return self + + # in general, we would need to implement the method quantify(self, instances) but, since this method is of + # type aggregative, we can simply implement the method aggregate, which has the following interface + def aggregate(self, classif_predictions: np.ndarray): + # the posterior probabilities have already been generated by the quantify method; we only need to + # specify what to do with them + positive_probabilities = classif_predictions[:, 1] + crisp_decisions = positive_probabilities > self.alpha + pos_prev = crisp_decisions.mean() + neg_prev = 1-pos_prev + return np.asarray([neg_prev, pos_prev]) + + +if __name__ == '__main__': + + qp.environ['SAMPLE_SIZE'] = 100 + + # define an instance of our custom quantifier + quantifier = MyQuantifier(LogisticRegression(), alpha=0.5) + + # load the IMDb dataset + train, test = qp.datasets.fetch_reviews('imdb', tfidf=True, min_df=5).train_test + + # model selection + # let us assume we want to explore our hyperparameter alpha along with one hyperparameter of the classifier + train, val = train.split_stratified(train_prop=0.75) + param_grid = { + 'alpha': np.linspace(0, 1, 11), # quantifier-dependent hyperparameter + 'classifier__C': np.logspace(-2, 2, 5) # classifier-dependent hyperparameter + } + quantifier = GridSearchQ(quantifier, param_grid, protocol=APP(val), n_jobs=-1, verbose=True).fit(train) + + # evaluation + mae = qp.evaluation.evaluate(quantifier, protocol=APP(test), error_metric='mae') + + print(f'MAE = {mae:.4f}') + + # final remarks: this method is only for demonstration purposes and makes little sense in general. The method relies + # on an hyperparameter alpha for binarizing the posterior probabilities. A much better way for fulfilling this + # goal would be to calibrate the classifier (LogisticRegression is already reasonably well calibrated) and then + # simply cut at 0.5. + diff --git a/examples/explicit_loss_minimization.py b/examples/explicit_loss_minimization.py new file mode 100644 index 0000000..fcc07f3 --- /dev/null +++ b/examples/explicit_loss_minimization.py @@ -0,0 +1,72 @@ +import quapy as qp +from quapy.method.aggregative import newELM +from quapy.method.base import newOneVsAll +from quapy.model_selection import GridSearchQ +from quapy.protocol import UPP + +""" +In this example, we will show hoy to define a quantifier based on explicit loss minimization (ELM). +ELM is a family of quantification methods relying on structured output learning. In particular, we will +showcase how to instantiate SVM(Q) as proposed by `Barranquero et al. 2015 +`_, and SVM(KLD) and SVM(nKLD) as proposed by +`Esuli et al. 2015 `_. + +All ELM quantifiers rely on SVMperf for optimizing a structured loss function (Q, KLD, or nKLD). Since these are +not part of the original SVMperf package by Joachims, you have to first download the SVMperf package, apply the +patch svm-perf-quantification-ext.patch (provided with QuaPy library), and compile the sources. +The script prepare_svmperf.sh does all the job. Simply run: + +>>> ./prepare_svmperf.sh + +Note that ELM quantifiers are nothing but a classify and count (CC) model instantiated with SVMperf as the +underlying classifier. E.g., SVM(Q) comes down to: + +>>> CC(SVMperf(svmperf_base, loss='q')) + +this means that ELM are aggregative quantifiers (since CC is an aggregative quantifier). QuaPy provides some helper +functions for simplify this; for example: + +>>> newSVMQ(svmperf_base) + +returns an instance of SVM(Q) (i.e., an instance of CC properly set to work with SVMperf optimizing for Q. + +Since we wan to explore the losses, we will instead use newELM. For this example we will create a quantifier for tweet +sentiment analysis considering three classes: negative, neutral, and positive. Since SVMperf is a binary classifier, +our quantifier will be binary as well. We will use a one-vs-all approach to work in multiclass model. +For more details about how one-vs-all works, we refer to the example "one_vs_all.py" and to the API documentation. +""" + +qp.environ['SAMPLE_SIZE'] = 100 +qp.environ['N_JOBS'] = -1 +qp.environ['SVMPERF_HOME'] = '../svm_perf_quantification' + +quantifier = newOneVsAll(newELM()) +print(f'the quantifier is an instance of {quantifier.__class__.__name__}') + +# load a ternary dataset +train_modsel, val = qp.datasets.fetch_twitter('hcr', for_model_selection=True, pickle=True).train_test + +""" +model selection: +We explore the classifier's loss and the classifier's C hyperparameters. +Since our model is actually an instance of OneVsAllAggregative, we need to add the prefix "binary_quantifier", and +since our binary quantifier is an instance of CC, we need to add the prefix "classifier". +""" +param_grid = { + 'binary_quantifier__classifier__loss': ['q', 'kld', 'mae'], # classifier-dependent hyperparameter + 'binary_quantifier__classifier__C': [0.01, 1, 100], # classifier-dependent hyperparameter +} +print('starting model selection') +model_selection = GridSearchQ(quantifier, param_grid, protocol=UPP(val), verbose=True, refit=False) +quantifier = model_selection.fit(train_modsel).best_model() + +print('training on the whole training set') +train, test = qp.datasets.fetch_twitter('hcr', for_model_selection=False, pickle=True).train_test +quantifier.fit(train) + +# evaluation +mae = qp.evaluation.evaluate(quantifier, protocol=UPP(test), error_metric='mae') + +print(f'MAE = {mae:.4f}') + + diff --git a/examples/lequa2022_experiments.py b/examples/lequa2022_experiments.py new file mode 100644 index 0000000..f3eec55 --- /dev/null +++ b/examples/lequa2022_experiments.py @@ -0,0 +1,53 @@ +import numpy as np +from sklearn.linear_model import LogisticRegression +import quapy as qp +import quapy.functional as F +from quapy.data.datasets import LEQUA2022_SAMPLE_SIZE, fetch_lequa2022 +from quapy.evaluation import evaluation_report +from quapy.method.aggregative import EMQ +from quapy.model_selection import GridSearchQ +import pandas as pd + +""" +This example shows hoy to use the LeQua datasets (new in v0.1.7). For more information about the datasets, and the +LeQua competition itself, check: +https://lequa2022.github.io/index (the site of the competition) +https://ceur-ws.org/Vol-3180/paper-146.pdf (the overview paper) +""" + +# there are 4 tasks (T1A, T1B, T2A, T2B) +task = 'T1A' + +# set the sample size in the environment. The sample size is task-dendendent and can be consulted by doing: +qp.environ['SAMPLE_SIZE'] = LEQUA2022_SAMPLE_SIZE[task] +qp.environ['N_JOBS'] = -1 + +# the fetch method returns a training set (an instance of LabelledCollection) and two generators: one for the +# validation set and another for the test sets. These generators are both instances of classes that extend +# AbstractProtocol (i.e., classes that implement sampling generation procedures) and, in particular, are instances +# of SamplesFromDir, a protocol that simply iterates over pre-generated samples (those provided for the competition) +# stored in a directory. +training, val_generator, test_generator = fetch_lequa2022(task=task) + +# define the quantifier +quantifier = EMQ(classifier=LogisticRegression()) + +# model selection +param_grid = { + 'classifier__C': np.logspace(-3, 3, 7), # classifier-dependent: inverse of regularization strength + 'classifier__class_weight': ['balanced', None], # classifier-dependent: weights of each class + 'recalib': ['bcts', 'platt', None] # quantifier-dependent: recalibration method (new in v0.1.7) +} +model_selection = GridSearchQ(quantifier, param_grid, protocol=val_generator, error='mrae', refit=False, verbose=True) +quantifier = model_selection.fit(training) + +# evaluation +report = evaluation_report(quantifier, protocol=test_generator, error_metrics=['mae', 'mrae', 'mkld'], verbose=True) + +# printing results +pd.set_option('display.expand_frame_repr', False) +report['estim-prev'] = report['estim-prev'].map(F.strprev) +print(report) + +print('Averaged values:') +print(report.mean()) diff --git a/examples/lequa2022_experiments_recalib.py b/examples/lequa2022_experiments_recalib.py new file mode 100644 index 0000000..a5a0e05 --- /dev/null +++ b/examples/lequa2022_experiments_recalib.py @@ -0,0 +1,63 @@ +import numpy as np +from abstention.calibration import NoBiasVectorScaling, VectorScaling, TempScaling +from sklearn.calibration import CalibratedClassifierCV +from sklearn.linear_model import LogisticRegression +import quapy as qp +import quapy.functional as F +from classification.calibration import RecalibratedProbabilisticClassifierBase, NBVSCalibration, \ + BCTSCalibration +from data.datasets import LEQUA2022_SAMPLE_SIZE, fetch_lequa2022 +from evaluation import evaluation_report +from method.aggregative import EMQ +from model_selection import GridSearchQ +import pandas as pd + +for task in ['T1A', 'T1B']: + + # calibration = TempScaling(verbose=False, bias_positions='all') + + qp.environ['SAMPLE_SIZE'] = LEQUA2022_SAMPLE_SIZE[task] + training, val_generator, test_generator = fetch_lequa2022(task=task) + + # define the quantifier + # learner = BCTSCalibration(LogisticRegression(), n_jobs=-1) + # learner = CalibratedClassifierCV(LogisticRegression()) + learner = LogisticRegression() + quantifier = EMQ(classifier=learner) + + # model selection + param_grid = { + 'classifier__C': np.logspace(-3, 3, 7), + 'classifier__class_weight': ['balanced', None], + 'recalib': ['platt', 'ts', 'vs', 'nbvs', 'bcts', None], + 'exact_train_prev': [False, True] + } + model_selection = GridSearchQ(quantifier, param_grid, protocol=val_generator, error='mrae', n_jobs=-1, refit=False, verbose=True) + quantifier = model_selection.fit(training) + + # evaluation + report = evaluation_report(quantifier, protocol=test_generator, error_metrics=['mae', 'mrae', 'mkld'], verbose=True) + + # import os + # os.makedirs(f'./out', exist_ok=True) + # with open(f'./out/EMQ_{calib}_{task}.txt', 'wt') as foo: + # estim_prev = report['estim-prev'].values + # nclasses = len(estim_prev[0]) + # foo.write(f'id,'+','.join([str(x) for x in range(nclasses)])+'\n') + # for id, prev in enumerate(estim_prev): + # foo.write(f'{id},'+','.join([f'{p:.5f}' for p in prev])+'\n') + # + # #os.makedirs(f'./errors/{task}', exist_ok=True) + # with open(f'./out/EMQ_{calib}_{task}_errors.txt', 'wt') as foo: + # maes, mraes = report['mae'].values, report['mrae'].values + # foo.write(f'id,AE,RAE\n') + # for id, (ae_i, rae_i) in enumerate(zip(maes, mraes)): + # foo.write(f'{id},{ae_i:.5f},{rae_i:.5f}\n') + + # printing results + pd.set_option('display.expand_frame_repr', False) + report['estim-prev'] = report['estim-prev'].map(F.strprev) + print(report) + + print('Averaged values:') + print(report.mean()) diff --git a/examples/model_selection.py b/examples/model_selection.py new file mode 100644 index 0000000..b9b4903 --- /dev/null +++ b/examples/model_selection.py @@ -0,0 +1,57 @@ +import quapy as qp +from quapy.protocol import APP +from quapy.method.aggregative import DistributionMatching +from sklearn.linear_model import LogisticRegression +import numpy as np + +""" +In this example, we show how to perform model selection on a DistributionMatching quantifier. +""" + +model = DistributionMatching(LogisticRegression()) + +qp.environ['SAMPLE_SIZE'] = 100 +qp.environ['N_JOBS'] = -1 + +training, test = qp.datasets.fetch_reviews('imdb', tfidf=True, min_df=5).train_test + +# The model will be returned by the fit method of GridSearchQ. +# Every combination of hyper-parameters will be evaluated by confronting the +# quantifier thus configured against a series of samples generated by means +# of a sample generation protocol. For this example, we will use the +# artificial-prevalence protocol (APP), that generates samples with prevalence +# values in the entire range of values from a grid (e.g., [0, 0.1, 0.2, ..., 1]). +# We devote 30% of the dataset for this exploration. +training, validation = training.split_stratified(train_prop=0.7) +protocol = APP(validation) + +# We will explore a classification-dependent hyper-parameter (e.g., the 'C' +# hyper-parameter of LogisticRegression) and a quantification-dependent hyper-parameter +# (e.g., the number of bins in a DistributionMatching quantifier. +# Classifier-dependent hyper-parameters have to be marked with a prefix "classifier__" +# in order to let the quantifier know this hyper-parameter belongs to its underlying +# classifier. +param_grid = { + 'classifier__C': np.logspace(-3,3,7), + 'nbins': [8, 16, 32, 64], +} + +model = qp.model_selection.GridSearchQ( + model=model, + param_grid=param_grid, + protocol=protocol, + error='mae', # the error to optimize is the MAE (a quantification-oriented loss) + refit=True, # retrain on the whole labelled set once done + verbose=True # show information as the process goes on +).fit(training) + +print(f'model selection ended: best hyper-parameters={model.best_params_}') +model = model.best_model_ + +# evaluation in terms of MAE +# we use the same evaluation protocol (APP) on the test set +mae_score = qp.evaluation.evaluate(model, protocol=APP(test), error_metric='mae') + +print(f'MAE={mae_score:.5f}') + + diff --git a/examples/one_vs_all.py b/examples/one_vs_all.py new file mode 100644 index 0000000..3f5c4ac --- /dev/null +++ b/examples/one_vs_all.py @@ -0,0 +1,54 @@ +import quapy as qp +from quapy.method.aggregative import MS2 +from quapy.method.base import newOneVsAll +from quapy.model_selection import GridSearchQ +from quapy.protocol import UPP +from sklearn.linear_model import LogisticRegression +import numpy as np + +""" +In this example, we will create a quantifier for tweet sentiment analysis considering three classes: negative, neutral, +and positive. We will use a one-vs-all approach using a binary quantifier for demonstration purposes. +""" + +qp.environ['SAMPLE_SIZE'] = 100 +qp.environ['N_JOBS'] = -1 + +""" +Any binary quantifier can be turned into a single-label quantifier by means of getOneVsAll function. +This function returns an instance of OneVsAll quantifier. Actually, it either returns the subclass OneVsAllGeneric +when the quantifier is an instance of BaseQuantifier, and it returns OneVsAllAggregative when the quantifier is +an instance of AggregativeQuantifier. Although OneVsAllGeneric works in all cases, using OneVsAllAggregative has +some additional advantages (namely, all the advantages that AggregativeQuantifiers enjoy, i.e., faster predictions +during evaluation). +""" +quantifier = newOneVsAll(MS2(LogisticRegression())) +print(f'the quantifier is an instance of {quantifier.__class__.__name__}') + +# load a ternary dataset +train_modsel, val = qp.datasets.fetch_twitter('hcr', for_model_selection=True, pickle=True).train_test + +""" +model selection: for this example, we are relying on the UPP protocol, i.e., a variant of the +artificial-prevalence protocol that generates random samples (100 in this case) for randomly picked priors +from the unit simplex. The priors are sampled using the Kraemer algorithm. Note this is in contrast to the +standard APP protocol, that instead explores a prefixed grid of prevalence values. +""" +param_grid = { + 'binary_quantifier__classifier__C': np.logspace(-2,2,5), # classifier-dependent hyperparameter + 'binary_quantifier__classifier__class_weight': ['balanced', None] # classifier-dependent hyperparameter +} +print('starting model selection') +model_selection = GridSearchQ(quantifier, param_grid, protocol=UPP(val), verbose=True, refit=False) +quantifier = model_selection.fit(train_modsel).best_model() + +print('training on the whole training set') +train, test = qp.datasets.fetch_twitter('hcr', for_model_selection=False, pickle=True).train_test +quantifier.fit(train) + +# evaluation +mae = qp.evaluation.evaluate(quantifier, protocol=UPP(test), error_metric='mae') + +print(f'MAE = {mae:.4f}') + + diff --git a/examples/quanet_example.py b/examples/quanet_example.py new file mode 100644 index 0000000..4be3132 --- /dev/null +++ b/examples/quanet_example.py @@ -0,0 +1,35 @@ +import quapy as qp +from quapy.classification.neural import CNNnet +from quapy.classification.neural import NeuralClassifierTrainer +from quapy.method.meta import QuaNet +import quapy.functional as F + +""" +This example shows how to train QuaNet. The internal classifier is a word-based CNN. +""" + +# set the sample size in the environment +qp.environ["SAMPLE_SIZE"] = 100 + +# the dataset is textual (Kindle reviews from Amazon), so we need to index terms, i.e., +# we need to convert distinct terms into numerical ids +dataset = qp.datasets.fetch_reviews('kindle', pickle=True) +qp.data.preprocessing.index(dataset, min_df=5, inplace=True) +train, test = dataset.train_test + +# train the text classifier: +cnn_module = CNNnet(dataset.vocabulary_size, dataset.training.n_classes) +cnn_classifier = NeuralClassifierTrainer(cnn_module, device='cuda') +cnn_classifier.fit(*dataset.training.Xy) + +# train QuaNet (alternatively, we can set fit_classifier=True and let QuaNet train the classifier) +quantifier = QuaNet(cnn_classifier, device='cuda') +quantifier.fit(train, fit_classifier=False) + +# prediction and evaluation +estim_prevalence = quantifier.quantify(test.instances) +mae = qp.error.mae(test.prevalence(), estim_prevalence) + +print(f'true prevalence: {F.strprev(test.prevalence())}') +print(f'estim prevalence: {F.strprev(estim_prevalence)}') +print(f'MAE = {mae:.4f}') \ No newline at end of file diff --git a/quapy/CHANGE_LOG.txt b/quapy/CHANGE_LOG.txt new file mode 100644 index 0000000..1e0908a --- /dev/null +++ b/quapy/CHANGE_LOG.txt @@ -0,0 +1,79 @@ +Change Log 0.1.7 +---------------- + +- Protocols are now abstracted as instances of AbstractProtocol. There is a new class extending AbstractProtocol called + AbstractStochasticSeededProtocol, which implements a seeding policy to allow replicate the series of samplings. + There are some examples of protocols, APP, NPP, UPP, DomainMixer (experimental). + The idea is to start the sample generation by simply calling the __call__ method. + This change has a great impact in the framework, since many functions in qp.evaluation, qp.model_selection, + and sampling functions in LabelledCollection relied of the old functions. E.g., the functionality of + qp.evaluation.artificial_prevalence_report or qp.evaluation.natural_prevalence_report is now obtained by means of + qp.evaluation.report which takes a protocol as an argument. I have not maintained compatibility with the old + interfaces because I did not really like them. Check the wiki guide and the examples for more details. + +- Exploration of hyperparameters in Model selection can now be run in parallel (there was a n_jobs argument in + QuaPy 0.1.6 but only the evaluation part for one specific hyperparameter was run in parallel). + +- The prediction function has been refactored, so it applies the optimization for aggregative quantifiers (that + consists in pre-classifying all instances, and then only invoking aggregate on the samples) only in cases in + which the total number of classifications would be smaller than the number of classifications with the standard + procedure. The user can now specify "force", "auto", True of False, in order to actively decide for applying it + or not. + +- examples directory created! + +- DyS, Topsoe distance and binary search (thanks to Pablo González) + +- Multi-thread reproducibility via seeding (thanks to Pablo González) + +- n_jobs is now taken from the environment if set to None + +- ACC, PACC, Forman's threshold variants have been parallelized. + +- cross_val_predict (for quantification) added to model_selection: would be nice to allow the user specifies a + test protocol maybe, or None for bypassing it? + +- Bugfix: adding two labelled collections (with +) now checks for consistency in the classes + +- newer versions of numpy raise a warning when accessing types (e.g., np.float). I have replaced all such instances + with the plain python type (e.g., float). + +- new dependency "abstention" (to add to the project requirements and setup). Calibration methods from + https://github.com/kundajelab/abstention added. + +- the internal classifier of aggregative methods is now called "classifier" instead of "learner" + +- when optimizing the hyperparameters of an aggregative quantifier, the classifier's specific hyperparameters + should be marked with a "classifier__" prefix (just like in scikit-learn with estimators), while the quantifier's + specific hyperparameters are named directly. For example, PCC(LogisticRegression()) quantifier has hyperparameters + "classifier__C", "classifier__class_weight", etc., instead of "C" and "class_weight" as in v0.1.6. + +- hyperparameters yielding to inconsistent runs raise a ValueError exception, while hyperparameter combinations + yielding to internal errors of surrogate functions are reported and skipped, without stopping the grid search. + +- DistributionMatching methods added. This is a general framework for distribution matching methods that catters for + multiclass quantification. That is to say, one could get a multiclass variant of the (originally binary) HDy + method aligned with the Firat's formulation. + +- internal method properties "binary", "aggregative", and "probabilistic" have been removed; these conditions are + checked via isinstance + +- quantifiers (i.e., classes that inherit from BaseQuantifier) are not forced to implement classes_ or n_classes; + these can be used anyway internally, but the framework will not suppose (nor impose) that a quantifier implements + them + +- qp.evaluation.prediction has been optimized so that, if a quantifier is of type aggregative, and if the evaluation + protocol is of type OnLabelledCollection, then the computation is faster. In this specific case, the predictions + are issued only once and for all, and not for each sample. An exception to this (which is implement also), is + when the number of instances across all samples is anyway smaller than the number of instances in the original + labelled collection; in this case the heuristic is of no help, and is therefore not applied. + +- the distinction between "classify" and "posterior_probabilities" has been removed in Aggregative quantifiers, + so that probabilistic classifiers return posterior probabilities, while non-probabilistic quantifiers + return crisp decisions. + +- OneVsAll fixed. There are now two classes: a generic one OneVsAllGeneric that works with any quantifier (e.g., + any instance of BaseQuantifier), and a subclass of it called OneVsAllAggregative which implements the + classify / aggregate interface. Both are instances of OneVsAll. There is a method getOneVsAll that returns the + best instance based on the type of quantifier. + diff --git a/quapy/__init__.py b/quapy/__init__.py index a1ccee4..47a7388 100644 --- a/quapy/__init__.py +++ b/quapy/__init__.py @@ -2,15 +2,15 @@ from . import error from . import data from quapy.data import datasets from . import functional -from . import method +# from . import method from . import evaluation +from . import protocol from . import plot from . import util from . import model_selection from . import classification -from quapy.method.base import isprobabilistic, isaggregative -__version__ = '0.1.6' +__version__ = '0.1.7' environ = { 'SAMPLE_SIZE': None, @@ -18,8 +18,33 @@ environ = { 'UNK_INDEX': 0, 'PAD_TOKEN': '[PAD]', 'PAD_INDEX': 1, - 'SVMPERF_HOME': './svm_perf_quantification' + 'SVMPERF_HOME': './svm_perf_quantification', + 'N_JOBS': 1 } -def isbinary(x): - return x.binary \ No newline at end of file + +def _get_njobs(n_jobs): + """ + If `n_jobs` is None, then it returns `environ['N_JOBS']`; if otherwise, returns `n_jobs`. + + :param n_jobs: the number of `n_jobs` or None if not specified + :return: int + """ + return environ['N_JOBS'] if n_jobs is None else n_jobs + + +def _get_sample_size(sample_size): + """ + If `sample_size` is None, then it returns `environ['SAMPLE_SIZE']`; if otherwise, returns `sample_size`. + If none of these are set, then a ValueError exception is raised. + + :param sample_size: integer or None + :return: int + """ + sample_size = environ['SAMPLE_SIZE'] if sample_size is None else sample_size + if sample_size is None: + raise ValueError('neither sample_size nor qp.environ["SAMPLE_SIZE"] have been specified') + return sample_size + + + diff --git a/quapy/classification/calibration.py b/quapy/classification/calibration.py new file mode 100644 index 0000000..a3f1543 --- /dev/null +++ b/quapy/classification/calibration.py @@ -0,0 +1,215 @@ +from copy import deepcopy + +from abstention.calibration import NoBiasVectorScaling, TempScaling, VectorScaling +from sklearn.base import BaseEstimator, clone +from sklearn.model_selection import cross_val_predict, train_test_split +import numpy as np + + +# Wrappers of calibration defined by Alexandari et al. in paper +# requires "pip install abstension" +# see https://github.com/kundajelab/abstention + + +class RecalibratedProbabilisticClassifier: + """ + Abstract class for (re)calibration method from `abstention.calibration`, as defined in + `Alexandari, A., Kundaje, A., & Shrikumar, A. (2020, November). Maximum likelihood with bias-corrected calibration + is hard-to-beat at label shift adaptation. In International Conference on Machine Learning (pp. 222-232). PMLR. + `_: + """ + pass + + +class RecalibratedProbabilisticClassifierBase(BaseEstimator, RecalibratedProbabilisticClassifier): + """ + Applies a (re)calibration method from `abstention.calibration`, as defined in + `Alexandari et al. paper `_: + + :param classifier: a scikit-learn probabilistic classifier + :param calibrator: the calibration object (an instance of abstention.calibration.CalibratorFactory) + :param val_split: indicate an integer k for performing kFCV to obtain the posterior probabilities, or a float p + in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the + training instances (the rest is used for training). In any case, the classifier is retrained in the whole + training set afterwards. Default value is 5. + :param n_jobs: indicate the number of parallel workers (only when val_split is an integer); default=None + :param verbose: whether or not to display information in the standard output + """ + + def __init__(self, classifier, calibrator, val_split=5, n_jobs=None, verbose=False): + self.classifier = classifier + self.calibrator = calibrator + self.val_split = val_split + self.n_jobs = n_jobs + self.verbose = verbose + + def fit(self, X, y): + """ + Fits the calibration for the probabilistic classifier. + + :param X: array-like of shape `(n_samples, n_features)` with the data instances + :param y: array-like of shape `(n_samples,)` with the class labels + :return: self + """ + k = self.val_split + if isinstance(k, int): + if k < 2: + raise ValueError('wrong value for val_split: the number of folds must be > 2') + return self.fit_cv(X, y) + elif isinstance(k, float): + if not (0 < k < 1): + raise ValueError('wrong value for val_split: the proportion of validation documents must be in (0,1)') + return self.fit_cv(X, y) + + def fit_cv(self, X, y): + """ + Fits the calibration in a cross-validation manner, i.e., it generates posterior probabilities for all + training instances via cross-validation, and then retrains the classifier on all training instances. + The posterior probabilities thus generated are used for calibrating the outputs of the classifier. + + :param X: array-like of shape `(n_samples, n_features)` with the data instances + :param y: array-like of shape `(n_samples,)` with the class labels + :return: self + """ + posteriors = cross_val_predict( + self.classifier, X, y, cv=self.val_split, n_jobs=self.n_jobs, verbose=self.verbose, method='predict_proba' + ) + self.classifier.fit(X, y) + nclasses = len(np.unique(y)) + self.calibration_function = self.calibrator(posteriors, np.eye(nclasses)[y], posterior_supplied=True) + return self + + def fit_tr_val(self, X, y): + """ + Fits the calibration in a train/val-split manner, i.e.t, it partitions the training instances into a + training and a validation set, and then uses the training samples to learn classifier which is then used + to generate posterior probabilities for the held-out validation data. These posteriors are used to calibrate + the classifier. The classifier is not retrained on the whole dataset. + + :param X: array-like of shape `(n_samples, n_features)` with the data instances + :param y: array-like of shape `(n_samples,)` with the class labels + :return: self + """ + Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=self.val_split, stratify=y) + self.classifier.fit(Xtr, ytr) + posteriors = self.classifier.predict_proba(Xva) + nclasses = len(np.unique(yva)) + self.calibrator = self.calibrator(posteriors, np.eye(nclasses)[yva], posterior_supplied=True) + return self + + def predict(self, X): + """ + Predicts class labels for the data instances in `X` + + :param X: array-like of shape `(n_samples, n_features)` with the data instances + :return: array-like of shape `(n_samples,)` with the class label predictions + """ + return self.classifier.predict(X) + + def predict_proba(self, X): + """ + Generates posterior probabilities for the data instances in `X` + + :param X: array-like of shape `(n_samples, n_features)` with the data instances + :return: array-like of shape `(n_samples, n_classes)` with posterior probabilities + """ + posteriors = self.classifier.predict_proba(X) + return self.calibration_function(posteriors) + + @property + def classes_(self): + """ + Returns the classes on which the classifier has been trained on + + :return: array-like of shape `(n_classes)` + """ + return self.classifier.classes_ + + +class NBVSCalibration(RecalibratedProbabilisticClassifierBase): + """ + Applies the No-Bias Vector Scaling (NBVS) calibration method from `abstention.calibration`, as defined in + `Alexandari et al. paper `_: + + :param classifier: a scikit-learn probabilistic classifier + :param val_split: indicate an integer k for performing kFCV to obtain the posterior prevalences, or a float p + in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the + training instances (the rest is used for training). In any case, the classifier is retrained in the whole + training set afterwards. Default value is 5. + :param n_jobs: indicate the number of parallel workers (only when val_split is an integer) + :param verbose: whether or not to display information in the standard output + """ + + def __init__(self, classifier, val_split=5, n_jobs=None, verbose=False): + self.classifier = classifier + self.calibrator = NoBiasVectorScaling(verbose=verbose) + self.val_split = val_split + self.n_jobs = n_jobs + self.verbose = verbose + + +class BCTSCalibration(RecalibratedProbabilisticClassifierBase): + """ + Applies the Bias-Corrected Temperature Scaling (BCTS) calibration method from `abstention.calibration`, as defined in + `Alexandari et al. paper `_: + + :param classifier: a scikit-learn probabilistic classifier + :param val_split: indicate an integer k for performing kFCV to obtain the posterior prevalences, or a float p + in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the + training instances (the rest is used for training). In any case, the classifier is retrained in the whole + training set afterwards. Default value is 5. + :param n_jobs: indicate the number of parallel workers (only when val_split is an integer) + :param verbose: whether or not to display information in the standard output + """ + + def __init__(self, classifier, val_split=5, n_jobs=None, verbose=False): + self.classifier = classifier + self.calibrator = TempScaling(verbose=verbose, bias_positions='all') + self.val_split = val_split + self.n_jobs = n_jobs + self.verbose = verbose + + +class TSCalibration(RecalibratedProbabilisticClassifierBase): + """ + Applies the Temperature Scaling (TS) calibration method from `abstention.calibration`, as defined in + `Alexandari et al. paper `_: + + :param classifier: a scikit-learn probabilistic classifier + :param val_split: indicate an integer k for performing kFCV to obtain the posterior prevalences, or a float p + in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the + training instances (the rest is used for training). In any case, the classifier is retrained in the whole + training set afterwards. Default value is 5. + :param n_jobs: indicate the number of parallel workers (only when val_split is an integer) + :param verbose: whether or not to display information in the standard output + """ + + def __init__(self, classifier, val_split=5, n_jobs=None, verbose=False): + self.classifier = classifier + self.calibrator = TempScaling(verbose=verbose) + self.val_split = val_split + self.n_jobs = n_jobs + self.verbose = verbose + + +class VSCalibration(RecalibratedProbabilisticClassifierBase): + """ + Applies the Vector Scaling (VS) calibration method from `abstention.calibration`, as defined in + `Alexandari et al. paper `_: + + :param classifier: a scikit-learn probabilistic classifier + :param val_split: indicate an integer k for performing kFCV to obtain the posterior prevalences, or a float p + in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the + training instances (the rest is used for training). In any case, the classifier is retrained in the whole + training set afterwards. Default value is 5. + :param n_jobs: indicate the number of parallel workers (only when val_split is an integer) + :param verbose: whether or not to display information in the standard output + """ + + def __init__(self, classifier, val_split=5, n_jobs=None, verbose=False): + self.classifier = classifier + self.calibrator = VectorScaling(verbose=verbose) + self.val_split = val_split + self.n_jobs = n_jobs + self.verbose = verbose + diff --git a/quapy/classification/neural.py b/quapy/classification/neural.py index 0d576c5..8d78d7c 100644 --- a/quapy/classification/neural.py +++ b/quapy/classification/neural.py @@ -42,7 +42,7 @@ class NeuralClassifierTrainer: batch_size=64, batch_size_test=512, padding_length=300, - device='cpu', + device='cuda', checkpointpath='../checkpoint/classifier_net.dat'): super().__init__() @@ -62,7 +62,6 @@ class NeuralClassifierTrainer: } self.learner_hyperparams = self.net.get_params() self.checkpointpath = checkpointpath - self.classes_ = np.asarray([0, 1]) print(f'[NeuralNetwork running on {device}]') os.makedirs(Path(checkpointpath).parent, exist_ok=True) @@ -174,6 +173,7 @@ class NeuralClassifierTrainer: :return: """ train, val = LabelledCollection(instances, labels).split_stratified(1-val_split) + self.classes_ = train.classes_ opt = self.trainer_hyperparams checkpoint = self.checkpointpath self.reset_net_params(self.vocab_size, train.n_classes) @@ -229,11 +229,11 @@ class NeuralClassifierTrainer: self.net.eval() opt = self.trainer_hyperparams with torch.no_grad(): - positive_probs = [] + posteriors = [] for xi in TorchDataset(instances).asDataloader( opt['batch_size_test'], shuffle=False, pad_length=opt['padding_length'], device=opt['device']): - positive_probs.append(self.net.predict_proba(xi)) - return np.concatenate(positive_probs) + posteriors.append(self.net.predict_proba(xi)) + return np.concatenate(posteriors) def transform(self, instances): """ diff --git a/quapy/classification/svmperf.py b/quapy/classification/svmperf.py index 2f6ad90..6c85084 100644 --- a/quapy/classification/svmperf.py +++ b/quapy/classification/svmperf.py @@ -1,5 +1,7 @@ import random +import shutil import subprocess +import tempfile from os import remove, makedirs from os.path import join, exists from subprocess import PIPE, STDOUT @@ -23,26 +25,34 @@ class SVMperf(BaseEstimator, ClassifierMixin): :param C: trade-off between training error and margin (default 0.01) :param verbose: set to True to print svm-perf std outputs :param loss: the loss to optimize for. Available losses are "01", "f1", "kld", "nkld", "q", "qacc", "qf1", "qgm", "mae", "mrae". + :param host_folder: directory where to store the trained model; set to None (default) for using a tmp directory + (temporal directories are automatically deleted) """ # losses with their respective codes in svm_perf implementation valid_losses = {'01':0, 'f1':1, 'kld':12, 'nkld':13, 'q':22, 'qacc':23, 'qf1':24, 'qgm':25, 'mae':26, 'mrae':27} - def __init__(self, svmperf_base, C=0.01, verbose=False, loss='01'): + def __init__(self, svmperf_base, C=0.01, verbose=False, loss='01', host_folder=None): assert exists(svmperf_base), f'path {svmperf_base} does not seem to point to a valid path' self.svmperf_base = svmperf_base self.C = C self.verbose = verbose self.loss = loss + self.host_folder = host_folder - def set_params(self, **parameters): - """ - Set the hyper-parameters for svm-perf. Currently, only the `C` parameter is supported - - :param parameters: a `**kwargs` dictionary `{'C': }` - """ - assert list(parameters.keys()) == ['C'], 'currently, only the C parameter is supported' - self.C = parameters['C'] + # def set_params(self, **parameters): + # """ + # Set the hyper-parameters for svm-perf. Currently, only the `C` and `loss` parameters are supported + # + # :param parameters: a `**kwargs` dictionary `{'C': }` + # """ + # assert sorted(list(parameters.keys())) == ['C', 'loss'], \ + # 'currently, only the C and loss parameters are supported' + # self.C = parameters.get('C', self.C) + # self.loss = parameters.get('loss', self.loss) + # + # def get_params(self, deep=True): + # return {'C': self.C, 'loss': self.loss} def fit(self, X, y): """ @@ -65,14 +75,14 @@ class SVMperf(BaseEstimator, ClassifierMixin): local_random = random.Random() # this would allow to run parallel instances of predict - random_code = '-'.join(str(local_random.randint(0,1000000)) for _ in range(5)) - # self.tmpdir = tempfile.TemporaryDirectory(suffix=random_code) - # tmp dir are removed after the fit terminates in multiprocessing... moving to regular directories + __del__ - self.tmpdir = '.svmperf-' + random_code + random_code = 'svmperfprocess'+'-'.join(str(local_random.randint(0, 1000000)) for _ in range(5)) + if self.host_folder is None: + # tmp dir are removed after the fit terminates in multiprocessing... + self.tmpdir = tempfile.TemporaryDirectory(suffix=random_code).name + else: + self.tmpdir = join(self.host_folder, '.' + random_code) makedirs(self.tmpdir, exist_ok=True) - # self.model = join(self.tmpdir.name, 'model-'+random_code) - # traindat = join(self.tmpdir.name, f'train-{random_code}.dat') self.model = join(self.tmpdir, 'model-'+random_code) traindat = join(self.tmpdir, f'train-{random_code}.dat') @@ -94,6 +104,7 @@ class SVMperf(BaseEstimator, ClassifierMixin): def predict(self, X): """ Predicts labels for the instances `X` + :param X: array-like of shape `(n_samples, n_features)` instances to classify :return: a `numpy` array of length `n` containing the label predictions, where `n` is the number of instances in `X` @@ -119,8 +130,6 @@ class SVMperf(BaseEstimator, ClassifierMixin): # in order to allow for parallel runs of predict, a random code is assigned local_random = random.Random() random_code = '-'.join(str(local_random.randint(0, 1000000)) for _ in range(5)) - # predictions_path = join(self.tmpdir.name, 'predictions'+random_code+'.dat') - # testdat = join(self.tmpdir.name, 'test'+random_code+'.dat') predictions_path = join(self.tmpdir, 'predictions' + random_code + '.dat') testdat = join(self.tmpdir, 'test' + random_code + '.dat') dump_svmlight_file(X, y, testdat, zero_based=False) @@ -141,5 +150,5 @@ class SVMperf(BaseEstimator, ClassifierMixin): def __del__(self): if hasattr(self, 'tmpdir'): - pass # shutil.rmtree(self.tmpdir, ignore_errors=True) + shutil.rmtree(self.tmpdir, ignore_errors=True) diff --git a/quapy/data/_lequa2022.py b/quapy/data/_lequa2022.py new file mode 100644 index 0000000..449eab6 --- /dev/null +++ b/quapy/data/_lequa2022.py @@ -0,0 +1,169 @@ +from typing import Tuple, Union +import pandas as pd +import numpy as np +import os + +from quapy.protocol import AbstractProtocol + +DEV_SAMPLES = 1000 +TEST_SAMPLES = 5000 + +ERROR_TOL = 1E-3 + + +def load_category_map(path): + cat2code = {} + with open(path, 'rt') as fin: + for line in fin: + category, code = line.split() + cat2code[category] = int(code) + code2cat = [cat for cat, code in sorted(cat2code.items(), key=lambda x: x[1])] + return cat2code, code2cat + + +def load_raw_documents(path): + df = pd.read_csv(path) + documents = list(df["text"].values) + labels = None + if "label" in df.columns: + labels = df["label"].values.astype(int) + return documents, labels + + +def load_vector_documents(path): + D = pd.read_csv(path).to_numpy(dtype=float) + labelled = D.shape[1] == 301 + if labelled: + X, y = D[:, 1:], D[:, 0].astype(int).flatten() + else: + X, y = D, None + return X, y + + +class SamplesFromDir(AbstractProtocol): + + def __init__(self, path_dir:str, ground_truth_path:str, load_fn): + self.path_dir = path_dir + self.load_fn = load_fn + self.true_prevs = ResultSubmission.load(ground_truth_path) + + def __call__(self): + for id, prevalence in self.true_prevs.iterrows(): + sample, _ = self.load_fn(os.path.join(self.path_dir, f'{id}.txt')) + yield sample, prevalence + + +class ResultSubmission: + + def __init__(self): + self.df = None + + def __init_df(self, categories: int): + if not isinstance(categories, int) or categories < 2: + raise TypeError('wrong format for categories: an int (>=2) was expected') + df = pd.DataFrame(columns=list(range(categories))) + df.index.set_names('id', inplace=True) + self.df = df + + @property + def n_categories(self): + return len(self.df.columns.values) + + def add(self, sample_id: int, prevalence_values: np.ndarray): + if not isinstance(sample_id, int): + raise TypeError(f'error: expected int for sample_sample, found {type(sample_id)}') + if not isinstance(prevalence_values, np.ndarray): + raise TypeError(f'error: expected np.ndarray for prevalence_values, found {type(prevalence_values)}') + if self.df is None: + self.__init_df(categories=len(prevalence_values)) + if sample_id in self.df.index.values: + raise ValueError(f'error: prevalence values for "{sample_id}" already added') + if prevalence_values.ndim != 1 and prevalence_values.size != self.n_categories: + raise ValueError(f'error: wrong shape found for prevalence vector {prevalence_values}') + if (prevalence_values < 0).any() or (prevalence_values > 1).any(): + raise ValueError(f'error: prevalence values out of range [0,1] for "{sample_id}"') + if np.abs(prevalence_values.sum() - 1) > ERROR_TOL: + raise ValueError(f'error: prevalence values do not sum up to one for "{sample_id}"' + f'(error tolerance {ERROR_TOL})') + + self.df.loc[sample_id] = prevalence_values + + def __len__(self): + return len(self.df) + + @classmethod + def load(cls, path: str) -> 'ResultSubmission': + df = ResultSubmission.check_file_format(path) + r = ResultSubmission() + r.df = df + return r + + def dump(self, path: str): + ResultSubmission.check_dataframe_format(self.df) + self.df.to_csv(path) + + def prevalence(self, sample_id: int): + sel = self.df.loc[sample_id] + if sel.empty: + return None + else: + return sel.values.flatten() + + def iterrows(self): + for index, row in self.df.iterrows(): + prevalence = row.values.flatten() + yield index, prevalence + + @classmethod + def check_file_format(cls, path) -> Union[pd.DataFrame, Tuple[pd.DataFrame, str]]: + try: + df = pd.read_csv(path, index_col=0) + except Exception as e: + print(f'the file {path} does not seem to be a valid csv file. ') + print(e) + return ResultSubmission.check_dataframe_format(df, path=path) + + @classmethod + def check_dataframe_format(cls, df, path=None) -> Union[pd.DataFrame, Tuple[pd.DataFrame, str]]: + hint_path = '' # if given, show the data path in the error message + if path is not None: + hint_path = f' in {path}' + + if df.index.name != 'id' or len(df.columns) < 2: + raise ValueError(f'wrong header{hint_path}, ' + f'the format of the header should be "id,0,...,n-1", ' + f'where n is the number of categories') + if [int(ci) for ci in df.columns.values] != list(range(len(df.columns))): + raise ValueError(f'wrong header{hint_path}, category ids should be 0,1,2,...,n-1, ' + f'where n is the number of categories') + if df.empty: + raise ValueError(f'error{hint_path}: results file is empty') + elif len(df) != DEV_SAMPLES and len(df) != TEST_SAMPLES: + raise ValueError(f'wrong number of prevalence values found{hint_path}; ' + f'expected {DEV_SAMPLES} for development sets and ' + f'{TEST_SAMPLES} for test sets; found {len(df)}') + + ids = set(df.index.values) + expected_ids = set(range(len(df))) + if ids != expected_ids: + missing = expected_ids - ids + if missing: + raise ValueError(f'there are {len(missing)} missing ids{hint_path}: {sorted(missing)}') + unexpected = ids - expected_ids + if unexpected: + raise ValueError(f'there are {len(missing)} unexpected ids{hint_path}: {sorted(unexpected)}') + + for category_id in df.columns: + if (df[category_id] < 0).any() or (df[category_id] > 1).any(): + raise ValueError(f'error{hint_path} column "{category_id}" contains values out of range [0,1]') + + prevs = df.values + round_errors = np.abs(prevs.sum(axis=-1) - 1.) > ERROR_TOL + if round_errors.any(): + raise ValueError(f'warning: prevalence values in rows with id {np.where(round_errors)[0].tolist()} ' + f'do not sum up to 1 (error tolerance {ERROR_TOL}), ' + f'probably due to some rounding errors.') + + return df + + diff --git a/quapy/data/base.py b/quapy/data/base.py index bbf8a8d..9cc6441 100644 --- a/quapy/data/base.py +++ b/quapy/data/base.py @@ -1,24 +1,29 @@ +import itertools +from functools import cached_property +from typing import Iterable + import numpy as np from scipy.sparse import issparse from scipy.sparse import vstack from sklearn.model_selection import train_test_split, RepeatedStratifiedKFold - -from quapy.functional import artificial_prevalence_sampling, strprev +from numpy.random import RandomState +from quapy.functional import strprev +from quapy.util import temp_seed class LabelledCollection: """ - A LabelledCollection is a set of objects each with a label associated to it. This class implements many sampling - routines. - + A LabelledCollection is a set of objects each with a label attached to each of them. + This class implements several sampling routines and other utilities. + :param instances: array-like (np.ndarray, list, or csr_matrix are supported) :param labels: array-like with the same length of instances - :param classes_: optional, list of classes from which labels are taken. If not specified, the classes are inferred + :param classes: optional, list of classes from which labels are taken. If not specified, the classes are inferred from the labels. The classes must be indicated in cases in which some of the labels might have no examples (i.e., a prevalence of 0) """ - def __init__(self, instances, labels, classes_=None): + def __init__(self, instances, labels, classes=None): if issparse(instances): self.instances = instances elif isinstance(instances, list) and len(instances) > 0 and isinstance(instances[0], str): @@ -28,14 +33,14 @@ class LabelledCollection: self.instances = np.asarray(instances) self.labels = np.asarray(labels) n_docs = len(self) - if classes_ is None: + if classes is None: self.classes_ = np.unique(self.labels) self.classes_.sort() else: - self.classes_ = np.unique(np.asarray(classes_)) + self.classes_ = np.unique(np.asarray(classes)) self.classes_.sort() - if len(set(self.labels).difference(set(classes_))) > 0: - raise ValueError(f'labels ({set(self.labels)}) contain values not included in classes_ ({set(classes_)})') + if len(set(self.labels).difference(set(classes))) > 0: + raise ValueError(f'labels ({set(self.labels)}) contain values not included in classes_ ({set(classes)})') self.index = {class_: np.arange(n_docs)[self.labels == class_] for class_ in self.classes_} @classmethod @@ -65,7 +70,7 @@ class LabelledCollection: def prevalence(self): """ - Returns the prevalence, or relative frequency, of the classes of interest. + Returns the prevalence, or relative frequency, of the classes in the codeframe. :return: a np.ndarray of shape `(n_classes)` with the relative frequencies of each class, in the same order as listed by `self.classes_` @@ -74,7 +79,7 @@ class LabelledCollection: def counts(self): """ - Returns the number of instances for each of the classes of interest. + Returns the number of instances for each of the classes in the codeframe. :return: a np.ndarray of shape `(n_classes)` with the number of instances of each class, in the same order as listed by `self.classes_` @@ -99,7 +104,7 @@ class LabelledCollection: """ return self.n_classes == 2 - def sampling_index(self, size, *prevs, shuffle=True): + def sampling_index(self, size, *prevs, shuffle=True, random_state=None): """ Returns an index to be used to extract a random sample of desired size and desired prevalence values. If the prevalence values are not specified, then returns the index of a uniform sampling. @@ -111,50 +116,72 @@ class LabelledCollection: it is constrained. E.g., for binary collections, only the prevalence `p` for the first class (as listed in `self.classes_` can be specified, while the other class takes prevalence value `1-p` :param shuffle: if set to True (default), shuffles the index before returning it + :param random_state: seed for reproducing sampling :return: a np.ndarray of shape `(size)` with the indexes """ if len(prevs) == 0: # no prevalence was indicated; returns an index for uniform sampling - return self.uniform_sampling_index(size) + return self.uniform_sampling_index(size, random_state=random_state) if len(prevs) == self.n_classes - 1: prevs = prevs + (1 - sum(prevs),) assert len(prevs) == self.n_classes, 'unexpected number of prevalences' assert sum(prevs) == 1, f'prevalences ({prevs}) wrong range (sum={sum(prevs)})' - taken = 0 - indexes_sample = [] - for i, class_ in enumerate(self.classes_): - if i == self.n_classes - 1: - n_requested = size - taken - else: - n_requested = int(size * prevs[i]) + # Decide how many instances should be taken for each class in order to satisfy the requested prevalence + # accurately, and the number of instances in the sample (exactly). If int(size * prevs[i]) (which is + # <= size * prevs[i]) examples are drawn from class i, there could be a remainder number of instances to take + # to satisfy the size constrain. The remainder is distributed along the classes with probability = prevs. + # (This aims at avoiding the remainder to be placed in a class for which the prevalence requested is 0.) + n_requests = {class_: round(size * prevs[i]) for i, class_ in enumerate(self.classes_)} + remainder = size - sum(n_requests.values()) + with temp_seed(random_state): + # due to rounding, the remainder can be 0, >0, or <0 + if remainder > 0: + # when the remainder is >0 we randomly add 1 to the requests for each class; + # more prevalent classes are more likely to be taken in order to minimize the impact in the final prevalence + for rand_class in np.random.choice(self.classes_, size=remainder, p=prevs): + n_requests[rand_class] += 1 + elif remainder < 0: + # when the remainder is <0 we randomly remove 1 from the requests, unless the request is 0 for a chosen + # class; we repeat until remainder==0 + while remainder!=0: + rand_class = np.random.choice(self.classes_, p=prevs) + if n_requests[rand_class] > 0: + n_requests[rand_class] -= 1 + remainder += 1 - n_candidates = len(self.index[class_]) - index_sample = self.index[class_][ - np.random.choice(n_candidates, size=n_requested, replace=(n_requested > n_candidates)) - ] if n_requested > 0 else [] + indexes_sample = [] + for class_, n_requested in n_requests.items(): + n_candidates = len(self.index[class_]) + index_sample = self.index[class_][ + np.random.choice(n_candidates, size=n_requested, replace=(n_requested > n_candidates)) + ] if n_requested > 0 else [] - indexes_sample.append(index_sample) - taken += n_requested + indexes_sample.append(index_sample) - indexes_sample = np.concatenate(indexes_sample).astype(int) + indexes_sample = np.concatenate(indexes_sample).astype(int) - if shuffle: - indexes_sample = np.random.permutation(indexes_sample) + if shuffle: + indexes_sample = np.random.permutation(indexes_sample) return indexes_sample - def uniform_sampling_index(self, size): + def uniform_sampling_index(self, size, random_state=None): """ Returns an index to be used to extract a uniform sample of desired size. The sampling is drawn with replacement if the requested size is greater than the number of instances, or without replacement otherwise. :param size: integer, the size of the uniform sample + :param random_state: if specified, guarantees reproducibility of the split. :return: a np.ndarray of shape `(size)` with the indexes """ - return np.random.choice(len(self), size, replace=False) + if random_state is not None: + ng = RandomState(seed=random_state) + else: + ng = np.random + return ng.choice(len(self), size, replace=size > len(self)) - def sampling(self, size, *prevs, shuffle=True): + def sampling(self, size, *prevs, shuffle=True, random_state=None): """ Return a random sample (an instance of :class:`LabelledCollection`) of desired size and desired prevalence values. For each class, the sampling is drawn without replacement if the requested prevalence is larger than @@ -165,22 +192,24 @@ class LabelledCollection: it is constrained. E.g., for binary collections, only the prevalence `p` for the first class (as listed in `self.classes_` can be specified, while the other class takes prevalence value `1-p` :param shuffle: if set to True (default), shuffles the index before returning it + :param random_state: seed for reproducing sampling :return: an instance of :class:`LabelledCollection` with length == `size` and prevalence close to `prevs` (or prevalence == `prevs` if the exact prevalence values can be met as proportions of instances) """ - prev_index = self.sampling_index(size, *prevs, shuffle=shuffle) + prev_index = self.sampling_index(size, *prevs, shuffle=shuffle, random_state=random_state) return self.sampling_from_index(prev_index) - def uniform_sampling(self, size): + def uniform_sampling(self, size, random_state=None): """ Returns a uniform sample (an instance of :class:`LabelledCollection`) of desired size. The sampling is drawn with replacement if the requested size is greater than the number of instances, or without replacement otherwise. :param size: integer, the requested size + :param random_state: if specified, guarantees reproducibility of the split. :return: an instance of :class:`LabelledCollection` with length == `size` """ - unif_index = self.uniform_sampling_index(size) + unif_index = self.uniform_sampling_index(size, random_state=random_state) return self.sampling_from_index(unif_index) def sampling_from_index(self, index): @@ -193,7 +222,7 @@ class LabelledCollection: """ documents = self.instances[index] labels = self.labels[index] - return LabelledCollection(documents, labels, classes_=self.classes_) + return LabelledCollection(documents, labels, classes=self.classes_) def split_stratified(self, train_prop=0.6, random_state=None): """ @@ -207,92 +236,91 @@ class LabelledCollection: :return: two instances of :class:`LabelledCollection`, the first one with `train_prop` elements, and the second one with `1-train_prop` elements """ - tr_docs, te_docs, tr_labels, te_labels = \ - train_test_split(self.instances, self.labels, train_size=train_prop, stratify=self.labels, - random_state=random_state) - return LabelledCollection(tr_docs, tr_labels), LabelledCollection(te_docs, te_labels) + tr_docs, te_docs, tr_labels, te_labels = train_test_split( + self.instances, self.labels, train_size=train_prop, stratify=self.labels, random_state=random_state + ) + training = LabelledCollection(tr_docs, tr_labels, classes=self.classes_) + test = LabelledCollection(te_docs, te_labels, classes=self.classes_) + return training, test - def artificial_sampling_generator(self, sample_size, n_prevalences=101, repeats=1): + def split_random(self, train_prop=0.6, random_state=None): """ - A generator of samples that implements the artificial prevalence protocol (APP). - The APP consists of exploring a grid of prevalence values containing `n_prevalences` points (e.g., - [0, 0.05, 0.1, 0.15, ..., 1], if `n_prevalences=21`), and generating all valid combinations of - prevalence values for all classes (e.g., for 3 classes, samples with [0, 0, 1], [0, 0.05, 0.95], ..., - [1, 0, 0] prevalence values of size `sample_size` will be yielded). The number of samples for each valid - combination of prevalence values is indicated by `repeats`. + Returns two instances of :class:`LabelledCollection` split randomly from this collection, at desired + proportion. - :param sample_size: the number of instances in each sample - :param n_prevalences: the number of prevalence points to be taken from the [0,1] interval (including the - limits {0,1}). E.g., if `n_prevalences=11`, then the prevalence points to take are [0, 0.1, 0.2, ..., 1] - :param repeats: the number of samples to generate for each valid combination of prevalence values (default 1) - :return: yield samples generated at artificially controlled prevalence values + :param train_prop: the proportion of elements to include in the left-most returned collection (typically used + as the training collection). The rest of elements are included in the right-most returned collection + (typically used as a test collection). + :param random_state: if specified, guarantees reproducibility of the split. + :return: two instances of :class:`LabelledCollection`, the first one with `train_prop` elements, and the + second one with `1-train_prop` elements """ - dimensions = self.n_classes - for prevs in artificial_prevalence_sampling(dimensions, n_prevalences, repeats): - yield self.sampling(sample_size, *prevs) - - def artificial_sampling_index_generator(self, sample_size, n_prevalences=101, repeats=1): - """ - A generator of sample indexes implementing the artificial prevalence protocol (APP). - The APP consists of exploring - a grid of prevalence values (e.g., [0, 0.05, 0.1, 0.15, ..., 1]), and generating all valid combinations of - prevalence values for all classes (e.g., for 3 classes, samples with [0, 0, 1], [0, 0.05, 0.95], ..., - [1, 0, 0] prevalence values of size `sample_size` will be yielded). The number of sample indexes for each valid - combination of prevalence values is indicated by `repeats` - - :param sample_size: the number of instances in each sample (i.e., length of each index) - :param n_prevalences: the number of prevalence points to be taken from the [0,1] interval (including the - limits {0,1}). E.g., if `n_prevalences=11`, then the prevalence points to take are [0, 0.1, 0.2, ..., 1] - :param repeats: the number of samples to generate for each valid combination of prevalence values (default 1) - :return: yield the indexes that generate the samples according to APP - """ - dimensions = self.n_classes - for prevs in artificial_prevalence_sampling(dimensions, n_prevalences, repeats): - yield self.sampling_index(sample_size, *prevs) - - def natural_sampling_generator(self, sample_size, repeats=100): - """ - A generator of samples that implements the natural prevalence protocol (NPP). The NPP consists of drawing - samples uniformly at random, therefore approximately preserving the natural prevalence of the collection. - - :param sample_size: integer, the number of instances in each sample - :param repeats: the number of samples to generate - :return: yield instances of :class:`LabelledCollection` - """ - for _ in range(repeats): - yield self.uniform_sampling(sample_size) - - def natural_sampling_index_generator(self, sample_size, repeats=100): - """ - A generator of sample indexes according to the natural prevalence protocol (NPP). The NPP consists of drawing - samples uniformly at random, therefore approximately preserving the natural prevalence of the collection. - - :param sample_size: integer, the number of instances in each sample (i.e., the length of each index) - :param repeats: the number of indexes to generate - :return: yield `repeats` instances of np.ndarray with shape `(sample_size,)` - """ - for _ in range(repeats): - yield self.uniform_sampling_index(sample_size) + indexes = np.random.RandomState(seed=random_state).permutation(len(self)) + if isinstance(train_prop, int): + assert train_prop < len(self), \ + 'argument train_prop cannot be greater than the number of elements in the collection' + splitpoint = train_prop + elif isinstance(train_prop, float): + assert 0 < train_prop < 1, \ + 'argument train_prop out of range (0,1)' + splitpoint = int(np.round(len(self)*train_prop)) + left, right = indexes[:splitpoint], indexes[splitpoint:] + training = self.sampling_from_index(left) + test = self.sampling_from_index(right) + return training, test def __add__(self, other): """ - Returns a new :class:`LabelledCollection` as the union of this collection with another collection + Returns a new :class:`LabelledCollection` as the union of this collection with another collection. + Both labelled collections must have the same classes. :param other: another :class:`LabelledCollection` :return: a :class:`LabelledCollection` representing the union of both collections """ - if other is None: - return self - elif issparse(self.instances) and issparse(other.instances): - join_instances = vstack([self.instances, other.instances]) - elif isinstance(self.instances, list) and isinstance(other.instances, list): - join_instances = self.instances + other.instances - elif isinstance(self.instances, np.ndarray) and isinstance(other.instances, np.ndarray): - join_instances = np.concatenate([self.instances, other.instances]) + if not all(np.sort(self.classes_)==np.sort(other.classes_)): + raise NotImplementedError(f'unsupported operation for collections on different classes; ' + f'expected {self.classes_}, found {other.classes_}') + return LabelledCollection.join(self, other) + + @classmethod + def join(cls, *args: Iterable['LabelledCollection']): + """ + Returns a new :class:`LabelledCollection` as the union of the collections given in input. + + :param args: instances of :class:`LabelledCollection` + :return: a :class:`LabelledCollection` representing the union of both collections + """ + + args = [lc for lc in args if lc is not None] + assert len(args) > 0, 'empty list is not allowed for mix' + + assert all([isinstance(lc, LabelledCollection) for lc in args]), \ + 'only instances of LabelledCollection allowed' + + first_instances = args[0].instances + first_type = type(first_instances) + assert all([type(lc.instances)==first_type for lc in args[1:]]), \ + 'not all the collections are of instances of the same type' + + if issparse(first_instances) or isinstance(first_instances, np.ndarray): + first_ndim = first_instances.ndim + assert all([lc.instances.ndim == first_ndim for lc in args[1:]]), \ + 'not all the ndarrays are of the same dimension' + if first_ndim > 1: + first_shape = first_instances.shape[1:] + assert all([lc.instances.shape[1:] == first_shape for lc in args[1:]]), \ + 'not all the ndarrays are of the same shape' + if issparse(first_instances): + instances = vstack([lc.instances for lc in args]) + else: + instances = np.concatenate([lc.instances for lc in args]) + elif isinstance(first_instances, list): + instances = list(itertools.chain(lc.instances for lc in args)) else: raise NotImplementedError('unsupported operation for collection types') - labels = np.concatenate([self.labels, other.labels]) - return LabelledCollection(join_instances, labels) + labels = np.concatenate([lc.labels for lc in args]) + classes = np.unique(labels).sort() + return LabelledCollection(instances, labels, classes=classes) @property def Xy(self): @@ -305,6 +333,44 @@ class LabelledCollection: """ return self.instances, self.labels + @property + def Xp(self): + """ + Gets the instances and the true prevalence. This is useful when implementing evaluation protocols from + a :class:`LabelledCollection` object. + + :return: a tuple `(instances, prevalence)` from this collection + """ + return self.instances, self.prevalence() + + @property + def X(self): + """ + An alias to self.instances + + :return: self.instances + """ + return self.instances + + @property + def y(self): + """ + An alias to self.labels + + :return: self.labels + """ + return self.labels + + @property + def p(self): + """ + An alias to self.prevalence() + + :return: self.prevalence() + """ + return self.prevalence() + + def stats(self, show=True): """ Returns (and eventually prints) a dictionary with some stats of this collection. E.g.,: @@ -337,7 +403,7 @@ class LabelledCollection: f'#classes={stats_["classes"]}, prevs={stats_["prevs"]}') return stats_ - def kFCV(self, nfolds=5, nrepeats=1, random_state=0): + def kFCV(self, nfolds=5, nrepeats=1, random_state=None): """ Generator of stratified folds to be used in k-fold cross validation. @@ -439,7 +505,17 @@ class Dataset: """ return len(self.vocabulary) - def stats(self, show): + @property + def train_test(self): + """ + Alias to `self.training` and `self.test` + + :return: the training and test collections + :return: the training and test collections + """ + return self.training, self.test + + def stats(self, show=True): """ Returns (and eventually prints) a dictionary with some stats of this dataset. E.g.,: @@ -477,13 +553,14 @@ class Dataset: yield Dataset(train, test, name=f'fold {(i % nfolds) + 1}/{nfolds} (round={(i // nfolds) + 1})') -def isbinary(data): - """ - Returns True if `data` is either a binary :class:`Dataset` or a binary :class:`LabelledCollection` + def reduce(self, n_train=100, n_test=100): + """ + Reduce the number of instances in place for quick experiments. Preserves the prevalence of each set. - :param data: a :class:`Dataset` or a :class:`LabelledCollection` object - :return: True if labelled according to two classes - """ - if isinstance(data, Dataset) or isinstance(data, LabelledCollection): - return data.binary - return False + :param n_train: number of training documents to keep (default 100) + :param n_test: number of test documents to keep (default 100) + :return: self + """ + self.training = self.training.sampling(n_train, *self.training.prevalence()) + self.test = self.test.sampling(n_test, *self.test.prevalence()) + return self \ No newline at end of file diff --git a/quapy/data/datasets.py b/quapy/data/datasets.py index 74e2a3e..5c5eb99 100644 --- a/quapy/data/datasets.py +++ b/quapy/data/datasets.py @@ -6,12 +6,14 @@ import os import zipfile from os.path import join import pandas as pd +import scipy from quapy.data.base import Dataset, LabelledCollection from quapy.data.preprocessing import text2tfidf, reduce_columns from quapy.data.reader import * from quapy.util import download_file_if_not_exists, download_file, get_quapy_home, pickled_resource + REVIEWS_SENTIMENT_DATASETS = ['hp', 'kindle', 'imdb'] TWITTER_SENTIMENT_DATASETS_TEST = ['gasp', 'hcr', 'omd', 'sanders', 'semeval13', 'semeval14', 'semeval15', 'semeval16', @@ -43,6 +45,22 @@ UCI_DATASETS = ['acute.a', 'acute.b', 'wine-q-red', 'wine-q-white', 'yeast'] +LEQUA2022_TASKS = ['T1A', 'T1B', 'T2A', 'T2B'] + +_TXA_SAMPLE_SIZE = 250 +_TXB_SAMPLE_SIZE = 1000 + +LEQUA2022_SAMPLE_SIZE = { + 'TXA': _TXA_SAMPLE_SIZE, + 'TXB': _TXB_SAMPLE_SIZE, + 'T1A': _TXA_SAMPLE_SIZE, + 'T1B': _TXB_SAMPLE_SIZE, + 'T2A': _TXA_SAMPLE_SIZE, + 'T2B': _TXB_SAMPLE_SIZE, + 'binary': _TXA_SAMPLE_SIZE, + 'multiclass': _TXB_SAMPLE_SIZE +} + def fetch_reviews(dataset_name, tfidf=False, min_df=None, data_home=None, pickle=False) -> Dataset: """ @@ -532,4 +550,77 @@ def fetch_UCILabelledCollection(dataset_name, data_home=None, verbose=False) -> def _df_replace(df, col, repl={'yes': 1, 'no':0}, astype=float): - df[col] = df[col].apply(lambda x:repl[x]).astype(astype, copy=False) \ No newline at end of file + df[col] = df[col].apply(lambda x:repl[x]).astype(astype, copy=False) + + +def fetch_lequa2022(task, data_home=None): + """ + Loads the official datasets provided for the `LeQua `_ competition. + In brief, there are 4 tasks (T1A, T1B, T2A, T2B) having to do with text quantification + problems. Tasks T1A and T1B provide documents in vector form, while T2A and T2B provide raw documents instead. + Tasks T1A and T2A are binary sentiment quantification problems, while T2A and T2B are multiclass quantification + problems consisting of estimating the class prevalence values of 28 different merchandise products. + We refer to the `Esuli, A., Moreo, A., Sebastiani, F., & Sperduti, G. (2022). + A Detailed Overview of LeQua@ CLEF 2022: Learning to Quantify. + `_ for a detailed description + on the tasks and datasets. + + The datasets are downloaded only once, and stored for fast reuse. + + See `lequa2022_experiments.py` provided in the example folder, that can serve as a guide on how to use these + datasets. + + + :param task: a string representing the task name; valid ones are T1A, T1B, T2A, and T2B + :param data_home: specify the quapy home directory where collections will be dumped (leave empty to use the default + ~/quay_data/ directory) + :return: a tuple `(train, val_gen, test_gen)` where `train` is an instance of + :class:`quapy.data.base.LabelledCollection`, `val_gen` and `test_gen` are instances of + :class:`quapy.protocol.SamplesFromDir`, i.e., are sampling protocols that return a series of samples + labelled by prevalence. + """ + + from quapy.data._lequa2022 import load_raw_documents, load_vector_documents, SamplesFromDir + + assert task in LEQUA2022_TASKS, \ + f'Unknown task {task}. Valid ones are {LEQUA2022_TASKS}' + if data_home is None: + data_home = get_quapy_home() + + URL_TRAINDEV=f'https://zenodo.org/record/6546188/files/{task}.train_dev.zip' + URL_TEST=f'https://zenodo.org/record/6546188/files/{task}.test.zip' + URL_TEST_PREV=f'https://zenodo.org/record/6546188/files/{task}.test_prevalences.zip' + + lequa_dir = join(data_home, 'lequa2022') + os.makedirs(lequa_dir, exist_ok=True) + + def download_unzip_and_remove(unzipped_path, url): + tmp_path = join(lequa_dir, task + '_tmp.zip') + download_file_if_not_exists(url, tmp_path) + with zipfile.ZipFile(tmp_path) as file: + file.extractall(unzipped_path) + os.remove(tmp_path) + + if not os.path.exists(join(lequa_dir, task)): + download_unzip_and_remove(lequa_dir, URL_TRAINDEV) + download_unzip_and_remove(lequa_dir, URL_TEST) + download_unzip_and_remove(lequa_dir, URL_TEST_PREV) + + if task in ['T1A', 'T1B']: + load_fn = load_vector_documents + elif task in ['T2A', 'T2B']: + load_fn = load_raw_documents + + tr_path = join(lequa_dir, task, 'public', 'training_data.txt') + train = LabelledCollection.load(tr_path, loader_func=load_fn) + + val_samples_path = join(lequa_dir, task, 'public', 'dev_samples') + val_true_prev_path = join(lequa_dir, task, 'public', 'dev_prevalences.txt') + val_gen = SamplesFromDir(val_samples_path, val_true_prev_path, load_fn=load_fn) + + test_samples_path = join(lequa_dir, task, 'public', 'test_samples') + test_true_prev_path = join(lequa_dir, task, 'public', 'test_prevalences.txt') + test_gen = SamplesFromDir(test_samples_path, test_true_prev_path, load_fn=load_fn) + + return train, val_gen, test_gen + diff --git a/quapy/data/preprocessing.py b/quapy/data/preprocessing.py index f04f010..9aa8f8b 100644 --- a/quapy/data/preprocessing.py +++ b/quapy/data/preprocessing.py @@ -88,7 +88,7 @@ def standardize(dataset: Dataset, inplace=False): :param dataset: a :class:`quapy.data.base.Dataset` object :param inplace: set to True if the transformation is to be applied inplace, or to False (default) if a new :class:`quapy.data.base.Dataset` is to be returned - :return: + :return: an instance of :class:`quapy.data.base.Dataset` """ s = StandardScaler(copy=not inplace) training = s.fit_transform(dataset.training.instances) @@ -110,7 +110,7 @@ def index(dataset: Dataset, min_df=5, inplace=False, **kwargs): :param min_df: minimum number of occurrences below which the term is replaced by a `UNK` index :param inplace: whether or not to apply the transformation inplace (True), or to a new copy (False, default) :param kwargs: the rest of parameters of the transformation (as for sklearn's - `CountVectorizer _`) + `CountVectorizer _`) :return: a new :class:`quapy.data.base.Dataset` (if inplace=False) or a reference to the current :class:`quapy.data.base.Dataset` (inplace=True) consisting of lists of integer values representing indices. """ @@ -121,6 +121,9 @@ def index(dataset: Dataset, min_df=5, inplace=False, **kwargs): training_index = indexer.fit_transform(dataset.training.instances) test_index = indexer.transform(dataset.test.instances) + training_index = np.asarray(training_index, dtype=object) + test_index = np.asarray(test_index, dtype=object) + if inplace: dataset.training = LabelledCollection(training_index, dataset.training.labels, dataset.classes_) dataset.test = LabelledCollection(test_index, dataset.test.labels, dataset.classes_) @@ -147,7 +150,8 @@ class IndexTransformer: contains, and that would be generated by sklearn's `CountVectorizer `_ - :param kwargs: keyworded arguments from `CountVectorizer `_ + :param kwargs: keyworded arguments from + `CountVectorizer `_ """ def __init__(self, **kwargs): @@ -169,7 +173,7 @@ class IndexTransformer: self.pad = self.add_word(qp.environ['PAD_TOKEN'], qp.environ['PAD_INDEX']) return self - def transform(self, X, n_jobs=-1): + def transform(self, X, n_jobs=None): """ Transforms the strings in `X` as lists of numerical ids @@ -179,14 +183,15 @@ class IndexTransformer: """ # given the number of tasks and the number of jobs, generates the slices for the parallel processes assert self.unk != -1, 'transform called before fit' - indexed = map_parallel(func=self._index, args=X, n_jobs=n_jobs) - return np.asarray(indexed) + n_jobs = qp._get_njobs(n_jobs) + return map_parallel(func=self._index, args=X, n_jobs=n_jobs) + def _index(self, documents): vocab = self.vocabulary_.copy() - return [[vocab.prevalence(word, self.unk) for word in self.analyzer(doc)] for doc in tqdm(documents, 'indexing')] + return [[vocab.get(word, self.unk) for word in self.analyzer(doc)] for doc in tqdm(documents, 'indexing')] - def fit_transform(self, X, n_jobs=-1): + def fit_transform(self, X, n_jobs=None): """ Fits the transform on `X` and transforms it. diff --git a/quapy/data/reader.py b/quapy/data/reader.py index 8f8bc79..88791e3 100644 --- a/quapy/data/reader.py +++ b/quapy/data/reader.py @@ -102,7 +102,7 @@ def reindex_labels(y): y = np.asarray(y) classnames = np.asarray(sorted(np.unique(y))) label2index = {label: index for index, label in enumerate(classnames)} - indexed = np.empty(y.shape, dtype=np.int) + indexed = np.empty(y.shape, dtype=int) for label in classnames: indexed[y==label] = label2index[label] return indexed, classnames @@ -121,7 +121,7 @@ def binarize(y, pos_class): 0 otherwise """ y = np.asarray(y) - ybin = np.zeros(y.shape, dtype=np.int) + ybin = np.zeros(y.shape, dtype=int) ybin[y == pos_class] = 1 return ybin diff --git a/quapy/error.py b/quapy/error.py index a71ed46..c1a8e7f 100644 --- a/quapy/error.py +++ b/quapy/error.py @@ -11,11 +11,6 @@ def from_name(err_name): """ assert err_name in ERROR_NAMES, f'unknown error {err_name}' callable_error = globals()[err_name] - if err_name in QUANTIFICATION_ERROR_SMOOTH_NAMES: - eps = __check_eps() - def bound_callable_error(y_true, y_pred): - return callable_error(y_true, y_pred, eps) - return bound_callable_error return callable_error @@ -215,12 +210,14 @@ def __check_eps(eps=None): CLASSIFICATION_ERROR = {f1e, acce} -QUANTIFICATION_ERROR = {mae, mrae, mse, mkld, mnkld, ae, rae, se, kld, nkld} +QUANTIFICATION_ERROR = {mae, mrae, mse, mkld, mnkld} +QUANTIFICATION_ERROR_SINGLE = {ae, rae, se, kld, nkld} QUANTIFICATION_ERROR_SMOOTH = {kld, nkld, rae, mkld, mnkld, mrae} CLASSIFICATION_ERROR_NAMES = {func.__name__ for func in CLASSIFICATION_ERROR} QUANTIFICATION_ERROR_NAMES = {func.__name__ for func in QUANTIFICATION_ERROR} +QUANTIFICATION_ERROR_SINGLE_NAMES = {func.__name__ for func in QUANTIFICATION_ERROR_SINGLE} QUANTIFICATION_ERROR_SMOOTH_NAMES = {func.__name__ for func in QUANTIFICATION_ERROR_SMOOTH} -ERROR_NAMES = CLASSIFICATION_ERROR_NAMES | QUANTIFICATION_ERROR_NAMES +ERROR_NAMES = CLASSIFICATION_ERROR_NAMES | QUANTIFICATION_ERROR_NAMES | QUANTIFICATION_ERROR_SINGLE_NAMES f1_error = f1e acc_error = acce diff --git a/quapy/evaluation.py b/quapy/evaluation.py index 936b83c..c198115 100644 --- a/quapy/evaluation.py +++ b/quapy/evaluation.py @@ -1,296 +1,122 @@ from typing import Union, Callable, Iterable import numpy as np from tqdm import tqdm -import inspect - import quapy as qp -from quapy.data import LabelledCollection +from quapy.protocol import AbstractProtocol, OnLabelledCollectionProtocol, IterateProtocol from quapy.method.base import BaseQuantifier -from quapy.util import temp_seed, _check_sample_size -import quapy.functional as F import pandas as pd -def artificial_prevalence_prediction( +def prediction( model: BaseQuantifier, - test: LabelledCollection, - sample_size=None, - n_prevpoints=101, - n_repetitions=1, - eval_budget: int = None, - n_jobs=1, - random_seed=42, + protocol: AbstractProtocol, + aggr_speedup: Union[str, bool] = 'auto', verbose=False): """ - Performs the predictions for all samples generated according to the Artificial Prevalence Protocol (APP). - The APP consists of exploring a grid of prevalence values containing `n_prevalences` points (e.g., - [0, 0.05, 0.1, 0.15, ..., 1], if `n_prevalences=21`), and generating all valid combinations of - prevalence values for all classes (e.g., for 3 classes, samples with [0, 0, 1], [0, 0.05, 0.95], ..., - [1, 0, 0] prevalence values of size `sample_size` will be considered). The number of samples for each valid - combination of prevalence values is indicated by `repeats`. + Uses a quantification model to generate predictions for the samples generated via a specific protocol. + This function is central to all evaluation processes, and is endowed with an optimization to speed-up the + prediction of protocols that generate samples from a large collection. The optimization applies to aggregative + quantifiers only, and to OnLabelledCollectionProtocol protocols, and comes down to generating the classification + predictions once and for all, and then generating samples over the classification predictions (instead of over + the raw instances), so that the classifier prediction is never called again. This behaviour is obtained by + setting `aggr_speedup` to 'auto' or True, and is only carried out if the overall process is convenient in terms + of computations (e.g., if the number of classification predictions needed for the original collection exceed the + number of classification predictions needed for all samples, then the optimization is not undertaken). - :param model: the model in charge of generating the class prevalence estimations - :param test: the test set on which to perform APP - :param sample_size: integer, the size of the samples; if None, then the sample size is - taken from qp.environ['SAMPLE_SIZE'] - :param n_prevpoints: integer, the number of different prevalences to sample (or set to None if eval_budget - is specified; default 101, i.e., steps of 1%) - :param n_repetitions: integer, the number of repetitions for each prevalence (default 1) - :param eval_budget: integer, if specified, sets a ceil on the number of evaluations to perform. For example, if - there are 3 classes, `repeats=1`, and `eval_budget=20`, then `n_prevpoints` will be set to 5, since this - will generate 15 different prevalence vectors ([0, 0, 1], [0, 0.25, 0.75], [0, 0.5, 0.5] ... [1, 0, 0]) and - since setting `n_prevpoints=6` would produce more than 20 evaluations. - :param n_jobs: integer, number of jobs to be run in parallel (default 1) - :param random_seed: integer, allows to replicate the samplings. The seed is local to the method and does not affect - any other random process (default 42) - :param verbose: if True, shows a progress bar - :return: a tuple containing two `np.ndarrays` of shape `(m,n,)` with `m` the number of samples - `(n_prevpoints*repeats)` and `n` the number of classes. The first one contains the true prevalence values - for the samples generated while the second one contains the prevalence estimations + :param model: a quantifier, instance of :class:`quapy.method.base.BaseQuantifier` + :param protocol: :class:`quapy.protocol.AbstractProtocol`; if this object is also instance of + :class:`quapy.protocol.OnLabelledCollectionProtocol`, then the aggregation speed-up can be run. This is the protocol + in charge of generating the samples for which the model has to issue class prevalence predictions. + :param aggr_speedup: whether or not to apply the speed-up. Set to "force" for applying it even if the number of + instances in the original collection on which the protocol acts is larger than the number of instances + in the samples to be generated. Set to True or "auto" (default) for letting QuaPy decide whether it is + convenient or not. Set to False to deactivate. + :param verbose: boolean, show or not information in stdout + :return: a tuple `(true_prevs, estim_prevs)` in which each element in the tuple is an array of shape + `(n_samples, n_classes)` containing the true, or predicted, prevalence values for each sample """ + assert aggr_speedup in [False, True, 'auto', 'force'], 'invalid value for aggr_speedup' - sample_size = _check_sample_size(sample_size) - n_prevpoints, _ = qp.evaluation._check_num_evals(test.n_classes, n_prevpoints, eval_budget, n_repetitions, verbose) + sout = lambda x: print(x) if verbose else None - with temp_seed(random_seed): - indexes = list(test.artificial_sampling_index_generator(sample_size, n_prevpoints, n_repetitions)) + apply_optimization = False - return _predict_from_indexes(indexes, model, test, n_jobs, verbose) + if aggr_speedup in [True, 'auto', 'force']: + # checks whether the prediction can be made more efficiently; this check consists in verifying if the model is + # of type aggregative, if the protocol is based on LabelledCollection, and if the total number of documents to + # classify using the protocol would exceed the number of test documents in the original collection + from quapy.method.aggregative import AggregativeQuantifier + if isinstance(model, AggregativeQuantifier) and isinstance(protocol, OnLabelledCollectionProtocol): + if aggr_speedup == 'force': + apply_optimization = True + sout(f'forcing aggregative speedup') + elif hasattr(protocol, 'sample_size'): + nD = len(protocol.get_labelled_collection()) + samplesD = protocol.total() * protocol.sample_size + if nD < samplesD: + apply_optimization = True + sout(f'speeding up the prediction for the aggregative quantifier, ' + f'total classifications {nD} instead of {samplesD}') - -def natural_prevalence_prediction( - model: BaseQuantifier, - test: LabelledCollection, - sample_size=None, - repeats=100, - n_jobs=1, - random_seed=42, - verbose=False): - """ - Performs the predictions for all samples generated according to the Natural Prevalence Protocol (NPP). - The NPP consists of drawing samples uniformly at random, therefore approximately preserving the natural - prevalence of the collection. - - :param model: the model in charge of generating the class prevalence estimations - :param test: the test set on which to perform NPP - :param sample_size: integer, the size of the samples; if None, then the sample size is - taken from qp.environ['SAMPLE_SIZE'] - :param repeats: integer, the number of samples to generate (default 100) - :param n_jobs: integer, number of jobs to be run in parallel (default 1) - :param random_seed: allows to replicate the samplings. The seed is local to the method and does not affect - any other random process (default 42) - :param verbose: if True, shows a progress bar - :return: a tuple containing two `np.ndarrays` of shape `(m,n,)` with `m` the number of samples - `(repeats)` and `n` the number of classes. The first one contains the true prevalence values - for the samples generated while the second one contains the prevalence estimations - """ - - sample_size = _check_sample_size(sample_size) - with temp_seed(random_seed): - indexes = list(test.natural_sampling_index_generator(sample_size, repeats)) - - return _predict_from_indexes(indexes, model, test, n_jobs, verbose) - - -def gen_prevalence_prediction(model: BaseQuantifier, gen_fn: Callable, eval_budget=None): - """ - Generates prevalence predictions for a custom protocol defined as a generator function that yields - samples at each iteration. The sequence of samples is processed exhaustively if `eval_budget=None` - or up to the `eval_budget` iterations if specified. - - :param model: the model in charge of generating the class prevalence estimations - :param gen_fn: a generator function yielding one sample at each iteration - :param eval_budget: a maximum number of evaluations to run. Set to None (default) for exploring the - entire sequence - :return: a tuple containing two `np.ndarrays` of shape `(m,n,)` with `m` the number of samples - generated and `n` the number of classes. The first one contains the true prevalence values - for the samples generated while the second one contains the prevalence estimations - """ - if not inspect.isgenerator(gen_fn()): - raise ValueError('param "gen_fun" is not a callable returning a generator') - - if not isinstance(eval_budget, int): - eval_budget = -1 - - true_prevalences, estim_prevalences = [], [] - for sample_instances, true_prev in gen_fn(): - true_prevalences.append(true_prev) - estim_prevalences.append(model.quantify(sample_instances)) - eval_budget -= 1 - if eval_budget == 0: - break - - true_prevalences = np.asarray(true_prevalences) - estim_prevalences = np.asarray(estim_prevalences) - - return true_prevalences, estim_prevalences - - -def _predict_from_indexes( - indexes, - model: BaseQuantifier, - test: LabelledCollection, - n_jobs=1, - verbose=False): - - if model.aggregative: #isinstance(model, qp.method.aggregative.AggregativeQuantifier): - # print('\tinstance of aggregative-quantifier') - quantification_func = model.aggregate - if model.probabilistic: # isinstance(model, qp.method.aggregative.AggregativeProbabilisticQuantifier): - # print('\t\tinstance of probabilitstic-aggregative-quantifier') - preclassified_instances = model.posterior_probabilities(test.instances) - else: - # print('\t\tinstance of hard-aggregative-quantifier') - preclassified_instances = model.classify(test.instances) - test = LabelledCollection(preclassified_instances, test.labels) + if apply_optimization: + pre_classified = model.classify(protocol.get_labelled_collection().instances) + protocol_with_predictions = protocol.on_preclassified_instances(pre_classified) + return __prediction_helper(model.aggregate, protocol_with_predictions, verbose) else: - # print('\t\tinstance of base-quantifier') - quantification_func = model.quantify - - def _predict_prevalences(index): - sample = test.sampling_from_index(index) - true_prevalence = sample.prevalence() - estim_prevalence = quantification_func(sample.instances) - return true_prevalence, estim_prevalence - - pbar = tqdm(indexes, desc='[artificial sampling protocol] generating predictions') if verbose else indexes - results = qp.util.parallel(_predict_prevalences, pbar, n_jobs=n_jobs) - - true_prevalences, estim_prevalences = zip(*results) - true_prevalences = np.asarray(true_prevalences) - estim_prevalences = np.asarray(estim_prevalences) - - return true_prevalences, estim_prevalences + return __prediction_helper(model.quantify, protocol, verbose) -def artificial_prevalence_report( - model: BaseQuantifier, - test: LabelledCollection, - sample_size=None, - n_prevpoints=101, - n_repetitions=1, - eval_budget: int = None, - n_jobs=1, - random_seed=42, - error_metrics:Iterable[Union[str,Callable]]='mae', - verbose=False): +def __prediction_helper(quantification_fn, protocol: AbstractProtocol, verbose=False): + true_prevs, estim_prevs = [], [] + for sample_instances, sample_prev in tqdm(protocol(), total=protocol.total(), desc='predicting') if verbose else protocol(): + estim_prevs.append(quantification_fn(sample_instances)) + true_prevs.append(sample_prev) + + true_prevs = np.asarray(true_prevs) + estim_prevs = np.asarray(estim_prevs) + + return true_prevs, estim_prevs + + +def evaluation_report(model: BaseQuantifier, + protocol: AbstractProtocol, + error_metrics: Iterable[Union[str,Callable]] = 'mae', + aggr_speedup: Union[str, bool] = 'auto', + verbose=False): """ - Generates an evaluation report for all samples generated according to the Artificial Prevalence Protocol (APP). - The APP consists of exploring a grid of prevalence values containing `n_prevalences` points (e.g., - [0, 0.05, 0.1, 0.15, ..., 1], if `n_prevalences=21`), and generating all valid combinations of - prevalence values for all classes (e.g., for 3 classes, samples with [0, 0, 1], [0, 0.05, 0.95], ..., - [1, 0, 0] prevalence values of size `sample_size` will be considered). The number of samples for each valid - combination of prevalence values is indicated by `repeats`. - Te report takes the form of a - pandas' `dataframe `_ - in which the rows correspond to different samples, and the columns inform of the true prevalence values, - the estimated prevalence values, and the score obtained by each of the evaluation measures indicated. + Generates a report (a pandas' DataFrame) containing information of the evaluation of the model as according + to a specific protocol and in terms of one or more evaluation metrics (errors). - :param model: the model in charge of generating the class prevalence estimations - :param test: the test set on which to perform APP - :param sample_size: integer, the size of the samples; if None, then the sample size is - taken from qp.environ['SAMPLE_SIZE'] - :param n_prevpoints: integer, the number of different prevalences to sample (or set to None if eval_budget - is specified; default 101, i.e., steps of 1%) - :param n_repetitions: integer, the number of repetitions for each prevalence (default 1) - :param eval_budget: integer, if specified, sets a ceil on the number of evaluations to perform. For example, if - there are 3 classes, `repeats=1`, and `eval_budget=20`, then `n_prevpoints` will be set to 5, since this - will generate 15 different prevalence vectors ([0, 0, 1], [0, 0.25, 0.75], [0, 0.5, 0.5] ... [1, 0, 0]) and - since setting `n_prevpoints=6` would produce more than 20 evaluations. - :param n_jobs: integer, number of jobs to be run in parallel (default 1) - :param random_seed: integer, allows to replicate the samplings. The seed is local to the method and does not affect - any other random process (default 42) - :param error_metrics: a string indicating the name of the error (as defined in :mod:`quapy.error`) or a - callable error function; optionally, a list of strings or callables can be indicated, if the results - are to be evaluated with more than one error metric. Default is "mae" - :param verbose: if True, shows a progress bar - :return: pandas' dataframe with rows corresponding to different samples, and with columns informing of the - true prevalence values, the estimated prevalence values, and the score obtained by each of the evaluation - measures indicated. + + :param model: a quantifier, instance of :class:`quapy.method.base.BaseQuantifier` + :param protocol: :class:`quapy.protocol.AbstractProtocol`; if this object is also instance of + :class:`quapy.protocol.OnLabelledCollectionProtocol`, then the aggregation speed-up can be run. This is the protocol + in charge of generating the samples in which the model is evaluated. + :param error_metrics: a string, or list of strings, representing the name(s) of an error function in `qp.error` + (e.g., 'mae', the default value), or a callable function, or a list of callable functions, implementing + the error function itself. + :param aggr_speedup: whether or not to apply the speed-up. Set to "force" for applying it even if the number of + instances in the original collection on which the protocol acts is larger than the number of instances + in the samples to be generated. Set to True or "auto" (default) for letting QuaPy decide whether it is + convenient or not. Set to False to deactivate. + :param verbose: boolean, show or not information in stdout + :return: a pandas' DataFrame containing the columns 'true-prev' (the true prevalence of each sample), + 'estim-prev' (the prevalence estimated by the model for each sample), and as many columns as error metrics + have been indicated, each displaying the score in terms of that metric for every sample. """ - true_prevs, estim_prevs = artificial_prevalence_prediction( - model, test, sample_size, n_prevpoints, n_repetitions, eval_budget, n_jobs, random_seed, verbose - ) + true_prevs, estim_prevs = prediction(model, protocol, aggr_speedup=aggr_speedup, verbose=verbose) return _prevalence_report(true_prevs, estim_prevs, error_metrics) -def natural_prevalence_report( - model: BaseQuantifier, - test: LabelledCollection, - sample_size=None, - repeats=100, - n_jobs=1, - random_seed=42, - error_metrics:Iterable[Union[str,Callable]]='mae', - verbose=False): - """ - Generates an evaluation report for all samples generated according to the Natural Prevalence Protocol (NPP). - The NPP consists of drawing samples uniformly at random, therefore approximately preserving the natural - prevalence of the collection. - Te report takes the form of a - pandas' `dataframe `_ - in which the rows correspond to different samples, and the columns inform of the true prevalence values, - the estimated prevalence values, and the score obtained by each of the evaluation measures indicated. - - :param model: the model in charge of generating the class prevalence estimations - :param test: the test set on which to perform NPP - :param sample_size: integer, the size of the samples; if None, then the sample size is - taken from qp.environ['SAMPLE_SIZE'] - :param repeats: integer, the number of samples to generate (default 100) - :param n_jobs: integer, number of jobs to be run in parallel (default 1) - :param random_seed: allows to replicate the samplings. The seed is local to the method and does not affect - any other random process (default 42) - :param error_metrics: a string indicating the name of the error (as defined in :mod:`quapy.error`) or a - callable error function; optionally, a list of strings or callables can be indicated, if the results - are to be evaluated with more than one error metric. Default is "mae" - :param verbose: if True, shows a progress bar - :return: a tuple containing two `np.ndarrays` of shape `(m,n,)` with `m` the number of samples - `(repeats)` and `n` the number of classes. The first one contains the true prevalence values - for the samples generated while the second one contains the prevalence estimations - - """ - sample_size = _check_sample_size(sample_size) - true_prevs, estim_prevs = natural_prevalence_prediction( - model, test, sample_size, repeats, n_jobs, random_seed, verbose - ) - return _prevalence_report(true_prevs, estim_prevs, error_metrics) - - -def gen_prevalence_report(model: BaseQuantifier, gen_fn: Callable, eval_budget=None, - error_metrics:Iterable[Union[str,Callable]]='mae'): - """ - GGenerates an evaluation report for a custom protocol defined as a generator function that yields - samples at each iteration. The sequence of samples is processed exhaustively if `eval_budget=None` - or up to the `eval_budget` iterations if specified. - Te report takes the form of a - pandas' `dataframe `_ - in which the rows correspond to different samples, and the columns inform of the true prevalence values, - the estimated prevalence values, and the score obtained by each of the evaluation measures indicated. - - :param model: the model in charge of generating the class prevalence estimations - :param gen_fn: a generator function yielding one sample at each iteration - :param eval_budget: a maximum number of evaluations to run. Set to None (default) for exploring the - entire sequence - :return: a tuple containing two `np.ndarrays` of shape `(m,n,)` with `m` the number of samples - generated. The first one contains the true prevalence values - for the samples generated while the second one contains the prevalence estimations - """ - true_prevs, estim_prevs = gen_prevalence_prediction(model, gen_fn, eval_budget) - return _prevalence_report(true_prevs, estim_prevs, error_metrics) - - -def _prevalence_report( - true_prevs, - estim_prevs, - error_metrics: Iterable[Union[str, Callable]] = 'mae'): +def _prevalence_report(true_prevs, estim_prevs, error_metrics: Iterable[Union[str, Callable]] = 'mae'): if isinstance(error_metrics, str): error_metrics = [error_metrics] - error_names = [e if isinstance(e, str) else e.__name__ for e in error_metrics] error_funcs = [qp.error.from_name(e) if isinstance(e, str) else e for e in error_metrics] assert all(hasattr(e, '__call__') for e in error_funcs), 'invalid error functions' + error_names = [e.__name__ for e in error_funcs] df = pd.DataFrame(columns=['true-prev', 'estim-prev'] + error_names) for true_prev, estim_prev in zip(true_prevs, estim_prevs): @@ -303,145 +129,59 @@ def _prevalence_report( return df -def artificial_prevalence_protocol( +def evaluate( model: BaseQuantifier, - test: LabelledCollection, - sample_size=None, - n_prevpoints=101, - repeats=1, - eval_budget: int = None, - n_jobs=1, - random_seed=42, - error_metric:Union[str,Callable]='mae', + protocol: AbstractProtocol, + error_metric: Union[str, Callable], + aggr_speedup: Union[str, bool] = 'auto', verbose=False): """ - Generates samples according to the Artificial Prevalence Protocol (APP). - The APP consists of exploring a grid of prevalence values containing `n_prevalences` points (e.g., - [0, 0.05, 0.1, 0.15, ..., 1], if `n_prevalences=21`), and generating all valid combinations of - prevalence values for all classes (e.g., for 3 classes, samples with [0, 0, 1], [0, 0.05, 0.95], ..., - [1, 0, 0] prevalence values of size `sample_size` will be considered). The number of samples for each valid - combination of prevalence values is indicated by `repeats`. + Evaluates a quantification model according to a specific sample generation protocol and in terms of one + evaluation metric (error). - :param model: the model in charge of generating the class prevalence estimations - :param test: the test set on which to perform APP - :param sample_size: integer, the size of the samples; if None, then the sample size is - taken from qp.environ['SAMPLE_SIZE'] - :param n_prevpoints: integer, the number of different prevalences to sample (or set to None if eval_budget - is specified; default 101, i.e., steps of 1%) - :param repeats: integer, the number of repetitions for each prevalence (default 1) - :param eval_budget: integer, if specified, sets a ceil on the number of evaluations to perform. For example, if - there are 3 classes, `repeats=1`, and `eval_budget=20`, then `n_prevpoints` will be set to 5, since this - will generate 15 different prevalence vectors ([0, 0, 1], [0, 0.25, 0.75], [0, 0.5, 0.5] ... [1, 0, 0]) and - since setting `n_prevpoints=6` would produce more than 20 evaluations. - :param n_jobs: integer, number of jobs to be run in parallel (default 1) - :param random_seed: integer, allows to replicate the samplings. The seed is local to the method and does not affect - any other random process (default 42) - :param error_metric: a string indicating the name of the error (as defined in :mod:`quapy.error`) or a - callable error function - :param verbose: set to True (default False) for displaying some information on standard output - :return: yields one sample at a time + :param model: a quantifier, instance of :class:`quapy.method.base.BaseQuantifier` + :param protocol: :class:`quapy.protocol.AbstractProtocol`; if this object is also instance of + :class:`quapy.protocol.OnLabelledCollectionProtocol`, then the aggregation speed-up can be run. This is the + protocol in charge of generating the samples in which the model is evaluated. + :param error_metric: a string representing the name(s) of an error function in `qp.error` + (e.g., 'mae'), or a callable function implementing the error function itself. + :param aggr_speedup: whether or not to apply the speed-up. Set to "force" for applying it even if the number of + instances in the original collection on which the protocol acts is larger than the number of instances + in the samples to be generated. Set to True or "auto" (default) for letting QuaPy decide whether it is + convenient or not. Set to False to deactivate. + :param verbose: boolean, show or not information in stdout + :return: if the error metric is not averaged (e.g., 'ae', 'rae'), returns an array of shape `(n_samples,)` with + the error scores for each sample; if the error metric is averaged (e.g., 'mae', 'mrae') then returns + a single float """ if isinstance(error_metric, str): error_metric = qp.error.from_name(error_metric) - - assert hasattr(error_metric, '__call__'), 'invalid error function' - - true_prevs, estim_prevs = artificial_prevalence_prediction( - model, test, sample_size, n_prevpoints, repeats, eval_budget, n_jobs, random_seed, verbose - ) - + true_prevs, estim_prevs = prediction(model, protocol, aggr_speedup=aggr_speedup, verbose=verbose) return error_metric(true_prevs, estim_prevs) -def natural_prevalence_protocol( +def evaluate_on_samples( model: BaseQuantifier, - test: LabelledCollection, - sample_size=None, - repeats=100, - n_jobs=1, - random_seed=42, - error_metric:Union[str,Callable]='mae', + samples: Iterable[qp.data.LabelledCollection], + error_metric: Union[str, Callable], verbose=False): """ - Generates samples according to the Natural Prevalence Protocol (NPP). - The NPP consists of drawing samples uniformly at random, therefore approximately preserving the natural - prevalence of the collection. + Evaluates a quantification model on a given set of samples and in terms of one evaluation metric (error). - :param model: the model in charge of generating the class prevalence estimations - :param test: the test set on which to perform NPP - :param sample_size: integer, the size of the samples; if None, then the sample size is - taken from qp.environ['SAMPLE_SIZE'] - :param repeats: integer, the number of samples to generate - :param n_jobs: integer, number of jobs to be run in parallel (default 1) - :param random_seed: allows to replicate the samplings. The seed is local to the method and does not affect - any other random process (default 42) - :param error_metric: a string indicating the name of the error (as defined in :mod:`quapy.error`) or a - callable error function - :param verbose: if True, shows a progress bar - :return: yields one sample at a time + :param model: a quantifier, instance of :class:`quapy.method.base.BaseQuantifier` + :param samples: a list of samples on which the quantifier is to be evaluated + :param error_metric: a string representing the name(s) of an error function in `qp.error` + (e.g., 'mae'), or a callable function implementing the error function itself. + :param verbose: boolean, show or not information in stdout + :return: if the error metric is not averaged (e.g., 'ae', 'rae'), returns an array of shape `(n_samples,)` with + the error scores for each sample; if the error metric is averaged (e.g., 'mae', 'mrae') then returns + a single float """ - if isinstance(error_metric, str): - error_metric = qp.error.from_name(error_metric) - - assert hasattr(error_metric, '__call__'), 'invalid error function' - - true_prevs, estim_prevs = natural_prevalence_prediction( - model, test, sample_size, repeats, n_jobs, random_seed, verbose - ) - - return error_metric(true_prevs, estim_prevs) + return evaluate(model, IterateProtocol(samples), error_metric, aggr_speedup=False, verbose=verbose) -def evaluate(model: BaseQuantifier, test_samples:Iterable[LabelledCollection], error_metric:Union[str, Callable], n_jobs:int=-1): - """ - Evaluates a model on a sequence of test samples in terms of a given error metric. - - :param model: the model in charge of generating the class prevalence estimations - :param test_samples: an iterable yielding one sample at a time - :param error_metric: a string indicating the name of the error (as defined in :mod:`quapy.error`) or a - callable error function - :param n_jobs: integer, number of jobs to be run in parallel (default 1) - :return: the score obtained using `error_metric` - """ - if isinstance(error_metric, str): - error_metric = qp.error.from_name(error_metric) - scores = qp.util.parallel(_delayed_eval, ((model, Ti, error_metric) for Ti in test_samples), n_jobs=n_jobs) - return np.mean(scores) -def _delayed_eval(args): - model, test, error = args - prev_estim = model.quantify(test.instances) - prev_true = test.prevalence() - return error(prev_true, prev_estim) - - -def _check_num_evals(n_classes, n_prevpoints=None, eval_budget=None, repeats=1, verbose=False): - if n_prevpoints is None and eval_budget is None: - raise ValueError('either n_prevpoints or eval_budget has to be specified') - elif n_prevpoints is None: - assert eval_budget > 0, 'eval_budget must be a positive integer' - n_prevpoints = F.get_nprevpoints_approximation(eval_budget, n_classes, repeats) - eval_computations = F.num_prevalence_combinations(n_prevpoints, n_classes, repeats) - if verbose: - print(f'setting n_prevpoints={n_prevpoints} so that the number of ' - f'evaluations ({eval_computations}) does not exceed the evaluation ' - f'budget ({eval_budget})') - elif eval_budget is None: - eval_computations = F.num_prevalence_combinations(n_prevpoints, n_classes, repeats) - if verbose: - print(f'{eval_computations} evaluations will be performed for each ' - f'combination of hyper-parameters') - else: - eval_computations = F.num_prevalence_combinations(n_prevpoints, n_classes, repeats) - if eval_computations > eval_budget: - n_prevpoints = F.get_nprevpoints_approximation(eval_budget, n_classes, repeats) - new_eval_computations = F.num_prevalence_combinations(n_prevpoints, n_classes, repeats) - if verbose: - print(f'the budget of evaluations would be exceeded with ' - f'n_prevpoints={n_prevpoints}. Chaning to n_prevpoints={n_prevpoints}. This will produce ' - f'{new_eval_computations} evaluation computations for each hyper-parameter combination.') - return n_prevpoints, eval_computations diff --git a/quapy/functional.py b/quapy/functional.py index a8b17f6..a1f0ba2 100644 --- a/quapy/functional.py +++ b/quapy/functional.py @@ -4,37 +4,6 @@ import scipy import numpy as np -def artificial_prevalence_sampling(dimensions, n_prevalences=21, repeat=1, return_constrained_dim=False): - """ - Generates vectors of prevalence values artificially drawn from an exhaustive grid of prevalence values. The - number of prevalence values explored for each dimension depends on `n_prevalences`, so that, if, for example, - `n_prevalences=11` then the prevalence values of the grid are taken from [0, 0.1, 0.2, ..., 0.9, 1]. Only - valid prevalence distributions are returned, i.e., vectors of prevalence values that sum up to 1. For each - valid vector of prevalence values, `repeat` copies are returned. The vector of prevalence values can be - implicit (by setting `return_constrained_dim=False`), meaning that the last dimension (which is constrained - to 1 - sum of the rest) is not returned (note that, quite obviously, in this case the vector does not sum up to 1). - - :param dimensions: the number of classes - :param n_prevalences: the number of equidistant prevalence points to extract from the [0,1] interval for the grid - (default is 21) - :param repeat: number of copies for each valid prevalence vector (default is 1) - :param return_constrained_dim: set to True to return all dimensions, or to False (default) for ommitting the - constrained dimension - :return: a `np.ndarray` of shape `(n, dimensions)` if `return_constrained_dim=True` or of shape `(n, dimensions-1)` - if `return_constrained_dim=False`, where `n` is the number of valid combinations found in the grid multiplied - by `repeat` - """ - s = np.linspace(0., 1., n_prevalences, endpoint=True) - s = [s] * (dimensions - 1) - prevs = [p for p in itertools.product(*s, repeat=1) if sum(p)<=1] - if return_constrained_dim: - prevs = [p+(1-sum(p),) for p in prevs] - prevs = np.asarray(prevs).reshape(len(prevs), -1) - if repeat>1: - prevs = np.repeat(prevs, repeat, axis=0) - return prevs - - def prevalence_linspace(n_prevalences=21, repeats=1, smooth_limits_epsilon=0.01): """ Produces an array of uniformly separated values of prevalence. @@ -70,7 +39,7 @@ def prevalence_from_labels(labels, classes): raise ValueError(f'param labels does not seem to be a ndarray of label predictions') unique, counts = np.unique(labels, return_counts=True) by_class = defaultdict(lambda:0, dict(zip(unique, counts))) - prevalences = np.asarray([by_class[class_] for class_ in classes], dtype=np.float) + prevalences = np.asarray([by_class[class_] for class_ in classes], dtype=float) prevalences /= prevalences.sum() return prevalences @@ -101,7 +70,7 @@ def HellingerDistance(P, Q): The HD for two discrete distributions of `k` bins is defined as: .. math:: - HD(P,Q) = \\frac{ 1 }{ \\sqrt{ 2 } } \\sqrt{ \sum_{i=1}^k ( \\sqrt{p_i} - \\sqrt{q_i} )^2 } + HD(P,Q) = \\frac{ 1 }{ \\sqrt{ 2 } } \\sqrt{ \\sum_{i=1}^k ( \\sqrt{p_i} - \\sqrt{q_i} )^2 } :param P: real-valued array-like of shape `(k,)` representing a discrete distribution :param Q: real-valued array-like of shape `(k,)` representing a discrete distribution @@ -110,6 +79,22 @@ def HellingerDistance(P, Q): return np.sqrt(np.sum((np.sqrt(P) - np.sqrt(Q))**2)) +def TopsoeDistance(P, Q, epsilon=1e-20): + """ + Topsoe distance between two (discretized) distributions `P` and `Q`. + The Topsoe distance for two discrete distributions of `k` bins is defined as: + + .. math:: + Topsoe(P,Q) = \\sum_{i=1}^k \\left( p_i \\log\\left(\\frac{ 2 p_i + \\epsilon }{ p_i+q_i+\\epsilon }\\right) + + q_i \\log\\left(\\frac{ 2 q_i + \\epsilon }{ p_i+q_i+\\epsilon }\\right) \\right) + + :param P: real-valued array-like of shape `(k,)` representing a discrete distribution + :param Q: real-valued array-like of shape `(k,)` representing a discrete distribution + :return: float + """ + return np.sum(P*np.log((2*P+epsilon)/(P+Q+epsilon)) + Q*np.log((2*Q+epsilon)/(P+Q+epsilon))) + + def uniform_prevalence_sampling(n_classes, size=1): """ Implements the `Kraemer algorithm `_ @@ -161,7 +146,6 @@ def adjusted_quantification(prevalence_estim, tpr, fpr, clip=True): .. math:: ACC(p) = \\frac{ p - fpr }{ tpr - fpr } - :param prevalence_estim: float, the estimated value for the positive class :param tpr: float, the true positive rate of the classifier :param fpr: float, the false positive rate of the classifier @@ -209,7 +193,7 @@ def __num_prevalence_combinations_depr(n_prevpoints:int, n_classes:int, n_repeat :param n_prevpoints: integer, number of prevalence points. :param n_repeats: integer, number of repetitions for each prevalence combination :return: The number of possible combinations. For example, if n_classes=2, n_prevpoints=5, n_repeats=1, then the - number of possible combinations are 5, i.e.: [0,1], [0.25,0.75], [0.50,0.50], [0.75,0.25], and [1.0,0.0] + number of possible combinations are 5, i.e.: [0,1], [0.25,0.75], [0.50,0.50], [0.75,0.25], and [1.0,0.0] """ __cache={} def __f(nc,np): @@ -241,7 +225,7 @@ def num_prevalence_combinations(n_prevpoints:int, n_classes:int, n_repeats:int=1 :param n_prevpoints: integer, number of prevalence points. :param n_repeats: integer, number of repetitions for each prevalence combination :return: The number of possible combinations. For example, if n_classes=2, n_prevpoints=5, n_repeats=1, then the - number of possible combinations are 5, i.e.: [0,1], [0.25,0.75], [0.50,0.50], [0.75,0.25], and [1.0,0.0] + number of possible combinations are 5, i.e.: [0,1], [0.25,0.75], [0.50,0.50], [0.75,0.25], and [1.0,0.0] """ N = n_prevpoints-1 C = n_classes @@ -255,7 +239,7 @@ def get_nprevpoints_approximation(combinations_budget:int, n_classes:int, n_repe that the number of valid prevalence values generated as combinations of prevalence points (points in a `n_classes`-dimensional simplex) do not exceed combinations_budget. - :param combinations_budget: integer, maximum number of combinatios allowed + :param combinations_budget: integer, maximum number of combinations allowed :param n_classes: integer, number of classes :param n_repeats: integer, number of repetitions for each prevalence combination :return: the largest number of prevalence points that generate less than combinations_budget valid prevalences @@ -269,3 +253,26 @@ def get_nprevpoints_approximation(combinations_budget:int, n_classes:int, n_repe else: n_prevpoints += 1 + +def check_prevalence_vector(p, raise_exception=False, toleranze=1e-08): + """ + Checks that p is a valid prevalence vector, i.e., that it contains values in [0,1] and that the values sum up to 1. + + :param p: the prevalence vector to check + :return: True if `p` is valid, False otherwise + """ + p = np.asarray(p) + if not all(p>=0): + if raise_exception: + raise ValueError('the prevalence vector contains negative numbers') + return False + if not all(p<=1): + if raise_exception: + raise ValueError('the prevalence vector contains values >1') + return False + if not np.isclose(p.sum(), 1, atol=toleranze): + if raise_exception: + raise ValueError('the prevalence vector does not sum up to 1') + return False + return True + diff --git a/quapy/method/__init__.py b/quapy/method/__init__.py index ddd7b26..39205de 100644 --- a/quapy/method/__init__.py +++ b/quapy/method/__init__.py @@ -3,15 +3,6 @@ from . import base from . import meta from . import non_aggregative -EXPLICIT_LOSS_MINIMIZATION_METHODS = { - aggregative.ELM, - aggregative.SVMQ, - aggregative.SVMAE, - aggregative.SVMKLD, - aggregative.SVMRAE, - aggregative.SVMNKLD -} - AGGREGATIVE_METHODS = { aggregative.CC, aggregative.ACC, @@ -19,12 +10,14 @@ AGGREGATIVE_METHODS = { aggregative.PACC, aggregative.EMQ, aggregative.HDy, + aggregative.DyS, + aggregative.SMM, aggregative.X, aggregative.T50, aggregative.MAX, aggregative.MS, aggregative.MS2, -} | EXPLICIT_LOSS_MINIMIZATION_METHODS +} NON_AGGREGATIVE_METHODS = { diff --git a/quapy/method/aggregative.py b/quapy/method/aggregative.py index 19969c6..3410999 100644 --- a/quapy/method/aggregative.py +++ b/quapy/method/aggregative.py @@ -1,20 +1,18 @@ from abc import abstractmethod from copy import deepcopy -from typing import Union - +from typing import Callable, Union import numpy as np -from joblib import Parallel, delayed +from scipy import optimize from sklearn.base import BaseEstimator from sklearn.calibration import CalibratedClassifierCV from sklearn.metrics import confusion_matrix -from sklearn.model_selection import StratifiedKFold -from tqdm import tqdm - +from sklearn.model_selection import cross_val_predict import quapy as qp import quapy.functional as F +from quapy.classification.calibration import NBVSCalibration, BCTSCalibration, TSCalibration, VSCalibration from quapy.classification.svmperf import SVMperf from quapy.data import LabelledCollection -from quapy.method.base import BaseQuantifier, BinaryQuantifier +from quapy.method.base import BaseQuantifier, BinaryQuantifier, OneVsAllGeneric # Abstract classes @@ -23,50 +21,52 @@ from quapy.method.base import BaseQuantifier, BinaryQuantifier class AggregativeQuantifier(BaseQuantifier): """ Abstract class for quantification methods that base their estimations on the aggregation of classification - results. Aggregative Quantifiers thus implement a :meth:`classify` method and maintain a :attr:`learner` attribute. - Subclasses of this abstract class must implement the method :meth:`aggregate` which computes the aggregation - of label predictions. The method :meth:`quantify` comes with a default implementation based on - :meth:`classify` and :meth:`aggregate`. + results. Aggregative Quantifiers thus implement a :meth:`classify` method and maintain a :attr:`classifier` + attribute. Subclasses of this abstract class must implement the method :meth:`aggregate` which computes the + aggregation of label predictions. The method :meth:`quantify` comes with a default implementation based on + :meth:`classify` and :meth:`aggregate`. """ @abstractmethod - def fit(self, data: LabelledCollection, fit_learner=True): + def fit(self, data: LabelledCollection, fit_classifier=True): """ Trains the aggregative quantifier :param data: a :class:`quapy.data.base.LabelledCollection` consisting of the training data - :param fit_learner: whether or not to train the learner (default is True). Set to False if the + :param fit_classifier: whether or not to train the learner (default is True). Set to False if the learner has been trained outside the quantifier. :return: self """ ... @property - def learner(self): + def classifier(self): """ Gives access to the classifier :return: the classifier (typically an sklearn's Estimator) """ - return self.learner_ + return self.classifier_ - @learner.setter - def learner(self, classifier): + @classifier.setter + def classifier(self, classifier): """ Setter for the classifier :param classifier: the classifier """ - self.learner_ = classifier + self.classifier_ = classifier def classify(self, instances): """ - Provides the label predictions for the given instances. + Provides the label predictions for the given instances. The predictions should respect the format expected by + :meth:`aggregate`, i.e., posterior probabilities for probabilistic quantifiers, or crisp predictions for + non-probabilistic quantifiers :param instances: array-like :return: np.ndarray of shape `(n_instances,)` with label predictions """ - return self.learner.predict(instances) + return self.classifier.predict(instances) def quantify(self, instances): """ @@ -74,7 +74,7 @@ class AggregativeQuantifier(BaseQuantifier): by the classifier. :param instances: array-like - :return: `np.ndarray` of shape `(self.n_classes_,)` with class prevalence estimates. + :return: `np.ndarray` of shape `(n_classes)` with class prevalence estimates. """ classif_predictions = self.classify(instances) return self.aggregate(classif_predictions) @@ -85,29 +85,10 @@ class AggregativeQuantifier(BaseQuantifier): Implements the aggregation of label predictions. :param classif_predictions: `np.ndarray` of label predictions - :return: `np.ndarray` of shape `(self.n_classes_,)` with class prevalence estimates. + :return: `np.ndarray` of shape `(n_classes,)` with class prevalence estimates. """ ... - def get_params(self, deep=True): - """ - Return the current parameters of the quantifier. - - :param deep: for compatibility with sklearn - :return: a dictionary of param-value pairs - """ - - return self.learner.get_params() - - def set_params(self, **parameters): - """ - Set the parameters of the quantifier. - - :param parameters: dictionary of param-value pairs - """ - - self.learner.set_params(**parameters) - @property def classes_(self): """ @@ -116,17 +97,7 @@ class AggregativeQuantifier(BaseQuantifier): :return: array-like """ - return self.learner.classes_ - - @property - def aggregative(self): - """ - Returns True, indicating the quantifier is of type aggregative. - - :return: True - """ - - return True + return self.classifier.classes_ class AggregativeProbabilisticQuantifier(AggregativeQuantifier): @@ -137,39 +108,31 @@ class AggregativeProbabilisticQuantifier(AggregativeQuantifier): probabilities. """ - def posterior_probabilities(self, instances): - return self.learner.predict_proba(instances) - - def predict_proba(self, instances): - return self.posterior_probabilities(instances) - - def quantify(self, instances): - classif_posteriors = self.posterior_probabilities(instances) - return self.aggregate(classif_posteriors) - - def set_params(self, **parameters): - if isinstance(self.learner, CalibratedClassifierCV): - parameters = {'base_estimator__' + k: v for k, v in parameters.items()} - self.learner.set_params(**parameters) - - @property - def probabilistic(self): - return True + def classify(self, instances): + return self.classifier.predict_proba(instances) # Helper # ------------------------------------ -def _training_helper(learner, +def _ensure_probabilistic(classifier): + if not hasattr(classifier, 'predict_proba'): + print(f'The learner {classifier.__class__.__name__} does not seem to be probabilistic. ' + f'The learner will be calibrated.') + classifier = CalibratedClassifierCV(classifier, cv=5) + return classifier + + +def _training_helper(classifier, data: LabelledCollection, - fit_learner: bool = True, + fit_classifier: bool = True, ensure_probabilistic=False, val_split: Union[LabelledCollection, float] = None): """ Training procedure common to all Aggregative Quantifiers. - :param learner: the learner to be fit + :param classifier: the learner to be fit :param data: the data on which to fit the learner. If requested, the data will be split before fitting the learner. - :param fit_learner: whether or not to fit the learner (if False, then bypasses any action) + :param fit_classifier: whether or not to fit the learner (if False, then bypasses any action) :param ensure_probabilistic: if True, guarantees that the resulting classifier implements predict_proba (if the learner is not probabilistic, then a CalibratedCV instance of it is trained) :param val_split: if specified as a float, indicates the proportion of training instances that will define the @@ -178,12 +141,9 @@ def _training_helper(learner, :return: the learner trained on the training set, and the unused data (a _LabelledCollection_ if train_val_split>0 or None otherwise) to be used as a validation set for any subsequent parameter fitting """ - if fit_learner: + if fit_classifier: if ensure_probabilistic: - if not hasattr(learner, 'predict_proba'): - print(f'The learner {learner.__class__.__name__} does not seem to be probabilistic. ' - f'The learner will be calibrated.') - learner = CalibratedClassifierCV(learner, cv=5) + classifier = _ensure_probabilistic(classifier) if val_split is not None: if isinstance(val_split, float): if not (0 < val_split < 1): @@ -199,19 +159,58 @@ def _training_helper(learner, else: train, unused = data, None - if isinstance(learner, BaseQuantifier): - learner.fit(train) + if isinstance(classifier, BaseQuantifier): + classifier.fit(train) else: - learner.fit(*train.Xy) + classifier.fit(*train.Xy) else: if ensure_probabilistic: - if not hasattr(learner, 'predict_proba'): - raise AssertionError('error: the learner cannot be calibrated since fit_learner is set to False') + if not hasattr(classifier, 'predict_proba'): + raise AssertionError('error: the learner cannot be calibrated since fit_classifier is set to False') unused = None if isinstance(val_split, LabelledCollection): unused = val_split - return learner, unused + return classifier, unused + + +def cross_generate_predictions( + data, + classifier, + val_split, + probabilistic, + fit_classifier, + n_jobs +): + + n_jobs = qp._get_njobs(n_jobs) + + if isinstance(val_split, int): + assert fit_classifier == True, \ + 'the parameters for the adjustment cannot be estimated with kFCV with fit_classifier=False' + + if probabilistic: + classifier = _ensure_probabilistic(classifier) + predict = 'predict_proba' + else: + predict = 'predict' + y_pred = cross_val_predict(classifier, *data.Xy, cv=val_split, n_jobs=n_jobs, method=predict) + class_count = data.counts() + + # fit the learner on all data + classifier.fit(*data.Xy) + y = data.y + classes = data.classes_ + else: + classifier, val_data = _training_helper( + classifier, data, fit_classifier, ensure_probabilistic=probabilistic, val_split=val_split + ) + y_pred = classifier.predict_proba(val_data.instances) if probabilistic else classifier.predict(val_data.instances) + y = val_data.labels + classes = val_data.classes_ + class_count = val_data.counts() + + return classifier, y, y_pred, classes, class_count # Methods @@ -221,22 +220,22 @@ class CC(AggregativeQuantifier): The most basic Quantification method. One that simply classifies all instances and counts how many have been attributed to each of the classes in order to compute class prevalence estimates. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier """ - def __init__(self, learner: BaseEstimator): - self.learner = learner + def __init__(self, classifier: BaseEstimator): + self.classifier = classifier - def fit(self, data: LabelledCollection, fit_learner=True): + def fit(self, data: LabelledCollection, fit_classifier=True): """ - Trains the Classify & Count method unless `fit_learner` is False, in which case, the classifier is assumed to + Trains the Classify & Count method unless `fit_classifier` is False, in which case, the classifier is assumed to be already fit and there is nothing else to do. :param data: a :class:`quapy.data.base.LabelledCollection` consisting of the training data - :param fit_learner: if False, the classifier is assumed to be fit + :param fit_classifier: if False, the classifier is assumed to be fit :return: self """ - self.learner, _ = _training_helper(self.learner, data, fit_learner) + self.classifier, _ = _training_helper(self.classifier, data, fit_classifier) return self def aggregate(self, classif_predictions: np.ndarray): @@ -244,7 +243,7 @@ class CC(AggregativeQuantifier): Computes class prevalence estimates by counting the prevalence of each of the predicted labels. :param classif_predictions: array-like with label predictions - :return: `np.ndarray` of shape `(self.n_classes_,)` with class prevalence estimates. + :return: `np.ndarray` of shape `(n_classes,)` with class prevalence estimates. """ return F.prevalence_from_labels(classif_predictions, self.classes_) @@ -255,7 +254,7 @@ class ACC(AggregativeQuantifier): the "adjusted" variant of :class:`CC`, that corrects the predictions of CC according to the `misclassification rates`. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier :param val_split: indicates the proportion of data to be used as a stratified held-out validation set in which the misclassification rates are to be estimated. This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of @@ -264,54 +263,33 @@ class ACC(AggregativeQuantifier): :class:`quapy.data.base.LabelledCollection` (the split itself). """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - self.learner = learner + def __init__(self, classifier: BaseEstimator, val_split=0.4, n_jobs=None): + self.classifier = classifier self.val_split = val_split + self.n_jobs = qp._get_njobs(n_jobs) - def fit(self, data: LabelledCollection, fit_learner=True, val_split: Union[float, int, LabelledCollection] = None): + def fit(self, data: LabelledCollection, fit_classifier=True, val_split: Union[float, int, LabelledCollection] = None): """ Trains a ACC quantifier. :param data: the training set - :param fit_learner: set to False to bypass the training (the learner is assumed to be already fit) + :param fit_classifier: set to False to bypass the training (the learner is assumed to be already fit) :param val_split: either a float in (0,1) indicating the proportion of training instances to use for validation (e.g., 0.3 for using 30% of the training set as validation data), or a LabelledCollection indicating the validation set itself, or an int indicating the number `k` of folds to be used in `k`-fold cross validation to estimate the parameters :return: self """ + if val_split is None: val_split = self.val_split - if isinstance(val_split, int): - assert fit_learner == True, \ - 'the parameters for the adjustment cannot be estimated with kFCV with fit_learner=False' - # kFCV estimation of parameters - y, y_ = [], [] - kfcv = StratifiedKFold(n_splits=val_split) - pbar = tqdm(kfcv.split(*data.Xy), total=val_split) - for k, (training_idx, validation_idx) in enumerate(pbar): - pbar.set_description(f'{self.__class__.__name__} fitting fold {k}') - training = data.sampling_from_index(training_idx) - validation = data.sampling_from_index(validation_idx) - learner, val_data = _training_helper(self.learner, training, fit_learner, val_split=validation) - y_.append(learner.predict(val_data.instances)) - y.append(val_data.labels) - y = np.concatenate(y) - y_ = np.concatenate(y_) - class_count = data.counts() + self.classifier, y, y_, classes, class_count = cross_generate_predictions( + data, self.classifier, val_split, probabilistic=False, fit_classifier=fit_classifier, n_jobs=self.n_jobs + ) - # fit the learner on all data - self.learner, _ = _training_helper(self.learner, data, fit_learner, val_split=None) - - else: - self.learner, val_data = _training_helper(self.learner, data, fit_learner, val_split=val_split) - y_ = self.learner.predict(val_data.instances) - y = val_data.labels - - self.cc = CC(self.learner) - - self.Pte_cond_estim_ = self.getPteCondEstim(data.classes_, y, y_) + self.cc = CC(self.classifier) + self.Pte_cond_estim_ = self.getPteCondEstim(self.classifier.classes_, y, y_) return self @@ -320,7 +298,7 @@ class ACC(AggregativeQuantifier): # estimate the matrix with entry (i,j) being the estimate of P(yi|yj), that is, the probability that a # document that belongs to yj ends up being classified as belonging to yi conf = confusion_matrix(y, y_, labels=classes).T - conf = conf.astype(np.float) + conf = conf.astype(float) class_counts = conf.sum(axis=0) for i, _ in enumerate(classes): if class_counts[i] == 0: @@ -363,14 +341,14 @@ class PCC(AggregativeProbabilisticQuantifier): `Probabilistic Classify & Count `_, the probabilistic variant of CC that relies on the posterior probabilities returned by a probabilistic classifier. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier """ - def __init__(self, learner: BaseEstimator): - self.learner = learner + def __init__(self, classifier: BaseEstimator): + self.classifier = classifier - def fit(self, data: LabelledCollection, fit_learner=True): - self.learner, _ = _training_helper(self.learner, data, fit_learner, ensure_probabilistic=True) + def fit(self, data: LabelledCollection, fit_classifier=True): + self.classifier, _ = _training_helper(self.classifier, data, fit_classifier, ensure_probabilistic=True) return self def aggregate(self, classif_posteriors): @@ -382,74 +360,42 @@ class PACC(AggregativeProbabilisticQuantifier): `Probabilistic Adjusted Classify & Count `_, the probabilistic variant of ACC that relies on the posterior probabilities returned by a probabilistic classifier. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier :param val_split: indicates the proportion of data to be used as a stratified held-out validation set in which the misclassification rates are to be estimated. This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of validation data, or as an integer, indicating that the misclassification rates should be estimated via `k`-fold cross validation (this integer stands for the number of folds `k`), or as a :class:`quapy.data.base.LabelledCollection` (the split itself). + :param n_jobs: number of parallel workers """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - self.learner = learner + def __init__(self, classifier: BaseEstimator, val_split=0.4, n_jobs=None): + self.classifier = classifier self.val_split = val_split + self.n_jobs = qp._get_njobs(n_jobs) - def fit(self, data: LabelledCollection, fit_learner=True, val_split: Union[float, int, LabelledCollection] = None): + def fit(self, data: LabelledCollection, fit_classifier=True, val_split: Union[float, int, LabelledCollection] = None): """ Trains a PACC quantifier. :param data: the training set - :param fit_learner: set to False to bypass the training (the learner is assumed to be already fit) + :param fit_classifier: set to False to bypass the training (the learner is assumed to be already fit) :param val_split: either a float in (0,1) indicating the proportion of training instances to use for validation (e.g., 0.3 for using 30% of the training set as validation data), or a LabelledCollection indicating the validation set itself, or an int indicating the number k of folds to be used in kFCV to estimate the parameters :return: self """ + if val_split is None: val_split = self.val_split - if isinstance(val_split, int): - assert fit_learner == True, \ - 'the parameters for the adjustment cannot be estimated with kFCV with fit_learner=False' - # kFCV estimation of parameters - y, y_ = [], [] - kfcv = StratifiedKFold(n_splits=val_split) - pbar = tqdm(kfcv.split(*data.Xy), total=val_split) - for k, (training_idx, validation_idx) in enumerate(pbar): - pbar.set_description(f'{self.__class__.__name__} fitting fold {k}') - training = data.sampling_from_index(training_idx) - validation = data.sampling_from_index(validation_idx) - learner, val_data = _training_helper( - self.learner, training, fit_learner, ensure_probabilistic=True, val_split=validation) - y_.append(learner.predict_proba(val_data.instances)) - y.append(val_data.labels) - - y = np.concatenate(y) - y_ = np.vstack(y_) - - # fit the learner on all data - self.learner, _ = _training_helper(self.learner, data, fit_learner, ensure_probabilistic=True, - val_split=None) - classes = data.classes_ - - else: - self.learner, val_data = _training_helper( - self.learner, data, fit_learner, ensure_probabilistic=True, val_split=val_split) - y_ = self.learner.predict_proba(val_data.instances) - y = val_data.labels - classes = val_data.classes_ - - self.pcc = PCC(self.learner) - - # estimate the matrix with entry (i,j) being the estimate of P(yi|yj), that is, the probability that a - # document that belongs to yj ends up being classified as belonging to yi - n_classes = len(classes) - confusion = np.empty(shape=(n_classes, n_classes)) - for i, class_ in enumerate(classes): - confusion[i] = y_[y == class_].mean(axis=0) + self.classifier, y, y_, classes, class_count = cross_generate_predictions( + data, self.classifier, val_split, probabilistic=True, fit_classifier=fit_classifier, n_jobs=self.n_jobs + ) + self.pcc = PCC(self.classifier) self.Pte_cond_estim_ = self.getPteCondEstim(classes, y, y_) return self @@ -482,54 +428,74 @@ class EMQ(AggregativeProbabilisticQuantifier): EMQ consists of using the well-known `Expectation Maximization algorithm` to iteratively update the posterior probabilities generated by a probabilistic classifier and the class prevalence estimates obtained via maximum-likelihood estimation, in a mutually recursive way, until convergence. - The `transform_prior` callback allows you to introduce ad-hoc regularizations which are not part of the - original EMQ algorithm. This callback can, for instance, enhance or diminish small class prevalences if - sparse or dense solutions should be promoted. - The original method is described in: - Saerens, M., Latinne, P., and Decaestecker, C. (2002). - Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. - Neural Computation, 14(1): 21–41. - - :param learner: a sklearn's Estimator that generates a classifier - :param transform_prior: an optional function :math:`R^c -> R^c` that transforms each intermediate estimate + :param classifier: a sklearn's Estimator that generates a classifier + :param exact_train_prev: set to True (default) for using, as the initial observation, the true training prevalence; + or set to False for computing the training prevalence as an estimate, akin to PCC, i.e., as the expected + value of the posterior probabilities of the training instances as suggested in + `Alexandari et al. paper `_: + :param recalib: a string indicating the method of recalibration. Available choices include "nbvs" (No-Bias Vector + Scaling), "bcts" (Bias-Corrected Temperature Scaling), "ts" (Temperature Scaling), and "vs" (Vector Scaling). + The default value is None, indicating no recalibration. """ MAX_ITER = 1000 EPSILON = 1e-4 - def __init__(self, learner: BaseEstimator, transform_prior=None): - self.learner = learner - self.transform_prior = transform_prior + def __init__(self, classifier: BaseEstimator, exact_train_prev=True, recalib=None): + self.classifier = classifier + self.exact_train_prev = exact_train_prev + self.recalib = recalib - def fit(self, data: LabelledCollection, fit_learner=True): - self.learner, _ = _training_helper(self.learner, data, fit_learner, ensure_probabilistic=True) - self.train_prevalence = F.prevalence_from_labels(data.labels, self.classes_) + def fit(self, data: LabelledCollection, fit_classifier=True): + if self.recalib is not None: + if self.recalib == 'nbvs': + self.classifier = NBVSCalibration(self.classifier) + elif self.recalib == 'bcts': + self.classifier = BCTSCalibration(self.classifier) + elif self.recalib == 'ts': + self.classifier = TSCalibration(self.classifier) + elif self.recalib == 'vs': + self.classifier = VSCalibration(self.classifier) + elif self.recalib == 'platt': + self.classifier = CalibratedClassifierCV(self.classifier, ensemble=False) + else: + raise ValueError('invalid param argument for recalibration method; available ones are ' + '"nbvs", "bcts", "ts", and "vs".') + self.classifier, _ = _training_helper(self.classifier, data, fit_classifier, ensure_probabilistic=True) + if self.exact_train_prev: + self.train_prevalence = F.prevalence_from_labels(data.labels, self.classes_) + else: + self.train_prevalence = qp.model_selection.cross_val_predict( + quantifier=PCC(deepcopy(self.classifier)), + data=data, + nfolds=3, + random_state=0 + ) return self def aggregate(self, classif_posteriors, epsilon=EPSILON): - priors, posteriors = self.EM(self.train_prevalence, classif_posteriors, epsilon, self.transform_prior) + priors, posteriors = self.EM(self.train_prevalence, classif_posteriors, epsilon) return priors def predict_proba(self, instances, epsilon=EPSILON): - classif_posteriors = self.learner.predict_proba(instances) - priors, posteriors = self.EM(self.train_prevalence, classif_posteriors, epsilon, self.transform_prior) + classif_posteriors = self.classifier.predict_proba(instances) + priors, posteriors = self.EM(self.train_prevalence, classif_posteriors, epsilon) return posteriors @classmethod - def EM(cls, tr_prev, posterior_probabilities, epsilon=EPSILON, transform_prior=None): + def EM(cls, tr_prev, posterior_probabilities, epsilon=EPSILON): """ Computes the `Expectation Maximization` routine. + :param tr_prev: array-like, the training prevalence :param posterior_probabilities: `np.ndarray` of shape `(n_instances, n_classes,)` with the posterior probabilities :param epsilon: float, the threshold different between two consecutive iterations to reach before stopping the loop - :param transform_prior: an optional function :math:`R^c -> R^c` that transforms each intermediate estimate :return: a tuple with the estimated prevalence values (shape `(n_classes,)`) and the corrected posterior probabilities (shape `(n_instances, n_classes,)`) """ - Px = posterior_probabilities Ptr = np.copy(tr_prev) qs = np.copy(Ptr) # qs (the running estimate) is initialized as the training prevalence @@ -550,17 +516,12 @@ class EMQ(AggregativeProbabilisticQuantifier): qs_prev_ = qs s += 1 - # transformation of intermediate estimates - if transform_prior is not None and not converged: - qs = transform_prior(qs) - if not converged: print('[warning] the method has reached the maximum number of iterations; it might have not converged') return qs, ps - class HDy(AggregativeProbabilisticQuantifier, BinaryQuantifier): """ `Hellinger Distance y `_ (HDy). @@ -571,21 +532,21 @@ class HDy(AggregativeProbabilisticQuantifier, BinaryQuantifier): class-conditional distributions of the posterior probabilities returned for the positive and negative validation examples, respectively. The parameters of the mixture thus represent the estimates of the class prevalence values. - :param learner: a sklearn's Estimator that generates a binary classifier + :param classifier: a sklearn's Estimator that generates a binary classifier :param val_split: a float in range (0,1) indicating the proportion of data to be used as a stratified held-out validation distribution, or a :class:`quapy.data.base.LabelledCollection` (the split itself). """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - self.learner = learner + def __init__(self, classifier: BaseEstimator, val_split=0.4): + self.classifier = classifier self.val_split = val_split - def fit(self, data: LabelledCollection, fit_learner=True, val_split: Union[float, LabelledCollection] = None): + def fit(self, data: LabelledCollection, fit_classifier=True, val_split: Union[float, LabelledCollection] = None): """ Trains a HDy quantifier. :param data: the training set - :param fit_learner: set to False to bypass the training (the learner is assumed to be already fit) + :param fit_classifier: set to False to bypass the training (the learner is assumed to be already fit) :param val_split: either a float in (0,1) indicating the proportion of training instances to use for validation (e.g., 0.3 for using 30% of the training set as validation data), or a :class:`quapy.data.base.LabelledCollection` indicating the validation set itself @@ -595,11 +556,11 @@ class HDy(AggregativeProbabilisticQuantifier, BinaryQuantifier): val_split = self.val_split self._check_binary(data, self.__class__.__name__) - self.learner, validation = _training_helper( - self.learner, data, fit_learner, ensure_probabilistic=True, val_split=val_split) - Px = self.posterior_probabilities(validation.instances)[:, 1] # takes only the P(y=+1|x) - self.Pxy1 = Px[validation.labels == self.learner.classes_[1]] - self.Pxy0 = Px[validation.labels == self.learner.classes_[0]] + self.classifier, validation = _training_helper( + self.classifier, data, fit_classifier, ensure_probabilistic=True, val_split=val_split) + Px = self.classify(validation.instances)[:, 1] # takes only the P(y=+1|x) + self.Pxy1 = Px[validation.labels == self.classifier.classes_[1]] + self.Pxy0 = Px[validation.labels == self.classifier.classes_[0]] # pre-compute the histogram for positive and negative examples self.bins = np.linspace(10, 110, 11, dtype=int) # [10, 20, 30, ..., 100, 110] self.Pxy1_density = {bins: np.histogram(self.Pxy1, bins=bins, range=(0, 1), density=True)[0] for bins in @@ -637,120 +598,378 @@ class HDy(AggregativeProbabilisticQuantifier, BinaryQuantifier): return np.asarray([1 - class1_prev, class1_prev]) -class ELM(AggregativeQuantifier, BinaryQuantifier): +def _get_divergence(divergence: Union[str, Callable]): + if isinstance(divergence, str): + if divergence=='HD': + return F.HellingerDistance + elif divergence=='topsoe': + return F.TopsoeDistance + else: + raise ValueError(f'unknown divergence {divergence}') + elif callable(divergence): + return divergence + else: + raise ValueError(f'argument "divergence" not understood; use a str or a callable function') + + +class DyS(AggregativeProbabilisticQuantifier, BinaryQuantifier): """ - Class of Explicit Loss Minimization (ELM) quantifiers. + `DyS framework `_ (DyS). + DyS is a generalization of HDy method, using a Ternary Search in order to find the prevalence that + minimizes the distance between distributions. + Details for the ternary search have been got from + + :param classifier: a sklearn's Estimator that generates a binary classifier + :param val_split: a float in range (0,1) indicating the proportion of data to be used as a stratified held-out + validation distribution, or a :class:`quapy.data.base.LabelledCollection` (the split itself). + :param n_bins: an int with the number of bins to use to compute the histograms. + :param divergence: a str indicating the name of divergence (currently supported ones are "HD" or "topsoe"), or a + callable function computes the divergence between two distributions (two equally sized arrays). + :param tol: a float with the tolerance for the ternary search algorithm. + """ + + def __init__(self, classifier: BaseEstimator, val_split=0.4, n_bins=8, divergence: Union[str, Callable]= 'HD', tol=1e-05): + self.classifier = classifier + self.val_split = val_split + self.tol = tol + self.divergence = divergence + self.n_bins = n_bins + + def _ternary_search(self, f, left, right, tol): + """ + Find maximum of unimodal function f() within [left, right] + """ + while abs(right - left) >= tol: + left_third = left + (right - left) / 3 + right_third = right - (right - left) / 3 + + if f(left_third) > f(right_third): + left = left_third + else: + right = right_third + + # Left and right are the current bounds; the maximum is between them + return (left + right) / 2 + + def fit(self, data: LabelledCollection, fit_classifier=True, val_split: Union[float, LabelledCollection] = None): + if val_split is None: + val_split = self.val_split + + self._check_binary(data, self.__class__.__name__) + self.classifier, validation = _training_helper( + self.classifier, data, fit_classifier, ensure_probabilistic=True, val_split=val_split) + Px = self.classify(validation.instances)[:, 1] # takes only the P(y=+1|x) + self.Pxy1 = Px[validation.labels == self.classifier.classes_[1]] + self.Pxy0 = Px[validation.labels == self.classifier.classes_[0]] + self.Pxy1_density = np.histogram(self.Pxy1, bins=self.n_bins, range=(0, 1), density=True)[0] + self.Pxy0_density = np.histogram(self.Pxy0, bins=self.n_bins, range=(0, 1), density=True)[0] + return self + + def aggregate(self, classif_posteriors): + Px = classif_posteriors[:, 1] # takes only the P(y=+1|x) + + Px_test = np.histogram(Px, bins=self.n_bins, range=(0, 1), density=True)[0] + divergence = _get_divergence(self.divergence) + + def distribution_distance(prev): + Px_train = prev * self.Pxy1_density + (1 - prev) * self.Pxy0_density + return divergence(Px_train, Px_test) + + class1_prev = self._ternary_search(f=distribution_distance, left=0, right=1, tol=self.tol) + return np.asarray([1 - class1_prev, class1_prev]) + + +class SMM(AggregativeProbabilisticQuantifier, BinaryQuantifier): + """ + `SMM method `_ (SMM). + SMM is a simplification of matching distribution methods where the representation of the examples + is created using the mean instead of a histogram. + + :param classifier: a sklearn's Estimator that generates a binary classifier. + :param val_split: a float in range (0,1) indicating the proportion of data to be used as a stratified held-out + validation distribution, or a :class:`quapy.data.base.LabelledCollection` (the split itself). + """ + + def __init__(self, classifier: BaseEstimator, val_split=0.4): + self.classifier = classifier + self.val_split = val_split + + def fit(self, data: LabelledCollection, fit_classifier=True, val_split: Union[float, LabelledCollection] = None): + if val_split is None: + val_split = self.val_split + + self._check_binary(data, self.__class__.__name__) + self.classifier, validation = _training_helper( + self.classifier, data, fit_classifier, ensure_probabilistic=True, val_split=val_split) + Px = self.classify(validation.instances)[:, 1] # takes only the P(y=+1|x) + self.Pxy1 = Px[validation.labels == self.classifier.classes_[1]] + self.Pxy0 = Px[validation.labels == self.classifier.classes_[0]] + self.Pxy1_mean = np.mean(self.Pxy1) + self.Pxy0_mean = np.mean(self.Pxy0) + return self + + def aggregate(self, classif_posteriors): + Px = classif_posteriors[:, 1] # takes only the P(y=+1|x) + Px_mean = np.mean(Px) + + class1_prev = (Px_mean - self.Pxy0_mean)/(self.Pxy1_mean - self.Pxy0_mean) + class1_prev = np.clip(class1_prev, 0, 1) + + return np.asarray([1 - class1_prev, class1_prev]) + + +class DistributionMatching(AggregativeProbabilisticQuantifier): + """ + Generic Distribution Matching quantifier for binary or multiclass quantification. + This implementation takes the number of bins, the divergence, and the possibility to work on CDF as hyperparameters. + + :param classifier: a `sklearn`'s Estimator that generates a probabilistic classifier + :param val_split: indicates the proportion of data to be used as a stratified held-out validation set to model the + validation distribution. + This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of + validation data, or as an integer, indicating that the validation distribution should be estimated via + `k`-fold cross validation (this integer stands for the number of folds `k`), or as a + :class:`quapy.data.base.LabelledCollection` (the split itself). + :param nbins: number of bins used to discretize the distributions (default 8) + :param divergence: a string representing a divergence measure (currently, "HD" and "topsoe" are implemented) + or a callable function taking two ndarrays of the same dimension as input (default "HD", meaning Hellinger + Distance) + :param cdf: whether or not to use CDF instead of PDF (default False) + :param n_jobs: number of parallel workers (default None) + """ + + def __init__(self, classifier, val_split=0.4, nbins=8, divergence: Union[str, Callable]='HD', cdf=False, n_jobs=None): + self.classifier = classifier + self.val_split = val_split + self.nbins = nbins + self.divergence = divergence + self.cdf = cdf + self.n_jobs = n_jobs + + def __get_distributions(self, posteriors): + histograms = [] + post_dims = posteriors.shape[1] + if post_dims == 2: + # in binary quantification we can use only one class, since the other one is its complement + post_dims = 1 + for dim in range(post_dims): + hist = np.histogram(posteriors[:, dim], bins=self.nbins, range=(0, 1))[0] + histograms.append(hist) + + counts = np.vstack(histograms) + distributions = counts/counts.sum(axis=1)[:,np.newaxis] + if self.cdf: + distributions = np.cumsum(distributions, axis=1) + return distributions + + def fit(self, data: LabelledCollection, fit_classifier=True, val_split: Union[float, LabelledCollection] = None): + """ + Trains the classifier (if requested) and generates the validation distributions out of the training data. + The validation distributions have shape `(n, ch, nbins)`, with `n` the number of classes, `ch` the number of + channels, and `nbins` the number of bins. In particular, let `V` be the validation distributions; `di=V[i]` + are the distributions obtained from training data labelled with class `i`; `dij = di[j]` is the discrete + distribution of posterior probabilities `P(Y=j|X=x)` for training data labelled with class `i`, and `dij[k]` + is the fraction of instances with a value in the `k`-th bin. + + :param data: the training set + :param fit_classifier: set to False to bypass the training (the learner is assumed to be already fit) + :param val_split: either a float in (0,1) indicating the proportion of training instances to use for + validation (e.g., 0.3 for using 30% of the training set as validation data), or a LabelledCollection + indicating the validation set itself, or an int indicating the number k of folds to be used in kFCV + to estimate the parameters + """ + if val_split is None: + val_split = self.val_split + + self.classifier, y, posteriors, classes, class_count = cross_generate_predictions( + data, self.classifier, val_split, probabilistic=True, fit_classifier=fit_classifier, n_jobs=self.n_jobs + ) + + self.validation_distribution = np.asarray( + [self.__get_distributions(posteriors[y==cat]) for cat in range(data.n_classes)] + ) + + return self + + def aggregate(self, posteriors: np.ndarray): + """ + Searches for the mixture model parameter (the sought prevalence values) that yields a validation distribution + (the mixture) that best matches the test distribution, in terms of the divergence measure of choice. + In the multiclass case, with `n` the number of classes, the test and mixture distributions contain + `n` channels (proper distributions of binned posterior probabilities), on which the divergence is computed + independently. The matching is computed as an average of the divergence across all channels. + + :param instances: instances in the sample + :return: a vector of class prevalence estimates + """ + test_distribution = self.__get_distributions(posteriors) + divergence = _get_divergence(self.divergence) + n_classes, n_channels, nbins = self.validation_distribution.shape + def match(prev): + prev = np.expand_dims(prev, axis=0) + mixture_distribution = (prev @ self.validation_distribution.reshape(n_classes,-1)).reshape(n_channels, -1) + divs = [divergence(test_distribution[ch], mixture_distribution[ch]) for ch in range(n_channels)] + return np.mean(divs) + + # the initial point is set as the uniform distribution + uniform_distribution = np.full(fill_value=1 / n_classes, shape=(n_classes,)) + + # solutions are bounded to those contained in the unit-simplex + bounds = tuple((0, 1) for x in range(n_classes)) # values in [0,1] + constraints = ({'type': 'eq', 'fun': lambda x: 1 - sum(x)}) # values summing up to 1 + r = optimize.minimize(match, x0=uniform_distribution, method='SLSQP', bounds=bounds, constraints=constraints) + return r.x + + +def newELM(svmperf_base=None, loss='01', C=1): + """ + Explicit Loss Minimization (ELM) quantifiers. Quantifiers based on ELM represent a family of methods based on structured output learning; these quantifiers rely on classifiers that have been optimized using a quantification-oriented loss measure. This implementation relies on `Joachims’ SVM perf `_ structured output learning algorithm, which has to be installed and patched for the purpose (see this `script `_). + This function equivalent to: - :param svmperf_base: path to the folder containing the binary files of `SVM perf` + >>> CC(SVMperf(svmperf_base, loss, C)) + + :param svmperf_base: path to the folder containing the binary files of `SVM perf`; if set to None (default) + this path will be obtained from qp.environ['SVMPERF_HOME'] :param loss: the loss to optimize (see :attr:`quapy.classification.svmperf.SVMperf.valid_losses`) - :param kwargs: rest of SVM perf's parameters + :param C: trade-off between training error and margin (default 0.01) + :return: returns an instance of CC set to work with SVMperf (with loss and C set properly) as the + underlying classifier """ - - def __init__(self, svmperf_base=None, loss='01', **kwargs): - self.svmperf_base = svmperf_base if svmperf_base is not None else qp.environ['SVMPERF_HOME'] - self.loss = loss - self.kwargs = kwargs - self.learner = SVMperf(self.svmperf_base, loss=self.loss, **self.kwargs) - - def fit(self, data: LabelledCollection, fit_learner=True): - self._check_binary(data, self.__class__.__name__) - assert fit_learner, 'the method requires that fit_learner=True' - self.learner.fit(data.instances, data.labels) - return self - - def aggregate(self, classif_predictions: np.ndarray): - return F.prevalence_from_labels(classif_predictions, self.classes_) - - def classify(self, X, y=None): - return self.learner.predict(X) + if svmperf_base is None: + svmperf_base = qp.environ['SVMPERF_HOME'] + assert svmperf_base is not None, \ + 'param svmperf_base was not specified, and the variable SVMPERF_HOME has not been set in the environment' + return CC(SVMperf(svmperf_base, loss=loss, C=C)) -class SVMQ(ELM): +def newSVMQ(svmperf_base=None, C=1): """ - SVM(Q), which attempts to minimize the `Q` loss combining a classification-oriented loss and a - quantification-oriented loss, as proposed by + SVM(Q) is an Explicit Loss Minimization (ELM) quantifier set to optimize for the `Q` loss combining a + classification-oriented loss and a quantification-oriented loss, as proposed by `Barranquero et al. 2015 `_. Equivalent to: - >>> ELM(svmperf_base, loss='q', **kwargs) + >>> CC(SVMperf(svmperf_base, loss='q', C=C)) - :param svmperf_base: path to the folder containing the binary files of `SVM perf` - :param kwargs: rest of SVM perf's parameters + Quantifiers based on ELM represent a family of methods based on structured output learning; + these quantifiers rely on classifiers that have been optimized using a quantification-oriented loss + measure. This implementation relies on + `Joachims’ SVM perf `_ structured output + learning algorithm, which has to be installed and patched for the purpose (see this + `script `_). + This function is a wrapper around CC(SVMperf(svmperf_base, loss, C)) + + :param svmperf_base: path to the folder containing the binary files of `SVM perf`; if set to None (default) + this path will be obtained from qp.environ['SVMPERF_HOME'] + :param C: trade-off between training error and margin (default 0.01) + :return: returns an instance of CC set to work with SVMperf (with loss and C set properly) as the + underlying classifier """ + return newELM(svmperf_base, loss='q', C=C) - def __init__(self, svmperf_base=None, **kwargs): - super(SVMQ, self).__init__(svmperf_base, loss='q', **kwargs) - - -class SVMKLD(ELM): +def newSVMKLD(svmperf_base=None, C=1): """ - SVM(KLD), which attempts to minimize the Kullback-Leibler Divergence as proposed by + SVM(KLD) is an Explicit Loss Minimization (ELM) quantifier set to optimize for the Kullback-Leibler Divergence + as proposed by `Esuli et al. 2015 `_. + Equivalent to: + + >>> CC(SVMperf(svmperf_base, loss='kld', C=C)) + + Quantifiers based on ELM represent a family of methods based on structured output learning; + these quantifiers rely on classifiers that have been optimized using a quantification-oriented loss + measure. This implementation relies on + `Joachims’ SVM perf `_ structured output + learning algorithm, which has to be installed and patched for the purpose (see this + `script `_). + This function is a wrapper around CC(SVMperf(svmperf_base, loss, C)) + + :param svmperf_base: path to the folder containing the binary files of `SVM perf`; if set to None (default) + this path will be obtained from qp.environ['SVMPERF_HOME'] + :param C: trade-off between training error and margin (default 0.01) + :return: returns an instance of CC set to work with SVMperf (with loss and C set properly) as the + underlying classifier + """ + return newELM(svmperf_base, loss='kld', C=C) + + +def newSVMKLD(svmperf_base=None, C=1): + """ + SVM(KLD) is an Explicit Loss Minimization (ELM) quantifier set to optimize for the Kullback-Leibler Divergence + normalized via the logistic function, as proposed by `Esuli et al. 2015 `_. Equivalent to: - >>> ELM(svmperf_base, loss='kld', **kwargs) + >>> CC(SVMperf(svmperf_base, loss='nkld', C=C)) - :param svmperf_base: path to the folder containing the binary files of `SVM perf` - :param kwargs: rest of SVM perf's parameters + Quantifiers based on ELM represent a family of methods based on structured output learning; + these quantifiers rely on classifiers that have been optimized using a quantification-oriented loss + measure. This implementation relies on + `Joachims’ SVM perf `_ structured output + learning algorithm, which has to be installed and patched for the purpose (see this + `script `_). + This function is a wrapper around CC(SVMperf(svmperf_base, loss, C)) + + :param svmperf_base: path to the folder containing the binary files of `SVM perf`; if set to None (default) + this path will be obtained from qp.environ['SVMPERF_HOME'] + :param C: trade-off between training error and margin (default 0.01) + :return: returns an instance of CC set to work with SVMperf (with loss and C set properly) as the + underlying classifier """ + return newELM(svmperf_base, loss='nkld', C=C) - def __init__(self, svmperf_base=None, **kwargs): - super(SVMKLD, self).__init__(svmperf_base, loss='kld', **kwargs) - - -class SVMNKLD(ELM): +def newSVMAE(svmperf_base=None, C=1): """ - SVM(NKLD), which attempts to minimize a version of the the Kullback-Leibler Divergence normalized - via the logistic function, as proposed by - `Esuli et al. 2015 `_. - Equivalent to: - - >>> ELM(svmperf_base, loss='nkld', **kwargs) - - :param svmperf_base: path to the folder containing the binary files of `SVM perf` - :param kwargs: rest of SVM perf's parameters - """ - - def __init__(self, svmperf_base=None, **kwargs): - super(SVMNKLD, self).__init__(svmperf_base, loss='nkld', **kwargs) - - -class SVMAE(ELM): - """ - SVM(AE), which attempts to minimize Absolute Error as first used by + SVM(KLD) is an Explicit Loss Minimization (ELM) quantifier set to optimize for the Absolute Error as first used by `Moreo and Sebastiani, 2021 `_. Equivalent to: - >>> ELM(svmperf_base, loss='mae', **kwargs) + >>> CC(SVMperf(svmperf_base, loss='mae', C=C)) - :param svmperf_base: path to the folder containing the binary files of `SVM perf` - :param kwargs: rest of SVM perf's parameters + Quantifiers based on ELM represent a family of methods based on structured output learning; + these quantifiers rely on classifiers that have been optimized using a quantification-oriented loss + measure. This implementation relies on + `Joachims’ SVM perf `_ structured output + learning algorithm, which has to be installed and patched for the purpose (see this + `script `_). + This function is a wrapper around CC(SVMperf(svmperf_base, loss, C)) + + :param svmperf_base: path to the folder containing the binary files of `SVM perf`; if set to None (default) + this path will be obtained from qp.environ['SVMPERF_HOME'] + :param C: trade-off between training error and margin (default 0.01) + :return: returns an instance of CC set to work with SVMperf (with loss and C set properly) as the + underlying classifier """ + return newELM(svmperf_base, loss='mae', C=C) - def __init__(self, svmperf_base=None, **kwargs): - super(SVMAE, self).__init__(svmperf_base, loss='mae', **kwargs) - - -class SVMRAE(ELM): +def newSVMRAE(svmperf_base=None, C=1): """ - SVM(RAE), which attempts to minimize Relative Absolute Error as first used by - `Moreo and Sebastiani, 2021 `_. + SVM(KLD) is an Explicit Loss Minimization (ELM) quantifier set to optimize for the Relative Absolute Error as first + used by `Moreo and Sebastiani, 2021 `_. Equivalent to: - >>> ELM(svmperf_base, loss='mrae', **kwargs) + >>> CC(SVMperf(svmperf_base, loss='mrae', C=C)) - :param svmperf_base: path to the folder containing the binary files of `SVM perf` - :param kwargs: rest of SVM perf's parameters + Quantifiers based on ELM represent a family of methods based on structured output learning; + these quantifiers rely on classifiers that have been optimized using a quantification-oriented loss + measure. This implementation relies on + `Joachims’ SVM perf `_ structured output + learning algorithm, which has to be installed and patched for the purpose (see this + `script `_). + This function is a wrapper around CC(SVMperf(svmperf_base, loss, C)) + + :param svmperf_base: path to the folder containing the binary files of `SVM perf`; if set to None (default) + this path will be obtained from qp.environ['SVMPERF_HOME'] + :param C: trade-off between training error and margin (default 0.01) + :return: returns an instance of CC set to work with SVMperf (with loss and C set properly) as the + underlying classifier """ - - def __init__(self, svmperf_base=None, **kwargs): - super(SVMRAE, self).__init__(svmperf_base, loss='mrae', **kwargs) + return newELM(svmperf_base, loss='mrae', C=C) class ThresholdOptimization(AggregativeQuantifier, BinaryQuantifier): @@ -763,7 +982,7 @@ class ThresholdOptimization(AggregativeQuantifier, BinaryQuantifier): that would allow for more true positives and many more false positives, on the grounds this would deliver larger denominators. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier :param val_split: indicates the proportion of data to be used as a stratified held-out validation set in which the misclassification rates are to be estimated. This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of @@ -772,44 +991,24 @@ class ThresholdOptimization(AggregativeQuantifier, BinaryQuantifier): :class:`quapy.data.base.LabelledCollection` (the split itself). """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - self.learner = learner + def __init__(self, classifier: BaseEstimator, val_split=0.4, n_jobs=None): + self.classifier = classifier self.val_split = val_split + self.n_jobs = qp._get_njobs(n_jobs) - def fit(self, data: LabelledCollection, fit_learner=True, val_split: Union[float, int, LabelledCollection] = None): + def fit(self, data: LabelledCollection, fit_classifier=True, val_split: Union[float, int, LabelledCollection] = None): self._check_binary(data, "Threshold Optimization") if val_split is None: val_split = self.val_split - if isinstance(val_split, int): - assert fit_learner == True, \ - 'the parameters for the adjustment cannot be estimated with kFCV with fit_learner=False' - # kFCV estimation of parameters - y, probabilities = [], [] - kfcv = StratifiedKFold(n_splits=val_split) - pbar = tqdm(kfcv.split(*data.Xy), total=val_split) - for k, (training_idx, validation_idx) in enumerate(pbar): - pbar.set_description(f'{self.__class__.__name__} fitting fold {k}') - training = data.sampling_from_index(training_idx) - validation = data.sampling_from_index(validation_idx) - learner, val_data = _training_helper(self.learner, training, fit_learner, val_split=validation) - probabilities.append(learner.predict_proba(val_data.instances)) - y.append(val_data.labels) - y = np.concatenate(y) - probabilities = np.concatenate(probabilities) + self.classifier, y, y_, classes, class_count = cross_generate_predictions( + data, self.classifier, val_split, probabilistic=True, fit_classifier=fit_classifier, n_jobs=self.n_jobs + ) - # fit the learner on all data - self.learner, _ = _training_helper(self.learner, data, fit_learner, val_split=None) + self.cc = CC(self.classifier) - else: - self.learner, val_data = _training_helper(self.learner, data, fit_learner, val_split=val_split) - probabilities = self.learner.predict_proba(val_data.instances) - y = val_data.labels - - self.cc = CC(self.learner) - - self.tpr, self.fpr = self._optimize_threshold(y, probabilities) + self.tpr, self.fpr = self._optimize_threshold(y, y_) return self @@ -868,7 +1067,7 @@ class ThresholdOptimization(AggregativeQuantifier, BinaryQuantifier): def _compute_tpr(self, TP, FP): if TP + FP == 0: - return 0 + return 1 return TP / (TP + FP) def _compute_fpr(self, FP, TN): @@ -885,7 +1084,7 @@ class T50(ThresholdOptimization): for the threshold that makes `tpr` cosest to 0.5. The goal is to bring improved stability to the denominator of the adjustment. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier :param val_split: indicates the proportion of data to be used as a stratified held-out validation set in which the misclassification rates are to be estimated. This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of @@ -894,8 +1093,8 @@ class T50(ThresholdOptimization): :class:`quapy.data.base.LabelledCollection` (the split itself). """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - super().__init__(learner, val_split) + def __init__(self, classifier: BaseEstimator, val_split=0.4): + super().__init__(classifier, val_split) def _condition(self, tpr, fpr) -> float: return abs(tpr - 0.5) @@ -909,7 +1108,7 @@ class MAX(ThresholdOptimization): for the threshold that maximizes `tpr-fpr`. The goal is to bring improved stability to the denominator of the adjustment. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier :param val_split: indicates the proportion of data to be used as a stratified held-out validation set in which the misclassification rates are to be estimated. This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of @@ -918,8 +1117,8 @@ class MAX(ThresholdOptimization): :class:`quapy.data.base.LabelledCollection` (the split itself). """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - super().__init__(learner, val_split) + def __init__(self, classifier: BaseEstimator, val_split=0.4): + super().__init__(classifier, val_split) def _condition(self, tpr, fpr) -> float: # MAX strives to maximize (tpr - fpr), which is equivalent to minimize (fpr - tpr) @@ -934,7 +1133,7 @@ class X(ThresholdOptimization): for the threshold that yields `tpr=1-fpr`. The goal is to bring improved stability to the denominator of the adjustment. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier :param val_split: indicates the proportion of data to be used as a stratified held-out validation set in which the misclassification rates are to be estimated. This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of @@ -943,8 +1142,8 @@ class X(ThresholdOptimization): :class:`quapy.data.base.LabelledCollection` (the split itself). """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - super().__init__(learner, val_split) + def __init__(self, classifier: BaseEstimator, val_split=0.4): + super().__init__(classifier, val_split) def _condition(self, tpr, fpr) -> float: return abs(1 - (tpr + fpr)) @@ -958,7 +1157,7 @@ class MS(ThresholdOptimization): class prevalence estimates for all decision thresholds and returns the median of them all. The goal is to bring improved stability to the denominator of the adjustment. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier :param val_split: indicates the proportion of data to be used as a stratified held-out validation set in which the misclassification rates are to be estimated. This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of @@ -966,8 +1165,8 @@ class MS(ThresholdOptimization): `k`-fold cross validation (this integer stands for the number of folds `k`), or as a :class:`quapy.data.base.LabelledCollection` (the split itself). """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - super().__init__(learner, val_split) + def __init__(self, classifier: BaseEstimator, val_split=0.4): + super().__init__(classifier, val_split) def _condition(self, tpr, fpr) -> float: pass @@ -995,7 +1194,7 @@ class MS2(MS): which `tpr-fpr>0.25` The goal is to bring improved stability to the denominator of the adjustment. - :param learner: a sklearn's Estimator that generates a classifier + :param classifier: a sklearn's Estimator that generates a classifier :param val_split: indicates the proportion of data to be used as a stratified held-out validation set in which the misclassification rates are to be estimated. This parameter can be indicated as a real value (between 0 and 1, default 0.4), representing a proportion of @@ -1003,8 +1202,8 @@ class MS2(MS): `k`-fold cross validation (this integer stands for the number of folds `k`), or as a :class:`quapy.data.base.LabelledCollection` (the split itself). """ - def __init__(self, learner: BaseEstimator, val_split=0.4): - super().__init__(learner, val_split) + def __init__(self, classifier: BaseEstimator, val_split=0.4): + super().__init__(classifier, val_split) def _optimize_threshold(self, y, probabilities): tprs = [0, 1] @@ -1028,12 +1227,11 @@ ProbabilisticAdjustedClassifyAndCount = PACC ExpectationMaximizationQuantifier = EMQ SLD = EMQ HellingerDistanceY = HDy -ExplicitLossMinimisation = ELM MedianSweep = MS MedianSweep2 = MS2 -class OneVsAll(AggregativeQuantifier): +class OneVsAllAggregative(OneVsAllGeneric, AggregativeQuantifier): """ Allows any binary quantifier to perform quantification on single-label datasets. The method maintains one binary quantifier for each class, and then l1-normalizes the outputs so that the @@ -1041,124 +1239,52 @@ class OneVsAll(AggregativeQuantifier): This variant was used, along with the :class:`EMQ` quantifier, in `Gao and Sebastiani, 2016 `_. - :param learner: a sklearn's Estimator that generates a binary classifier + :param binary_quantifier: a quantifier (binary) that will be employed to work on multiclass model in a + one-vs-all manner :param n_jobs: number of parallel workers + :param parallel_backend: the parallel backend for joblib (default "loky"); this is helpful for some quantifiers + (e.g., ELM-based ones) that cannot be run with multiprocessing, since the temp dir they create during fit will + is removed and no longer available at predict time. """ - def __init__(self, binary_quantifier, n_jobs=-1): - self.binary_quantifier = binary_quantifier - self.n_jobs = n_jobs - - def fit(self, data: LabelledCollection, fit_learner=True): - assert not data.binary, \ - f'{self.__class__.__name__} expect non-binary data' - assert isinstance(self.binary_quantifier, BaseQuantifier), \ + def __init__(self, binary_quantifier, n_jobs=None, parallel_backend='multiprocessing'): + assert isinstance(binary_quantifier, BaseQuantifier), \ f'{self.binary_quantifier} does not seem to be a Quantifier' - assert fit_learner == True, 'fit_learner must be True' - - self.dict_binary_quantifiers = {c: deepcopy(self.binary_quantifier) for c in data.classes_} - self.__parallel(self._delayed_binary_fit, data) - return self + assert isinstance(binary_quantifier, AggregativeQuantifier), \ + f'{self.binary_quantifier} does not seem to be of type Aggregative' + self.binary_quantifier = binary_quantifier + self.n_jobs = qp._get_njobs(n_jobs) + self.parallel_backend = parallel_backend def classify(self, instances): """ - Returns a matrix of shape `(n,m,)` with `n` the number of instances and `m` the number of classes. The entry - `(i,j)` is a binary value indicating whether instance `i `belongs to class `j`. The binary classifications are - independent of each other, meaning that an instance can end up be attributed to 0, 1, or more classes. + If the base quantifier is not probabilistic, returns a matrix of shape `(n,m,)` with `n` the number of + instances and `m` the number of classes. The entry `(i,j)` is a binary value indicating whether instance + `i `belongs to class `j`. The binary classifications are independent of each other, meaning that an instance + can end up be attributed to 0, 1, or more classes. + If the base quantifier is probabilistic, returns a matrix of shape `(n,m,2)` with `n` the number of instances + and `m` the number of classes. The entry `(i,j,1)` (resp. `(i,j,0)`) is a value in [0,1] indicating the + posterior probability that instance `i` belongs (resp. does not belong) to class `j`. The posterior + probabilities are independent of each other, meaning that, in general, they do not sum up to one. :param instances: array-like :return: `np.ndarray` """ - classif_predictions_bin = self.__parallel(self._delayed_binary_classification, instances) - return classif_predictions_bin.T - - def posterior_probabilities(self, instances): - """ - Returns a matrix of shape `(n,m,2)` with `n` the number of instances and `m` the number of classes. The entry - `(i,j,1)` (resp. `(i,j,0)`) is a value in [0,1] indicating the posterior probability that instance `i` belongs - (resp. does not belong) to class `j`. - The posterior probabilities are independent of each other, meaning that, in general, they do not sum - up to one. - - :param instances: array-like - :return: `np.ndarray` - """ - - if not self.binary_quantifier.probabilistic: - raise NotImplementedError(f'{self.__class__.__name__} does not implement posterior_probabilities because ' - f'the base quantifier {self.binary_quantifier.__class__.__name__} is not ' - f'probabilistic') - posterior_predictions_bin = self.__parallel(self._delayed_binary_posteriors, instances) - return np.swapaxes(posterior_predictions_bin, 0, 1) - - def aggregate(self, classif_predictions_bin): - if self.probabilistic: - assert classif_predictions_bin.shape[1] == self.n_classes and classif_predictions_bin.shape[2] == 2, \ - 'param classif_predictions_bin does not seem to be a valid matrix (ndarray) of posterior ' \ - 'probabilities (2 dimensions) for each document (row) and class (columns)' + classif_predictions = self._parallel(self._delayed_binary_classification, instances) + if isinstance(self.binary_quantifier, AggregativeProbabilisticQuantifier): + return np.swapaxes(classif_predictions, 0, 1) else: - assert set(np.unique(classif_predictions_bin)).issubset({0, 1}), \ - 'param classif_predictions_bin does not seem to be a valid matrix (ndarray) of binary ' \ - 'predictions for each document (row) and class (columns)' - prevalences = self.__parallel(self._delayed_binary_aggregate, classif_predictions_bin) + return classif_predictions.T + + def aggregate(self, classif_predictions): + prevalences = self._parallel(self._delayed_binary_aggregate, classif_predictions) return F.normalize_prevalence(prevalences) - def quantify(self, X): - if self.probabilistic: - predictions = self.posterior_probabilities(X) - else: - predictions = self.classify(X) - return self.aggregate(predictions) - - def __parallel(self, func, *args, **kwargs): - return np.asarray( - # some quantifiers (in particular, ELM-based ones) cannot be run with multiprocess, since the temp dir they - # create during the fit will be removed and be no longer available for the predict... - Parallel(n_jobs=self.n_jobs, backend='threading')( - delayed(func)(c, *args, **kwargs) for c in self.classes_ - ) - ) - - @property - def classes_(self): - return sorted(self.dict_binary_quantifiers.keys()) - - def set_params(self, **parameters): - self.binary_quantifier.set_params(**parameters) - - def get_params(self, deep=True): - return self.binary_quantifier.get_params() - def _delayed_binary_classification(self, c, X): return self.dict_binary_quantifiers[c].classify(X) - def _delayed_binary_posteriors(self, c, X): - return self.dict_binary_quantifiers[c].posterior_probabilities(X) - def _delayed_binary_aggregate(self, c, classif_predictions): # the estimation for the positive class prevalence return self.dict_binary_quantifiers[c].aggregate(classif_predictions[:, c])[1] - def _delayed_binary_fit(self, c, data): - bindata = LabelledCollection(data.instances, data.labels == c, classes_=[False, True]) - self.dict_binary_quantifiers[c].fit(bindata) - - @property - def binary(self): - """ - Informs that the classifier is not binary - - :return: False - """ - return False - - @property - def probabilistic(self): - """ - Indicates if the classifier is probabilistic or not (depending on the nature of the base classifier). - - :return: boolean - """ - - return self.binary_quantifier.probabilistic \ No newline at end of file diff --git a/quapy/method/base.py b/quapy/method/base.py index 4a4962a..e0363f1 100644 --- a/quapy/method/base.py +++ b/quapy/method/base.py @@ -1,11 +1,17 @@ from abc import ABCMeta, abstractmethod +from copy import deepcopy +from joblib import Parallel, delayed +from sklearn.base import BaseEstimator + +import quapy as qp from quapy.data import LabelledCollection +import numpy as np # Base Quantifier abstract class # ------------------------------------ -class BaseQuantifier(metaclass=ABCMeta): +class BaseQuantifier(BaseEstimator): """ Abstract Quantifier. A quantifier is defined as an object of a class that implements the method :meth:`fit` on :class:`quapy.data.base.LabelledCollection`, the method :meth:`quantify`, and the :meth:`set_params` and @@ -28,79 +34,10 @@ class BaseQuantifier(metaclass=ABCMeta): Generate class prevalence estimates for the sample's instances :param instances: array-like - :return: `np.ndarray` of shape `(self.n_classes_,)` with class prevalence estimates. + :return: `np.ndarray` of shape `(n_classes,)` with class prevalence estimates. """ ... - @abstractmethod - def set_params(self, **parameters): - """ - Set the parameters of the quantifier. - - :param parameters: dictionary of param-value pairs - """ - ... - - @abstractmethod - def get_params(self, deep=True): - """ - Return the current parameters of the quantifier. - - :param deep: for compatibility with sklearn - :return: a dictionary of param-value pairs - """ - ... - - @property - @abstractmethod - def classes_(self): - """ - Class labels, in the same order in which class prevalence values are to be computed. - - :return: array-like - """ - ... - - @property - def n_classes(self): - """ - Returns the number of classes - - :return: integer - """ - return len(self.classes_) - - # these methods allows meta-learners to reimplement the decision based on their constituents, and not - # based on class structure - @property - def binary(self): - """ - Indicates whether the quantifier is binary or not. - - :return: False (to be overridden) - """ - return False - - @property - def aggregative(self): - """ - Indicates whether the quantifier is of type aggregative or not - - :return: False (to be overridden) - """ - - return False - - @property - def probabilistic(self): - """ - Indicates whether the quantifier is of type probabilistic or not - - :return: False (to be overridden) - """ - - return False - class BinaryQuantifier(BaseQuantifier): """ @@ -112,90 +49,61 @@ class BinaryQuantifier(BaseQuantifier): assert data.binary, f'{quantifier_name} works only on problems of binary classification. ' \ f'Use the class OneVsAll to enable {quantifier_name} work on single-label data.' + +class OneVsAll: + pass + + +def newOneVsAll(binary_quantifier, n_jobs=None): + assert isinstance(binary_quantifier, BaseQuantifier), \ + f'{binary_quantifier} does not seem to be a Quantifier' + if isinstance(binary_quantifier, qp.method.aggregative.AggregativeQuantifier): + return qp.method.aggregative.OneVsAllAggregative(binary_quantifier, n_jobs) + else: + return OneVsAllGeneric(binary_quantifier, n_jobs) + + +class OneVsAllGeneric(OneVsAll,BaseQuantifier): + """ + Allows any binary quantifier to perform quantification on single-label datasets. The method maintains one binary + quantifier for each class, and then l1-normalizes the outputs so that the class prevelence values sum up to 1. + """ + + def __init__(self, binary_quantifier, n_jobs=None): + assert isinstance(binary_quantifier, BaseQuantifier), \ + f'{binary_quantifier} does not seem to be a Quantifier' + if isinstance(binary_quantifier, qp.method.aggregative.AggregativeQuantifier): + print('[warning] the quantifier seems to be an instance of qp.method.aggregative.AggregativeQuantifier; ' + f'you might prefer instantiating {qp.method.aggregative.OneVsAllAggregative.__name__}') + self.binary_quantifier = binary_quantifier + self.n_jobs = qp._get_njobs(n_jobs) + + def fit(self, data: LabelledCollection, fit_classifier=True): + assert not data.binary, f'{self.__class__.__name__} expect non-binary data' + assert fit_classifier == True, 'fit_classifier must be True' + + self.dict_binary_quantifiers = {c: deepcopy(self.binary_quantifier) for c in data.classes_} + self._parallel(self._delayed_binary_fit, data) + return self + + def _parallel(self, func, *args, **kwargs): + return np.asarray( + Parallel(n_jobs=self.n_jobs, backend='threading')( + delayed(func)(c, *args, **kwargs) for c in self.classes_ + ) + ) + + def quantify(self, instances): + prevalences = self._parallel(self._delayed_binary_predict, instances) + return qp.functional.normalize_prevalence(prevalences) + @property - def binary(self): - """ - Informs that the quantifier is binary - - :return: True - """ - return True - - -def isbinary(model:BaseQuantifier): - """ - Alias for property `binary` - - :param model: the model - :return: True if the model is binary, False otherwise - """ - return model.binary - - -def isaggregative(model:BaseQuantifier): - """ - Alias for property `aggregative` - - :param model: the model - :return: True if the model is aggregative, False otherwise - """ - - return model.aggregative - - -def isprobabilistic(model:BaseQuantifier): - """ - Alias for property `probabilistic` - - :param model: the model - :return: True if the model is probabilistic, False otherwise - """ - - return model.probabilistic - - -# class OneVsAll: -# """ -# Allows any binary quantifier to perform quantification on single-label datasets. The method maintains one binary -# quantifier for each class, and then l1-normalizes the outputs so that the class prevelences sum up to 1. -# """ -# -# def __init__(self, binary_method, n_jobs=-1): -# self.binary_method = binary_method -# self.n_jobs = n_jobs -# -# def fit(self, data: LabelledCollection, **kwargs): -# assert not data.binary, f'{self.__class__.__name__} expect non-binary data' -# assert isinstance(self.binary_method, BaseQuantifier), f'{self.binary_method} does not seem to be a Quantifier' -# self.class_method = {c: deepcopy(self.binary_method) for c in data.classes_} -# Parallel(n_jobs=self.n_jobs, backend='threading')( -# delayed(self._delayed_binary_fit)(c, self.class_method, data, **kwargs) for c in data.classes_ -# ) -# return self -# -# def quantify(self, X, *args): -# prevalences = np.asarray( -# Parallel(n_jobs=self.n_jobs, backend='threading')( -# delayed(self._delayed_binary_predict)(c, self.class_method, X) for c in self.classes -# ) -# ) -# return F.normalize_prevalence(prevalences) -# -# @property -# def classes(self): -# return sorted(self.class_method.keys()) -# -# def set_params(self, **parameters): -# self.binary_method.set_params(**parameters) -# -# def get_params(self, deep=True): -# return self.binary_method.get_params() -# -# def _delayed_binary_predict(self, c, learners, X): -# return learners[c].quantify(X)[:,1] # the mean is the estimation for the positive class prevalence -# -# def _delayed_binary_fit(self, c, learners, data, **kwargs): -# bindata = LabelledCollection(data.instances, data.labels == c, n_classes=2) -# learners[c].fit(bindata, **kwargs) + def classes_(self): + return sorted(self.dict_binary_quantifiers.keys()) + def _delayed_binary_predict(self, c, X): + return self.dict_binary_quantifiers[c].quantify(X)[1] + def _delayed_binary_fit(self, c, data): + bindata = LabelledCollection(data.instances, data.labels == c, classes=[False, True]) + self.dict_binary_quantifiers[c].fit(bindata) diff --git a/quapy/method/meta.py b/quapy/method/meta.py index 3504301..2bb8af7 100644 --- a/quapy/method/meta.py +++ b/quapy/method/meta.py @@ -9,7 +9,6 @@ from tqdm import tqdm import quapy as qp from quapy import functional as F from quapy.data import LabelledCollection -from quapy.evaluation import evaluate from quapy.model_selection import GridSearchQ try: @@ -73,7 +72,7 @@ class Ensemble(BaseQuantifier): policy='ave', max_sample_size=None, val_split:Union[qp.data.LabelledCollection, float]=None, - n_jobs=1, + n_jobs=None, verbose=False): assert policy in Ensemble.VALID_POLICIES, \ f'unknown policy={policy}; valid are {Ensemble.VALID_POLICIES}' @@ -85,7 +84,7 @@ class Ensemble(BaseQuantifier): self.red_size = red_size self.policy = policy self.val_split = val_split - self.n_jobs = n_jobs + self.n_jobs = qp._get_njobs(n_jobs) self.post_proba_fn = None self.verbose = verbose self.max_sample_size = max_sample_size @@ -147,15 +146,15 @@ class Ensemble(BaseQuantifier): This function should not be used within :class:`quapy.model_selection.GridSearchQ` (is here for compatibility with the abstract class). Instead, use `Ensemble(GridSearchQ(q),...)`, with `q` a Quantifier (recommended), or - `Ensemble(Q(GridSearchCV(l)))` with `Q` a quantifier class that has a learner `l` optimized for - classification (not recommended). + `Ensemble(Q(GridSearchCV(l)))` with `Q` a quantifier class that has a classifier `l` optimized for + classification (not recommended). :param parameters: dictionary :return: raises an Exception """ raise NotImplementedError(f'{self.__class__.__name__} should not be used within GridSearchQ; ' f'instead, use Ensemble(GridSearchQ(q),...), with q a Quantifier (recommended), ' - f'or Ensemble(Q(GridSearchCV(l))) with Q a quantifier class that has a learner ' + f'or Ensemble(Q(GridSearchCV(l))) with Q a quantifier class that has a classifier ' f'l optimized for classification (not recommended).') def get_params(self, deep=True): @@ -163,11 +162,13 @@ class Ensemble(BaseQuantifier): This function should not be used within :class:`quapy.model_selection.GridSearchQ` (is here for compatibility with the abstract class). Instead, use `Ensemble(GridSearchQ(q),...)`, with `q` a Quantifier (recommended), or - `Ensemble(Q(GridSearchCV(l)))` with `Q` a quantifier class that has a learner `l` optimized for - classification (not recommended). + `Ensemble(Q(GridSearchCV(l)))` with `Q` a quantifier class that has a classifier `l` optimized for + classification (not recommended). + :param deep: for compatibility with scikit-learn :return: raises an Exception """ + raise NotImplementedError() def _accuracy_policy(self, error_name): @@ -176,11 +177,12 @@ class Ensemble(BaseQuantifier): For each model in the ensemble, the performance is measured in terms of _error_name_ on the quantification of the samples used for training the rest of the models in the ensemble. """ + from quapy.evaluation import evaluate_on_samples error = qp.error.from_name(error_name) tests = [m[3] for m in self.ensemble] scores = [] for i, model in enumerate(self.ensemble): - scores.append(evaluate(model[0], tests[:i] + tests[i + 1:], error, self.n_jobs)) + scores.append(evaluate_on_samples(model[0], tests[:i] + tests[i + 1:], error)) order = np.argsort(scores) self.ensemble = _select_k(self.ensemble, order, k=self.red_size) @@ -234,19 +236,6 @@ class Ensemble(BaseQuantifier): order = np.argsort(dist) return _select_k(predictions, order, k=self.red_size) - @property - def classes_(self): - return self.base_quantifier.classes_ - - @property - def binary(self): - """ - Returns a boolean indicating whether the base quantifiers are binary or not - - :return: boolean - """ - return self.base_quantifier.binary - @property def aggregative(self): """ @@ -339,18 +328,18 @@ def _draw_simplex(ndim, min_val, max_trials=100): f'>= {min_val} is unlikely (it failed after {max_trials} trials)') -def _instantiate_ensemble(learner, base_quantifier_class, param_grid, optim, param_model_sel, **kwargs): +def _instantiate_ensemble(classifier, base_quantifier_class, param_grid, optim, param_model_sel, **kwargs): if optim is None: - base_quantifier = base_quantifier_class(learner) + base_quantifier = base_quantifier_class(classifier) elif optim in qp.error.CLASSIFICATION_ERROR: if optim == qp.error.f1e: scoring = make_scorer(f1_score) elif optim == qp.error.acce: scoring = make_scorer(accuracy_score) - learner = GridSearchCV(learner, param_grid, scoring=scoring) - base_quantifier = base_quantifier_class(learner) + classifier = GridSearchCV(classifier, param_grid, scoring=scoring) + base_quantifier = base_quantifier_class(classifier) else: - base_quantifier = GridSearchQ(base_quantifier_class(learner), + base_quantifier = GridSearchQ(base_quantifier_class(classifier), param_grid=param_grid, **param_model_sel, error=optim) @@ -370,7 +359,7 @@ def _check_error(error): f'the name of an error function in {qp.error.ERROR_NAMES}') -def ensembleFactory(learner, base_quantifier_class, param_grid=None, optim=None, param_model_sel: dict = None, +def ensembleFactory(classifier, base_quantifier_class, param_grid=None, optim=None, param_model_sel: dict = None, **kwargs): """ Ensemble factory. Provides a unified interface for instantiating ensembles that can be optimized (via model @@ -403,7 +392,7 @@ def ensembleFactory(learner, base_quantifier_class, param_grid=None, optim=None, >>> >>> ensembleFactory(LogisticRegression(), PACC, optim='mae', policy='mae', **common) - :param learner: sklearn's Estimator that generates a classifier + :param classifier: sklearn's Estimator that generates a classifier :param base_quantifier_class: a class of quantifiers :param param_grid: a dictionary with the grid of parameters to optimize for :param optim: a valid quantification or classification error, or a string name of it @@ -418,21 +407,21 @@ def ensembleFactory(learner, base_quantifier_class, param_grid=None, optim=None, if param_model_sel is None: raise ValueError(f'param_model_sel is None but optim was requested.') error = _check_error(optim) - return _instantiate_ensemble(learner, base_quantifier_class, param_grid, error, param_model_sel, **kwargs) + return _instantiate_ensemble(classifier, base_quantifier_class, param_grid, error, param_model_sel, **kwargs) -def ECC(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): +def ECC(classifier, param_grid=None, optim=None, param_mod_sel=None, **kwargs): """ Implements an ensemble of :class:`quapy.method.aggregative.CC` quantifiers, as used by `Pérez-Gállego et al., 2019 `_. Equivalent to: - >>> ensembleFactory(learner, CC, param_grid, optim, param_mod_sel, **kwargs) + >>> ensembleFactory(classifier, CC, param_grid, optim, param_mod_sel, **kwargs) See :meth:`ensembleFactory` for further details. - :param learner: sklearn's Estimator that generates a classifier + :param classifier: sklearn's Estimator that generates a classifier :param param_grid: a dictionary with the grid of parameters to optimize for :param optim: a valid quantification or classification error, or a string name of it :param param_model_sel: a dictionary containing any keyworded argument to pass to @@ -441,21 +430,21 @@ def ECC(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): :return: an instance of :class:`Ensemble` """ - return ensembleFactory(learner, CC, param_grid, optim, param_mod_sel, **kwargs) + return ensembleFactory(classifier, CC, param_grid, optim, param_mod_sel, **kwargs) -def EACC(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): +def EACC(classifier, param_grid=None, optim=None, param_mod_sel=None, **kwargs): """ Implements an ensemble of :class:`quapy.method.aggregative.ACC` quantifiers, as used by `Pérez-Gállego et al., 2019 `_. Equivalent to: - >>> ensembleFactory(learner, ACC, param_grid, optim, param_mod_sel, **kwargs) + >>> ensembleFactory(classifier, ACC, param_grid, optim, param_mod_sel, **kwargs) See :meth:`ensembleFactory` for further details. - :param learner: sklearn's Estimator that generates a classifier + :param classifier: sklearn's Estimator that generates a classifier :param param_grid: a dictionary with the grid of parameters to optimize for :param optim: a valid quantification or classification error, or a string name of it :param param_model_sel: a dictionary containing any keyworded argument to pass to @@ -464,20 +453,20 @@ def EACC(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): :return: an instance of :class:`Ensemble` """ - return ensembleFactory(learner, ACC, param_grid, optim, param_mod_sel, **kwargs) + return ensembleFactory(classifier, ACC, param_grid, optim, param_mod_sel, **kwargs) -def EPACC(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): +def EPACC(classifier, param_grid=None, optim=None, param_mod_sel=None, **kwargs): """ Implements an ensemble of :class:`quapy.method.aggregative.PACC` quantifiers. Equivalent to: - >>> ensembleFactory(learner, PACC, param_grid, optim, param_mod_sel, **kwargs) + >>> ensembleFactory(classifier, PACC, param_grid, optim, param_mod_sel, **kwargs) See :meth:`ensembleFactory` for further details. - :param learner: sklearn's Estimator that generates a classifier + :param classifier: sklearn's Estimator that generates a classifier :param param_grid: a dictionary with the grid of parameters to optimize for :param optim: a valid quantification or classification error, or a string name of it :param param_model_sel: a dictionary containing any keyworded argument to pass to @@ -486,21 +475,21 @@ def EPACC(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): :return: an instance of :class:`Ensemble` """ - return ensembleFactory(learner, PACC, param_grid, optim, param_mod_sel, **kwargs) + return ensembleFactory(classifier, PACC, param_grid, optim, param_mod_sel, **kwargs) -def EHDy(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): +def EHDy(classifier, param_grid=None, optim=None, param_mod_sel=None, **kwargs): """ Implements an ensemble of :class:`quapy.method.aggregative.HDy` quantifiers, as used by `Pérez-Gállego et al., 2019 `_. Equivalent to: - >>> ensembleFactory(learner, HDy, param_grid, optim, param_mod_sel, **kwargs) + >>> ensembleFactory(classifier, HDy, param_grid, optim, param_mod_sel, **kwargs) See :meth:`ensembleFactory` for further details. - :param learner: sklearn's Estimator that generates a classifier + :param classifier: sklearn's Estimator that generates a classifier :param param_grid: a dictionary with the grid of parameters to optimize for :param optim: a valid quantification or classification error, or a string name of it :param param_model_sel: a dictionary containing any keyworded argument to pass to @@ -509,20 +498,20 @@ def EHDy(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): :return: an instance of :class:`Ensemble` """ - return ensembleFactory(learner, HDy, param_grid, optim, param_mod_sel, **kwargs) + return ensembleFactory(classifier, HDy, param_grid, optim, param_mod_sel, **kwargs) -def EEMQ(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): +def EEMQ(classifier, param_grid=None, optim=None, param_mod_sel=None, **kwargs): """ Implements an ensemble of :class:`quapy.method.aggregative.EMQ` quantifiers. Equivalent to: - >>> ensembleFactory(learner, EMQ, param_grid, optim, param_mod_sel, **kwargs) + >>> ensembleFactory(classifier, EMQ, param_grid, optim, param_mod_sel, **kwargs) See :meth:`ensembleFactory` for further details. - :param learner: sklearn's Estimator that generates a classifier + :param classifier: sklearn's Estimator that generates a classifier :param param_grid: a dictionary with the grid of parameters to optimize for :param optim: a valid quantification or classification error, or a string name of it :param param_model_sel: a dictionary containing any keyworded argument to pass to @@ -531,4 +520,4 @@ def EEMQ(learner, param_grid=None, optim=None, param_mod_sel=None, **kwargs): :return: an instance of :class:`Ensemble` """ - return ensembleFactory(learner, EMQ, param_grid, optim, param_mod_sel, **kwargs) + return ensembleFactory(classifier, EMQ, param_grid, optim, param_mod_sel, **kwargs) diff --git a/quapy/method/neural.py b/quapy/method/neural.py index bf1f375..aeb8a7d 100644 --- a/quapy/method/neural.py +++ b/quapy/method/neural.py @@ -6,6 +6,7 @@ import torch from torch.nn import MSELoss from torch.nn.functional import relu +from quapy.protocol import UPP from quapy.method.aggregative import * from quapy.util import EarlyStop @@ -31,17 +32,18 @@ class QuaNetTrainer(BaseQuantifier): >>> >>> # the text classifier is a CNN trained by NeuralClassifierTrainer >>> cnn = CNNnet(dataset.vocabulary_size, dataset.n_classes) - >>> learner = NeuralClassifierTrainer(cnn, device='cuda') + >>> classifier = NeuralClassifierTrainer(cnn, device='cuda') >>> >>> # train QuaNet (QuaNet is an alias to QuaNetTrainer) - >>> model = QuaNet(learner, qp.environ['SAMPLE_SIZE'], device='cuda') + >>> model = QuaNet(classifier, qp.environ['SAMPLE_SIZE'], device='cuda') >>> model.fit(dataset.training) >>> estim_prevalence = model.quantify(dataset.test.instances) - :param learner: an object implementing `fit` (i.e., that can be trained on labelled data), + :param classifier: an object implementing `fit` (i.e., that can be trained on labelled data), `predict_proba` (i.e., that can generate posterior probabilities of unlabelled examples) and `transform` (i.e., that can generate embedded representations of the unlabelled instances). - :param sample_size: integer, the sample size + :param sample_size: integer, the sample size; default is None, meaning that the sample size should be + taken from qp.environ["SAMPLE_SIZE"] :param n_epochs: integer, maximum number of training epochs :param tr_iter_per_poch: integer, number of training iterations before considering an epoch complete :param va_iter_per_poch: integer, number of validation iterations to perform after each epoch @@ -60,8 +62,8 @@ class QuaNetTrainer(BaseQuantifier): """ def __init__(self, - learner, - sample_size, + classifier, + sample_size=None, n_epochs=100, tr_iter_per_poch=500, va_iter_per_poch=100, @@ -76,14 +78,14 @@ class QuaNetTrainer(BaseQuantifier): checkpointname=None, device='cuda'): - assert hasattr(learner, 'transform'), \ - f'the learner {learner.__class__.__name__} does not seem to be able to produce document embeddings ' \ + assert hasattr(classifier, 'transform'), \ + f'the classifier {classifier.__class__.__name__} does not seem to be able to produce document embeddings ' \ f'since it does not implement the method "transform"' - assert hasattr(learner, 'predict_proba'), \ - f'the learner {learner.__class__.__name__} does not seem to be able to produce posterior probabilities ' \ + assert hasattr(classifier, 'predict_proba'), \ + f'the classifier {classifier.__class__.__name__} does not seem to be able to produce posterior probabilities ' \ f'since it does not implement the method "predict_proba"' - self.learner = learner - self.sample_size = sample_size + self.classifier = classifier + self.sample_size = qp._get_sample_size(sample_size) self.n_epochs = n_epochs self.tr_iter = tr_iter_per_poch self.va_iter = va_iter_per_poch @@ -105,26 +107,26 @@ class QuaNetTrainer(BaseQuantifier): self.checkpoint = os.path.join(checkpointdir, checkpointname) self.device = torch.device(device) - self.__check_params_colision(self.quanet_params, self.learner.get_params()) + self.__check_params_colision(self.quanet_params, self.classifier.get_params()) self._classes_ = None - def fit(self, data: LabelledCollection, fit_learner=True): + def fit(self, data: LabelledCollection, fit_classifier=True): """ Trains QuaNet. - :param data: the training data on which to train QuaNet. If `fit_learner=True`, the data will be split in + :param data: the training data on which to train QuaNet. If `fit_classifier=True`, the data will be split in 40/40/20 for training the classifier, training QuaNet, and validating QuaNet, respectively. If - `fit_learner=False`, the data will be split in 66/34 for training QuaNet and validating it, respectively. - :param fit_learner: if True, trains the classifier on a split containing 40% of the data + `fit_classifier=False`, the data will be split in 66/34 for training QuaNet and validating it, respectively. + :param fit_classifier: if True, trains the classifier on a split containing 40% of the data :return: self """ self._classes_ = data.classes_ os.makedirs(self.checkpointdir, exist_ok=True) - if fit_learner: + if fit_classifier: classifier_data, unused_data = data.split_stratified(0.4) train_data, valid_data = unused_data.split_stratified(0.66) # 0.66 split of 60% makes 40% and 20% - self.learner.fit(*classifier_data.Xy) + self.classifier.fit(*classifier_data.Xy) else: classifier_data = None train_data, valid_data = data.split_stratified(0.66) @@ -133,21 +135,21 @@ class QuaNetTrainer(BaseQuantifier): self.tr_prev = data.prevalence() # compute the posterior probabilities of the instances - valid_posteriors = self.learner.predict_proba(valid_data.instances) - train_posteriors = self.learner.predict_proba(train_data.instances) + valid_posteriors = self.classifier.predict_proba(valid_data.instances) + train_posteriors = self.classifier.predict_proba(train_data.instances) # turn instances' original representations into embeddings - valid_data_embed = LabelledCollection(self.learner.transform(valid_data.instances), valid_data.labels, self._classes_) - train_data_embed = LabelledCollection(self.learner.transform(train_data.instances), train_data.labels, self._classes_) + valid_data_embed = LabelledCollection(self.classifier.transform(valid_data.instances), valid_data.labels, self._classes_) + train_data_embed = LabelledCollection(self.classifier.transform(train_data.instances), train_data.labels, self._classes_) self.quantifiers = { - 'cc': CC(self.learner).fit(None, fit_learner=False), - 'acc': ACC(self.learner).fit(None, fit_learner=False, val_split=valid_data), - 'pcc': PCC(self.learner).fit(None, fit_learner=False), - 'pacc': PACC(self.learner).fit(None, fit_learner=False, val_split=valid_data), + 'cc': CC(self.classifier).fit(None, fit_classifier=False), + 'acc': ACC(self.classifier).fit(None, fit_classifier=False, val_split=valid_data), + 'pcc': PCC(self.classifier).fit(None, fit_classifier=False), + 'pacc': PACC(self.classifier).fit(None, fit_classifier=False, val_split=valid_data), } if classifier_data is not None: - self.quantifiers['emq'] = EMQ(self.learner).fit(classifier_data, fit_learner=False) + self.quantifiers['emq'] = EMQ(self.classifier).fit(classifier_data, fit_classifier=False) self.status = { 'tr-loss': -1, @@ -191,7 +193,7 @@ class QuaNetTrainer(BaseQuantifier): label_predictions = np.argmax(posteriors, axis=-1) prevs_estim = [] for quantifier in self.quantifiers.values(): - predictions = posteriors if quantifier.probabilistic else label_predictions + predictions = posteriors if isinstance(quantifier, AggregativeProbabilisticQuantifier) else label_predictions prevs_estim.extend(quantifier.aggregate(predictions)) # there is no real need for adding static estims like the TPR or FPR from training since those are constant @@ -199,8 +201,8 @@ class QuaNetTrainer(BaseQuantifier): return prevs_estim def quantify(self, instances): - posteriors = self.learner.predict_proba(instances) - embeddings = self.learner.transform(instances) + posteriors = self.classifier.predict_proba(instances) + embeddings = self.classifier.transform(instances) quant_estims = self._get_aggregative_estims(posteriors) self.quanet.eval() with torch.no_grad(): @@ -216,16 +218,13 @@ class QuaNetTrainer(BaseQuantifier): self.quanet.train(mode=train) losses = [] mae_errors = [] - if train==False: - prevpoints = F.get_nprevpoints_approximation(iterations, self.quanet.n_classes) - iterations = F.num_prevalence_combinations(prevpoints, self.quanet.n_classes) - with qp.util.temp_seed(0): - sampling_index_gen = data.artificial_sampling_index_generator(self.sample_size, prevpoints) - else: - sampling_index_gen = [data.sampling_index(self.sample_size, *prev) for prev in - F.uniform_simplex_sampling(data.n_classes, iterations)] - pbar = tqdm(sampling_index_gen, total=iterations) if train else sampling_index_gen - + sampler = UPP( + data, + sample_size=self.sample_size, + repeats=iterations, + random_state=None if train else 0 # different samples during train, same samples during validation + ) + pbar = tqdm(sampler.samples_parameters(), total=sampler.total()) for it, index in enumerate(pbar): sample_data = data.sampling_from_index(index) sample_posteriors = posteriors[index] @@ -264,7 +263,7 @@ class QuaNetTrainer(BaseQuantifier): f'patience={early_stop.patience}/{early_stop.PATIENCE_LIMIT}') def get_params(self, deep=True): - return {**self.learner.get_params(), **self.quanet_params} + return {**self.classifier.get_params(), **self.quanet_params} def set_params(self, **parameters): learner_params = {} @@ -273,7 +272,7 @@ class QuaNetTrainer(BaseQuantifier): self.quanet_params[key] = val else: learner_params[key] = val - self.learner.set_params(**learner_params) + self.classifier.set_params(**learner_params) def __check_params_colision(self, quanet_params, learner_params): quanet_keys = set(quanet_params.keys()) @@ -281,7 +280,7 @@ class QuaNetTrainer(BaseQuantifier): intersection = quanet_keys.intersection(learner_keys) if len(intersection) > 0: raise ValueError(f'the use of parameters {intersection} is ambiguous sine those can refer to ' - f'the parameters of QuaNet or the learner {self.learner.__class__.__name__}') + f'the parameters of QuaNet or the learner {self.classifier.__class__.__name__}') def clean_checkpoint(self): """ diff --git a/quapy/method/non_aggregative.py b/quapy/method/non_aggregative.py index f70a0c6..0a8680d 100644 --- a/quapy/method/non_aggregative.py +++ b/quapy/method/non_aggregative.py @@ -21,7 +21,6 @@ class MaximumLikelihoodPrevalenceEstimation(BaseQuantifier): :param data: the training sample :return: self """ - self._classes_ = data.classes_ self.estimated_prevalence = data.prevalence() return self @@ -34,29 +33,3 @@ class MaximumLikelihoodPrevalenceEstimation(BaseQuantifier): """ return self.estimated_prevalence - @property - def classes_(self): - """ - Number of classes - - :return: integer - """ - - return self._classes_ - - def get_params(self, deep=True): - """ - Does nothing, since this learner has no parameters. - - :param deep: for compatibility with sklearn - :return: `None` - """ - return None - - def set_params(self, **parameters): - """ - Does nothing, since this learner has no parameters. - - :param parameters: dictionary of param-value pairs (ignored) - """ - pass diff --git a/quapy/model_selection.py b/quapy/model_selection.py index 86e79f3..40b89c8 100644 --- a/quapy/model_selection.py +++ b/quapy/model_selection.py @@ -4,14 +4,14 @@ from copy import deepcopy from typing import Union, Callable import numpy as np +from sklearn import clone import quapy as qp +from quapy import evaluation +from quapy.protocol import AbstractProtocol, OnLabelledCollectionProtocol from quapy.data.base import LabelledCollection -from quapy.evaluation import artificial_prevalence_prediction, natural_prevalence_prediction, gen_prevalence_prediction from quapy.method.aggregative import BaseQuantifier -import inspect - -from util import _check_sample_size +from time import time class GridSearchQ(BaseQuantifier): @@ -23,33 +23,11 @@ class GridSearchQ(BaseQuantifier): :param model: the quantifier to optimize :type model: BaseQuantifier :param param_grid: a dictionary with keys the parameter names and values the list of values to explore - :param sample_size: the size of the samples to extract from the validation set (ignored if protocl='gen') - :param protocol: either 'app' for the artificial prevalence protocol, 'npp' for the natural prevalence - protocol, or 'gen' for using a custom sampling generator function - :param n_prevpoints: if specified, indicates the number of equally distant points to extract from the interval - [0,1] in order to define the prevalences of the samples; e.g., if n_prevpoints=5, then the prevalences for - each class will be explored in [0.00, 0.25, 0.50, 0.75, 1.00]. If not specified, then eval_budget is requested. - Ignored if protocol!='app'. - :param n_repetitions: the number of repetitions for each combination of prevalences. This parameter is ignored - for the protocol='app' if eval_budget is set and is lower than the number of combinations that would be - generated using the value assigned to n_prevpoints (for the current number of classes and n_repetitions). - Ignored for protocol='npp' and protocol='gen' (use eval_budget for setting a maximum number of samples in - those cases). - :param eval_budget: if specified, sets a ceil on the number of evaluations to perform for each hyper-parameter - combination. For example, if protocol='app', there are 3 classes, n_repetitions=1 and eval_budget=20, then - n_prevpoints will be set to 5, since this will generate 15 different prevalences, i.e., [0, 0, 1], - [0, 0.25, 0.75], [0, 0.5, 0.5] ... [1, 0, 0], and since setting it to 6 would generate more than - 20. When protocol='gen', indicates the maximum number of samples to generate, but less samples will be - generated if the generator yields less samples. + :param protocol: a sample generation protocol, an instance of :class:`quapy.protocol.AbstractProtocol` :param error: an error function (callable) or a string indicating the name of an error function (valid ones - are those in qp.error.QUANTIFICATION_ERROR + are those in :class:`quapy.error.QUANTIFICATION_ERROR` :param refit: whether or not to refit the model on the whole labelled collection (training+validation) with the best chosen hyperparameter combination. Ignored if protocol='gen' - :param val_split: either a LabelledCollection on which to test the performance of the different settings, or - a float in [0,1] indicating the proportion of labelled data to extract from the training set, or a callable - returning a generator function each time it is invoked (only for protocol='gen'). - :param n_jobs: number of parallel jobs - :param random_seed: set the seed of the random generator to replicate experiments. Ignored if protocol='gen'. :param timeout: establishes a timer (in seconds) for each of the hyperparameters configurations being tested. Whenever a run takes longer than this timer, that configuration will be ignored. If all configurations end up being ignored, a TimeoutError exception is raised. If -1 (default) then no time bound is set. @@ -59,65 +37,27 @@ class GridSearchQ(BaseQuantifier): def __init__(self, model: BaseQuantifier, param_grid: dict, - sample_size: Union[int, None] = None, - protocol='app', - n_prevpoints: int = None, - n_repetitions: int = 1, - eval_budget: int = None, + protocol: AbstractProtocol, error: Union[Callable, str] = qp.error.mae, refit=True, - val_split=0.4, - n_jobs=1, - random_seed=42, timeout=-1, + n_jobs=None, verbose=False): self.model = model self.param_grid = param_grid - self.sample_size = sample_size - self.protocol = protocol.lower() - self.n_prevpoints = n_prevpoints - self.n_repetitions = n_repetitions - self.eval_budget = eval_budget + self.protocol = protocol self.refit = refit - self.val_split = val_split - self.n_jobs = n_jobs - self.random_seed = random_seed self.timeout = timeout + self.n_jobs = qp._get_njobs(n_jobs) self.verbose = verbose self.__check_error(error) - assert self.protocol in {'app', 'npp', 'gen'}, \ - 'unknown protocol: valid ones are "app" or "npp" for the "artificial" or the "natural" prevalence ' \ - 'protocols. Use protocol="gen" when passing a generator function thorough val_split that yields a ' \ - 'sample (instances) and their prevalence (ndarray) at each iteration.' - assert self.eval_budget is None or isinstance(self.eval_budget, int) - if self.protocol in ['npp', 'gen']: - if self.protocol=='npp' and (self.eval_budget is None or self.eval_budget <= 0): - raise ValueError(f'when protocol="npp" the parameter eval_budget should be ' - f'indicated (and should be >0).') - if self.n_repetitions != 1: - print('[warning] n_repetitions has been set and will be ignored for the selected protocol') + assert isinstance(protocol, AbstractProtocol), 'unknown protocol' def _sout(self, msg): if self.verbose: print(f'[{self.__class__.__name__}]: {msg}') - def __check_training_validation(self, training, validation): - if isinstance(validation, LabelledCollection): - return training, validation - elif isinstance(validation, float): - assert 0. < validation < 1., 'validation proportion should be in (0,1)' - training, validation = training.split_stratified(train_prop=1 - validation, random_state=self.random_seed) - return training, validation - elif self.protocol=='gen' and inspect.isgenerator(validation()): - return training, validation - else: - raise ValueError(f'"validation" must either be a LabelledCollection or a float in (0,1) indicating the' - f'proportion of training documents to extract (type found: {type(validation)}). ' - f'Optionally, "validation" can be a callable function returning a generator that yields ' - f'the sample instances along with their true prevalence at each iteration by ' - f'setting protocol="gen".') - def __check_error(self, error): if error in qp.error.QUANTIFICATION_ERROR: self.error = error @@ -129,95 +69,103 @@ class GridSearchQ(BaseQuantifier): raise ValueError(f'unexpected error type; must either be a callable function or a str representing\n' f'the name of an error function in {qp.error.QUANTIFICATION_ERROR_NAMES}') - def __generate_predictions(self, model, val_split): - commons = { - 'n_repetitions': self.n_repetitions, - 'n_jobs': self.n_jobs, - 'random_seed': self.random_seed, - 'verbose': False - } - if self.protocol == 'app': - return artificial_prevalence_prediction( - model, val_split, self.sample_size, - n_prevpoints=self.n_prevpoints, - eval_budget=self.eval_budget, - **commons - ) - elif self.protocol == 'npp': - return natural_prevalence_prediction( - model, val_split, self.sample_size, - **commons) - elif self.protocol == 'gen': - return gen_prevalence_prediction(model, gen_fn=val_split, eval_budget=self.eval_budget) - else: - raise ValueError('unknown protocol') - - def fit(self, training: LabelledCollection, val_split: Union[LabelledCollection, float, Callable] = None): + def fit(self, training: LabelledCollection): """ Learning routine. Fits methods with all combinations of hyperparameters and selects the one minimizing the error metric. :param training: the training set on which to optimize the hyperparameters - :param val_split: either a LabelledCollection on which to test the performance of the different settings, or - a float in [0,1] indicating the proportion of labelled data to extract from the training set :return: self """ - if val_split is None: - val_split = self.val_split - training, val_split = self.__check_training_validation(training, val_split) - if self.protocol != 'gen': - self.sample_size = _check_sample_size(self.sample_size) - params_keys = list(self.param_grid.keys()) params_values = list(self.param_grid.values()) - model = self.model + protocol = self.protocol + + self.param_scores_ = {} + self.best_score_ = None + + tinit = time() + + hyper = [dict({k: val[i] for i, k in enumerate(params_keys)}) for val in itertools.product(*params_values)] + self._sout(f'starting model selection with {self.n_jobs =}') + #pass a seed to parallel so it is set in clild processes + scores = qp.util.parallel( + self._delayed_eval, + ((params, training) for params in hyper), + seed=qp.environ.get('_R_SEED', None), + n_jobs=self.n_jobs + ) + + for params, score, model in scores: + if score is not None: + if self.best_score_ is None or score < self.best_score_: + self.best_score_ = score + self.best_params_ = params + self.best_model_ = model + self.param_scores_[str(params)] = score + else: + self.param_scores_[str(params)] = 'timeout' + + tend = time()-tinit + + if self.best_score_ is None: + raise TimeoutError('no combination of hyperparameters seem to work') + + self._sout(f'optimization finished: best params {self.best_params_} (score={self.best_score_:.5f}) ' + f'[took {tend:.4f}s]') + + if self.refit: + if isinstance(protocol, OnLabelledCollectionProtocol): + self._sout(f'refitting on the whole development set') + self.best_model_.fit(training + protocol.get_labelled_collection()) + else: + raise RuntimeWarning(f'"refit" was requested, but the protocol does not ' + f'implement the {OnLabelledCollectionProtocol.__name__} interface') + + return self + + def _delayed_eval(self, args): + params, training = args + + protocol = self.protocol + error = self.error if self.timeout > 0: def handler(signum, frame): - self._sout('timeout reached') raise TimeoutError() signal.signal(signal.SIGALRM, handler) - self.param_scores_ = {} - self.best_score_ = None - some_timeouts = False - for values in itertools.product(*params_values): - params = dict({k: values[i] for i, k in enumerate(params_keys)}) + tinit = time() + + if self.timeout > 0: + signal.alarm(self.timeout) + + try: + model = deepcopy(self.model) + # overrides default parameters with the parameters being explored at this iteration + model.set_params(**params) + model.fit(training) + score = evaluation.evaluate(model, protocol=protocol, error_metric=error) + + ttime = time()-tinit + self._sout(f'hyperparams={params}\t got {error.__name__} score {score:.5f} [took {ttime:.4f}s]') if self.timeout > 0: - signal.alarm(self.timeout) + signal.alarm(0) + except TimeoutError: + self._sout(f'timeout ({self.timeout}s) reached for config {params}') + score = None + except ValueError as e: + self._sout(f'the combination of hyperparameters {params} is invalid') + raise e + except Exception as e: + self._sout(f'something went wrong for config {params}; skipping:') + self._sout(f'\tException: {e}') + score = None - try: - # overrides default parameters with the parameters being explored at this iteration - model.set_params(**params) - model.fit(training) - true_prevalences, estim_prevalences = self.__generate_predictions(model, val_split) - score = self.error(true_prevalences, estim_prevalences) + return params, score, model - self._sout(f'checking hyperparams={params} got {self.error.__name__} score {score:.5f}') - if self.best_score_ is None or score < self.best_score_: - self.best_score_ = score - self.best_params_ = params - self.best_model_ = deepcopy(model) - self.param_scores_[str(params)] = score - - if self.timeout > 0: - signal.alarm(0) - except TimeoutError: - print(f'timeout reached for config {params}') - some_timeouts = True - - if self.best_score_ is None and some_timeouts: - raise TimeoutError('all jobs took more than the timeout time to end') - - self._sout(f'optimization finished: best params {self.best_params_} (score={self.best_score_:.5f})') - - if self.refit: - self._sout(f'refitting on the whole development set') - self.best_model_.fit(training + val_split) - - return self def quantify(self, instances): """Estimate class prevalence values using the best model found after calling the :meth:`fit` method. @@ -229,14 +177,6 @@ class GridSearchQ(BaseQuantifier): assert hasattr(self, 'best_model_'), 'quantify called before fit' return self.best_model().quantify(instances) - @property - def classes_(self): - """ - Classes on which the quantifier has been trained on. - :return: a ndarray of shape `(n_classes)` with the class identifiers - """ - return self.best_model().classes_ - def set_params(self, **parameters): """Sets the hyper-parameters to explore. @@ -262,3 +202,30 @@ class GridSearchQ(BaseQuantifier): if hasattr(self, 'best_model_'): return self.best_model_ raise ValueError('best_model called before fit') + + + + +def cross_val_predict(quantifier: BaseQuantifier, data: LabelledCollection, nfolds=3, random_state=0): + """ + Akin to `scikit-learn's cross_val_predict `_ + but for quantification. + + :param quantifier: a quantifier issuing class prevalence values + :param data: a labelled collection + :param nfolds: number of folds for k-fold cross validation generation + :param random_state: random seed for reproducibility + :return: a vector of class prevalence values + """ + + total_prev = np.zeros(shape=data.n_classes) + + for train, test in data.kFCV(nfolds=nfolds, random_state=random_state): + quantifier.fit(train) + fold_prev = quantifier.quantify(test.X) + rel_size = len(test.X)/len(data) + total_prev += fold_prev*rel_size + + return total_prev + + diff --git a/quapy/plot.py b/quapy/plot.py index cafe520..6552765 100644 --- a/quapy/plot.py +++ b/quapy/plot.py @@ -4,6 +4,8 @@ from matplotlib.cm import get_cmap import numpy as np from matplotlib import cm from scipy.stats import ttest_ind_from_stats +from matplotlib.ticker import ScalarFormatter +import math import quapy as qp @@ -49,9 +51,10 @@ def binary_diagonal(method_names, true_prevs, estim_prevs, pos_class=1, title=No table = {method_name:[true_prev, estim_prev] for method_name, true_prev, estim_prev in order} order = [(method_name, *table[method_name]) for method_name in method_order] - #cm = plt.get_cmap('tab20') - #NUM_COLORS = len(method_names) - #ax.set_prop_cycle(color=[cm(1. * i / NUM_COLORS) for i in range(NUM_COLORS)]) + NUM_COLORS = len(method_names) + if NUM_COLORS>10: + cm = plt.get_cmap('tab20') + ax.set_prop_cycle(color=[cm(1. * i / NUM_COLORS) for i in range(NUM_COLORS)]) for method, true_prev, estim_prev in order: true_prev = true_prev[:,pos_class] estim_prev = estim_prev[:,pos_class] @@ -74,13 +77,12 @@ def binary_diagonal(method_names, true_prevs, estim_prevs, pos_class=1, title=No ax.set_xlim(0, 1) if legend: + ax.legend(loc='center left', bbox_to_anchor=(1, 0.5)) # box = ax.get_position() # ax.set_position([box.x0, box.y0, box.width * 0.8, box.height]) - ax.legend(loc='center left', bbox_to_anchor=(1, 0.5)) - # ax.set_position([box.x0, box.y0, box.width * 0.8, box.height]) - #ax.legend(loc='lower center', - # bbox_to_anchor=(1, -0.5), - # ncol=(len(method_names)+1)//2) + # ax.legend(loc='lower center', + # bbox_to_anchor=(1, -0.5), + # ncol=(len(method_names)+1)//2) _save_or_show(savepath) @@ -212,6 +214,7 @@ def binary_bias_bins(method_names, true_prevs, estim_prevs, pos_class=1, title=N def error_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, n_bins=20, error_name='ae', show_std=False, show_density=True, + show_legend=True, logscale=False, title=f'Quantification error as a function of distribution shift', vlines=None, @@ -234,6 +237,7 @@ def error_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, :param error_name: a string representing the name of an error function (as defined in `quapy.error`, default is "ae") :param show_std: whether or not to show standard deviations as color bands (default is False) :param show_density: whether or not to display the distribution of experiments for each bin (default is True) + :param show_density: whether or not to display the legend of the chart (default is True) :param logscale: whether or not to log-scale the y-error measure (default is False) :param title: title of the plot (default is "Quantification error as a function of distribution shift") :param vlines: array-like list of values (default is None). If indicated, highlights some regions of the space @@ -254,6 +258,9 @@ def error_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, # x_error function) and 'y' is the estim-test shift (computed as according to y_error) data = _join_data_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, x_error, y_error, method_order) + if method_order is None: + method_order = method_names + _set_colors(ax, n_methods=len(method_order)) bins = np.linspace(0, 1, n_bins+1) @@ -264,7 +271,10 @@ def error_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, tr_test_drifts = data[method]['x'] method_drifts = data[method]['y'] if logscale: - method_drifts=np.log(1+method_drifts) + ax.set_yscale("log") + ax.yaxis.set_major_formatter(ScalarFormatter()) + ax.yaxis.get_major_formatter().set_scientific(False) + ax.minorticks_off() inds = np.digitize(tr_test_drifts, bins, right=True) @@ -295,9 +305,14 @@ def error_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, ax.fill_between(xs, ys-ystds, ys+ystds, alpha=0.25) if show_density: - ax.bar([ind * binwidth-binwidth/2 for ind in range(len(bins))], - max_y*npoints/np.max(npoints), alpha=0.15, color='g', width=binwidth, label='density') - + ax2 = ax.twinx() + densities = npoints/np.sum(npoints) + ax2.bar([ind * binwidth-binwidth/2 for ind in range(len(bins))], + densities, alpha=0.15, color='g', width=binwidth, label='density') + ax2.set_ylim(0,max(densities)) + ax2.spines['right'].set_color('g') + ax2.tick_params(axis='y', colors='g') + ax.set(xlabel=f'Distribution shift between training set and test sample', ylabel=f'{error_name.upper()} (true distribution, predicted distribution)', title=title) @@ -306,9 +321,18 @@ def error_by_drift(method_names, true_prevs, estim_prevs, tr_prevs, if vlines: for vline in vlines: ax.axvline(vline, 0, 1, linestyle='--', color='k') - ax.set_xlim(0, max_x) - ax.legend(loc='center left', bbox_to_anchor=(1, 0.5)) + ax.set_xlim(min_x, max_x) + if logscale: + #nice scale for the logaritmic axis + ax.set_ylim(0,10 ** math.ceil(math.log10(max_y))) + + + if show_legend: + fig.legend(loc='lower center', + bbox_to_anchor=(1, 0.5), + ncol=(len(method_names)+1)//2) + _save_or_show(savepath) @@ -370,7 +394,7 @@ def brokenbar_supremacy_by_drift(method_names, true_prevs, estim_prevs, tr_prevs bins[-1] += 0.001 # we use this to keep track of how many datapoits contribute to each bin - inds_histogram_global = np.zeros(n_bins, dtype=np.float) + inds_histogram_global = np.zeros(n_bins, dtype=float) n_methods = len(method_order) buckets = np.zeros(shape=(n_methods, n_bins, 3)) for i, method in enumerate(method_order): diff --git a/quapy/protocol.py b/quapy/protocol.py new file mode 100644 index 0000000..9361f1d --- /dev/null +++ b/quapy/protocol.py @@ -0,0 +1,490 @@ +from copy import deepcopy +import quapy as qp +import numpy as np +import itertools +from contextlib import ExitStack +from abc import ABCMeta, abstractmethod +from quapy.data import LabelledCollection +import quapy.functional as F +from os.path import exists +from glob import glob + + +class AbstractProtocol(metaclass=ABCMeta): + """ + Abstract parent class for sample generation protocols. + """ + + @abstractmethod + def __call__(self): + """ + Implements the protocol. Yields one sample at a time along with its prevalence + + :return: yields a tuple `(sample, prev) at a time, where `sample` is a set of instances + and in which `prev` is an `nd.array` with the class prevalence values + """ + ... + + def total(self): + """ + Indicates the total number of samples that the protocol generates. + + :return: The number of samples to generate if known, or `None` otherwise. + """ + return None + + +class IterateProtocol(AbstractProtocol): + """ + A very simple protocol which simply iterates over a list of previously generated samples + + :param samples: a list of :class:`quapy.data.base.LabelledCollection` + """ + def __init__(self, samples: [LabelledCollection]): + self.samples = samples + + def __call__(self): + """ + Yields one sample from the initial list at a time + + :return: yields a tuple `(sample, prev) at a time, where `sample` is a set of instances + and in which `prev` is an `nd.array` with the class prevalence values + """ + for sample in self.samples: + yield sample.Xp + + def total(self): + """ + Returns the number of samples in this protocol + + :return: int + """ + return len(self.samples) + + +class AbstractStochasticSeededProtocol(AbstractProtocol): + """ + An `AbstractStochasticSeededProtocol` is a protocol that generates, via any random procedure (e.g., + via random sampling), sequences of :class:`quapy.data.base.LabelledCollection` samples. + The protocol abstraction enforces + the object to be instantiated using a seed, so that the sequence can be fully replicated. + In order to make this functionality possible, the classes extending this abstraction need to + implement only two functions, :meth:`samples_parameters` which generates all the parameters + needed for extracting the samples, and :meth:`sample` that, given some parameters as input, + deterministically generates a sample. + + :param random_state: the seed for allowing to replicate any sequence of samples. Default is 0, meaning that + the sequence will be consistent every time the protocol is called. + """ + + _random_state = -1 # means "not set" + + def __init__(self, random_state=0): + self.random_state = random_state + + @property + def random_state(self): + return self._random_state + + @random_state.setter + def random_state(self, random_state): + self._random_state = random_state + + @abstractmethod + def samples_parameters(self): + """ + This function has to return all the necessary parameters to replicate the samples + + :return: a list of parameters, each of which serves to deterministically generate a sample + """ + ... + + @abstractmethod + def sample(self, params): + """ + Extract one sample determined by the given parameters + + :param params: all the necessary parameters to generate a sample + :return: one sample (the same sample has to be generated for the same parameters) + """ + ... + + def __call__(self): + """ + Yields one sample at a time. The type of object returned depends on the `collator` function. The + default behaviour returns tuples of the form `(sample, prevalence)`. + + :return: a tuple `(sample, prevalence)` if return_type='sample_prev', or an instance of + :class:`qp.data.LabelledCollection` if return_type='labelled_collection' + """ + with ExitStack() as stack: + if self.random_state == -1: + raise ValueError('The random seed has never been initialized. ' + 'Set it to None not to impose replicability.') + if self.random_state is not None: + stack.enter_context(qp.util.temp_seed(self.random_state)) + for params in self.samples_parameters(): + yield self.collator(self.sample(params)) + + def collator(self, sample, *args): + """ + The collator prepares the sample to accommodate the desired output format before returning the output. + This collator simply returns the sample as it is. Classes inheriting from this abstract class can + implement their custom collators. + + :param sample: the sample to be returned + :param args: additional arguments + :return: the sample adhering to a desired output format (in this case, the sample is returned as it is) + """ + return sample + + +class OnLabelledCollectionProtocol: + """ + Protocols that generate samples from a :class:`qp.data.LabelledCollection` object. + """ + + RETURN_TYPES = ['sample_prev', 'labelled_collection', 'index'] + + def get_labelled_collection(self): + """ + Returns the labelled collection on which this protocol acts. + + :return: an object of type :class:`qp.data.LabelledCollection` + """ + return self.data + + def on_preclassified_instances(self, pre_classifications, in_place=False): + """ + Returns a copy of this protocol that acts on a modified version of the original + :class:`qp.data.LabelledCollection` in which the original instances have been replaced + with the outputs of a classifier for each instance. (This is convenient for speeding-up + the evaluation procedures for many samples, by pre-classifying the instances in advance.) + + :param pre_classifications: the predictions issued by a classifier, typically an array-like + with shape `(n_instances,)` when the classifier is a hard one, or with shape + `(n_instances, n_classes)` when the classifier is a probabilistic one. + :param in_place: whether or not to apply the modification in-place or in a new copy (default). + :return: a copy of this protocol + """ + assert len(pre_classifications) == len(self.data), \ + f'error: the pre-classified data has different shape ' \ + f'(expected {len(self.data)}, found {len(pre_classifications)})' + if in_place: + self.data.instances = pre_classifications + return self + else: + new = deepcopy(self) + return new.on_preclassified_instances(pre_classifications, in_place=True) + + @classmethod + def get_collator(cls, return_type='sample_prev'): + """ + Returns a collator function, i.e., a function that prepares the yielded data + + :param return_type: either 'sample_prev' (default) if the collator is requested to yield tuples of + `(sample, prevalence)`, or 'labelled_collection' when it is requested to yield instances of + :class:`qp.data.LabelledCollection` + :return: the collator function (a callable function that takes as input an instance of + :class:`qp.data.LabelledCollection`) + """ + assert return_type in cls.RETURN_TYPES, \ + f'unknown return type passed as argument; valid ones are {cls.RETURN_TYPES}' + if return_type=='sample_prev': + return lambda lc:lc.Xp + elif return_type=='labelled_collection': + return lambda lc:lc + + +class APP(AbstractStochasticSeededProtocol, OnLabelledCollectionProtocol): + """ + Implementation of the artificial prevalence protocol (APP). + The APP consists of exploring a grid of prevalence values containing `n_prevalences` points (e.g., + [0, 0.05, 0.1, 0.15, ..., 1], if `n_prevalences=21`), and generating all valid combinations of + prevalence values for all classes (e.g., for 3 classes, samples with [0, 0, 1], [0, 0.05, 0.95], ..., + [1, 0, 0] prevalence values of size `sample_size` will be yielded). The number of samples for each valid + combination of prevalence values is indicated by `repeats`. + + :param data: a `LabelledCollection` from which the samples will be drawn + :param sample_size: integer, number of instances in each sample; if None (default) then it is taken from + qp.environ["SAMPLE_SIZE"]. If this is not set, a ValueError exception is raised. + :param n_prevalences: the number of equidistant prevalence points to extract from the [0,1] interval for the + grid (default is 21) + :param repeats: number of copies for each valid prevalence vector (default is 10) + :param smooth_limits_epsilon: the quantity to add and subtract to the limits 0 and 1 + :param random_state: allows replicating samples across runs (default 0, meaning that the sequence of samples + will be the same every time the protocol is called) + :param return_type: set to "sample_prev" (default) to get the pairs of (sample, prevalence) at each iteration, or + to "labelled_collection" to get instead instances of LabelledCollection + """ + + def __init__(self, data:LabelledCollection, sample_size=None, n_prevalences=21, repeats=10, + smooth_limits_epsilon=0, random_state=0, return_type='sample_prev'): + super(APP, self).__init__(random_state) + self.data = data + self.sample_size = qp._get_sample_size(sample_size) + self.n_prevalences = n_prevalences + self.repeats = repeats + self.smooth_limits_epsilon = smooth_limits_epsilon + self.collator = OnLabelledCollectionProtocol.get_collator(return_type) + + def prevalence_grid(self): + """ + Generates vectors of prevalence values from an exhaustive grid of prevalence values. The + number of prevalence values explored for each dimension depends on `n_prevalences`, so that, if, for example, + `n_prevalences=11` then the prevalence values of the grid are taken from [0, 0.1, 0.2, ..., 0.9, 1]. Only + valid prevalence distributions are returned, i.e., vectors of prevalence values that sum up to 1. For each + valid vector of prevalence values, `repeat` copies are returned. The vector of prevalence values can be + implicit (by setting `return_constrained_dim=False`), meaning that the last dimension (which is constrained + to 1 - sum of the rest) is not returned (note that, quite obviously, in this case the vector does not sum up to + 1). Note that this method is deterministic, i.e., there is no random sampling anywhere. + + :return: a `np.ndarray` of shape `(n, dimensions)` if `return_constrained_dim=True` or of shape + `(n, dimensions-1)` if `return_constrained_dim=False`, where `n` is the number of valid combinations found + in the grid multiplied by `repeat` + """ + dimensions = self.data.n_classes + s = F.prevalence_linspace(self.n_prevalences, repeats=1, smooth_limits_epsilon=self.smooth_limits_epsilon) + s = [s] * (dimensions - 1) + prevs = [p for p in itertools.product(*s, repeat=1) if (sum(p) <= 1.0)] + prevs = np.asarray(prevs).reshape(len(prevs), -1) + if self.repeats > 1: + prevs = np.repeat(prevs, self.repeats, axis=0) + return prevs + + def samples_parameters(self): + """ + Return all the necessary parameters to replicate the samples as according to the APP protocol. + + :return: a list of indexes that realize the APP sampling + """ + indexes = [] + for prevs in self.prevalence_grid(): + index = self.data.sampling_index(self.sample_size, *prevs) + indexes.append(index) + return indexes + + def sample(self, index): + """ + Realizes the sample given the index of the instances. + + :param index: indexes of the instances to select + :return: an instance of :class:`qp.data.LabelledCollection` + """ + return self.data.sampling_from_index(index) + + def total(self): + """ + Returns the number of samples that will be generated + + :return: int + """ + return F.num_prevalence_combinations(self.n_prevalences, self.data.n_classes, self.repeats) + + +class NPP(AbstractStochasticSeededProtocol, OnLabelledCollectionProtocol): + """ + A generator of samples that implements the natural prevalence protocol (NPP). The NPP consists of drawing + samples uniformly at random, therefore approximately preserving the natural prevalence of the collection. + + :param data: a `LabelledCollection` from which the samples will be drawn + :param sample_size: integer, the number of instances in each sample; if None (default) then it is taken from + qp.environ["SAMPLE_SIZE"]. If this is not set, a ValueError exception is raised. + :param repeats: the number of samples to generate. Default is 100. + :param random_state: allows replicating samples across runs (default 0, meaning that the sequence of samples + will be the same every time the protocol is called) + :param return_type: set to "sample_prev" (default) to get the pairs of (sample, prevalence) at each iteration, or + to "labelled_collection" to get instead instances of LabelledCollection + """ + + def __init__(self, data:LabelledCollection, sample_size=None, repeats=100, random_state=0, + return_type='sample_prev'): + super(NPP, self).__init__(random_state) + self.data = data + self.sample_size = qp._get_sample_size(sample_size) + self.repeats = repeats + self.random_state = random_state + self.collator = OnLabelledCollectionProtocol.get_collator(return_type) + + def samples_parameters(self): + """ + Return all the necessary parameters to replicate the samples as according to the NPP protocol. + + :return: a list of indexes that realize the NPP sampling + """ + indexes = [] + for _ in range(self.repeats): + index = self.data.uniform_sampling_index(self.sample_size) + indexes.append(index) + return indexes + + def sample(self, index): + """ + Realizes the sample given the index of the instances. + + :param index: indexes of the instances to select + :return: an instance of :class:`qp.data.LabelledCollection` + """ + return self.data.sampling_from_index(index) + + def total(self): + """ + Returns the number of samples that will be generated (equals to "repeats") + + :return: int + """ + return self.repeats + + +class UPP(AbstractStochasticSeededProtocol, OnLabelledCollectionProtocol): + """ + A variant of :class:`APP` that, instead of using a grid of equidistant prevalence values, + relies on the Kraemer algorithm for sampling unit (k-1)-simplex uniformly at random, with + k the number of classes. This protocol covers the entire range of prevalence values in a + statistical sense, i.e., unlike APP there is no guarantee that it is covered precisely + equally for all classes, but it is preferred in cases in which the number of possible + combinations of the grid values of APP makes this endeavour intractable. + + :param data: a `LabelledCollection` from which the samples will be drawn + :param sample_size: integer, the number of instances in each sample; if None (default) then it is taken from + qp.environ["SAMPLE_SIZE"]. If this is not set, a ValueError exception is raised. + :param repeats: the number of samples to generate. Default is 100. + :param random_state: allows replicating samples across runs (default 0, meaning that the sequence of samples + will be the same every time the protocol is called) + :param return_type: set to "sample_prev" (default) to get the pairs of (sample, prevalence) at each iteration, or + to "labelled_collection" to get instead instances of LabelledCollection + """ + + def __init__(self, data: LabelledCollection, sample_size=None, repeats=100, random_state=0, + return_type='sample_prev'): + super(UPP, self).__init__(random_state) + self.data = data + self.sample_size = qp._get_sample_size(sample_size) + self.repeats = repeats + self.random_state = random_state + self.collator = OnLabelledCollectionProtocol.get_collator(return_type) + + def samples_parameters(self): + """ + Return all the necessary parameters to replicate the samples as according to the UPP protocol. + + :return: a list of indexes that realize the UPP sampling + """ + indexes = [] + for prevs in F.uniform_simplex_sampling(n_classes=self.data.n_classes, size=self.repeats): + index = self.data.sampling_index(self.sample_size, *prevs) + indexes.append(index) + return indexes + + def sample(self, index): + """ + Realizes the sample given the index of the instances. + + :param index: indexes of the instances to select + :return: an instance of :class:`qp.data.LabelledCollection` + """ + return self.data.sampling_from_index(index) + + def total(self): + """ + Returns the number of samples that will be generated (equals to "repeats") + + :return: int + """ + return self.repeats + + +class DomainMixer(AbstractStochasticSeededProtocol): + """ + Generates mixtures of two domains (A and B) at controlled rates, but preserving the original class prevalence. + + :param domainA: one domain, an object of :class:`qp.data.LabelledCollection` + :param domainB: another domain, an object of :class:`qp.data.LabelledCollection` + :param sample_size: integer, the number of instances in each sample; if None (default) then it is taken from + qp.environ["SAMPLE_SIZE"]. If this is not set, a ValueError exception is raised. + :param repeats: int, number of samples to draw for every mixture rate + :param prevalence: the prevalence to preserv along the mixtures. If specified, should be an array containing + one prevalence value (positive float) for each class and summing up to one. If not specified, the prevalence + will be taken from the domain A (default). + :param mixture_points: an integer indicating the number of points to take from a linear scale (e.g., 21 will + generate the mixture points [1, 0.95, 0.9, ..., 0]), or the array of mixture values itself. + the specific points + :param random_state: allows replicating samples across runs (default 0, meaning that the sequence of samples + will be the same every time the protocol is called) + """ + + def __init__( + self, + domainA: LabelledCollection, + domainB: LabelledCollection, + sample_size, + repeats=1, + prevalence=None, + mixture_points=11, + random_state=0, + return_type='sample_prev'): + super(DomainMixer, self).__init__(random_state) + self.A = domainA + self.B = domainB + self.sample_size = qp._get_sample_size(sample_size) + self.repeats = repeats + if prevalence is None: + self.prevalence = domainA.prevalence() + else: + self.prevalence = np.asarray(prevalence) + assert len(self.prevalence) == domainA.n_classes, \ + f'wrong shape for the vector prevalence (expected {domainA.n_classes})' + assert F.check_prevalence_vector(self.prevalence), \ + f'the prevalence vector is not valid (either it contains values outside [0,1] or does not sum up to 1)' + if isinstance(mixture_points, int): + self.mixture_points = np.linspace(0, 1, mixture_points)[::-1] + else: + self.mixture_points = np.asarray(mixture_points) + assert all(np.logical_and(self.mixture_points >= 0, self.mixture_points<=1)), \ + 'mixture_model datatype not understood (expected int or a sequence of real values in [0,1])' + self.random_state = random_state + self.collator = OnLabelledCollectionProtocol.get_collator(return_type) + + def samples_parameters(self): + """ + Return all the necessary parameters to replicate the samples as according to the this protocol. + + :return: a list of zipped indexes (from A and B) that realize the sampling + """ + indexesA, indexesB = [], [] + for propA in self.mixture_points: + for _ in range(self.repeats): + nA = int(np.round(self.sample_size * propA)) + nB = self.sample_size-nA + sampleAidx = self.A.sampling_index(nA, *self.prevalence) + sampleBidx = self.B.sampling_index(nB, *self.prevalence) + indexesA.append(sampleAidx) + indexesB.append(sampleBidx) + return list(zip(indexesA, indexesB)) + + def sample(self, indexes): + """ + Realizes the sample given a pair of indexes of the instances from A and B. + + :param indexes: indexes of the instances to select from A and B + :return: an instance of :class:`qp.data.LabelledCollection` + """ + indexesA, indexesB = indexes + sampleA = self.A.sampling_from_index(indexesA) + sampleB = self.B.sampling_from_index(indexesB) + return sampleA+sampleB + + def total(self): + """ + Returns the number of samples that will be generated (equals to "repeats * mixture_points") + + :return: int + """ + return self.repeats * len(self.mixture_points) + + +# aliases + +ArtificialPrevalenceProtocol = APP +NaturalPrevalenceProtocol = NPP +UniformPrevalenceProtocol = UPP \ No newline at end of file diff --git a/quapy/tests/test_datasets.py b/quapy/tests/test_datasets.py index 88209e8..b0c2f7a 100644 --- a/quapy/tests/test_datasets.py +++ b/quapy/tests/test_datasets.py @@ -1,7 +1,8 @@ import pytest from quapy.data.datasets import REVIEWS_SENTIMENT_DATASETS, TWITTER_SENTIMENT_DATASETS_TEST, \ - TWITTER_SENTIMENT_DATASETS_TRAIN, UCI_DATASETS, fetch_reviews, fetch_twitter, fetch_UCIDataset + TWITTER_SENTIMENT_DATASETS_TRAIN, UCI_DATASETS, LEQUA2022_TASKS, \ + fetch_reviews, fetch_twitter, fetch_UCIDataset, fetch_lequa2022 @pytest.mark.parametrize('dataset_name', REVIEWS_SENTIMENT_DATASETS) @@ -41,3 +42,11 @@ def test_fetch_UCIDataset(dataset_name): print('Training set stats') dataset.training.stats() print('Test set stats') + + +@pytest.mark.parametrize('dataset_name', LEQUA2022_TASKS) +def test_fetch_lequa2022(dataset_name): + train, gen_val, gen_test = fetch_lequa2022(dataset_name) + print(train.stats()) + print('Val:', gen_val.total()) + print('Test:', gen_test.total()) diff --git a/quapy/tests/test_evaluation.py b/quapy/tests/test_evaluation.py new file mode 100644 index 0000000..4992d86 --- /dev/null +++ b/quapy/tests/test_evaluation.py @@ -0,0 +1,84 @@ +import unittest + +import numpy as np + +import quapy as qp +from sklearn.linear_model import LogisticRegression +from time import time + +from error import QUANTIFICATION_ERROR_SINGLE, QUANTIFICATION_ERROR, QUANTIFICATION_ERROR_NAMES, \ + QUANTIFICATION_ERROR_SINGLE_NAMES +from quapy.method.aggregative import EMQ, PCC +from quapy.method.base import BaseQuantifier + + +class EvalTestCase(unittest.TestCase): + def test_eval_speedup(self): + + data = qp.datasets.fetch_reviews('hp', tfidf=True, min_df=10, pickle=True) + train, test = data.training, data.test + + protocol = qp.protocol.APP(test, sample_size=1000, n_prevalences=11, repeats=1, random_state=1) + + class SlowLR(LogisticRegression): + def predict_proba(self, X): + import time + time.sleep(1) + return super().predict_proba(X) + + emq = EMQ(SlowLR()).fit(train) + + tinit = time() + score = qp.evaluation.evaluate(emq, protocol, error_metric='mae', verbose=True, aggr_speedup='force') + tend_optim = time()-tinit + print(f'evaluation (with optimization) took {tend_optim}s [MAE={score:.4f}]') + + class NonAggregativeEMQ(BaseQuantifier): + + def __init__(self, cls): + self.emq = EMQ(cls) + + def quantify(self, instances): + return self.emq.quantify(instances) + + def fit(self, data): + self.emq.fit(data) + return self + + emq = NonAggregativeEMQ(SlowLR()).fit(train) + + tinit = time() + score = qp.evaluation.evaluate(emq, protocol, error_metric='mae', verbose=True) + tend_no_optim = time() - tinit + print(f'evaluation (w/o optimization) took {tend_no_optim}s [MAE={score:.4f}]') + + self.assertEqual(tend_no_optim>(tend_optim/2), True) + + def test_evaluation_output(self): + + data = qp.datasets.fetch_reviews('hp', tfidf=True, min_df=10, pickle=True) + train, test = data.training, data.test + + qp.environ['SAMPLE_SIZE']=100 + + protocol = qp.protocol.APP(test, random_state=0) + + q = PCC(LogisticRegression()).fit(train) + + single_errors = list(QUANTIFICATION_ERROR_SINGLE_NAMES) + averaged_errors = ['m'+e for e in single_errors] + single_errors = single_errors + [qp.error.from_name(e) for e in single_errors] + averaged_errors = averaged_errors + [qp.error.from_name(e) for e in averaged_errors] + for error_metric, averaged_error_metric in zip(single_errors, averaged_errors): + score = qp.evaluation.evaluate(q, protocol, error_metric=averaged_error_metric) + self.assertTrue(isinstance(score, float)) + + scores = qp.evaluation.evaluate(q, protocol, error_metric=error_metric) + self.assertTrue(isinstance(scores, np.ndarray)) + + self.assertEqual(scores.mean(), score) + + + +if __name__ == '__main__': + unittest.main() diff --git a/quapy/tests/test_hierarchy.py b/quapy/tests/test_hierarchy.py new file mode 100644 index 0000000..2ea3af5 --- /dev/null +++ b/quapy/tests/test_hierarchy.py @@ -0,0 +1,31 @@ +import unittest + +from sklearn.linear_model import LogisticRegression + +import quapy as qp +from quapy.method.aggregative import * + + + +class HierarchyTestCase(unittest.TestCase): + + def test_aggregative(self): + lr = LogisticRegression() + for m in [CC(lr), PCC(lr), ACC(lr), PACC(lr)]: + self.assertEqual(isinstance(m, AggregativeQuantifier), True) + + def test_binary(self): + lr = LogisticRegression() + for m in [HDy(lr)]: + self.assertEqual(isinstance(m, BinaryQuantifier), True) + + def test_probabilistic(self): + lr = LogisticRegression() + for m in [CC(lr), ACC(lr)]: + self.assertEqual(isinstance(m, AggregativeProbabilisticQuantifier), False) + for m in [PCC(lr), PACC(lr)]: + self.assertEqual(isinstance(m, AggregativeProbabilisticQuantifier), True) + + +if __name__ == '__main__': + unittest.main() diff --git a/quapy/tests/test_labelcollection.py b/quapy/tests/test_labelcollection.py new file mode 100644 index 0000000..f596e9a --- /dev/null +++ b/quapy/tests/test_labelcollection.py @@ -0,0 +1,65 @@ +import unittest +import numpy as np +from scipy.sparse import csr_matrix + +import quapy as qp + + +class LabelCollectionTestCase(unittest.TestCase): + def test_split(self): + x = np.arange(100) + y = np.random.randint(0,5,100) + data = qp.data.LabelledCollection(x,y) + tr, te = data.split_random(0.7) + check_prev = tr.prevalence()*0.7 + te.prevalence()*0.3 + + self.assertEqual(len(tr), 70) + self.assertEqual(len(te), 30) + self.assertEqual(np.allclose(check_prev, data.prevalence()), True) + self.assertEqual(len(tr+te), len(data)) + + def test_join(self): + x = np.arange(50) + y = np.random.randint(2, 5, 50) + data1 = qp.data.LabelledCollection(x, y) + + x = np.arange(200) + y = np.random.randint(0, 3, 200) + data2 = qp.data.LabelledCollection(x, y) + + x = np.arange(100) + y = np.random.randint(0, 6, 100) + data3 = qp.data.LabelledCollection(x, y) + + combined = qp.data.LabelledCollection.join(data1, data2, data3) + self.assertEqual(len(combined), len(data1)+len(data2)+len(data3)) + self.assertEqual(all(combined.classes_ == np.arange(6)), True) + + x = np.random.rand(10, 3) + y = np.random.randint(0, 1, 10) + data4 = qp.data.LabelledCollection(x, y) + with self.assertRaises(Exception): + combined = qp.data.LabelledCollection.join(data1, data2, data3, data4) + + x = np.random.rand(20, 3) + y = np.random.randint(0, 1, 20) + data5 = qp.data.LabelledCollection(x, y) + combined = qp.data.LabelledCollection.join(data4, data5) + self.assertEqual(len(combined), len(data4)+len(data5)) + + x = np.random.rand(10, 4) + y = np.random.randint(0, 1, 10) + data6 = qp.data.LabelledCollection(x, y) + with self.assertRaises(Exception): + combined = qp.data.LabelledCollection.join(data4, data5, data6) + + data4.instances = csr_matrix(data4.instances) + with self.assertRaises(Exception): + combined = qp.data.LabelledCollection.join(data4, data5) + data5.instances = csr_matrix(data5.instances) + combined = qp.data.LabelledCollection.join(data4, data5) + self.assertEqual(len(combined), len(data4) + len(data5)) + + +if __name__ == '__main__': + unittest.main() diff --git a/quapy/tests/test_methods.py b/quapy/tests/test_methods.py index bcf721c..4da5617 100644 --- a/quapy/tests/test_methods.py +++ b/quapy/tests/test_methods.py @@ -4,24 +4,28 @@ from sklearn.linear_model import LogisticRegression from sklearn.svm import LinearSVC import quapy as qp +from quapy.method.base import BinaryQuantifier from quapy.data import Dataset, LabelledCollection -from quapy.method import AGGREGATIVE_METHODS, NON_AGGREGATIVE_METHODS, EXPLICIT_LOSS_MINIMIZATION_METHODS +from quapy.method import AGGREGATIVE_METHODS, NON_AGGREGATIVE_METHODS from quapy.method.aggregative import ACC, PACC, HDy from quapy.method.meta import Ensemble -datasets = [pytest.param(qp.datasets.fetch_twitter('hcr'), id='hcr'), +datasets = [pytest.param(qp.datasets.fetch_twitter('hcr', pickle=True), id='hcr'), pytest.param(qp.datasets.fetch_UCIDataset('ionosphere'), id='ionosphere')] +tinydatasets = [pytest.param(qp.datasets.fetch_twitter('hcr', pickle=True).reduce(), id='tiny_hcr'), + pytest.param(qp.datasets.fetch_UCIDataset('ionosphere').reduce(), id='tiny_ionosphere')] + learners = [LogisticRegression, LinearSVC] @pytest.mark.parametrize('dataset', datasets) -@pytest.mark.parametrize('aggregative_method', AGGREGATIVE_METHODS.difference(EXPLICIT_LOSS_MINIMIZATION_METHODS)) +@pytest.mark.parametrize('aggregative_method', AGGREGATIVE_METHODS) @pytest.mark.parametrize('learner', learners) def test_aggregative_methods(dataset: Dataset, aggregative_method, learner): model = aggregative_method(learner()) - if model.binary and not dataset.binary: + if isinstance(model, BinaryQuantifier) and not dataset.binary: print(f'skipping the test of binary model {type(model)} on non-binary dataset {dataset}') return @@ -35,36 +39,12 @@ def test_aggregative_methods(dataset: Dataset, aggregative_method, learner): assert type(error) == numpy.float64 -@pytest.mark.parametrize('dataset', datasets) -@pytest.mark.parametrize('elm_method', EXPLICIT_LOSS_MINIMIZATION_METHODS) -def test_elm_methods(dataset: Dataset, elm_method): - try: - model = elm_method() - except AssertionError as ae: - if ae.args[0].find('does not seem to point to a valid path') > 0: - print('Missing SVMperf binary program, skipping test') - return - - if model.binary and not dataset.binary: - print(f'skipping the test of binary model {model} on non-binary dataset {dataset}') - return - - model.fit(dataset.training) - - estim_prevalences = model.quantify(dataset.test.instances) - - true_prevalences = dataset.test.prevalence() - error = qp.error.mae(true_prevalences, estim_prevalences) - - assert type(error) == numpy.float64 - - @pytest.mark.parametrize('dataset', datasets) @pytest.mark.parametrize('non_aggregative_method', NON_AGGREGATIVE_METHODS) def test_non_aggregative_methods(dataset: Dataset, non_aggregative_method): model = non_aggregative_method() - if model.binary and not dataset.binary: + if isinstance(model, BinaryQuantifier) and not dataset.binary: print(f'skipping the test of binary model {model} on non-binary dataset {dataset}') return @@ -78,16 +58,20 @@ def test_non_aggregative_methods(dataset: Dataset, non_aggregative_method): assert type(error) == numpy.float64 -@pytest.mark.parametrize('base_method', AGGREGATIVE_METHODS.difference(EXPLICIT_LOSS_MINIMIZATION_METHODS)) -@pytest.mark.parametrize('learner', learners) -@pytest.mark.parametrize('dataset', datasets) +@pytest.mark.parametrize('base_method', AGGREGATIVE_METHODS) +@pytest.mark.parametrize('learner', [LogisticRegression]) +@pytest.mark.parametrize('dataset', tinydatasets) @pytest.mark.parametrize('policy', Ensemble.VALID_POLICIES) def test_ensemble_method(base_method, learner, dataset: Dataset, policy): - qp.environ['SAMPLE_SIZE'] = len(dataset.training) - model = Ensemble(quantifier=base_method(learner()), size=5, policy=policy, n_jobs=-1) - if model.binary and not dataset.binary: - print(f'skipping the test of binary model {model} on non-binary dataset {dataset}') + qp.environ['SAMPLE_SIZE'] = 20 + base_quantifier=base_method(learner()) + if isinstance(base_quantifier, BinaryQuantifier) and not dataset.binary: + print(f'skipping the test of binary model {base_quantifier} on non-binary dataset {dataset}') return + if not dataset.binary and policy=='ds': + print(f'skipping the test of binary policy ds on non-binary dataset {dataset}') + return + model = Ensemble(quantifier=base_quantifier, size=5, policy=policy, n_jobs=-1) model.fit(dataset.training) @@ -106,21 +90,25 @@ def test_quanet_method(): print('skipping QuaNet test due to missing torch package') return + + qp.environ['SAMPLE_SIZE'] = 100 + + # load the kindle dataset as text, and convert words to numerical indexes dataset = qp.datasets.fetch_reviews('kindle', pickle=True) - dataset = Dataset(dataset.training.sampling(100, *dataset.training.prevalence()), - dataset.test.sampling(100, *dataset.test.prevalence())) + dataset = Dataset(dataset.training.sampling(200, *dataset.training.prevalence()), + dataset.test.sampling(200, *dataset.test.prevalence())) qp.data.preprocessing.index(dataset, min_df=5, inplace=True) from quapy.classification.neural import CNNnet - cnn = CNNnet(dataset.vocabulary_size, dataset.training.n_classes) + cnn = CNNnet(dataset.vocabulary_size, dataset.n_classes) from quapy.classification.neural import NeuralClassifierTrainer learner = NeuralClassifierTrainer(cnn, device='cuda') from quapy.method.meta import QuaNet - model = QuaNet(learner, sample_size=len(dataset.training), device='cuda') + model = QuaNet(learner, device='cuda') - if model.binary and not dataset.binary: + if isinstance(model, BinaryQuantifier) and not dataset.binary: print(f'skipping the test of binary model {model} on non-binary dataset {dataset}') return @@ -134,28 +122,15 @@ def test_quanet_method(): assert type(error) == numpy.float64 -def models_to_test_for_str_label_names(): - models = list() - learner = LogisticRegression - for method in AGGREGATIVE_METHODS.difference(EXPLICIT_LOSS_MINIMIZATION_METHODS): - models.append(method(learner())) - for method in NON_AGGREGATIVE_METHODS: - models.append(method()) - return models - - -@pytest.mark.parametrize('model', models_to_test_for_str_label_names()) -def test_str_label_names(model): - if type(model) in {ACC, PACC, HDy}: - print( - f'skipping the test of binary model {type(model)} because it currently does not support random seed control.') - return +def test_str_label_names(): + model = qp.method.aggregative.CC(LogisticRegression()) dataset = qp.datasets.fetch_reviews('imdb', pickle=True) dataset = Dataset(dataset.training.sampling(1000, *dataset.training.prevalence()), - dataset.test.sampling(1000, *dataset.test.prevalence())) + dataset.test.sampling(1000, 0.25, 0.75)) qp.data.preprocessing.text2tfidf(dataset, min_df=5, inplace=True) + numpy.random.seed(0) model.fit(dataset.training) int_estim_prevalences = model.quantify(dataset.test.instances) @@ -168,7 +143,8 @@ def test_str_label_names(model): ['one' if label == 1 else 'zero' for label in dataset.training.labels]), LabelledCollection(dataset.test.instances, ['one' if label == 1 else 'zero' for label in dataset.test.labels])) - + assert all(dataset_str.training.classes_ == dataset_str.test.classes_), 'wrong indexation' + numpy.random.seed(0) model.fit(dataset_str.training) str_estim_prevalences = model.quantify(dataset_str.test.instances) diff --git a/quapy/tests/test_modsel.py b/quapy/tests/test_modsel.py new file mode 100644 index 0000000..180f680 --- /dev/null +++ b/quapy/tests/test_modsel.py @@ -0,0 +1,108 @@ +import unittest + +import numpy as np +from sklearn.linear_model import LogisticRegression +from sklearn.svm import SVC + +import quapy as qp +from quapy.method.aggregative import PACC +from quapy.model_selection import GridSearchQ +from quapy.protocol import APP +import time + + +class ModselTestCase(unittest.TestCase): + + def test_modsel(self): + + q = PACC(LogisticRegression(random_state=1, max_iter=5000)) + + data = qp.datasets.fetch_reviews('imdb', tfidf=True, min_df=10) + training, validation = data.training.split_stratified(0.7, random_state=1) + + param_grid = {'classifier__C': np.logspace(-3,3,7)} + app = APP(validation, sample_size=100, random_state=1) + q = GridSearchQ( + q, param_grid, protocol=app, error='mae', refit=True, timeout=-1, verbose=True + ).fit(training) + print('best params', q.best_params_) + print('best score', q.best_score_) + + self.assertEqual(q.best_params_['classifier__C'], 10.0) + self.assertEqual(q.best_model().get_params()['classifier__C'], 10.0) + + def test_modsel_parallel(self): + + q = PACC(LogisticRegression(random_state=1, max_iter=5000)) + + data = qp.datasets.fetch_reviews('imdb', tfidf=True, min_df=10) + training, validation = data.training.split_stratified(0.7, random_state=1) + # test = data.test + + param_grid = {'classifier__C': np.logspace(-3,3,7)} + app = APP(validation, sample_size=100, random_state=1) + q = GridSearchQ( + q, param_grid, protocol=app, error='mae', refit=True, timeout=-1, n_jobs=-1, verbose=True + ).fit(training) + print('best params', q.best_params_) + print('best score', q.best_score_) + + self.assertEqual(q.best_params_['classifier__C'], 10.0) + self.assertEqual(q.best_model().get_params()['classifier__C'], 10.0) + + def test_modsel_parallel_speedup(self): + class SlowLR(LogisticRegression): + def fit(self, X, y, sample_weight=None): + time.sleep(1) + return super(SlowLR, self).fit(X, y, sample_weight) + + q = PACC(SlowLR(random_state=1, max_iter=5000)) + + data = qp.datasets.fetch_reviews('imdb', tfidf=True, min_df=10) + training, validation = data.training.split_stratified(0.7, random_state=1) + + param_grid = {'classifier__C': np.logspace(-3, 3, 7)} + app = APP(validation, sample_size=100, random_state=1) + + tinit = time.time() + GridSearchQ( + q, param_grid, protocol=app, error='mae', refit=False, timeout=-1, n_jobs=1, verbose=True + ).fit(training) + tend_nooptim = time.time()-tinit + + tinit = time.time() + GridSearchQ( + q, param_grid, protocol=app, error='mae', refit=False, timeout=-1, n_jobs=-1, verbose=True + ).fit(training) + tend_optim = time.time() - tinit + + print(f'parallel training took {tend_optim:.4f}s') + print(f'sequential training took {tend_nooptim:.4f}s') + + self.assertEqual(tend_optim < (0.5*tend_nooptim), True) + + def test_modsel_timeout(self): + + class SlowLR(LogisticRegression): + def fit(self, X, y, sample_weight=None): + import time + time.sleep(10) + super(SlowLR, self).fit(X, y, sample_weight) + + q = PACC(SlowLR()) + + data = qp.datasets.fetch_reviews('imdb', tfidf=True, min_df=10) + training, validation = data.training.split_stratified(0.7, random_state=1) + # test = data.test + + param_grid = {'classifier__C': np.logspace(-3,3,7)} + app = APP(validation, sample_size=100, random_state=1) + q = GridSearchQ( + q, param_grid, protocol=app, error='mae', refit=True, timeout=3, n_jobs=-1, verbose=True + ) + with self.assertRaises(TimeoutError): + q.fit(training) + + +if __name__ == '__main__': + unittest.main() diff --git a/quapy/tests/test_protocols.py b/quapy/tests/test_protocols.py new file mode 100644 index 0000000..6c76d4b --- /dev/null +++ b/quapy/tests/test_protocols.py @@ -0,0 +1,179 @@ +import unittest +import numpy as np +from quapy.data import LabelledCollection +from quapy.protocol import APP, NPP, UPP, DomainMixer, AbstractStochasticSeededProtocol + + +def mock_labelled_collection(prefix=''): + y = [0] * 250 + [1] * 250 + [2] * 250 + [3] * 250 + X = [prefix + str(i) + '-' + str(yi) for i, yi in enumerate(y)] + return LabelledCollection(X, y, classes=sorted(np.unique(y))) + + +def samples_to_str(protocol): + samples_str = "" + for instances, prev in protocol(): + samples_str += f'{instances}\t{prev}\n' + return samples_str + + +class TestProtocols(unittest.TestCase): + + def test_app_replicate(self): + data = mock_labelled_collection() + p = APP(data, sample_size=5, n_prevalences=11, random_state=42) + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertEqual(samples1, samples2) + + p = APP(data, sample_size=5, n_prevalences=11) # <- random_state is by default set to 0 + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertEqual(samples1, samples2) + + def test_app_not_replicate(self): + data = mock_labelled_collection() + p = APP(data, sample_size=5, n_prevalences=11, random_state=None) + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertNotEqual(samples1, samples2) + + p = APP(data, sample_size=5, n_prevalences=11, random_state=42) + samples1 = samples_to_str(p) + p = APP(data, sample_size=5, n_prevalences=11, random_state=0) + samples2 = samples_to_str(p) + + self.assertNotEqual(samples1, samples2) + + def test_app_number(self): + data = mock_labelled_collection() + p = APP(data, sample_size=100, n_prevalences=10, repeats=1) + + # surprisingly enough, for some n_prevalences the test fails, notwithstanding + # everything is correct. The problem is that in function APP.prevalence_grid() + # there is sometimes one rounding error that gets cumulated and + # surpasses 1.0 (by a very small float value, 0.0000000000002 or sthe like) + # so these tuples are mistakenly removed... I have tried with np.close, and + # other workarounds, but eventually happens that there is some negative probability + # in the sampling function... + + count = 0 + for _ in p(): + count+=1 + + self.assertEqual(count, p.total()) + + def test_npp_replicate(self): + data = mock_labelled_collection() + p = NPP(data, sample_size=5, repeats=5, random_state=42) + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertEqual(samples1, samples2) + + p = NPP(data, sample_size=5, repeats=5) # <- random_state is by default set to 0 + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertEqual(samples1, samples2) + + def test_npp_not_replicate(self): + data = mock_labelled_collection() + p = NPP(data, sample_size=5, repeats=5, random_state=None) + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertNotEqual(samples1, samples2) + + p = NPP(data, sample_size=5, repeats=5, random_state=42) + samples1 = samples_to_str(p) + p = NPP(data, sample_size=5, repeats=5, random_state=0) + samples2 = samples_to_str(p) + self.assertNotEqual(samples1, samples2) + + def test_kraemer_replicate(self): + data = mock_labelled_collection() + p = UPP(data, sample_size=5, repeats=10, random_state=42) + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertEqual(samples1, samples2) + + p = UPP(data, sample_size=5, repeats=10) # <- random_state is by default set to 0 + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertEqual(samples1, samples2) + + def test_kraemer_not_replicate(self): + data = mock_labelled_collection() + p = UPP(data, sample_size=5, repeats=10, random_state=None) + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertNotEqual(samples1, samples2) + + def test_covariate_shift_replicate(self): + dataA = mock_labelled_collection('domA') + dataB = mock_labelled_collection('domB') + p = DomainMixer(dataA, dataB, sample_size=10, mixture_points=11, random_state=1) + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertEqual(samples1, samples2) + + p = DomainMixer(dataA, dataB, sample_size=10, mixture_points=11) # <- random_state is by default set to 0 + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertEqual(samples1, samples2) + + def test_covariate_shift_not_replicate(self): + dataA = mock_labelled_collection('domA') + dataB = mock_labelled_collection('domB') + p = DomainMixer(dataA, dataB, sample_size=10, mixture_points=11, random_state=None) + + samples1 = samples_to_str(p) + samples2 = samples_to_str(p) + + self.assertNotEqual(samples1, samples2) + + def test_no_seed_init(self): + class NoSeedInit(AbstractStochasticSeededProtocol): + def __init__(self): + self.data = mock_labelled_collection() + + def samples_parameters(self): + # return a matrix containing sampling indexes in the rows + return np.random.randint(0, len(self.data), 10*10).reshape(10, 10) + + def sample(self, params): + index = np.unique(params) + return self.data.sampling_from_index(index) + + p = NoSeedInit() + + # this should raise a ValueError, since the class is said to be AbstractStochasticSeededProtocol but the + # random_seed has never been passed to super(NoSeedInit, self).__init__(random_seed) + with self.assertRaises(ValueError): + for sample in p(): + pass + print('done') + + +if __name__ == '__main__': + unittest.main() diff --git a/quapy/tests/test_replicability.py b/quapy/tests/test_replicability.py new file mode 100644 index 0000000..e89531a --- /dev/null +++ b/quapy/tests/test_replicability.py @@ -0,0 +1,78 @@ +import unittest +import quapy as qp +from quapy.data import LabelledCollection +from quapy.functional import strprev +from sklearn.linear_model import LogisticRegression + +from quapy.method.aggregative import PACC + + +class MyTestCase(unittest.TestCase): + def test_prediction_replicability(self): + + dataset = qp.datasets.fetch_UCIDataset('yeast') + + with qp.util.temp_seed(0): + lr = LogisticRegression(random_state=0, max_iter=10000) + pacc = PACC(lr) + prev = pacc.fit(dataset.training).quantify(dataset.test.X) + str_prev1 = strprev(prev, prec=5) + + with qp.util.temp_seed(0): + lr = LogisticRegression(random_state=0, max_iter=10000) + pacc = PACC(lr) + prev2 = pacc.fit(dataset.training).quantify(dataset.test.X) + str_prev2 = strprev(prev2, prec=5) + + self.assertEqual(str_prev1, str_prev2) # add assertion here + + def test_samping_replicability(self): + import numpy as np + + def equal_collections(c1, c2, value=True): + self.assertEqual(np.all(c1.X == c2.X), value) + self.assertEqual(np.all(c1.y == c2.y), value) + if value: + self.assertEqual(np.all(c1.classes_ == c2.classes_), value) + + X = list(map(str, range(100))) + y = np.random.randint(0, 2, 100) + data = LabelledCollection(instances=X, labels=y) + + sample1 = data.sampling(50) + sample2 = data.sampling(50) + equal_collections(sample1, sample2, False) + + sample1 = data.sampling(50, random_state=0) + sample2 = data.sampling(50, random_state=0) + equal_collections(sample1, sample2, True) + + sample1 = data.sampling(50, *[0.7, 0.3], random_state=0) + sample2 = data.sampling(50, *[0.7, 0.3], random_state=0) + equal_collections(sample1, sample2, True) + + with qp.util.temp_seed(0): + sample1 = data.sampling(50, *[0.7, 0.3]) + with qp.util.temp_seed(0): + sample2 = data.sampling(50, *[0.7, 0.3]) + equal_collections(sample1, sample2, True) + + sample1 = data.sampling(50, *[0.7, 0.3], random_state=0) + sample2 = data.sampling(50, *[0.7, 0.3], random_state=0) + equal_collections(sample1, sample2, True) + + sample1_tr, sample1_te = data.split_stratified(train_prop=0.7, random_state=0) + sample2_tr, sample2_te = data.split_stratified(train_prop=0.7, random_state=0) + equal_collections(sample1_tr, sample2_tr, True) + equal_collections(sample1_te, sample2_te, True) + + with qp.util.temp_seed(0): + sample1_tr, sample1_te = data.split_stratified(train_prop=0.7) + with qp.util.temp_seed(0): + sample2_tr, sample2_te = data.split_stratified(train_prop=0.7) + equal_collections(sample1_tr, sample2_tr, True) + equal_collections(sample1_te, sample2_te, True) + + +if __name__ == '__main__': + unittest.main() diff --git a/quapy/util.py b/quapy/util.py index 12ffc23..733fbb8 100644 --- a/quapy/util.py +++ b/quapy/util.py @@ -5,13 +5,14 @@ import os import pickle import urllib from pathlib import Path +from contextlib import ExitStack import quapy as qp import numpy as np from joblib import Parallel, delayed -def _get_parallel_slices(n_tasks, n_jobs=-1): +def _get_parallel_slices(n_tasks, n_jobs): if n_jobs == -1: n_jobs = multiprocessing.cpu_count() batch = int(n_tasks / n_jobs) @@ -21,8 +22,9 @@ def _get_parallel_slices(n_tasks, n_jobs=-1): def map_parallel(func, args, n_jobs): """ - Applies func to n_jobs slices of args. E.g., if args is an array of 99 items and n_jobs=2, then - func is applied in two parallel processes to args[0:50] and to args[50:99] + Applies func to n_jobs slices of args. E.g., if args is an array of 99 items and `n_jobs`=2, then + func is applied in two parallel processes to args[0:50] and to args[50:99]. func is a function + that already works with a list of arguments. :param func: function to be parallelized :param args: array-like of arguments to be passed to the function in different parallel calls @@ -36,7 +38,7 @@ def map_parallel(func, args, n_jobs): return list(itertools.chain.from_iterable(results)) -def parallel(func, args, n_jobs): +def parallel(func, args, n_jobs, seed=None): """ A wrapper of multiprocessing: @@ -44,32 +46,43 @@ def parallel(func, args, n_jobs): >>> delayed(func)(args_i) for args_i in args >>> ) - that takes the `quapy.environ` variable as input silently + that takes the `quapy.environ` variable as input silently. + Seeds the child processes to ensure reproducibility when n_jobs>1 """ - def func_dec(environ, *args): - qp.environ = environ - return func(*args) + def func_dec(environ, seed, *args): + qp.environ = environ.copy() + qp.environ['N_JOBS'] = 1 + #set a context with a temporal seed to ensure results are reproducibles in parallel + with ExitStack() as stack: + if seed is not None: + stack.enter_context(qp.util.temp_seed(seed)) + return func(*args) + return Parallel(n_jobs=n_jobs)( - delayed(func_dec)(qp.environ, args_i) for args_i in args + delayed(func_dec)(qp.environ, None if seed is None else seed+i, args_i) for i, args_i in enumerate(args) ) @contextlib.contextmanager -def temp_seed(seed): +def temp_seed(random_state): """ Can be used in a "with" context to set a temporal seed without modifying the outer numpy's current state. E.g.: >>> with temp_seed(random_seed): >>> pass # do any computation depending on np.random functionality - :param seed: the seed to set within the "with" context + :param random_state: the seed to set within the "with" context """ - state = np.random.get_state() - np.random.seed(seed) + if random_state is not None: + state = np.random.get_state() + #save the seed just in case is needed (for instance for setting the seed to child processes) + qp.environ['_R_SEED'] = random_state + np.random.seed(random_state) try: yield finally: - np.random.set_state(state) + if random_state is not None: + np.random.set_state(state) def download_file(url, archive_filename): @@ -117,6 +130,7 @@ def create_if_not_exist(path): def get_quapy_home(): """ Gets the home directory of QuaPy, i.e., the directory where QuaPy saves permanent data, such as dowloaded datasets. + This directory is `~/quapy_data` :return: a string representing the path """ @@ -151,7 +165,7 @@ def save_text_file(path, text): def pickled_resource(pickle_path:str, generation_func:callable, *args): """ - Allows for fast reuse of resources that are generated only once by calling generation_func(*args). The next times + Allows for fast reuse of resources that are generated only once by calling generation_func(\\*args). The next times this function is invoked, it loads the pickled resource. Example: >>> def some_array(n): # a mock resource created with one parameter (`n`) @@ -190,10 +204,6 @@ class EarlyStop: """ A class implementing the early-stopping condition typically used for training neural networks. - :param patience: the number of (consecutive) times that a monitored evaluation metric (typically obtaind in a - held-out validation split) can be found to be worse than the best one obtained so far, before flagging the - stopping condition. An instance of this class is `callable`, and is to be used as follows: - >>> earlystop = EarlyStop(patience=2, lower_is_better=True) >>> earlystop(0.9, epoch=0) >>> earlystop(0.7, epoch=1) @@ -205,14 +215,14 @@ class EarlyStop: >>> earlystop.best_epoch # is 1 >>> earlystop.best_score # is 0.7 - + :param patience: the number of (consecutive) times that a monitored evaluation metric (typically obtaind in a + held-out validation split) can be found to be worse than the best one obtained so far, before flagging the + stopping condition. An instance of this class is `callable`, and is to be used as follows: :param lower_is_better: if True (default) the metric is to be minimized. - :ivar best_score: keeps track of the best value seen so far :ivar best_epoch: keeps track of the epoch in which the best score was set :ivar STOP: flag (boolean) indicating the stopping condition :ivar IMPROVED: flag (boolean) indicating whether there was an improvement in the last call - """ def __init__(self, patience, lower_is_better=True): @@ -242,4 +252,5 @@ class EarlyStop: else: self.patience -= 1 if self.patience <= 0: - self.STOP = True \ No newline at end of file + self.STOP = True + diff --git a/setup.py b/setup.py index 898ff46..5bb39a4 100644 --- a/setup.py +++ b/setup.py @@ -14,6 +14,7 @@ def get_version(rel_path): return line.split(delim)[1] else: raise RuntimeError("Unable to find version string.") + # Arguments marked as "Required" below must be included for upload to PyPI. # Fields marked as "Optional" may be commented out. @@ -114,7 +115,7 @@ setup( python_requires='>=3.6, <4', - install_requires=['scikit-learn', 'pandas', 'tqdm', 'matplotlib'], + install_requires=['scikit-learn', 'pandas', 'tqdm', 'matplotlib', 'joblib', 'xlrd', 'abstention'], # List additional groups of dependencies here (e.g. development # dependencies). Users will be able to install these using the "extras" @@ -158,7 +159,8 @@ setup( project_urls={ # Optional 'Contributors': 'https://github.com/HLT-ISTI/QuaPy/graphs/contributors', 'Bug Reports': 'https://github.com/HLT-ISTI/QuaPy/issues', - 'Documentation': 'https://github.com/HLT-ISTI/QuaPy/wiki', + 'Wiki': 'https://github.com/HLT-ISTI/QuaPy/wiki', + 'Documentation': 'https://hlt-isti.github.io/QuaPy/build/html/index.html', 'Source': 'https://github.com/HLT-ISTI/QuaPy/', }, )