Classifier Ensemble Framework: a Diversity Based Approach
محورهای موضوعی : Data MiningHamid Parvin 1 , Hosein Alizadeh 2 , Mohsen Moshki 3
1 - 1Department of Computer Science, Nourabad Mamasani Branch, Islamic Azad University, Nourabad Mamasani, Iran
2 - Department of Computer Science, Nourabad Mamasani Branch, Islamic Azad University, Nourabad Mamasani, Iran
3 - Department of Computer Science, Nourabad Mamasani Branch, Islamic Azad University, Nourabad Mamasani,
کلید واژه: Terms&mdash, Classifier Ensemble, diversity, Linear Discriminant Analysis,
چکیده مقاله :
Pattern recognition systems are widely used in a host of different fields. Due to some reasons such as lack of knowledge about a method based on which the best classifier is detected for any arbitrary problem, and thanks to significant improvement in accuracy, researchers turn to ensemble methods in almost every task of pattern recognition. Classification as a major task in pattern recognition, have been subject to this transition. The classifier ensemble which uses a number of base classifiers is considered as meta-classifier to learn any classification problem in pattern recognition. Although some researchers think they are better than single classifiers, they will not be better if some conditions are not met. The most important condition among them is diversity of base classifiers. Generally in design of multiple classifier systems, the more diverse the results of the classifiers, the more appropriate the aggregated result. It has been shown that the necessary diversity for the ensemble can be achieved by manipulation of dataset features, manipulation of data points in dataset, different sub-samplings of dataset, and usage of different classification algorithms. We also propose a new method of creating this diversity. We use Linear Discriminant Analysis to manipulate the data points in dataset. Although the classifier ensemble produced by proposed method may not always outperform all of its base classifiers, it always possesses the diversity needed for creation of an ensemble, and consequently it always outperforms all of its base classifiers on average.
[1] L. Breiman,.: Bagging predictors. Machine Learning, 24(2): 123-140, 1996.
[2] E. Ghanbari, H. Beigy: Incremental RotBoost algorithm: An application for spam filtering. Intell. Data Anal. 19(2): 449-468, 2015.
[3] R.O Duda., P.E. Hart., and D.G. Stork.: Pattern Classification. 2nd ed. John Wiley & Sons, NY,.B. Smith, “An approach to graphs of linear forms (Unpublished work style),” unpublished. 2001.
[4] Q. Fu, S.X. Hu and S.Y. Zhao,.: A PSO-based approach for neural network ensemble. Journal of Zhejiang University (Engineering Science), 38(12): 1596-1600, 2004 (in Chinese).
[5] L. Breiman, Random forests, Mach. Learn. 45: 5–32, 2001.
[6] L. Breiman, Arcing classifiers, Annal. Stat. 26: 801–824, 1998.
[7] Y. Freund, and R.E. Schapire, “Experiments with a new boosting algorithm”, International Conference on Machine Learning, pp. 148–156, 1996.
[8] Y. Freund and R.E. Schapire ,”A decision-theoretic generalization of on-line learning and an application to boosting”, J. Comput. Sys. Sci. 55: 119–139, 1997.
[9] L.I. Kuncheva: Combining Pattern Classifiers, Methods and Algorithms, New York: Wiley, 2005.
[10] Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.
[11] A. Lazarevic, and Z. Obradovic,: Effective pruning of neural network classifier ensembles. Proc. International Joint Conference on Neural Networks, 2: 796-801, 2001.
[12] Y. Liu, X.Yao: Evolutionary ensembles with negative correlation learning. IEEE Trans. Evolutionary Computation, 4(4): 380-387, 2000.
[13] P. Melville and R. Mooney: Constructing Diverse Classifier Ensembles Using Artificial Training Examples. Proc. of the IJCAI-2003, p.505-510, 2003.
[14] B. Minaei-Bidgoli, G. Kortemeyer, and W.F Punch.: Optimizing Classification Ensembles via a Genetic Algorithm for a Web-based Educational System. Lecture Notes in Computer Science 3138: 397-406, 2004.
[15] B. Minaei-Bidgoli, H. Parvin, H. Alinejad-Rokny, H. Alizadeh, W.F Punch.: Effects of resampling method and adaptation on clustering ensemble efficacy. Artif. Intell. Rev. 41(1): 27-48 2014.
[16] H.D. Navone, P.F. Verdes, P.M. Granitto and H.A. Ceccatto: Selecting Diverse Members of Neural Network Ensembles. Proc. 16th Brazilian Symposium on Neural Networks, p.255-260, 2000.
[17] D. Opitz, and J. Shavlik,: Actively searching for an effective neural network ensemble. Connection Science, 8(3-4): 337-353, 1996.
[18] H. Parvin, H. Helmi, B. Minaei-Bidgoli, H. Alinejad-Rokny, and H. Shirgahi: Linkage Learning Based on Differences in Local Optimums of Building Blocks with One Optima. International Journal of the Physical Sciences 6(14): 3419–3425, 2011.
[19] H. Parvin, B. Minaei-Bidgoli, S. Ghatei, and H. Alinejad-Rokny,: An Innovative Combination of Particle Swarm Optimization, Learning Automaton and Great Deluge Algorithms for Dynamic Environments. International Journal of the Physical Sciences 6(22): 5121 – 5127, 2011.
[20] A. Krogh, and J. Vedelsdy,: Neural Network Ensembles Cross Validation, and Active Learning. Advances in Neural Information Processing Systems, 7: 231-238, 1995.
[21] H.R. Qodmanan, M. Nasiri, B. Minaei-Bidgoli: Multi objective association rule mining with genetic algorithm without specifying minimum support and minimum confidence, Expert Systems with Applications, 38(1): 288-298, 2011.
[22] F. Roli and J. Kittler, Proc. of 2nd International Workshop on Multiple Classifier Systems, Vol. 2096 of Lecture Notes in Computer Science LNCS Springer- Verlag, Cambridge, UK, 2001.
[23] F. Roli and J. Kittler, Proc. of 3rd Int. Workshop on Multiple Classifier Systems, Vol. 2364 of Lecture Notes in Computer Science LNCS Springer Verlag, Cagliari, Italy, 2002.
[24] B.E. Rosen,: Ensemble learning using decorrelated neural network. Connection Science, 8(3-4): 373-384, 1996.
[25] R.E. Schapire,: The strength of weak learn ability. Machine Learning, 5(2):1971-227, 1990.
[26] Z.H. Zhou, J.X. Wu, Y. Jiang, and S.F. Chen: Genetic algorithm based selective neural network ensemble. Proc. 17th International Joint Conference on Artificial Intelligence, 2: 797-802, 2001.
[27] H. Parvin, B. Minaei-Bidgoli: A clustering ensemble framework based on selection of fuzzy weighted clusters in a locally adaptive clustering algorithm. Pattern Anal. Appl. 18(1): 87-112, 2015.
[28] M.H. Fouladgar, B. Minaei-Bidgoli, and H. Parvin: On Possibility of Conditional Invariant Detection. 6881(2): 214-224, 2011.
[29] H. Parvin, B. Minaei-Bidgoli, and H. Alizadeh: Detection of Cancer Patients Using an Innovative Method for Learning at Imbalanced Datasets. LNCS 6954: 376-381, 2011.
[30] M. Daryabari, B. Minaei-Bidgoli, and H. Parvin: Localizing Program Logical Errors Using Extraction of Knowledge from Invariants. LNCS 6630: 124-135, 2011.
[31] H. Parvin, B. Minaei-Bidgoli and S. Parvin: A Metric to Evaluate a Cluster by Eliminating Effect of Complement Cluster. LNCS 7006: 246-254, 2011.
[32] H. Parvin, B. Minaei-Bidgoli: Linkage Learning Based on Local Optima. LNCS 6922(1): 163-172, 2011.
[33] H. Parvin, B. Minaei-Bidgoli, and H. Alizadeh and A. Beigi: A Novel Classifier Ensemble Method Based on Class Weightening in Huge Dataset. LNCS 6676 (2): 144-150, 2011.
[34] H. Parvin, B. Minaei-Bidgoli and H. Ghaffarian: An Innovative Feature Selection Using Fuzzy Entropy. LNCS 6677 (3): 576-585, 2011.
[35] H. Parvin, B. Minaei, H. Karshenas, and A. Beigi: A New N-gram Feature Extraction-Selection Method for Malicious Code. LNCS 6594(2): 98-107, 2011.
