Combining Classifiers for Foreign Pattern Rejection

Władysław Homenda 1 , 2 , Agnieszka Jastrzȩbska 1 , 3 , Witold Pedrycz 3 , 4  and Fusheng Yu 5
  • 1 The Faculty of Mathematics and Information Science, Poland
  • 2 The Faculty of Economics and Informatics in Vilnius, Vilnius, Lithuania
  • 3 The Systems Research Institute Polish Academy of Sciences, Warsaw, Poland
  • 4 The University of Alberta, Edmonton, Canada
  • 5 The Beijing Normal University, Beijing, China


In this paper, we look closely at the issue of contaminated data sets, where apart from legitimate (proper) patterns we encounter erroneous patterns. In a typical scenario, the classification of a contaminated data set is always negatively influenced by garbage patterns (referred to as foreign patterns). Ideally, we would like to remove them from the data set entirely. The paper is devoted to comparison and analysis of three different models capable to perform classification of proper patterns with rejection of foreign patterns. It should be stressed that the studied models are constructed using proper patterns only, and no knowledge about the characteristics of foreign patterns is needed. The methods are illustrated with a case study of handwritten digits recognition, but the proposed approach itself is formulated in a general manner. Therefore, it can be applied to different problems. We have distinguished three structures: global, local, and embedded, all capable to eliminate foreign patterns while performing classification of proper patterns at the same time. A comparison of the proposed models shows that the embedded structure provides the best results but at the cost of a relatively high model complexity. The local architecture provides satisfying results and at the same time is relatively simple.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] W. Homenda and A. Jastrzebska, Global, local and embedded architectures for multiclass classification with foreign elements rejection: an overview, Proc. of the 7th International Conference of Soft Computing and Pattern Recognition, pp. 89–94, 2015.

  • [2] F. J. Anscombe, Rejection of outliers, Technometrics, vol. 2, no. 2, pp. 123–147, 1960.

  • [3] V. Barnett and T. Lewis, Outliers in Statistical Data, 3rd ed. Wiley, 1994.

  • [4] M. P. Maples, D. E. Reichart, N. C. Konz, T. A. Berger, A. S. Trotter, J. R. Martin, D. A. Dutton, M. L. Paggen, R. E. Joyner, and C. P. Salemi, Robust chauvenet outlier rejection, The Astrophysical Journal Supplement Series, vol. 238, no. 1, p. 2, 2018.

  • [5] Z. Li, R. J. Baseman, Y. Zhu, F. A. Tipu, N. Slonim, and L. Shpigelman, A unified framework for outlier detection in trace data analysis, IEEE Transactions on Semiconductor Manufacturing, vol. 27, no. 1, pp. 95–103, 2014.

  • [6] G. Yuksel and M. Cetin, Outlier detection in a preliminary test estimator of the mean, Journal of Statistics and Management Systems, vol. 19, no. 4, pp. 605–615, 2016.

  • [7] M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko, A review of novelty detection, Signal Processing, vol. 99, pp. 215–249, 2014.

  • [8] R. Rocci, S. A. Gattone, and R. Di Mari, A data driven equivariant approach to constrained gaussian mixture modeling, Advances in Data Analysis and Classification, vol. 12, no. 2, pp. 235–260, 2018.

  • [9] A. Punzo, A. Mazza, and A. Maruotti, Fitting insurance and economic data with outliers: a flexible approach based on finite mixtures of contaminated gamma distributions, Journal of Applied Statistics, vol. 45, no. 14, pp. 2563–2584, 2018.

  • [10] L. Xiang, K. K. Yau, and A. H. Lee, The robust estimation method for a finite mixture of poisson mixed-effect models, Computational Statistics & Data Analysis, vol. 56, no. 6, pp. 1994–2005, 2012.

  • [11] H. Otneim and D. Tjøstheim, The locally gaussian density estimator for multivariate data, Statistics and Computing, vol. 27, no. 6, pp. 1595–1616, 2017.

  • [12] J. Zhang and H. Wang, Detecting outlying subspaces for high- dimensional data: the new task, and performance, Knowledge and Information Systems, vol. 3, no. 10, pp. 333–355, 2006.

  • [13] V. Hautamaki, I. Karkkainen, and P. Franti, Outlier detection using k-nearest neighbour graph, Proc. of the 17th International Conference on Pattern Recognition, vol. 3, pp. 430–433, 2004.

  • [14] M. M. Breunig, H. P. Kriegel, R. T. Ng, and J. Sander, Lof: identifying density- based local outliers, Proc. of the ACM SIGMOD International Conference on Management of Data, vol. 29, pp. 93–104, 2000.

  • [15] H. Izakian and W. Pedrycz, Anomaly detection in time series data using a fuzzy c-means clustering, Proc. of IFSA World Congress and NAFIPS Annual Meeting, pp. 1513–1518, 2013.

  • [16] F. de Morsier, D. Tuia, M. Borgeaud, V. Gass, and J.-P. Thiran, Cluster validity measure and merging system for hierarchical clustering considering outliers, Pattern Recognition, vol. 48, no. 4, pp. 1478–1489, 2015.

  • [17] B. Schölkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett, New support vector algorithms, Neural Computation, vol. 12, no. 5, pp. 1207–1245, 2000.

  • [18] B. Schölkopf, J. C. Platt, J. C. Shawe-Taylor, A. J. Smola, and R. C. Williamson, Estimating the support of a high-dimensional distribution, Neural Computation, vol. 13, no. 7, pp. 1443–1471, 2001.

  • [19] C. Gautam, R. Balaji, S. K., A. Tiwari, and K. Ahuja, Localized multiple kernel learning for anomaly detection: One-class classification, Knowledge-Based Systems, vol. 165, pp. 241–252, 2019.

  • [20] C. Desir, S. Bernard, C. Petitjean, and L. Heutte, One class random forests, Pattern Recognition, vol. 46, no. 12, pp. 3490–3506, 2013.

  • [21] D. M. J. Tax and R. P. W. Duin, Combining one-class classifiers, Proc. of Multiple Classifier Systems: Second International Workshop, pp. 299–308, 2001.

  • [22] W. Homenda, A. Jastrzebska, and W. Pedrycz, Rejecting foreign elements in pattern recognition problem. reinforced training of rejection level, Proc. of the 7th International Conference on Agents and Artificial Intelligence, pp. 90–99, 2015.

  • [23] Y. Shiraishia and K. Fukumizu, Statistical approaches to combining binary classifiers for multi-class classification, Neurocomputing, vol. 74, pp. 680–688, 2011.

  • [24] M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, and F. Herrera, An overview of ensemble methods for binary classifiers in multi-class problems: Experimental study on one-vs-one and one-vs-all schemes, Pattern Recognition, vol. 8, no. 44, pp. 1761–1776, 2011.

  • [25] Y. LeCun, C. Cortes, and C. J. Burges, The mnist database of handwritten digits,

  • [26] T. E. de Campos, B. R. Babu, and M. Varma, Character recognition in natural images, in Proc. of the International Conference on Computer Vision Theory and Applications, 2009. [Online]. Available:

  • [27] L. Breiman, Random forests, Machine Learning, vol. 1, no. 45, pp. 5–32, 2001.


Journal + Issues