Recursive-Rule Extraction Algorithm With J48graft And Applications To Generating Credit Scores

Yoichi Hayashi 1 , Yuki Tanaka 1 , Tomohiro Takagi 1 , Takamichi Saito 1 , Hideaki Iiduka 1 , Hiroaki Kikuchi 2 , Guido Bologna 3  and Sushmita Mitra 4
  • 1 Department of Computer Science, Meiji University Kawasaki 214-8571, Japan
  • 2 Department of Frontier Media Science,, Meiji University Nakano-ku, Tokyo 164-8525, Japan
  • 3 Department of Information Technology, University of Applied Sciences of Western Switzerland Rue de la prairie 4, 1204 Geneva, Switzerland
  • 4 Sushmita Mitra Machine Intelligence Unit, Indian Statistical Institute 203 B.T. Road, Kolkata 700 108, India


The purpose of this study was to generate more concise rule extraction from the Recursive-Rule Extraction (Re-RX) algorithm by replacing the C4.5 program currently employed in Re-RX with the J48graft algorithm. Experiments were subsequently conducted to determine rules for six different two-class mixed datasets having discrete and continuous attributes and to compare the resulting accuracy, comprehensibility and conciseness. When working with the CARD1, CARD2, CARD3, German, Bene1 and Bene2 datasets, Re-RX with J48graft provided more concise rules than the original Re-RX algorithm. The use of Re-RX with J48graft resulted in 43.2%, 37% and 21% reductions in rules in the case of the German, Bene1 and Bene2 datasets compared to Re-RX. Furthermore, the Re-RX with J48graft showed 8.87% better accuracy than the Re-RX algorithm for the German dataset. These results confirm that the application of Re-RX in conjunction with J48graft has the capacity to facilitate migration from existing data systems toward new concise analytic systems and Big Data.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] Marqus, A.I., Garca, V., & Snchez, J.S. On the suitability of resampling techniques for the class imbalance problem in credit scoring. Journal of the Operational Research Society 64, pp. 1060–1070, 2013.

  • [2] Zhao Z., Xu, S., Kang, B. H., Kabir, M. M. J., & Liu, Y. Investigation and improvement of multilayer perceptron neural networks for credit scoring Expert Systems with Applications 42, pp. 3508–3516, 2015

  • [3] Finlay, S. M. Multiple classifier architectures and their applications to credit risk assessment. European Journal of Operational Research 210, pp. 368–378, 2011.

  • [4] Quinlan, J.R. Programs for Machine Learning. Morgan Kaufmann Series in Machine Learning, San Mateo, CA, 1993, Morgan Kaufman.

  • [5] Martens, D., Baesens, B., Van Gestel, T., & Vanthienen, J. Comprehensible credit scoring models using support vector machines. European Journal of Operational Research 183, pp. 1497–1488, 2007.

  • [6] Baesens, B., et al. Using neural network rule extraction and decision tables for credit-risk evaluation. Management Science 49, No. 3, pp. 312–329, 2004.

  • [7] Abellan, J., and Mantas, C. Improving experimental studies about ensembles of classifiers for bankruptcy prediction and credit scoring. Expert Systems with Applications, pp. 3825–3830, 2014.

  • [8] Finlay, S. M. Multiple classifier architectures and their applications to credit risk assessment. European Journal of Operational Research 210, pp. 368–378, 2011.

  • [9] Braket, N., and Bradely, A.P. Rule extraction from support vector machine: a review. Neurocomputing 74, pp. 178–190, 2010.

  • [10] Setiono, R., and Liu, H. Neurolinear: From neural networks to oblique decision rules. Neurocomputing 17, No. 1, pp. 1–24, 1997.

  • [11] Setiono, R. and Liu, H. A connectionist approach to generating oblique decision trees. IEEE Trans. Syst., Man, Cybern. B, Cybern. Vol. 29, No. 3, pp. 440–444, Jun. 1999.

  • [12] Setiono, R., Baesens, B. & Mues, C. A note on knowledge discovery using neural Setiono networks and its application to credit card screening. European Journal of Operational Research 192, pp.326-332, 2009.

  • [13] Setiono, R., et al. Recursive neural network rule extraction for data with mixed attributes. IEEE Trans. Neural Netw. 19, No. 2, pp. 299–307, 2008.

  • [14] Setiono, R., and Liu, H. Symbolic representation of neural networks. IEEE Computer 29, No. 3, pp. 71–77, 1996.

  • [15] Gupta, A, Park, S. and Lam, S.M. Generalized analytic rule extraction for feedforward neural networks, IEEE Trans. Knowledge and Data Engineering, 11, pp.985-991, 1999.

  • [16] Etchell, T.A. and Lisboa, J.P.G., Orthogonal search-based rule extraction (OSRE) for trained neural-networks: A practical and efficient approach, IEEE Trans. Neural Networks 17, pp.374-384, 2006.

  • [17] Hansen, L.K., and Salamon, P., Neural network ensembles. IEEE Trans. Patter Analysis and Machine Intelligence 12, pp. 993–1001, 1990.

  • [18] Igelnik, S., Pao, Y.-H., LeClair, S. R., and Shen, C. Y. The ensemble approach to neural-network learning and generalization. IEEE Trans. Neural Networks 10, pp. 19–30, 1999.

  • [19] Liao, J.-J., Shih, C.-H., Chen, T.-F., and Hsu, M.-F. An example-based model for two-class imbalanced financial problem. Economic Modelling 37, pp. 175–183, 2014.

  • [20] Setiono R. et al., Rule extraction from minimal neural networks for credit card screening, Inter. J. of Neural Systems., Vol. 21, No. 4, pp. 265–276, 2011.

  • [21] Setiono R. et al., A note on knowledge discovery using neural networks and its application to credit card screening, European J. Operational Research, Vol. 192, pp. 326–332, 2009.

  • [22] Hayashi Y. et al., Understanding consumer heterogeneity: A business intelligence application of neural networks, Knowledge-Based Systems, Vol. 23, No. 8, pp. 856–863, 2010.

  • [23] Bologna G., Is it worth generating rules from neural network ensemble?, J. of Applied Logic, Vol. 2, pp. 325–348, 2004.

  • [24] Zhou, Z.-H. Extracting symbolic rules from trained neural network ensembles. AI Communications 16, pp. 3–15, 2003.

  • [25]

  • [26] Setiono R. et al., A penalty-function approach for pruning feedforward neural networks, Neural Comp., Vol. 9, No. 1, pp. 185–204, 1997.

  • [27] Witten, I.H. and Frank, E., Data Ming: Practical Machine Learning Tools With Java Implementations. San Francisco, CA: Morgan Kaufmann, 1999.

  • [28] Quinlan J.R., Induction of decision trees, Machine Learning, Vol.1, pp.81-106, 1986.

  • [29] Webb. G.I., Decision Tree Grafting from the All-Tests-But-One Partition, in Proc. 16th International Joint Conference on Artificial Intelligence (IJCAI), Vol. 2, pp. 702–707, 1999.

  • [30] Webb, G. I., Decision Tree Grafting, Learining, IJCAI’97 Proceedings of the 15th International Conference on Artificial Intelligence, Vol.2, pp. 846-885, 1997.

  • [31] Frank, A. & Asuncion, A. University of California. Irvine Machine Learning Repository., 2010

  • [32] Prechelt, L. Proben1 — A set of benchmarks and benchmarking rules for neural network training algorithms, Technical Report 21/94, Fakultt fr Informatik, Universitt Karlsruhe, Germany. Anonymous ftp available from ftp://pub/papers/techreport/1994/ on, 1994.

  • [33] Smith, M. Neural Networks for Statistical Modeling, New York: Van Nostrand Reinhold, 1993.


Journal + Issues