On the Optimum Architecture of the Biologically Inspired Hierarchical Temporal Memory Model Applied to the Hand-Written Digit Recognition
In the paper we describe basic functions of the Hierarchical Temporal Memory (HTM) network based on a novel biologically inspired model of the large-scale structure of the mammalian neocortex. The focus of this paper is in a systematic exploration of possibilities how to optimize important controlling parameters of the HTM model applied to the classification of hand-written digits from the USPS database. The statistical properties of this database are analyzed using the permutation test which employs a randomization distribution of the training and testing data. Based on a notion of the homogeneous usage of input image pixels, a methodology of the HTM parameter optimization is proposed. In order to study effects of two substantial parameters of the architecture: the patch size and the overlap in more details, we have restricted ourselves to the single-level HTM networks. A novel method for construction of the training sequences by ordering series of the static images is developed. A novel method for estimation of the parameter maxDist based on the box counting method is proposed. The parameter sigma of the inference Gaussian is optimized on the basis of the maximization of the belief distribution entropy. Both optimization algorithms can be equally applied to the multi-level HTM networks as well. The influences of the parameters transitionMemory and requestedGroupCount on the HTM network performance have been explored. Altogether, we have investigated 2736 different HTM network configurations. The obtained classification accuracy results have been benchmarked with the published results of several conventional classifiers.
Thornton, J. R., Gustafsson, T., Blumenstein, M., Hine, T. (2006). Robust character recognition using hierarchical Bayesian network. In: Proc. 19th Australian Joint Conf. on Artificial Intelligence, Hobart, Australia. pp. 1259-1264.
Thornton, J. R., Faichney, J., Blumenstein, M., Hine, T. (2008). Character recognition using hierarchical vector quantization and temporal pooling. In: Wobcke, W., Zhang, M. (eds.) Proc 21st Australasian Joint Conf. Artificial Intelligence, Vol. Lecture Notes in Computer Science. pp. 562-572.
Goldberg, D. E. (1989). Genetic algorithms in search, optimization and machine learning. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
Wang, C. H., Srihari, S. N. (1988). A framework for object recognition in a visually complex environment and its application to locating address blocks on mail pieces. Int. Journal of Computer Vision 2(2), 125-151.
Dong, J., Krzyzak, A., Suen, C. Y. (2001). Statistical results of human performance on USPS database. Technical report, Centre of Pattern Recognition and Machine Intelligence, Concordia University.
Seewald, A. K. (2005). Digits-a dataset for hand-written digit recognition. Technical Report TR-2005-27, OFAI, Wien.
Hull, J. J. (1994). A database for hand-written text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence 16, 550-554.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., Jackel, L. D. (1989). Back-propagation applied to handwritten zip code recognition. Neural Computing 1(4), 541-551.
Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science 19(4), 676-685. DOI 10.1214/088342304000000396.
Schroeder, M. R. (1991). Fractals, chaos, power laws: minutes from an infinite paradise. W. H. Freeman, New York.
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal 27, 379-423.
Martin, K. J., Hirschberg, D. S. (1996). Small sample statistics for classification error rates II: confidence intervals and significance tests.