On the Optimum Architecture of the Biologically Inspired Hierarchical Temporal Memory Model Applied to the Hand-Written Digit Recognition

Open access

On the Optimum Architecture of the Biologically Inspired Hierarchical Temporal Memory Model Applied to the Hand-Written Digit Recognition

In the paper we describe basic functions of the Hierarchical Temporal Memory (HTM) network based on a novel biologically inspired model of the large-scale structure of the mammalian neocortex. The focus of this paper is in a systematic exploration of possibilities how to optimize important controlling parameters of the HTM model applied to the classification of hand-written digits from the USPS database. The statistical properties of this database are analyzed using the permutation test which employs a randomization distribution of the training and testing data. Based on a notion of the homogeneous usage of input image pixels, a methodology of the HTM parameter optimization is proposed. In order to study effects of two substantial parameters of the architecture: the patch size and the overlap in more details, we have restricted ourselves to the single-level HTM networks. A novel method for construction of the training sequences by ordering series of the static images is developed. A novel method for estimation of the parameter maxDist based on the box counting method is proposed. The parameter sigma of the inference Gaussian is optimized on the basis of the maximization of the belief distribution entropy. Both optimization algorithms can be equally applied to the multi-level HTM networks as well. The influences of the parameters transitionMemory and requestedGroupCount on the HTM network performance have been explored. Altogether, we have investigated 2736 different HTM network configurations. The obtained classification accuracy results have been benchmarked with the published results of several conventional classifiers.

Felleman, D., van Essen, D. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex (1), 1-47.

Serre, T., Oliva, A., Poggio, T. (2007). A feedforward architecture accounts for rapid categorization. In: Proc. National Academy of Sciences of the USA, Vol. 15. pp. 6424-6429.

Lee, T. S., Mumford, D. (2003). Hierarchical Bayesian inference in visual cortex. Journal of Optical Society of America A 20(7), 1434-1448.

Dean, T. (2006). Scalable inference in hierarchical generative models. In: Proc. 9th Int. Symp. on Artificial Intelligence and mathematics. pp. 1-9.

Hawkins, J., Blakeslee, S. (2004). On intelligence. Henry Holt and Company, New York.

George, D., Hawkins, J. (2009). Towards a mathematical theory of cortical micro-circuits. PLoS Computational Biology 5(10). DOI 10.1371/journal.pcbi.1000532.

George, D., Hawkins, J. (2005). Hierarchical Bayesian model of invariant pattern recognition in the visual cortex. In: Proc. Int. Joint Conf. on Neural Networks. Montreal, Canada.

Numenta (2007). Zeta1 algorithms reference. Document version 1.0.

Dong, J. (2001). Statistical results of human performance on USPS database. Technical report, CEN-PARMI, Concordia University.

Thornton, J. R., Gustafsson, T., Blumenstein, M., Hine, T. (2006). Robust character recognition using hierarchical Bayesian network. In: Proc. 19th Australian Joint Conf. on Artificial Intelligence, Hobart, Australia. pp. 1259-1264.

Thornton, J. R., Faichney, J., Blumenstein, M., Hine, T. (2008). Character recognition using hierarchical vector quantization and temporal pooling. In: Wobcke, W., Zhang, M. (eds.) Proc 21st Australasian Joint Conf. Artificial Intelligence, Vol. Lecture Notes in Computer Science. pp. 562-572.

Bobier, B. (2007). Hand-written digit recognition using Hierarchical Temporal Memory. http://arts.uwaterloo.ca/~cnrglab/?q=system/files/SoftComputingFinalProject.pdf

Numenta (2009). Numenta forum: benchmark with USPS handwritten digit dataset. http://www.numenta.com/phpBB2/viewtopic.php?t=224

Numenta (2008). Hierarchical temporal memory, concepts, theory, and terminology. Document version 1.8.0.

George, D. (2008). How the brain might work: a hierarchical and temporal model for learning and recognition. Ph.D. thesis, Dept. of Electrical Engineering, Stanford University, USA.

Numenta (2009). Numenta node algorithms guide, NuPIC 1.7.

Johnson, S. T. (1967). Hierarchical clustering schemes. Psychometrika 32, 241-254.

Numenta (2008). Vision framework guide, NuPIC 1.6.1.

Goldberg, D. E. (1989). Genetic algorithms in search, optimization and machine learning. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

Wang, C. H., Srihari, S. N. (1988). A framework for object recognition in a visually complex environment and its application to locating address blocks on mail pieces. Int. Journal of Computer Vision 2(2), 125-151.

Dong, J., Krzyzak, A., Suen, C. Y. (2001). Statistical results of human performance on USPS database. Technical report, Centre of Pattern Recognition and Machine Intelligence, Concordia University.

Seewald, A. K. (2005). Digits-a dataset for hand-written digit recognition. Technical Report TR-2005-27, OFAI, Wien.

Hull, J. J. (1994). A database for hand-written text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence 16, 550-554.

LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., Jackel, L. D. (1989). Back-propagation applied to handwritten zip code recognition. Neural Computing 1(4), 541-551.

Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science 19(4), 676-685. DOI 10.1214/088342304000000396.

Schroeder, M. R. (1991). Fractals, chaos, power laws: minutes from an infinite paradise. W. H. Freeman, New York.

Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal 27, 379-423.

Martin, K. J., Hirschberg, D. S. (1996). Small sample statistics for classification error rates II: confidence intervals and significance tests.

Measurement Science Review

The Journal of Institute of Measurement Science of Slovak Academy of Sciences

Journal Information

IMPACT FACTOR 2017: 1.345
5-year IMPACT FACTOR: 1.253

CiteScore 2016: 1.88

SCImago Journal Rank (SJR) 2016: 0.495
Source Normalized Impact per Paper (SNIP) 2016: 1.419


All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 22 22 8
PDF Downloads 7 7 4