Evolving Non-Dominated Parameter Sets for Computational Models from Multiple Experiments

Open access


Creating robust, reproducible and optimal computational models is a key challenge for theorists in many sciences. Psychology and cognitive science face particular challenges as large amounts of data are collected and many models are not amenable to analytical techniques for calculating parameter sets. Particular problems are to locate the full range of acceptable model parameters for a given dataset, and to confirm the consistency of model parameters across different datasets. Resolving these problems will provide a better understanding of the behaviour of computational models, and so support the development of general and robust models. In this article, we address these problems using evolutionary algorithms to develop parameters for computational models against multiple sets of experimental data; in particular, we propose the ‘speciated non-dominated sorting genetic algorithm’ for evolving models in several theories. We discuss the problem of developing a model of categorisation using twenty-nine sets of data and models drawn from four different theories. We find that the evolutionary algorithms generate high quality models, adapted to provide a good fit to all available data.

Anderson, J. R.; Bothell, D.; Byrne, M. D.; Douglass, S.; Lebi`ere, C.; and Qin, Y. L. 2004. An integrated theory of the mind. Psychological Review 111(4):1036-1060.

Burnham, K. P., and Anderson, D. R. 2002. Model selection and multimodel inference. Springer.

Coello, C. A. C. 2000. An updated survey of GA-based multiobjective optimization techniques. ACM Computing Surveys 32:109-143.

Coello, C. A. C. 2003. Recent trends in evolutionary multiobjective optimization. In Abraham, A.; Jain, L.; and Goldberg, R., eds., Evolutionary Multi-Objective Optimization. London, UK: Springer-Verlag. 7-32.

Cooper, R. P., and Shallice, T. 1995. Soar and the case for unified theories of cognition. Cognition 55:115-49.

Cooper, R. P.; Fox, J.; Farringdon, J.; and Shallice, T. 1996. A systematic methodology for cognitive modelling. Artificial Intelligence 85:3-44.

Cooper, R. P. 2002. Modelling high-level cognitive processes. Mahwah, NJ: Erlbaum.

Feigenbaum, E. A., and Simon, H. A. 1984. EPAM-like models of recognition and learning. Cognitive Science 8:305-336.

Fisher, D. H.; Pazzani, M. J.; and Langley, P. 1991. Concept formation: Knowledge and experience in unsupervised learning. San Mateo, CA: Morgan Kaufmann Publishers.

Frias-Martinez, E., and Gobet, F. 2007. Automatic generation of cognitive theories using genetic programming. Minds and Machines 17:287-309.

Gluck, K. A.; Staszewski, J. J.; Richman, H.; Simon, H. A.; and Delahanty, P. 2001. The right tool for the job: Information-processing analysis in categorization. In Moore, J. D., and Stenning, K., eds., Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, 330-335. Mahwah, NJ: Lawrence Erlbaum.

Gluck, K. A.; Stanley, C. T.; Moore, L. R.; Reitter, D.; and Halbrugge, M. 2010. Exploration for understanding in cognitive modeling. Journal of Artificial General Intelligence 2:88-107.

Gobet, F., and Lane, P. C. R. 2005. A distributed framework for semi-automatically developing architectures of brain and mind. In Proceedings of the First International Conference on e-Social Science.

Gobet, F., and Ritter, F. E. 2000. Individual data analysis and Unified Theories of Cognition: A methodological proposal. In Taatgen, N., and Aasman, J., eds., Proceedings of the Third International Conference on Cognitive Modelling, 150-57. Veenendaal, The Netherlands: Universal Press.

Gobet, F., and Schiller, M. 2011. A manifesto for cognitive models of problem gambling. In Kokinov, B.; Karmiloff-Smith, A.; and Nersessian, N. J., eds., European Perspectives on Cognitive Sciences - Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press, Sofia.

Gobet, F., andWaters, A. J. 2003. The role of constraints in expert memory. Journal of Experimental Psychology: Learning, Memory & Cognition 29:1082-1094.

Gobet, F.; Richman, H.; Staszewski, J.; and Simon, H. A. 1997. Goals, representations, and strategies in a concept attainment task: The EPAM model. The Psychology of Learning and Motivation 37:265-290.

Gobet, F.; Lane, P. C. R.; Croker, S. J.; Cheng, P. C.-H.; Jones, G.; Oliver, I.; and Pine, J. M. 2001. Chunking mechanisms in human learning. Trends in Cognitive Sciences 5:236-243.

Goldberg, D. E. 1989. Genetic Algorithms in Search Optimization and Machine Learning. Reading, MA: Addison-Wesley.

Grant, D. A. 1962. Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review 69:54-61.

Gunzelmann, G. 2008. Strategy generalization across orientation tasks: Testing a computational cognitive model. Cognitive Science 32:835-861.

Holland, J. H. 1975. Adaptation in natural and artificial systems. Ann Arbor: The University of Michigan Press.

Hsu, C.-W.; Chang, C.-C.; and Lin, C.-J. 2003. A Practical Guide to Support Vector Classification. Available from http://www.csie.ntu.edu.tw/˜cjlin/libsvm/ (accessed February 2013).

Kase, S. E.; Ritter, F. E.; and Schoelles, M. 2008. From modeler-free individual data fitting in 3-d parametric prediction landscapes: A research expedition. In Proceedings of the 30th Annual Conference of the Cognitive Science Society, 1398-1403. Austin, TX: Cognitive Science Society.

Lane, P. C. R., and Gobet, F. 2003. Developing reproducible and comprehensible computational models. Artificial Intelligence 144:251-63.

Lane, P. C. R., and Gobet, F. 2005. Multi-task learning and transfer: The effect of algorithm representation. In Giraud-Carrier, C.; Vilalta, R.; and Brazdil, P., eds., Proceedings of the ICML-2005 Workshop on Meta-Learning.

Lane, P. C. R., and Gobet, F. 2012. A theory-driven testing methodology for developing scientific software. Journal of Experimental and Theoretical Artificial Intelligence 24:421-56.

Leahy, K. 1994. The overfitting problem in perspective. AI Expert 9:35-36.

Maneeratana, K.; Boonlong, K.; and Chaiyaratana, N. 2004. Multi-objective optimisation by cooperative co-evolution. In Yao, X.; Burke, E.; Lozano, J. A.; Smith, J.; Merelo-Guervs, J. J.; Bullinaria, J. A.; Rowe, J.; Tino, P.; Kabn, A.; and Schwefel, J.-P., eds., The Eighth International Conference on Parallel Problem Solving from Nature, volume 3242 of Lecture Notes in Computer Science, 772-81. Springer Verlag.

Medin, D. L., and Schaffer, M. M. 1978. Context theory of classification learning. Psychological Review 85:207-238.

Medin, D. L., and Smith, E. E. 1981. Strategies and classification learning. Journal of Experimental Psychology: Human Learning and Memory 7:241-253.

Medin, D. L. 1989. Concept and conceptual structure. American Psychologist 44:1469-1481.

Moore Jr, L. R. 2011. Cognitive model exploration and optimization: A new challenge for computational science. Computational and Mathematical Organization Theory 17:296-313.

Murphy, G. L. 2002. The big book of concepts. Cambridge, MA: The MIT Press.

Myung, I. J., and Pitt, M. A. 2010. Cognitive modeling repository. In Proceedings of the Thirty- Second Annual Meeting of the Cognitive Science Society, 556. Portland, Oregon: Lawrence Erlbaum.

Newell, A., and Simon, H. A. 1972. Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall.

Newell, A. 1990. Unified Theories of Cognition. Cambridge, MA: Harvard University Press.

Nosofsky, R. M. 2000. Exemplar representation with generalization? Comment on Smith and Minda’s (2000) “Thirty Categorization Results in Search of a Model”. Journal of Experimental Psychology: Learning, Memory and Cognition 26:1735-1743.

Pew, R. W., and Mavor, A. S., eds. 1998. Modeling human and organizational behavior: Applications to military simulations. Washington, D. C.: National Academy Press.

Pitt, M. A.; Kim, W.; Navarro, D. J.; and Myung, J. I. 2006. Global model analysis by parameter space partitioning. Psychological Review 113:57-83.

Pitt, M. A.; Myung, J. I.; and Zhang, S. 2002. Toward a method of selecting among computational models of cognition. Psychological Review 109:472-491.

Richman, H. B., and Simon, H. A. 1989. Context effects in letter perception: Comparison of two theories. Psychological Review 3:417-432.

Richman, H. B.; Simon, H. A.; and Feigenbaum, E. A. 2002. Simulations of Paired Associate Learning using EPAM VI. Complex Information Processing, Working Paper #553.

Ritter, F. E.; Shadbolt, N. R.; Elliman, D.; Young, R. M.; Gobet, F.; and Baxter, G. D. 2003. Techniques for Modeling Human Performance in Synthetic Environments: A Supplementary Review. Wright-Patterson Air Force Base, Ohio: Human Systems Information Analysis Center.

Ritter, F. E.; Schoelles, M. J.; Quigley, K. S.; and Klein, L. C. 2011. Determining the number of model runs: Treating cognitive models as theories by not sampling their behaviour. In Rothrock, L., and Narayanan, S., eds., Human-in-the-loop simulations: Methods and practice. London: Springer-Verlag. 97-116.

Ritter, F. E. 1991. Towards Fair Comparisons of Connectionist Algorithms through Automatically Optimized Parameter Sets. In Hammond, K. J., and Gentner, D., eds., Proceedings of the Thirteenth Annual Conference of the Cognitive Science Society, 877-881. Hillsdale, NJ: Lawrence Erlbaum.

Roberts, S., and Pashler, H. 2000. How persuasive is a good fit? A comment on theory testing. Psychological Review 107:358-367.

Rumelhart, D. E., and McClelland, J. L., eds. 1986. Parallel Distributed Processing, volume 1 and 2. Cambridge, MA: MIT Press.

Schaffer, J. D. 1984. Some experiments in machine learning using vector evaluated genetic algorithms. Ph.D. Dissertation, Vanderbilt University, Nashville.

Schaffer, C. 1993. Overfitting avoidance as bias. Machine Learning 10:153-178.

Simon, H. A., and Gobet, F. 2000. Expertise effects in memory recall: Comments on Vicente and Wang. Psychological Review 107(3):593-600.

Smith, J. D., and Minda, J. P. 2000. Thirty Categorization Results in Search of a Model. Journal of Experimental Psychology: Learning, Memory and Cognition 26:3-27.

Srinivas, N., and Deb, K. 1994. Multiobjective optimization using nondominated sorting in genetic algorithms. Evolutionary Computation 2:221-248.

Stewart, T., andWest, R. 2010. Testing for equivalence: A methodology for computational cognitive modelling. Journal of Artificial General Intelligence 2:69-87.

Tor, K., and Ritter, F. E. 2004. Using a Genetic Algorithm to Optimize the Fit of Cognitive Models. In Proceedings of the Sixth International Conference on Cognitive Modeling, 308-313. Mahwah, NJ: Lawrence Erlbaum.