Keep it simple - A case study of model development in the context of the Dynamic Stocks and Flows (DSF) task
This paper describes the creation of a cognitive model submitted to the ‘Dynamic Stocks and Flows’ (DSF) modeling challenge. This challenge aims at comparing computational cognitive models for human behavior during an open ended control task. Participants in the modeling competition were provided with a simulation environment and training data for benchmarking their models while the actual specification of the competition task was withheld. To meet this challenge, the cognitive model described here was designed and optimized for generalizability. Only two simple assumptions about human problem solving were used to explain the empirical findings of the training data. In-depth analysis of the data set prior to the development of the model led to the dismissal of correlations or other parametric statistics as goodness-of-fit indicators. A new statistical measurement based on rank orders and sequence matching techniques is being proposed instead. This measurement, when being applied to the human sample, also identifies clusters of subjects that use different strategies for the task. The acceptability of the fits achieved by the model is verified using permutation tests.
completion times, Computers & Operations Research , 36 , 2009, 3031-3040.
 Ching C. Y., Liao, Wu C. J., Batching to minimize total production time for two part types, International Journal of Production Economics , 48 , 1997, 63-72.
 Drobouchevitch I. G., Strusevich V. A., Heuristics for the two-stage job shop scheduling problem with a bottleneck machine, European Journal of Operational Research , 123 , 2000, 229-240.
 Kyparisis G. J., Koulamas, C., Flow shop and open shop scheduling with a critical machine and
Andrzej Polańczyk, Michał Strzelecki, Tomasz Woźniak, Wojciech Szubert and Ludomir Stefańczyk
-Newtonian properties of blood on the flow in large arteries: steady flow in a carotid bifurcation model, Journal of Biomechanics, 32, 1999, 601-608.
 Gonzales R., Woods R., Digital Image Processing, Addison-Wesley, 1983.
 Hoi Y., Meng H., Woodward S.H., Bendok B.R., Hanel R.A., Guterman L.R., Hopkins L.N., Effects of arterial geometry on aneurysm growth: three-dimensional computational fluid dynamics study, Journal of Neurosurgery, 101, 2004, 676-681.
 Klepaczko A., Szczypiński P., Dwojakowski G., Strzelecki M., Materka A
Gonzalez, C., and Lebiere, C. 2005. Instance-based cognitive models of decision making. In Zizzo, D., and Courakis, A., eds., Transfer of knowledge in economic decision making. New York: Palgrave McMillan.
Gonzalez, C.; Lerch, F.; and Lebiere, C. 2003. Instance-based learning in dynamic decision making. Cognitive Science 27(4):591-635.
Halbrügge, M. 2010. Keep it simple-A case study of model development in the context of the Dynamic Stocks and Flows (DSF) task. Journal of Artificial General Intelligence (this
Kevin Gluck, Clayton Stanley, L. Moore, David Reitter and Marc Halbrügge
the Sixteenth Conference on Behavior Representation in Modeling and Simulation , 73-83, Orlando, FL: Simulation Interoperability Standards Organization.
Gonzalez, C., and Dutt, V. 2007. Learning to control a dynamic task: A system dynamics cognitive model of the slope effect. In Proceedings of the 8th International Conference on Cognitive Modeling , 61-66. Ann Arbor, MI.
Halbrügge, M. in press. Keep it simple - A case study of model development in the context of the Dynamic Stock and Flows (DSF) task. Journal of
 Fanjul-Peyro, L., Ruiz, R., Scheduling unrelated parallel machines with optional machines and jobs selection, Computers and Operational Research, 39 , 2012, 1745-1753.
 Finke, G., Lemaire, P., Proth, J. M., Queyranne, M., Minimizing the number of machines for minimum length schedules, European Journal of Operational Research, 199 , 2009, 702-705.
 Gerstl, E., Mosheiov, G., A two-stage flow shop scheduling with a critical machine and batch availability, Foundations of Computing and Decision Sciences, 37 , 2012, 39
Modelling Dynamic Decision Making with the ACT-R Cognitive Architecture
This paper describes a model of dynamic decision making in the Dynamic Stocks and Flows (DSF) task, developed using the ACT-R cognitive architecture. This task is a simple simulation of a water tank in which the water level must be kept constant whilst the inflow and outflow changes at varying rates. The basic functions of the model are based around three steps. Firstly, the model predicts the water level in the next cycle by adding the current water level to the predicted net inflow of water. Secondly, based on this projection, the net outflow of the water is adjusted to bring the water level back to the target. Thirdly, the predicted net inflow of water is adjusted to improve its accuracy in the future. If the prediction has overestimated net inflow then it is reduced, if it has underestimated net inflow it is increased. The model was entered into a model comparison competition—the Dynamic Stocks and Flows Challenge—to model human performance on four conditions of the DSF task and then subject the model to testing on five unseen transfer conditions. The model reproduced the main features of the development data reasonably well but did not reproduce human performance well under the transfer conditions. This suggests that the principles underlying human performance across the different conditions differ considerably despite their apparent similarity. Further lessons for the future development of our model and model comparison challenges are considered.
Testing for Equivalence: A Methodology for Computational Cognitive Modelling
The equivalence test (Stewart and West, 2007; Stewart, 2007) is a statistical measure for evaluating the similarity between a model and the system being modelled. It is designed to avoid over-fitting and to generate an easily interpretable summary of the quality of a model. We apply the equivalence test to two tasks: Repeated Binary Choice (Erev et al., 2010) and Dynamic Stocks and Flows (Gonzalez and Dutt, 2007). In the first case, we find a broad range of statistically equivalent models (and win a prediction competition) while identifying particular aspects of the task that are not yet adequately captured. In the second case, we re-evaluate results from the Dynamic Stocks and Flows challenge, demonstrating how our method emphasizes the breadth of coverage of a model and how it can be used for comparing different models. We argue that the explanatory power of models hinges on numerical similarity to empirical data over a broad set of measures.
Christian Lebiere, Cleotilde Gonzalez and Walter Warwick
Editorial: Cognitive Architectures, Model Comparison and AGI
Cognitive Science and Artificial Intelligence share compatible goals of understanding and possibly generating broadly intelligent behavior. In order to determine if progress is made, it is essential to be able to evaluate the behavior of complex computational models, especially those built on general cognitive architectures, and compare it to benchmarks of intelligent behavior such as human performance. Significant methodological challenges arise, however, when trying to extend approaches used to compare model and human performance from tightly controlled laboratory tasks to complex tasks involving more open-ended behavior. This paper describes a model comparison challenge built around a dynamic control task, the Dynamic Stocks and Flows. We present and discuss distinct approaches to evaluating performance and comparing models. Lessons drawn from this challenge are discussed in light of the challenge of using cognitive architectures to achieve Artificial General Intelligence.
.A.; and Smith, S.S. 2001.Using brain MERMER testing to detect knowledge despite efforts to conceal. J Forensic Sci. 46 (1): 135-143.
Fiebelkorn, I.C.; Snyder, A.C.; and et al. 2013. Cortical cross-frequency coupling predicts perceptual outcomes. Neuroimage. 1;69:126-37. Fleury, V. 2011. A change in boundary conditions induces a discontinuity of tissue flow in chicken embryos and the formation of the cephalic fold. The European Physical Journal E. 34(7) Freeman, W. J.; and Kozma, R. 2010. Freeman's mass action Scholarpedia, 5(1):8040 Available