Learning Structures of Conceptual Models from Observed Dynamics Using Evolutionary Echo State Networks

Open access


Conceptual or explanatory models are a key element in the process of complex system modelling. They not only provide an intuitive way for modellers to comprehend and scope the complex phenomena under investigation through an abstract representation but also pave the way for the later development of detailed and higher-resolution simulation models. An evolutionary echo state network-based method for supporting the development of such models, which can help to expedite the generation of alternative models for explaining the underlying phenomena and potentially reduce the manual effort required, is proposed. It relies on a customised echo state neural network for learning sparse conceptual model representations from the observed data. In this paper, three evolutionary algorithms, a genetic algorithm, differential evolution and particle swarm optimisation are applied to optimize the network design in order to improve model learning. The proposed methodology is tested on four examples of problems that represent complex system models in the economic, ecological and physical domains. The empirical analysis shows that the proposed technique can learn models which are both sparse and effective for generating the output that matches the observed behaviour.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] J. D. Sterman Business Dynamics: Systems Thinking and Modeling for a Complex World vol. 19. Irwin/McGraw-Hill Boston 2000.

  • [2] F. C. Billari Agent-based computational modelling: applications in demography social economic and environmental sciences. Taylor & Francis 2006.

  • [3] R. A. Howard and J. E. Matheson Influence diagrams Decis. Anal. vol. 2 no. 3 pp. 127–143 2005.

  • [4] F.-R. Lin M.-C. Yang and Y.-H. Pai A generic structure for business process modeling Bus. Process Manag. J. vol. 8 no. 1 pp. 19–41 2002.

  • [5] L. Schruben Simulation modeling with event graphs Commun. ACM vol. 26 no. 11 pp. 957–963 1983.

  • [6] S. Robinson Simulation: the practice of model development and use. Palgrave Macmillan 2014.

  • [7] J. Ryan and C. Heavey Requirements gathering for simulation in Proceedings of the 3rd Operational Research Society Simulation Workshop. The Operational Research Society Birmingham UK 175-184 2006.

  • [8] A. Medina-Borja and K. S. Pasupathy Uncovering complex relationships in system dynamics modeling: Exploring the use of CART CHAID and SEM in Proceedings of the 25th International Conference of the System Dynamics Society (Boston USA) pp. 1–24 2007.

  • [9] V. Quiñones-Avila and A. Medina-Borja Universal healthcare: key behavioural factors affecting providers and recipients value propositions: a structural causal model of the puerto rico experience Int. J. of Behav. and Hlthc. Res. vol. 3 no. 1 pp. 25–45 2012.

  • [10] M. Drobek W. Gilani T. Molka and D. Soban Automated equation formulation for causal loop diagrams Lecture Notes in Business Information Processing vol. 208 pp. 38–49 2015.

  • [11] E. Pruyt S. Cunningham J. Kwakkel and J. De Bruijn From data-poor to data-rich: system dynamics in the era of big data in Proceedings of the 32nd International Conference of the System Dynamics Society Delft The Netherlands 20-24 July 2014.

  • [12] H. Jaeger The ’echo state’ approach to analysing and training recurrent neural networks-with an erratum note Bonn Germany: German National Research Center for Information Technology GMD Technical Report vol. 148 p. 34 2001.

  • [13] H. Abdelbari and K. Shafi Learning causal loop diagram-like structures for system dynamics modeling using echo state networks Syst. Dynam. Rev. - In Press 2017.

  • [14] D. E. Goldberg Genetic algorithms. Pearson Education India 2006.

  • [15] R. Storn and K. Price Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces J. Global. Optim. vol. 11 no. 4 pp. 341–359 1997.

  • [16] J. Kennedy Particle swarm optimization in Encyclopedia of machine learning pp. 760–766 Springer 2011.

  • [17] Z. Wang J. Zhang J. Ren and M. N. Aslam A geometric singular perturbation approach for planar stationary shock waves Physica D vol. 310 pp. 19–36 2015.

  • [18] C. K. Jones R. Marangell P. D. Miller and R. G. Plaza On the stability analysis of periodic sine–gordon traveling waves Physica D vol. 251 pp. 63–74 2013.

  • [19] V. V. Gursky J. Reinitz and A. M. Samsonov How gap genes make their domains: An analytical study based on data driven approximations Chaos vol. 11 no. 1 pp. 132–141 2001.

  • [20] P. Young Data-based mechanistic modelling of environmental ecological economic and engineering systems Environ. Modell. Softw. vol. 13 no. 2 pp. 105–122 1998.

  • [21] Y. Zhao T. Weng and M. Small Response of the parameters of a neural network to pseudoperiodic time series Physica D vol. 268 pp. 79–90 2014.

  • [22] Y. Feng Y. Liu X. Tong M. Liu and S. Deng Modeling dynamic urban growth using cellular automata and particle swarm optimization rules Landscape Urban Plan. vol. 102 no. 3 pp. 188–196 2011.

  • [23] N. Petrov and A. Gegov Model optimization for complex systems using fuzzy networks theory in Proceedings of the 8th WSEAS international conference on Artificial intelligence knowledge engineering and data bases pp. 116–121 World Scientific and Engineering Academy and Society (WSEAS) 2009.

  • [24] I. M. Greca and M. A. Moreira Mental models conceptual models and modelling Int. J. Sci. Educ vol. 22 no. 1 pp. 1–11 2000.

  • [25] J. D. Sterman Systems dynamics modeling: tools for learning in a complex world IEEE Eng. Manag. Rev. vol. 30 no. 1 pp. 42–42 2002.

  • [26] G. Desthieux F. Joerin and M. Lebreton Ulysse: a qualitative tool for eliciting mental models of complex systems Syst. Dynam. Rev. vol. 26 no. 2 pp. 163–192 2010.

  • [27] K.-i. Funahashi and Y. Nakamura Approximation of dynamical systems by continuous time recurrent neural networks Neural networks vol. 6 no. 6 pp. 801–806 1993.

  • [28] H. Jaeger Tutorial on training recurrent neural networks covering BPPT RTRL EKF and the” echo state network” approach Tech. Rep. 159 Fraunhofer Institute for Autonomous Intelligent Systems (AIS) 2002b.

  • [29] D. Koryakin J. Lohmann and M. V. Butz Balanced echo state networks Neural Networks vol. 36 pp. 35–45 2012.

  • [30] I. B. Yildiz H. Jaeger and S. J. Kiebel Re-visiting the echo state property Neural networks vol. 35 pp. 1–9 2012.

  • [31] M. Lukoševišius A practical guide to applying echo state networks in Neural Networks: Tricks of the Trade pp. 659–686 Springer 2012.

  • [32] C. E. Martin and J. A. Reggia Fusing swarm intelligence and self-assembly for optimizing echo state networks Comput. Intell. Neurosci. vol. 2015 p. 9 2015.

  • [33] A. A. Ferreira and T. B. Ludermir Comparing evolutionary methods for reservoir computing pretraining in Proceedings of the 2011 International Joint Conference on Neural Networks San Jose California USA pp. 283–290 July 31 - August 5 2011.

  • [34] A. Deihimi and A. Solat optimised echo state networks using a big bang–big crunch algorithm for distance protection of series-compensated transmission lines Int. J. Elec. Power. vol. 54 pp. 408–424 2014.

  • [35] A. A. Ferreira T. B. Ludermir and R. R. De Aquino An approach to reservoir computing design and training Expert. Syst. Appl. vol. 40 no. 10 pp. 4172–4182 2013.

  • [36] D. Liu J. Wang and H. Wang Short-term wind speed forecasting based on spectral clustering and optimised echo state networks Renew. Energ. vol. 78 pp. 599–608 2015.

  • [37] J. L. Gross and J. Yellen Handbook of graph theory. CRC press 2004.

  • [38] R. Tarjan Depth-first search and linear graph algorithms SIAM J. Comput. vol. 1 no. 2 pp. 146–160 1972.

  • [39] V. Petridis S. Kazarlis and A. Bakirtzis Varying fitness functions in genetic algorithm constrained optimization: the cutting stock and unit commitment problems IEEE Trans. Syst. Man Cybern. Part B: Cybern. vol. 28 no. 5 pp. 629–640 1998.

  • [40] A. E. Smith and D. M. Tate Genetic optimization using a penalty function in Proceedings of the 5th international conference on genetic algorithms pp. 499–505 Morgan Kaufmann Publishers Inc. 1993.

  • [41] K. Langfield-Smith and A. Wirth Measuring differences between cognitive maps J. Oper. Res. Soc. pp. 1135–1150 1992.

  • [42] Y.-C. Chuang C.-T. Chen and C. Hwang A simple and efficient real-coded genetic algorithm for constrained optimization Appl. Soft. Comput. vol. 38 pp. 87–105 2016.

  • [43] J. Lane A. Engelbrecht and J. Gain Particle swarm optimization with spatially meaningful neighbours in Swarm Intelligence Symposium 2008. SIS 2008. IEEE pp. 1–8 IEEE 2008.

  • [44] R. C. Eberhart and Y. Shi Comparing inertia weights and constriction factors in particle swarm optimization in Proceedings of the 2000 Congress on Evolutionary Computation vol. 1 pp. 84–88 IEEE 2000.

  • [45] S. N. Grösser and M. Schaffernicht Mental models of dynamic systems: taking stock and looking ahead Syst. Dynam. Rev. vol. 28 no. 1 pp. 46–68 2012.

  • [46] E. M. Aylward P. A. Parrilo and J.-J. E. Slotine Stability and robustness analysis of nonlinear systems via contraction metrics and sos programming Automatica vol. 44 no. 8 pp. 2163–2170 2008.

  • [47] M. Rafferty Butterflies and buffers in Proceedings of the 27th International Conference of the System Dynamics Society Albuquerque Mexico USA July 26-30 2009.

  • [48] E. Theodorsson-Norheim Friedman and quade tests: Basic computer program to perform nonparametric two-way analysis of variance and multiple comparisons on ranks of several related samples Comput. Biol. Med. vol. 17 no. 2 pp. 85–99 1987.

  • [49] M. R. Stoline The status of multiple comparisons: simultaneous estimation of all pairwise comparisons in one-way anova designs Am. Stat. vol. 35 no. 3 pp. 134–141 1981.

  • [50] K. Deb A. Pratap S. Agarwal and T. Meyarivan A fast and elitist multiobjective genetic algorithm: NSGA-II IEEE Trans. Evol. Comput. vol. 6 no. 2 pp. 182–197 2002.

Journal information
Impact Factor

CiteScore 2018: 4.70

SCImago Journal Rank (SJR) 2018: 0.351
Source Normalized Impact per Paper (SNIP) 2018: 4.066

Cited By
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 439 258 7
PDF Downloads 220 112 3