<abstract xmlns="http://www.w3.org/1999/xhtml"><p>In test-based problems, solutions produced by search algorithms are typically assessed using average outcomes of interactions with multiple tests. This aggregation leads to information loss, which can render different solutions apparently indifferent and hinder comparison of search algorithms. In this paper we introduce the <italic>performance profile</italic>, a generic, domain-independent, multi-criteria performance evaluation method that mitigates this problem by characterizing the performance of a solution by a <italic>vector</italic> of outcomes of interactions with tests of various difficulty. To demonstrate the usefulness of this gauge, we employ it to analyze the behavior of Othello and Iterated Prisoner’s Dilemma players produced by five (co)evolutionary algorithms as well as players known from previous publications. Performance profiles reveal interesting differences between the players, which escape the attention of the scalar performance measure of the expected utility. In particular, they allow us to observe that evolution with random sampling produces players coping well against the mediocre opponents, while the coevolutionary and temporal difference learning strategies play better against the high-grade opponents. We postulate that performance profiles improve our understanding of characteristics of search algorithms applied to arbitrary test-based problems, and can prospectively help design better methods for interactive domains.</p></abstract>

In test-based problems, solutions produced by search algorithms are typically assessed using average outcomes of interactions with multiple tests. This aggregation leads to information loss, which can render different solutions apparently indifferent and hinder comparison of search algorithms. In this paper we introduce the performance profile, a generic, domain-independent, multi-criteria performance evaluation method that mitigates this problem by characterizing the performance of a solution by a vector of outcomes of interactions with tests of various difficulty. To demonstrate the usefulness of this gauge, we employ it to analyze the behavior of Othello and Iterated Prisoner’s Dilemma players produced by five (co)evolutionary algorithms as well as players known from previous publications. Performance profiles reveal interesting differences between the players, which escape the attention of the scalar performance measure of the expected utility. In particular, they allow us to observe that evolution with random sampling produces players coping well against the mediocre opponents, while the coevolutionary and temporal difference learning strategies play better against the high-grade opponents. We postulate that performance profiles improve our understanding of characteristics of search algorithms applied to arbitrary test-based problems, and can prospectively help design better methods for interactive domains.

The performance profile: A multi–criteria performance evaluation method for test–based problems

Institute of Computing Science, Poznań University of Technology, ul. Piotrowo 2, 60-965 Poznań, Poland

International Journal of Applied Mathematics and Computer Science

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

In test-based problems, solutions produced by search algorithms are typically assessed using average outcomes of interactions with multiple tests. This...

The performance profile: A multi–criteria performance evaluation method for test–based problems

Published Online: Mar 31, 2016

Page range: 215 - 229

Received: Feb 13, 2015

DOI: https://doi.org/10.1515/amcs-2016-0015

Keywords
coevolutionary algorithms, evolution strategies, Othello, Reversi, games, multi-objective analysis

© 2016 Wojciech Jaśkowski et al., published by De Gruyter Open

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

The performance profile: A multi–criteria performance evaluation method for test–based problems

Published Online: Mar 31, 2016

Page range: 215 - 229

Received: Feb 13, 2015

DOI: https://doi.org/10.1515/amcs-2016-0015

Keywordscoevolutionary algorithms, evolution strategies, Othello, Reversi, games, multi-objective analysis

© 2016 Wojciech Jaśkowski et al., published by De Gruyter Open

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Keywords
coevolutionary algorithms, evolution strategies, Othello, Reversi, games, multi-objective analysis