<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>Measuring progress in the field of Artificial General Intelligence (AGI) can be difficult without commonly accepted methods of evaluation. An AGI benchmark would allow evaluation and comparison of the many computational intelligence algorithms that have been developed. In this paper I propose that a benchmark for natural world interaction would possess seven key characteristics: fitness, breadth, specificity, low cost, simplicity, range, and task focus. I also outline two benchmark examples that meet most of these criteria. In the first, the direction task, a human coach directs a machine to perform a novel task in an unfamiliar environment. The direction task is extremely broad, but may be idealistic. In the second, the AGI battery, AGI candidates are evaluated based on their performance on a collection of more specific tasks. The AGI battery is designed to be appropriate to the capabilities of currently existing systems. Both the direction task and the AGI battery would require further definition before implementing. The paper concludes with a description of a task that might be included in the AGI battery: the search and retrieve task.</p></abstract>

Measuring progress in the field of Artificial General Intelligence (AGI) can be difficult without commonly accepted methods of evaluation. An AGI benchmark would allow evaluation and comparison of the many computational intelligence algorithms that have been developed. In this paper I propose that a benchmark for natural world interaction would possess seven key characteristics: fitness, breadth, specificity, low cost, simplicity, range, and task focus. I also outline two benchmark examples that meet most of these criteria. In the first, the direction task, a human coach directs a machine to perform a novel task in an unfamiliar environment. The direction task is extremely broad, but may be idealistic. In the second, the AGI battery, AGI candidates are evaluated based on their performance on a collection of more specific tasks. The AGI battery is designed to be appropriate to the capabilities of currently existing systems. Both the direction task and the AGI battery would require further definition before implementing. The paper concludes with a description of a task that might be included in the AGI battery: the search and retrieve task.

Accelerating progress in Artificial General Intelligence: Choosing a benchmark for natural world interaction

Sandia National Laboratories, MS 1010, PO Box 5800 Albuquerque, NM 87185-1010, USA

Journal of Artificial General Intelligence

Measuring progress in the field of Artificial General Intelligence (AGI) can be difficult without commonly accepted methods of evaluation. An AGI benchmark...

Accelerating progress in Artificial General Intelligence: Choosing a benchmark for natural world interaction

Published Online: Nov 23, 2011

Page range: 1 - 28

DOI: https://doi.org/10.2478/v10229-011-0005-5

Keywords
benchmark, metrics, roadmap, breadth, direction task, AGI battery, natural world interaction

This content is open access.

Accelerating progress in Artificial General Intelligence: Choosing a benchmark for natural world interaction

Published Online: Nov 23, 2011

Page range: 1 - 28

DOI: https://doi.org/10.2478/v10229-011-0005-5

Keywordsbenchmark, metrics, roadmap, breadth, direction task, AGI battery, natural world interaction

This content is open access.

Keywords
benchmark, metrics, roadmap, breadth, direction task, AGI battery, natural world interaction