<abstract xmlns="http://www.w3.org/1999/xhtml"><p>A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.</p></abstract>

A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.

Encouraging an appropriate representation simplifies training of neural networks

Acta Universitatis Sapientiae, Informatica

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

{"article-title":"Encouraging an appropriate representation simplifies training of neural networks"}

A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this...

Encouraging an appropriate representation simplifies training of neural networks

Published Online: Jul 16, 2020

Page range: 102 - 111

Received: Mar 26, 2020

Accepted: Jun 07, 2020

DOI: https://doi.org/10.2478/ausi-2020-0007

Keywords
neural networks, representation, integration of domain knowledge

© 2020 Krisztian Buza, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Encouraging an appropriate representation simplifies training of neural networks

Published Online: Jul 16, 2020

Page range: 102 - 111

Received: Mar 26, 2020

Accepted: Jun 07, 2020

DOI: https://doi.org/10.2478/ausi-2020-0007

Keywordsneural networks, representation, integration of domain knowledge

© 2020 Krisztian Buza, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Keywords
neural networks, representation, integration of domain knowledge