Disclosing and Evaluating Artistic Research

The rapid institutionalization of artistic research—or indeed its somewhat artificial introduction as a policy concept and practiced scholarly activity—by the introduction of higher arts education to the university sector (Lesage, 2009; Wilson, 2016) has generated considerable debate these past decades (e.g. Borgdorff, 2012; Busch, 2009; Gielen, 2013; Lesage, 2009; Schwarzenbach & Hackett, 2015; Wissler, 1996). Discussions have focused on the role played by the academization of higher arts education in the output and autonomy of professional artists and faculty members now conceived of as academics (e.g. Gielen, 2013; Jewesbury, 2009), on the parallels or discontinuities between the practices of scientific research and the arts (e.g. Klein, 2010; O'Riley, 2011; Rust, 2007; Sheikh, 2009; Sullivan, 2006) and on difficulties and tensions associated with the quality assessment of artistic research output (e.g. Biggs & Karlsson, 2011; Vanhaesebrouck, 2018; Wilson, 2016; Wissler, 2018). Whereas artistic research is now an institutional reality in many national contexts—embedded in funding schemes (Borgdorff, 2012; Jewesbury, 2009) or included in university performance evaluations (e.g. Lewandowska & Stano, 2018)—there is still little clarity on what exactly constitutes artistic research, how it is distinct from professional art practice in general, and which parameters and indicators could guide the quality assessment of its output. Authors like Wissler (1997), Biggs and Karlsson (2011) or Hellström (2010) emphasize that the field would benefit from disciplinary meta-reflection to generate a “new paradigm” by which to conceive of artistic research as an autonomous discipline. Acknowledging the potential hereto of the collaborative design of quality indicators and assessment models, these authors forward the notion of an evaluation culture or a “community of judgement” (Wissler, 1997) as an indispensable dimension to generate such a paradigm.

In this paper, we first expand on the conditions wherein artistic research was introduced as a policy concept and subsequently institutionalized in Europe—pointing to the ramifications of the 1999 Bologna Declaration (Lesage, 2009) in particular. Without discrediting the potential value and contributions of artistic research, we illustrate its artificial nature—stemming from the sudden elevation of higher art education to university status and its associated expectations. Arguing that this precipitous introduction of higher art education into the traditional university sector has deprived the newly established notion of artistic research from the time needed to organize disciplinary demarcations and goals, we point to the difficulties of assuming a broader, more abstract perspective on the issue of evaluating and assessing of its output. Whereas authors have discussed the evaluation of autonomous forms of artistic research output—be it doctorates (e.g. Biggs & Karlsson, 2011; Lesage, 2009; Paltridge et al., 2011), funding applications and assessments (e.g. Hellström, 2010) or peer review processes for the dissemination of artistic research outcomes (e.g. Borgdorff, 2012), few have explored the potential of institutional performance-based research funding systems (PRFSs) as a platform to debate and discuss the definitions and demarcations of what constitutes artistic research.

Turning to the paper’s case study, we first establish how the situation in Flanders—the Dutch-speaking northern region of Belgium—differs from most European examples in that institutions for higher art education (IHAEs) are now in an association with universities, but have not been subsumed into their structures (Ysebaert & Martens, 2018). As autonomous institutions, they are not included in university PRFSs—contrasting contemporary evaluation frameworks in some other European contexts (e.g. Lewandowska & Stano, 2018; Jewesbury, 2009)—and will be evaluated with a model tailored specifically to artistic research output in the future. By expanding on the ongoing stakeholder-driven development of this model and a test case undertaken to gauge its performance, the paper shows how a culture of evaluation hinges on the generation of a culture of registration. This, in turn, offers crucial resources for disciplinary meta-reflection in the form of a centralized database of artistic research output. As the test case shows, the demand for registration and disclosure associated with quality assessment encourages the disclosure and accessibility of artistic research outcomes. The accessibility hereof is indispensable to the articulation of a new paradigm on artistic research. Simply put, the registration of artistic research outcomes for evaluation and subsequent funding allocation simultaneously furnishes a necessary overview of the output of artistic research as a field in Flanders, which benefits a deeper understanding of its disciplinary ontology.

At a time where the very notion of “artistic research” remains subject to scholarly and artistic debate in Europe (e.g. Gielen, 2013; Klein, 2010; Schwarzenbach & Hackett, 2015; Sheikh, 2009), the methods and practices associated with the assessment and evaluation of its output are disputed (Biggs & Karlsson, 2011; Hellström, 2010; Jewesbury, 2009; Lesage, 2009). Tensions between “artistic” and “scientific” perspectives on what exactly constitutes research or the production of legitimate knowledge problematize clear conceptualizations of quality parameters for the assessment of artistic research output—whether on an individual (e.g. doctoral research) or an aggregated (e.g. institutional research performance) level. In some national contexts—particularly in Nordic countries, but to an extent in Flanders too—this ongoing discussion has led to an intermediary, dualistic evaluation culture that expects both artistic and scientific excellence (Laermans, 2018). One particularly contested ramification hereof is the “double doctorate” (Biggs & Karlsson, 2011; Lesage, 2009 ), alluding to the demanding expectations cultivated in such a system and the inequity of imposing both artistic and scientific standards of performance to artistic researchers (Biggs & Karlsson, 2011). Put bluntly, the current lack of clarity on how artistic research should be evaluated and assessed means that researchers might have to put in twice the time and effort in their work— further exasperated by the presence of both artistic and academic assessors in the examination committees for artistic PhDs. In light of these challenges, authors like Wissler (1997), Biggs and Karlsson (2011) or Hellström (2010) argue for an approach that follows a “new paradigm” on artistic research. The current dualism is then mediated by conceiving of artistic research as a new, autonomous discipline. This fosters attention both for the implications of the introduction of a research component to artistic practices and for the heterogeneity of its perspectives and outcomes. In short, artistic research outcomes should not be assessed by purely artistic nor by purely scientific standards, but by standards particular to the field of artistic research.

Establishing artistic research as a new and autonomous discipline supposes not only a collaborative effort of those involved in artistic research—whether as practitioners, as assessors or as philosophers and critics engaged in theory and reflection. First and foremost, time and the space to engage in a collective effort of discipline-building in the first place is a stringent prerequisite. This simple observation notwithstanding, time and space for reflection is exactly what has been withheld from artistic research at the moment of its rapid institutionalization by the 1999

While the declaration was signed in 1999, the Bologna Process—stipulating among other things the harmonized three-tiered structure for higher education (i.e. Bachelor—Master—PhD)—followed a particular trajectory in the individual countries that ratified the declaration. The Netherlands, for instance, introduced its terms to national legislature in the 2002-2003 academic year, whereas this happened in Flanders in the 2004-2005 academic year.

Consequently, it is not far-fetched to characterize artistic research in Europe— both as a policy concept and an applied or even embodied practice—as an artificial construct (Vanhaesebrouck, 2018). It did not develop organically out of a perceived need for systematic reflection and academization at IHAEs—let alone the art world in general—but rather as a consequence of an inter-European effort towards the harmonization of higher education (Lesage, 2009; Gielen, 2013). “Artistic research” as it features in current European debates is the result of a sudden incorporation of IHAEs in universities (or their elevation to university status). This entails responsibilities with regards to “knowledge production” traditionally associated with science. Hence, reflection on artistic research has little resources to draw from but traditional, scientific conceptualizations of research and how it is to be evaluated—which is exemplified by many essays and papers exploring interlinkages and parallels between characterizations of scientific research and artistic practice (e.g. Jewesbury, 2009; Klein, 2010; O’Riley, 2011; Rust, 2007; Sheikh, 2009; Sullivan, 2006). Conversely, the notion of the “double doctorate” (Biggs & Karlsson, 2011) also testifies to the structuring role of scientific discourses on what constitutes legitimate research output, and the primacy hereof in the evaluation and assessment of the extent to which the requirements for the degree of doctor in artistic disciplines are met.

Nevertheless, neither the somewhat artificial status of artistic research nor the difficulties in conceptualizing what “research” constitutes in the arts downplay the necessity of reflecting on artistic research as a discipline—on the contrary. Given that artistic research has now become an institutionalized discipline, regardless of whether this evolution was desired by those now expected to conduct it, it merits from an exploration of the resources and conditions currently at hand to better understand and define its disciplinary orientations and demarcations. In recognition of the fact that artistic research cannot draw from an established tradition with regards to research as it is understood in the context of the university, research output evaluation cultures are indispensable modalities to explore and discuss the institutional future of artistic research in Europe (Biggs & Karlsson, 2011). However, a particularly tangible result of both the sudden institutionalization of artistic research and the discretion of a defined or formalized research tradition in artistic practice is the absence of an overarching perspective on the evaluation and assessment of artistic research output. Literature on the tensions between the “artistic” and “scientific” merits of artistic research generally departs from difficulties in the assessment of autonomous—that is, individual—forms of artistic research output— such as doctorates (e.g. Biggs & Karlsson, 2011; Lesage, 2009; Paltridge et al., 2011), the funding of individual artistic research networks or projects (e.g. Hellström, 2010) or the review process employed for the evaluation and eventual dissemination of artistic research in specialty journals (e.g. Borgdorff, 2012).

The value of these in-depth explorations of the assessment and evaluation of artistic research to disciplinary reflection is manifest, in that they locate particular tensions between certain quality expectations (e.g. Biggs & Karlsson, 2011; Lesage, 2009) or identify the logics and assumptions of particular evaluation cultures—both in terms of the evaluated and the evaluators (e.g. Hellström, 2010; Wilson, 2016). Hence, these studies discuss the challenges faced by artistic research in its current form, and testify to the necessity of establishing it as an autonomous discipline to mediate these challenges (Biggs & Karlsson, 2011). How artistic research can be articulated as an autonomous discipline, and what the prerequisites are to do so, are understandably beyond their scope. Here, the adoption of a wider, macro perspective seems a necessary avenue to explore, seeing that recognition as an autonomous discipline hinges on the aggregation of a body of works generally accepted as “artistic research outcomes”. Unlike individual assessments and evaluations—be it doctorates, research projects or publications, the potential of institutional performance-based research funding systems (PRFSs) for discipline-building is rarely explored. These systems, which organize and allocate the research funding of institutions like IHAEs are rarely discussed with regards to artistic research. When mentioned, they are generally criticized for their homogenizing and constraining influence on its envisioned outcomes (e.g. Gielen, 2013; Laermans, 2018; Jewesbury, 2009). Indeed, the expectation for individual faculty members to register research outcomes to facilitate institutional research performance evaluation seems to evoke dystopian images of neoliberal rationalization efforts (e.g. Gielen, 2013; Laermans, 2018; Vanhaesebrouck, 2018). This inhibits a candid exploration of how the registration and aggregation of artistic research outcomes could potentially produce a shared body of works recognized as representative outcomes of artistic research practices. Such views, however, cannot be dislodged from the fact that the Bologna Process has effectively subjected higher art education to university standards in many European countries—leaving little to no room for those involved to establish artistic research as a discipline on its own, autonomous terms.

In Flanders, the view that higher art education was effectively subjected to university standards by the implementation of the Bologna Process is not entirely justifiable—notwithstanding that some do characterize the situation as such (e.g. Laermans, 2018; Vanhaesebrouck, 2018), and the academization of higher art education in general has garnered mixed responses at best. Nevertheless, the various institutions for higher art education did not dissolve into existing universities, but engaged in an association with them as Schools of Arts

For reasons of clarity, the abbreviation used to refer to the Schools of Arts in the remainder of this paper is the same used for international examples: IHAEs [Institutions for Higher Art Education].

Hence, as is the case in Nordic countries or Australia (Biggs & Karlsson, 2011; Wilson, 2016), the individual assessment of whether the research output generated in the context of a doctoral program fulfills the requirements for the degree is conducted by both artistic and academic experts in Flanders. As was mentioned before, this had led to inequities particular to such a dualistic evaluation culture, with the “double doctorate” as a central issue (Lesage, 2009). Indeed, it could be argued that the intermediary status of Flemish IHAEs is a defining factor to this problem, seeing that academic assessors are generally absent from the doctoral program in the arts, but do reside in the doctoral examination committee as a sine qua non to grant the title of doctor in the arts. While the framework for doctoral programs tends to differ from association to association (i.e. between particular IHAEs and universities), this situation can represent added difficulties to the notion of the “double doctorate” particular to Flanders. In some cases, it deprives doctoral candidates from direct access to the expectations and considerations fostered by academic members in their examination committees. If the role of academic members of the examination committee is limited or possibly non-existent throughout the doctoral program, arts & design researchers might consider their representation in examination committees disproportionate.

There is another side of the coin to this intermediary solution for the academization of higher art education, however. Among other implications less important to the scope of this paper, this model precludes the output of Flemish IHAEs from the PRFSs used for the allocation of institutional research funding used for Flemish universities—which tend to emphasize quantitative, scientometric evaluation methods and conceptualize research output exclusively as publications (Engels et al., 2018). The research funding model for the Schools of Arts is independent from that of universities, meaning that the former are not expected to conform to the standards of the latter. This differs crucially from, for instance, the REF-system used in the United Kingdom, where research funding for faculties of arts is allocated by using the same general quality indicators used for the university in general— notwithstanding that the evaluation itself is based on peer reviews (Nelson, 2013) and research outcomes are not necessarily expected to take the form of a traditional publication. In Flanders, the partial autonomy of the IHAEs supposes an autonomous model to assess their research outcomes, meaning that they are not held to the standards used to measure universities’ research performance.

At the time, Flemish policy makers recognized the specificity of artistic & design research and the undesirability of imposing an ill-suited assessment format, organizing the research funding of the Schools of Arts—that is, funding distinct from competitive grants awarded to individual researchers generating overhead for the institutions they are embedded in—in an ex-ante model for the time being. As such, the institutional component of research funding currently foregoes on the evaluation of the Schools of Arts’ research output, and instead uses student enrollments and delivered degrees (Higher Education Context, III)—in many ways similar to the current Norwegian approach (Sivertsen, 2018). Naturally, this approach has its detriments too, and questions regarding the fairness of research funding allocation based on student enrollments and delivered degrees are exceptionally pertinent in this regard. Currently, larger institutions—whose research output is not necessarily commensurate to their scale—clearly enjoy an unfair advantage to their smaller counterparts—whose research output might exceed that of institutions with greater student enrollments and delivered degrees.

This system furthermore disadvantages IHAEs constricted by a numerus fixus system. Seeing that student enrollments and delivered degrees are stable at such institutions, they currently cannot enjoy an increase in research funding induced by rising student numbers even though their research output might be growing.

naturally, the strategic decision to let an evaluation culture for artistic & design research output emerge bottom-up from stakeholders in Flanders does not entail a complete laisser-faire policy on behalf of the administration. The stakeholder-driven approach was facilitated and organized by the Vrije Universiteit Brussel (VUB— Free University of Brussels) branch of the Flemish Center for Research and Development Monitoring (ECOOM)—which is predominantly devoted to the development of a central registration format and evaluation design for art and design research outcomes (Ysebaert & Martens, 2018). The role of ECOOM-VUB and its researchers is therefore not that of decision makers, but rather that of facilitators and mediators. Therefore, the effort took the form of what Barré (2010) calls an “indicator platform”, whereby involved parties

The consulted stakeholders consist of Flemish IHAE faculty members with coordinating roles—such as research coordinators or directors.

Initially, this approach crystalized in the formation of a “Working Group on Content” (IW—Inhoudelijke Werkgroep)—comprising of representatives from the Flemish Schools of Arts—which met several times to discuss the most suitable method for the assessment of arts & design research output

Each Flemish IHAE has a representative in the IW, ensuring that the indicators decided upon reflect the diversity and pluralism of higher art education in Flanders—which covers all artistic disciplines.

This initially resulted in a design wherein three separate “research dimensions” are emphasized as indicative of a qualitative and meaningful arts & design research outcome: “quality”, “impact” and “contextuality”. The IW conceptualized research quality around notions of originality, rigor, transparency, process reflexivity and representation and/or dissemination of outcomes. Its characterization of research impact distinguishes between artistic impact, knowledge impact and societal impact. Research context, finally, mediates the wish of the IW members to allow quality assessment to be attentive for the consistency of the research outcomes with the individual research policies of the Flemish IHAEs—pointing to the ambition of the institutions themselves to articulate a research identity of their own—and for the possible ways in which the research outcome demonstrates meaningful interaction with relevant practitioners in the artistic field. Importantly, the dimensions decided upon by the IW articulate a conviction that arts & design research must be distinct from arts & design practice in general, but do not stipulate that the outcomes of arts & design research are necessarily distinct from those of artistic practice—meaning primarily that the tangible results of arts & design research might not differ substantially from the products of arts & design practice. While the artistic product—whether this product consists of one or more material artefacts or a singular, ephemeral performance—is the central node in the evaluation procedure, the indicators used to register and document the differing research quality dimensions served to elucidate and evocate the process underlying the product (see table 1: “Research context” & “Research impact”). In the view of the IW, the processes and procedures that inform, shape and guide the final artistic outcome are the primary criteria to differentiate between arts & design practice and arts & design research, and should therefore be central to the quality assessment of the output. The table below demonstrates how the considerations and decisions made by the IW were translated into the architecture of the FRIS database segment for artistic and design research outcomes:

As table 1 illustrates, the finality of the FRIS database segment for arts & design research outcomes cannot reside in quantification or metric approaches—seeing that the registration format does not facilitate the use of such methods. Rather, it discloses arts & design research output generated at Flemish IHAEs, in order to facilitate an evaluation culture based on qualitative peer assessment methods.

Up to this point, this remained an expressively abstract exercise—which nevertheless signaled a first collaborative effort towards the construction of a new critical paradigm for artistic research in Flanders (Ysebaert & Martens, 2018)— in need of practical testing. To this end, a test case was organized in 2016 for output entries eligible for evaluation for the Architecture & Design expert panel

The other expert panels to which artistic research output may be submitted are: Music, Stage Arts, Audiovisual Arts, Visual Arts and an Interdisciplinary panel. As is the case with the output indicators themselves, these panels—and the disciplinary contexts they represent—were decided upon by the stakeholders in the IW.

In practice, the test case entailed both a registration phase—implying that partaking institutions entered their output data into the FRIS-segment for artistic research output—and a peer evaluation phase. For this latter phase, international experts were chosen from a list of eligible candidates provided by the participating institutions with demonstrable expertise in architecture and design research. The evaluation by the expert panels was intended to have four points of focus: a concise research statement by participating institutions (1), the totality of entries in the FRIS-segment for artistic research output marked as Architecture & Design (3), a selection of three entries per institution decided upon by the expert panel for further discussion by participating institutions (3) and a selection of three entries chosen by the institutions themselves which they felt were representative for their take on artistic research (4). As such, the third and fourth evaluation exercises imply a direct interaction between the expert panels and participating institutions—rather than a separate or even anonymous peer review process. Admittedly, the results of this test case are all derived from the registration and evaluation of artistic research output submitted to the Architecture and Design expert panel. Despite this particular focus, they identify crucial elements pertinent to the generation of an autonomous paradigm on artistic research.

That contemporary discussions on the quality assessment of artistic research output tend to focus on individual examples of such outcomes (e.g. Biggs & Karlsson, 2011; Hellström, 2010; Lesage, 2009; Wilson, 2016) is understandable, but this should not be allowed to obscure the potential of reflecting on large-scale evaluation models for the articulation of a new paradigm on artistic research. While exploring quality assessment procedures used for the evaluation of doctorates, research projects or the dissemination of research outcomes identifies particular challenges—such as the discrepancies between artistic and scientific expectations of excellence or different conceptualizations regarding the generation of knowledge, approaching evaluation cultures from a large-scale perspective has particular merits. The Flemish test case discussed in this paper specifically demonstrates how the registration of artistic research outcomes is welcomed by stakeholders as a tool to map the field of artistic research. Although it would be unrealistic to expect that the registration of artistic research output has immediate effects for the establishment of a new paradigm on artistic research as an autonomous field, the collection and categorization of outcomes in one centralized database does retain the potential to serve as a useful tool for contextualization and reflection in the long run. Seeing that artistic research currently has little options but to draw from scientific discourses on what constitutes research and the production of knowledge, its disciplinary future is predicated on the generation of an autonomous body of outcomes that may serve as a theoretical and methodological context for future research. Herein, additional engagements with the quality assessment and evaluation of artistic research on a level that exceeds that of individual output are crucial interventions for a deeper and more nuanced understanding of artistic research as an autonomous discipline.