Disclosing and Evaluating Artistic Research

Open access

Abstract

Purpose

This study expands on the results of a stakeholder-driven research project on quality indicators and output assessment of art and design research in Flanders—the Northern, Dutch-speaking region of Belgium. Herein, it emphasizes the value of arts & design output registration as a modality to articulate the disciplinary demarcations of art and design research.

Design/methodology/approach

The particularity of art and design research in Flanders is first analyzed and compared to international examples. Hereafter, the results of the stakeholder-driven project on the creation of indicators for arts & design research output assessment are discussed.

Findings

The findings accentuate the importance of allowing an assessment culture to emerge from practitioners themselves, instead of imposing ill-suited methods borrowed from established scientific evaluation models (Biggs & Karlsson, 2011)—notwithstanding the practical difficulties it generates. They point to the potential of stakeholder-driven approaches for artistic research, which benefits from constructing a shared metadiscourse among its practitioners regarding the continuities and discontinuities between “artistic” and “traditional” research, and the communal goals and values that guide its knowledge production (Biggs & Karlsson, 2011; Hellström, 2010; Ysebaert & Martens, 2018).

Research limitation

The central limitation of the study is that it focuses exclusively on the “Architecture & Design” panel of the project, and does not account for intra-disciplinary complexities in output assessment.

Practical implications

The goal of the research project is to create a robust assessment system for arts & design research in Flanders, which may later guide similar international projects.

Originality/value

This study is currently the only one to consider the productive potential of (collaborative) PRFSs for artistic research.

1 Introduction

The rapid institutionalization of artistic research—or indeed its somewhat artificial introduction as a policy concept and practiced scholarly activity—by the introduction of higher arts education to the university sector (Lesage, 2009; Wilson, 2016) has generated considerable debate these past decades (e.g. Borgdorff, 2012; Busch, 2009; Gielen, 2013; Lesage, 2009; Schwarzenbach & Hackett, 2015; Wissler, 1996). Discussions have focused on the role played by the academization of higher arts education in the output and autonomy of professional artists and faculty members now conceived of as academics (e.g. Gielen, 2013; Jewesbury, 2009), on the parallels or discontinuities between the practices of scientific research and the arts (e.g. Klein, 2010; O'Riley, 2011; Rust, 2007; Sheikh, 2009; Sullivan, 2006) and on difficulties and tensions associated with the quality assessment of artistic research output (e.g. Biggs & Karlsson, 2011; Vanhaesebrouck, 2018; Wilson, 2016; Wissler, 2018). Whereas artistic research is now an institutional reality in many national contexts—embedded in funding schemes (Borgdorff, 2012; Jewesbury, 2009) or included in university performance evaluations (e.g. Lewandowska & Stano, 2018)—there is still little clarity on what exactly constitutes artistic research, how it is distinct from professional art practice in general, and which parameters and indicators could guide the quality assessment of its output. Authors like Wissler (1997), Biggs and Karlsson (2011) or Hellström (2010) emphasize that the field would benefit from disciplinary meta-reflection to generate a “new paradigm” by which to conceive of artistic research as an autonomous discipline. Acknowledging the potential hereto of the collaborative design of quality indicators and assessment models, these authors forward the notion of an evaluation culture or a “community of judgement” (Wissler, 1997) as an indispensable dimension to generate such a paradigm.

In this paper, we first expand on the conditions wherein artistic research was introduced as a policy concept and subsequently institutionalized in Europe—pointing to the ramifications of the 1999 Bologna Declaration (Lesage, 2009) in particular. Without discrediting the potential value and contributions of artistic research, we illustrate its artificial nature—stemming from the sudden elevation of higher art education to university status and its associated expectations. Arguing that this precipitous introduction of higher art education into the traditional university sector has deprived the newly established notion of artistic research from the time needed to organize disciplinary demarcations and goals, we point to the difficulties of assuming a broader, more abstract perspective on the issue of evaluating and assessing of its output. Whereas authors have discussed the evaluation of autonomous forms of artistic research output—be it doctorates (e.g. Biggs & Karlsson, 2011; Lesage, 2009; Paltridge et al., 2011), funding applications and assessments (e.g. Hellström, 2010) or peer review processes for the dissemination of artistic research outcomes (e.g. Borgdorff, 2012), few have explored the potential of institutional performance-based research funding systems (PRFSs) as a platform to debate and discuss the definitions and demarcations of what constitutes artistic research.

Turning to the paper’s case study, we first establish how the situation in Flanders—the Dutch-speaking northern region of Belgium—differs from most European examples in that institutions for higher art education (IHAEs) are now in an association with universities, but have not been subsumed into their structures (Ysebaert & Martens, 2018). As autonomous institutions, they are not included in university PRFSs—contrasting contemporary evaluation frameworks in some other European contexts (e.g. Lewandowska & Stano, 2018; Jewesbury, 2009)—and will be evaluated with a model tailored specifically to artistic research output in the future. By expanding on the ongoing stakeholder-driven development of this model and a test case undertaken to gauge its performance, the paper shows how a culture of evaluation hinges on the generation of a culture of registration. This, in turn, offers crucial resources for disciplinary meta-reflection in the form of a centralized database of artistic research output. As the test case shows, the demand for registration and disclosure associated with quality assessment encourages the disclosure and accessibility of artistic research outcomes. The accessibility hereof is indispensable to the articulation of a new paradigm on artistic research. Simply put, the registration of artistic research outcomes for evaluation and subsequent funding allocation simultaneously furnishes a necessary overview of the output of artistic research as a field in Flanders, which benefits a deeper understanding of its disciplinary ontology.

2 Artistic research and disciplinary meta-reflection

At a time where the very notion of “artistic research” remains subject to scholarly and artistic debate in Europe (e.g. Gielen, 2013; Klein, 2010; Schwarzenbach & Hackett, 2015; Sheikh, 2009), the methods and practices associated with the assessment and evaluation of its output are disputed (Biggs & Karlsson, 2011; Hellström, 2010; Jewesbury, 2009; Lesage, 2009). Tensions between “artistic” and “scientific” perspectives on what exactly constitutes research or the production of legitimate knowledge problematize clear conceptualizations of quality parameters for the assessment of artistic research output—whether on an individual (e.g. doctoral research) or an aggregated (e.g. institutional research performance) level. In some national contexts—particularly in Nordic countries, but to an extent in Flanders too—this ongoing discussion has led to an intermediary, dualistic evaluation culture that expects both artistic and scientific excellence (Laermans, 2018). One particularly contested ramification hereof is the “double doctorate” (Biggs & Karlsson, 2011; Lesage, 2009 ), alluding to the demanding expectations cultivated in such a system and the inequity of imposing both artistic and scientific standards of performance to artistic researchers (Biggs & Karlsson, 2011). Put bluntly, the current lack of clarity on how artistic research should be evaluated and assessed means that researchers might have to put in twice the time and effort in their work— further exasperated by the presence of both artistic and academic assessors in the examination committees for artistic PhDs. In light of these challenges, authors like Wissler (1997), Biggs and Karlsson (2011) or Hellström (2010) argue for an approach that follows a “new paradigm” on artistic research. The current dualism is then mediated by conceiving of artistic research as a new, autonomous discipline. This fosters attention both for the implications of the introduction of a research component to artistic practices and for the heterogeneity of its perspectives and outcomes. In short, artistic research outcomes should not be assessed by purely artistic nor by purely scientific standards, but by standards particular to the field of artistic research.

Establishing artistic research as a new and autonomous discipline supposes not only a collaborative effort of those involved in artistic research—whether as practitioners, as assessors or as philosophers and critics engaged in theory and reflection. First and foremost, time and the space to engage in a collective effort of discipline-building in the first place is a stringent prerequisite. This simple observation notwithstanding, time and space for reflection is exactly what has been withheld from artistic research at the moment of its rapid institutionalization by the 1999 Bologna Declaration (Lesage, 2009). Suddenly, most institutions offering tertiary artistic education were merged with established universities in Europe. Among other things, this implies that the former inadvertently became subject to the latter’s evaluation mechanisms—such as the United Kingdom’s REF (Research Excellence Framework) or Po land’s CESU (Comprehensive Evaluation of Scientific Units) as “faculties of art”—and the often science-centric approach they operate by (Jewesbury, 2009; Lewandowska & Stano, 2018). Many European institutions for higher arts education were met with the expectation to develop research agendas. Their faculty members precipitously entered a competitive funding environment based on peer-reviewed research output.

Consequently, it is not far-fetched to characterize artistic research in Europe— both as a policy concept and an applied or even embodied practice—as an artificial construct (Vanhaesebrouck, 2018). It did not develop organically out of a perceived need for systematic reflection and academization at IHAEs—let alone the art world in general—but rather as a consequence of an inter-European effort towards the harmonization of higher education (Lesage, 2009; Gielen, 2013). “Artistic research” as it features in current European debates is the result of a sudden incorporation of IHAEs in universities (or their elevation to university status). This entails responsibilities with regards to “knowledge production” traditionally associated with science. Hence, reflection on artistic research has little resources to draw from but traditional, scientific conceptualizations of research and how it is to be evaluated—which is exemplified by many essays and papers exploring interlinkages and parallels between characterizations of scientific research and artistic practice (e.g. Jewesbury, 2009; Klein, 2010; O’Riley, 2011; Rust, 2007; Sheikh, 2009; Sullivan, 2006). Conversely, the notion of the “double doctorate” (Biggs & Karlsson, 2011) also testifies to the structuring role of scientific discourses on what constitutes legitimate research output, and the primacy hereof in the evaluation and assessment of the extent to which the requirements for the degree of doctor in artistic disciplines are met.

Nevertheless, neither the somewhat artificial status of artistic research nor the difficulties in conceptualizing what “research” constitutes in the arts downplay the necessity of reflecting on artistic research as a discipline—on the contrary. Given that artistic research has now become an institutionalized discipline, regardless of whether this evolution was desired by those now expected to conduct it, it merits from an exploration of the resources and conditions currently at hand to better understand and define its disciplinary orientations and demarcations. In recognition of the fact that artistic research cannot draw from an established tradition with regards to research as it is understood in the context of the university, research output evaluation cultures are indispensable modalities to explore and discuss the institutional future of artistic research in Europe (Biggs & Karlsson, 2011). However, a particularly tangible result of both the sudden institutionalization of artistic research and the discretion of a defined or formalized research tradition in artistic practice is the absence of an overarching perspective on the evaluation and assessment of artistic research output. Literature on the tensions between the “artistic” and “scientific” merits of artistic research generally departs from difficulties in the assessment of autonomous—that is, individual—forms of artistic research output— such as doctorates (e.g. Biggs & Karlsson, 2011; Lesage, 2009; Paltridge et al., 2011), the funding of individual artistic research networks or projects (e.g. Hellström, 2010) or the review process employed for the evaluation and eventual dissemination of artistic research in specialty journals (e.g. Borgdorff, 2012).

The value of these in-depth explorations of the assessment and evaluation of artistic research to disciplinary reflection is manifest, in that they locate particular tensions between certain quality expectations (e.g. Biggs & Karlsson, 2011; Lesage, 2009) or identify the logics and assumptions of particular evaluation cultures—both in terms of the evaluated and the evaluators (e.g. Hellström, 2010; Wilson, 2016). Hence, these studies discuss the challenges faced by artistic research in its current form, and testify to the necessity of establishing it as an autonomous discipline to mediate these challenges (Biggs & Karlsson, 2011). How artistic research can be articulated as an autonomous discipline, and what the prerequisites are to do so, are understandably beyond their scope. Here, the adoption of a wider, macro perspective seems a necessary avenue to explore, seeing that recognition as an autonomous discipline hinges on the aggregation of a body of works generally accepted as “artistic research outcomes”. Unlike individual assessments and evaluations—be it doctorates, research projects or publications, the potential of institutional performance-based research funding systems (PRFSs) for discipline-building is rarely explored. These systems, which organize and allocate the research funding of institutions like IHAEs are rarely discussed with regards to artistic research. When mentioned, they are generally criticized for their homogenizing and constraining influence on its envisioned outcomes (e.g. Gielen, 2013; Laermans, 2018; Jewesbury, 2009). Indeed, the expectation for individual faculty members to register research outcomes to facilitate institutional research performance evaluation seems to evoke dystopian images of neoliberal rationalization efforts (e.g. Gielen, 2013; Laermans, 2018; Vanhaesebrouck, 2018). This inhibits a candid exploration of how the registration and aggregation of artistic research outcomes could potentially produce a shared body of works recognized as representative outcomes of artistic research practices. Such views, however, cannot be dislodged from the fact that the Bologna Process has effectively subjected higher art education to university standards in many European countries—leaving little to no room for those involved to establish artistic research as a discipline on its own, autonomous terms.

3 Arts & design research in flanders

In Flanders, the view that higher art education was effectively subjected to university standards by the implementation of the Bologna Process is not entirely justifiable—notwithstanding that some do characterize the situation as such (e.g. Laermans, 2018; Vanhaesebrouck, 2018), and the academization of higher art education in general has garnered mixed responses at best. Nevertheless, the various institutions for higher art education did not dissolve into existing universities, but engaged in an association with them as Schools of Arts—their new official designation from the start of the 2009–2010 academic year (Lesage, 2009; Ysebaert & Martens, 2018). This intermediary system allows Flemish universities to keep their monopoly on granting doctoral titles, while safeguarding the autonomy of the IHAEs in matters of educational and research policy (Ysebaert & Martens, 2018). In practice, this entails that candidates pursuing a PhD in the arts or design enroll in a doctoral program at one of the Flemish IHAEs—being supervised by the faculty of those institutions—but acquire their doctoral title from the university their institution is associated with—meaning that the examination committee includes both IHAE and university faculty members.

Hence, as is the case in Nordic countries or Australia (Biggs & Karlsson, 2011; Wilson, 2016), the individual assessment of whether the research output generated in the context of a doctoral program fulfills the requirements for the degree is conducted by both artistic and academic experts in Flanders. As was mentioned before, this had led to inequities particular to such a dualistic evaluation culture, with the “double doctorate” as a central issue (Lesage, 2009). Indeed, it could be argued that the intermediary status of Flemish IHAEs is a defining factor to this problem, seeing that academic assessors are generally absent from the doctoral program in the arts, but do reside in the doctoral examination committee as a sine qua non to grant the title of doctor in the arts. While the framework for doctoral programs tends to differ from association to association (i.e. between particular IHAEs and universities), this situation can represent added difficulties to the notion of the “double doctorate” particular to Flanders. In some cases, it deprives doctoral candidates from direct access to the expectations and considerations fostered by academic members in their examination committees. If the role of academic members of the examination committee is limited or possibly non-existent throughout the doctoral program, arts & design researchers might consider their representation in examination committees disproportionate.

There is another side of the coin to this intermediary solution for the academization of higher art education, however. Among other implications less important to the scope of this paper, this model precludes the output of Flemish IHAEs from the PRFSs used for the allocation of institutional research funding used for Flemish universities—which tend to emphasize quantitative, scientometric evaluation methods and conceptualize research output exclusively as publications (Engels et al., 2018). The research funding model for the Schools of Arts is independent from that of universities, meaning that the former are not expected to conform to the standards of the latter. This differs crucially from, for instance, the REF-system used in the United Kingdom, where research funding for faculties of arts is allocated by using the same general quality indicators used for the university in general— notwithstanding that the evaluation itself is based on peer reviews (Nelson, 2013) and research outcomes are not necessarily expected to take the form of a traditional publication. In Flanders, the partial autonomy of the IHAEs supposes an autonomous model to assess their research outcomes, meaning that they are not held to the standards used to measure universities’ research performance.

At the time, Flemish policy makers recognized the specificity of artistic & design research and the undesirability of imposing an ill-suited assessment format, organizing the research funding of the Schools of Arts—that is, funding distinct from competitive grants awarded to individual researchers generating overhead for the institutions they are embedded in—in an ex-ante model for the time being. As such, the institutional component of research funding currently foregoes on the evaluation of the Schools of Arts’ research output, and instead uses student enrollments and delivered degrees (Higher Education Context, III)—in many ways similar to the current Norwegian approach (Sivertsen, 2018). Naturally, this approach has its detriments too, and questions regarding the fairness of research funding allocation based on student enrollments and delivered degrees are exceptionally pertinent in this regard. Currently, larger institutions—whose research output is not necessarily commensurate to their scale—clearly enjoy an unfair advantage to their smaller counterparts—whose research output might exceed that of institutions with greater student enrollments and delivered degrees. This imbalance was and is recognized by policy makers, however, and the ex-ante model currently in place has always been intended as a provisional system meant to allocate research funding to institutions previously not engaged in research. Hence, the evaluation procedure informing the allocation of institutional research funding might possibly change to an ex-post PRFS in 2023 based on the assessment of non-traditional research output (Higher Education Codex, III). Importantly, the Flemish administration does not specify the criteria or indicators to be used in this future PRFS, and does not establish what is and what is not to be understood as “artistic & design research output”. Instead of a top-down model, where the administration imposes standards of quality, possibly tailored to or harmonized with existing models of artistic research output assessment, it was elected to let a pluralist and differentiated evaluation design emerge from collaboration among stakeholders involved in or with artistic & design research. This does not only offer a degree of freedom to the incumbent field of artistic & design research in Flanders to define the terms on which the quality of its output is to be evaluated and assessed, but the time to do so in a considerate manner too.

4 The development of quality indicators for artistic & design research in flanders

naturally, the strategic decision to let an evaluation culture for artistic & design research output emerge bottom-up from stakeholders in Flanders does not entail a complete laisser-faire policy on behalf of the administration. The stakeholder-driven approach was facilitated and organized by the Vrije Universiteit Brussel (VUB— Free University of Brussels) branch of the Flemish Center for Research and Development Monitoring (ECOOM)—which is predominantly devoted to the development of a central registration format and evaluation design for art and design research outcomes (Ysebaert & Martens, 2018). The role of ECOOM-VUB and its researchers is therefore not that of decision makers, but rather that of facilitators and mediators. Therefore, the effort took the form of what Barré (2010) calls an “indicator platform”, whereby involved parties debate the differing criteria that offer an indication of the quality of (artistic & design) research output without necessarily drawing from the various approaches used for assessment in other fields. Expected to consult the opinions of stakeholders on these indications, ECOOM-VUB explores how these views can be translated into indicators to subsequently operationalize them in the design of an assessment structure.

Initially, this approach crystalized in the formation of a “Working Group on Content” (IW—Inhoudelijke Werkgroep)—comprising of representatives from the Flemish Schools of Arts—which met several times to discuss the most suitable method for the assessment of arts & design research output. Conceiving of “arts & design research output” as an outcome—such as an artifact, a performance or other artistic products—resulting from a demonstrable research process with subsequent public dissemination, the IW formulated a distinction between the results of arts & design practice and arts & design research, of which only the latter is eligible for evaluation. Hereby, consensus befell on a dualistic model based on the one hand on the registration of output by researchers affiliated with institutions to provide the material for evaluation, and peer review by expert panels as the assessment method itself. While several Schools of Arts already registered research output in their institutional databases (Ysebaert & Martens, 2018), the IW decided on the creation of a central registration format—in order to harmonize the peer review process. This central registration format was then added as an autonomous segment to the FRIS (Flemish Research Information Space) database—meaning that this segment is based on entry fields that differ considerably from those used with traditional academic output (i.e. publications), semantically recognizing the pluralism and diversity of arts & design research outcomes. As such, the registration format aims to avoid a normative or scientometric registration design, balancing the necessary conceptual freedom for researchers with a certain degree of uniformity in the descriptive information collected.

This initially resulted in a design wherein three separate “research dimensions” are emphasized as indicative of a qualitative and meaningful arts & design research outcome: “quality”, “impact” and “contextuality”. The IW conceptualized research quality around notions of originality, rigor, transparency, process reflexivity and representation and/or dissemination of outcomes. Its characterization of research impact distinguishes between artistic impact, knowledge impact and societal impact. Research context, finally, mediates the wish of the IW members to allow quality assessment to be attentive for the consistency of the research outcomes with the individual research policies of the Flemish IHAEs—pointing to the ambition of the institutions themselves to articulate a research identity of their own—and for the possible ways in which the research outcome demonstrates meaningful interaction with relevant practitioners in the artistic field. Importantly, the dimensions decided upon by the IW articulate a conviction that arts & design research must be distinct from arts & design practice in general, but do not stipulate that the outcomes of arts & design research are necessarily distinct from those of artistic practice—meaning primarily that the tangible results of arts & design research might not differ substantially from the products of arts & design practice. While the artistic product—whether this product consists of one or more material artefacts or a singular, ephemeral performance—is the central node in the evaluation procedure, the indicators used to register and document the differing research quality dimensions served to elucidate and evocate the process underlying the product (see table 1: “Research context” & “Research impact”). In the view of the IW, the processes and procedures that inform, shape and guide the final artistic outcome are the primary criteria to differentiate between arts & design practice and arts & design research, and should therefore be central to the quality assessment of the output. The table below demonstrates how the considerations and decisions made by the IW were translated into the architecture of the FRIS database segment for artistic and design research outcomes:

Table 1

Initial design of the FRIS database segment for arts & design research outcomes.

ENTRY FIELDDESCRIPTION
TitleOpen entry field - MandatoryGive the title by which the research outcome is described

Researcher(s)/designer(s)/artist(s) involved in the generation of the research outcomeOpen entry field- MandatoryName all individuals involved in the production of the research outcome

Research output categoryLayered multiple-choice entry field - MandatorySelect the output category the artistic or design research outcome pertains to and describe its qualities (e.g. materials used; instruments included; number of performers):
- Design: research outcomes that have been disseminated in the form of a diagram/design/blueprint and precede (possible) materialization
- Artefact: research outcomes that have been disseminated in a material form
- Performance: research outcomes that have been disseminated live for an audience

Researcher roleLayered multiple-choice entry field - MandatorySelect the role taken up by the researcher(s) in the generation of the research outcome:
- Performer: the researcher has disseminated the knowledge generated in or by the outcome to a live audience
- Creator: the researcher has disseminated the knowledge generated in or by the outcome in a conceptual (i.e. a design) or material (i.e. an artefact) form
- Contributor: the researcher has made a demonstrable contribution to the creation or dissemination of knowledge generated by artistic research

DateOpen entry field - MandatoryEnter the entry and (if applicable) closing date wherein the outcome was publicly disseminated

Evaluation panelSingle-layer multiple-choice entry field - MandatorySelect the expert panel for which the research outcome is entered for peer evaluation:
- Stage Arts
- Music
- Architecture & Design
- Audiovisual Arts
- Visual Arts
- Interdisciplinary Outcomes (exclusively applicable if and when the outcome contributes innovatively to more than one artistic discipline)

Research ContextOpen entry field - OptionalDescribe:
1. The specific context and environment wherein the outcome was generated (e.g. research project; research mandate)
2. The research and/or artistic/design context wherein the outcome was generated with specific attention for how the outcome produces innovative insights/knowledge and therefore exceeds the status of an artistic or design product (e.g. research question; state of the art)

Research ImpactOpen entry field - OptionalDescribe how the outcome impacts:
- Artistic or design praxis (i.e. artistic impact) and/or
- Research praxis (i.e. knowledge impact) and/or
- Social dimensions (i.e. societal impact)

Impact referencesOpen entry field - OptionalInsert link(s) to material by which the expert panel may asses the different forms of impact generated by the research outcome

As table 1 illustrates, the finality of the FRIS database segment for arts & design research outcomes cannot reside in quantification or metric approaches—seeing that the registration format does not facilitate the use of such methods. Rather, it discloses arts & design research output generated at Flemish IHAEs, in order to facilitate an evaluation culture based on qualitative peer assessment methods.

Up to this point, this remained an expressively abstract exercise—which nevertheless signaled a first collaborative effort towards the construction of a new critical paradigm for artistic research in Flanders (Ysebaert & Martens, 2018)— in need of practical testing. To this end, a test case was organized in 2016 for output entries eligible for evaluation for the Architecture & Design expert panel. Importantly, while architecture and design programs became embedded in universities in 2013, their artistic research output is not included in the PRFS system used for the allocation of research funding for Flemish universities (Ysebaert & Martens, 2018). Hence, their artistic research funding is currently organized in the ex-ante model discussed above, and will be allocated by the artistic research PRFS in the future. Thus, although the test case was carried out with the cooperation of four faculties of architecture and design—those of UAntwerpen (University of Antwerp), KULeuven (Leuven University), UHasselt (Hasselt University) and VUB (Free University of Brussels)—which also enjoy traditional research funding, all participating institutions are projected to be included in the PRFS based on artistic research outcomes in the future.

In practice, the test case entailed both a registration phase—implying that partaking institutions entered their output data into the FRIS-segment for artistic research output—and a peer evaluation phase. For this latter phase, international experts were chosen from a list of eligible candidates provided by the participating institutions with demonstrable expertise in architecture and design research. The evaluation by the expert panels was intended to have four points of focus: a concise research statement by participating institutions (1), the totality of entries in the FRIS-segment for artistic research output marked as Architecture & Design (3), a selection of three entries per institution decided upon by the expert panel for further discussion by participating institutions (3) and a selection of three entries chosen by the institutions themselves which they felt were representative for their take on artistic research (4). As such, the third and fourth evaluation exercises imply a direct interaction between the expert panels and participating institutions—rather than a separate or even anonymous peer review process. Admittedly, the results of this test case are all derived from the registration and evaluation of artistic research output submitted to the Architecture and Design expert panel. Despite this particular focus, they identify crucial elements pertinent to the generation of an autonomous paradigm on artistic research.

4.1 The test case: the difficulties of evaluation

Notwithstanding that all parties involved—that is, both the members of the expert panel and the participating institutions—recognized the potential of registering research output into the FRIS database segment for the evaluation of artistic and design research outcomes, particularly tangible problems were identified during the test case. Overall, the exercise pointed to significant discrepancies in the quality of entries. Whereas the vast majority of researchers was diligent in describing the outcome itself, such as the materials used or its dimensions or scale (see table 1: “Research output category”), fields meant to indicate the quality of the differing research dimensions were often limited, incomplete or unaddressed altogether. Particularly, the entries concerning “research context” and “research impact” (see table 1) proved problematic categories, generating limited and fragmented results at best. Seeing that these categories pertain most expressively to how artistic researchers position themselves vis-à-vis the broader research context—by the formulation of a critical state of the art or a reflexive exploration of the various innovative contributions of the outcome—the test case points to the limited disciplinary self-awareness of artistic research in Flanders. On a similar note, a number of researchers supposedly working in a field broadly defined as “Architecture and Design” characterized their work as pertaining to “Visual Arts” or “Interdisciplinary Outcomes” (see table 1: “Evaluation Panel”), further suggesting a lack of disciplinary clarity in relation to the critical context wherein artistic research currently takes place, and the shapes its outcomes take. Naturally, this observation is not meant to discredit the work of those researchers in Architecture & Design programs who feel that their practices and outcomes relate more to other artistic fields. Rather, it again underscores the current absence of clarity regarding what artistic research is in general, how it is constituted by different subfields, and which expectations they articulate.

Hence, though it might be tempting to brush of the failure of indicators such as “Research Context” or “Research Impact”—emphasized by the IW as indicative for the quality of artistic research output—as a mere disinclination of researchers to collaborate in the test case, it should also be noted that these indicators suppose a degree of disciplinary consciousness. Outside of a defined research tradition, for instance, expecting an articulate research question or a state of the art (see table 1: “Research Context”) is rather demanding. Similarly, whereas it might seem appropriate to gauge the differing ways in which the research outcome and its underlying process impact artistic praxis, research praxis or the social (see table 1: “Research Impact”), it does so at a time characterized by a lack of epistemological traditions, an opacity of the relations between artistic production and artistic research and an absence of established discourses on the interlinkages between art, research and the social sphere. Considering that this entails that researchers cannot rely on an established vocabulary to articulate how their output relates to such issues, it is unsurprising that these indicators in particular failed to deliver the desired results—and proved unsuitable to proceed with the peer evaluation phase of the test case. Because the IW prioritizes the entry fields of “Research Context” and “Research Impact” as indicative of the reflexivity and rigor of the research process, and in turn defines the excellence of artistic research by these parameters, the expert panel was effectively hindered by the absence of material to engage in a quality assessment in the envisioned format.

4.2 The test case: the potential of registration

That the peer quality assessment phase of the test case could not take place due to a lack of pertinent material does not entail that the test case was a fruitless endeavor altogether. The registration of artistic research outcomes was broadly recognized by the participating institutions and the expert panel as a valuable modality to get a better grasp on the field of artistic research in Flanders. That literature on the evaluation of artistic research tends to focus on the assessment of individual research outcomes (e.g. Biggs & Karlsson, 2011; Borgdorff, 2012; HellStröm, 2010; Lesage, 2009; Paltridge et al., 2011; Wilson, 2016) testifies to the fact that there is currently little overview on the many forms such endeavors take. Quantitative insights into various parameters on artistic research output—such as artistic field, materials used or forms of dissemination—are scarce, for instance, notwithstanding the explorative value such knowledge would offer to reflection on artistic research as an autonomous discipline.

At the most basic level, the registration of arts & design research outcomes generated at Flemish IHAEs in the designated FRIS database segment has the potential to provide extensive descriptive information on the formal qualities of artistic research output in Flanders (see table 1: “Research output category”). This allows the field to be explored, mapped and understood, which negotiates the current discretion of artistic research output in the general artistic output generated by Flemish IHAEs—which generally disseminate not only their research outcomes, but also the artistic work of students and alumni via their communication channels. The registration of artistic & design research outcomes in one designated central database therefore facilitates a specific perspective on the state of the art in particular sub-disciplines on artistic research in Flanders. This benefits artistic & design researchers, who—by the indicators decided upon by the IW—are expected to operate reflexively, with particular attention for the interactivity of their artistic process with larger peer communities. Accordingly, both the members of the Architecture and Design expert panel and the participating institutions perceived the generation of a registration culture as a tangible and productive result of the test case. The registration and categorization of artistic & design research output may intuitively conjure an imaginary of quantification and measurement (Gielen, 2013; Van Haesebrouck, 2018), but proves a valuable tool to disclose and centralize the multifarious results of arts & design research in Flanders—which in turn facilitates discussion and interaction on its subjects, methods and outcomes. Advertent of the dual role the registration format might play, a new design of the FRIS database segment for artistic & design research outcomes was drafted:

This updated design of the FRIS database segment for arts & design research outcomes explicitly recognizes the multifarious potential of the registration and evaluation format. By grouping the desired information into two overarching categories (see table 2: “1. Outcome Description” & “2: Research Context”), the new registration format combines the functionality of the database segment as an instrument for quality assessment and as a dynamic repository of the multifarious forms and shapes artistic & design research output takes in Flanders. In doing so, it shifts from being a modality that benefits only the considerate allocation of institutional research funding to a space that allows individual researchers and institutions alike to survey the type of outcomes generated in the subfields of arts & design research, to consult the types of questions posed and the methods used to address them and to explicitly position individual outcomes or larger research projects in relation to others. Because the new format explicitly positions itself as one that centralizes artistic & design research outcomes in Flanders—complemented with additional information—its value transcends the field of arts & design research too. To researchers who work on rather than in artistic & design research—such as cultural sociologists, art historians or scholars of education—the FRIS database segment for arts & design research outcomes presents an opportunity to approach the subject from a wider perspective. It allows for the identification of trends and dynamics on an aggregated level, which in turn facilitates a form of disciplinary meta-reflection that departs from the outcomes themselves.

Table 2

Improved design of the FRIS database segment for arts & design research outcomes.

ENTRY FIELDDESCRIPTION
1. OUTCOME DESCRIPTIONOpen entry field - MandatoryDifferentiating between the description of the outcome and the description of its impact will improve the usefulness of the database segment as an explorative tool for artistic research in Flanders

Title + SubtitleOpen entry field - MandatoryGive the title by which the research outcome is described The addition of a subtitle will contribute to a better understanding of the type of outcome registered.

Researcher(s)/designer(s)/artist(s) involved in the generation of the research outcomeOpen entry field - MandatoryName all individuals involved in the production of the research outcome

Research output categoryLayered multiple-choice entry field - MandatorySelect the output category the artistic or design research outcome pertains to and describe its qualities (e.g. materials used; instruments included; number of performers):
- Design: research outcomes that have been disseminated intake the form of a diagram/design/blueprint and precede (possible) materialization
- Artefact: research outcomes that have been disseminated intake a material form
- Performance: research outcomes that have been disseminatedtake the form of a lecture, a performance, a showcase etc. for a live audience
By downplaying the previous focus on dissemination, researchers have more autonomy in describing exactly how the knowledge generated by their research outcomes circulate.

Researcher roleLayered multiple-choice entry field - MandatorySelect the role taken up by the researcher(s) in the generation of the research outcome:
- Performer: the researcher has disseminated the knowledge generated in or by the outcome to a live audience
- Creator: the researcher has disseminated the knowledge generated in or by the outcome in a conceptual (i.e. a design) or material (i.e. an artefact) form
- Contributor: the researcher has made a demonstrable contribution to the creation or dissemination of knowledge generated by artistic research

DateOpen entry field - MandatoryEnter the entry and (if applicable) closing date wherein the outcome was publicly disseminated

Abstract/SummaryOpen entry field - MandatoryIncluding a mandatory written description of the research outcome will both assist the researcher(s) in reflecting on their work and offer peer reviewers added insight into the outcome.

Peer review(s)Open entry field - MandatoryIncluding references to the differing ways wherein the outcomes has been peer reviewed (i.e. prizes; juries; double-blind peer review) allows to better understand the stakeholder’s conceptualization of quality and excellence.

Evaluation panelSingle-layer multiple-choice entry field - MandatorySelect the expert panel for which the research outcome is entered for peer evaluation:
- Stage Arts
- Music
- Architecture & Design
- Audiovisual Arts
- Visual Arts
- Interdisciplinary Outcomes (exclusively applicable if and when the outcome contributes innovatively to more than one artistic discipline)
To facilitate future peer evaluation, the selection of an evaluation panel should not be an entry in the registration process, but precede it. This way, peer assessors are not expected to differentiate between materials as they survey registered outcomes.

Sources/DocumentationOpen entry field - OptionalThe FRIS database-segment for artistic research should act as an archive for artistic research output in Flanders. Therefore, the documentation of the outcomes (visual or otherwise) should be embedded in its architecture and subsequently registered.

KeywordsOpen entry field - MandatoryTo maximize the potential of the FRIS database segment for artistic research output, a keywords-based system will allow for the aggregation of similar research outcomes (whether based on topic, method etc.)

2. RESEARCH CONTEXTBy explicitly distinguishing between the description of the outcome on the one hand, and the description of its context on the other, the importance of both is accentuated to the researchers.

Research ContextOpen entry field - MandatoryDescribe:
3. The specific context and environment wherein the outcome was generated (e.g. research project; research mandate)
4. The research and/or artistic/design context wherein the outcome was generated with specific attention for how the outcome produces innovative insights/knowledge and therefore exceeds the status of an artistic or design product (e.g. research question; state of the art)
Describe:
a. Research background: what field does the outcome pertain to, which question(s) does it pose and how do these relate to the wider context?
b. Research contribution: what contribution does the outcome make to knowledge?
c. Research significance:how was the outcome received in the wider field it features in? (include evidence of the resonance, reception or impact of the research outcome)

Research ImpactOpen entry field - OptionalDescribe how the outcome impacts:
- Artistic or design praxis (i.e. artistic impact) and/or
- Research praxis (i.e. knowledge impact) and/or
- Social dimensions (i.e. societal impact)

Impact referencesOpen entry field - OptionalInsert link(s) to material by which the expert panel may asses the different forms of impact generated by the research outcome

Related research output(s)Open entry field - OptionalIf applicable, what are the connections between the research outcome and other artistic or research outcomes? (e.g. an exhibition; urban redevelopment project)

Related projectOpen entry field - OptionalIf applicable, to which research project does the registered research outcome pertain?

5 Conclusion

That contemporary discussions on the quality assessment of artistic research output tend to focus on individual examples of such outcomes (e.g. Biggs & Karlsson, 2011; Hellström, 2010; Lesage, 2009; Wilson, 2016) is understandable, but this should not be allowed to obscure the potential of reflecting on large-scale evaluation models for the articulation of a new paradigm on artistic research. While exploring quality assessment procedures used for the evaluation of doctorates, research projects or the dissemination of research outcomes identifies particular challenges—such as the discrepancies between artistic and scientific expectations of excellence or different conceptualizations regarding the generation of knowledge, approaching evaluation cultures from a large-scale perspective has particular merits. The Flemish test case discussed in this paper specifically demonstrates how the registration of artistic research outcomes is welcomed by stakeholders as a tool to map the field of artistic research. Although it would be unrealistic to expect that the registration of artistic research output has immediate effects for the establishment of a new paradigm on artistic research as an autonomous field, the collection and categorization of outcomes in one centralized database does retain the potential to serve as a useful tool for contextualization and reflection in the long run. Seeing that artistic research currently has little options but to draw from scientific discourses on what constitutes research and the production of knowledge, its disciplinary future is predicated on the generation of an autonomous body of outcomes that may serve as a theoretical and methodological context for future research. Herein, additional engagements with the quality assessment and evaluation of artistic research on a level that exceeds that of individual output are crucial interventions for a deeper and more nuanced understanding of artistic research as an autonomous discipline.

Author ContributionsFlorian Vanlee (Florian.Hendrik.J.Vanlee@vub.be) conducted the literature review, analyzed the available materials generated by the stakeholder-driven research project and wrote the paper. Walter Ysebaert (Walter.Ysebaert@vub.be) analyzed the available materials generated by the stakeholder-driven research project and reviewed the paper.

References

  • Barre R. (2010). Towards socially robust S&T indicators: Indicators as debatable devices enabling collective learning. Research Evaluation 19(3) 227–231.

    • Crossref
    • Export Citation
  • Biggs M. & Karlsson H. (2011). Evaluating quality in artistic research. In Biggs M. & Karlsson H. (Eds.) The Routledge companion to research in the arts (pp. 405–424). London UK: Routledge.

  • Borgdorff H. A. H. (2012). The conflict of the faculties: Perspectives on artistic research and academia. Leiden NE: Leiden University Press.

  • Engels T. C. & Guns R. (2018). The Flemish performance-based research funding system: A unique variant of the Norwegian model. Journal of Data and Information Science 3(4) 44–59.

  • Flemish Community. (2014). Besluit van de Vlaamse regering betreffende de codificatie van de decretale bepalingen betreffende het Hoger Onderwijs (Higher Education Codex). Retrieved from http://data-onderwijs.vlaanderen.be/edulex/inhoudstafel.aspx?docid=14650

  • Gielen P. (2013). Artistic praxis and the neoliberalization of the educational space. Journal of Aesthetic Education 47(1) 58–71.

    • Crossref
    • Export Citation
  • Hellström T. (2010). Evaluation of artistic research. Research Evaluation 19(5) 306–316.

    • Crossref
    • Export Citation
  • Jewesbury D. (2009). Some problems with ‘research’ in UK Fine Art institutions. Art & Research 2(2) 1–3.

  • Klein J. (2010). What is artistic research?. Journal for Artistic Research. Retrieved from https://www.jar-online.net/what-artistic-research

  • Laermans R. (2018). Onderzoekskunst? Retrieved from https://www.dewitteraaf.be/artikel/detail/nl/4579

  • Lesage D. (2009). Who’s afraid of artistic research? On measuring artistic research output. Art & Research 2(2) 1–10.

  • Lewandowska K. & Stano P. M. (2018). Evaluation of research in the arts: Evidence from Poland. Research Evaluation 27(4) 323–334.

    • Crossref
    • Export Citation
  • Nelson R. (2013). Practice as research in the arts: Principles protocols pedagogies resistances. London UK: Palgrave MacMillan.

  • O’Riley T. (2011). A discrete continuity: On the relation between research and art practice. Journal of Research Practice 7(1) P1.

  • Paltridge B. Starfield S. Ravelli L. & Nicholson S. (2011). Doctoral writing in the visual and performing arts: Issues and debates. International Journal of Art & Design Education 30(2) 242–255.

    • Crossref
    • Export Citation
  • Rust C. (2007). Unstated contributions: How artistic inquiry can inform inter-disciplinary research. International Journal of Design 1(3) 69–76.

  • Schwarzenbach J. & Hackett P. (2015). Transatlantic reflections on the practice-based PhD in fine art. London UK: Routledge.

  • Sheikh S. (2009). Objects of study or commodification of knowledge? Remarks on artistic research. Art & Research 2(2) 1–8.

  • Sivertsen G. (2018). An attempt to create an indicator for performance-based funding for artistic research in Norway. In Ysebaert W. & Van Kerckhoven B. (Eds.) Evaluating art and design research: Reflections evaluation practices and research presentations (pp. 93–109). Brussels BE: VUB Press.

  • Sullivan G. (2006). Research acts in art practice. Studies in Art Education 48(1) 19–35.

    • Crossref
    • Export Citation
  • Vanhaesebrouck K. (2018). Measuring artistic research or the real agenda behind Bologna in higher arts education. In Ysebaert W. & Van Kerckhoven B. (Eds.) Evaluating art and design research: Reflections evaluation practices and research presentations (pp. 16–22). Brussels BE: VUB Press.

  • Wilson J. (2016). The white cube in the black box: Assessing artistic research quality in multidisciplinary academic panels. Assessment & Evaluation in Higher Education 41(8) 1223–1236.

    • Crossref
    • Export Citation
  • Wissler R. (1997). Artistic practice and research: Towards a community of judgment. In Double Dialogues: The Arts and their Theoretical Paradigms Conference Proceedings (pp. 88–105). Melbourne AU: Theatreworks.

  • Ysebaert W. & Martens B. (2018). The ECOOM-VUB stakeholder-driven evaluation design for art and design research outcomes. In Ysebaert W. & Van Kerckhoven B. (Eds.) Evaluating art and design research: Reflections evaluation practices and research presentations (pp. 93–109). Brussels BE: VUB Press.

Footnotes

While the declaration was signed in 1999, the Bologna Process—stipulating among other things the harmonized three-tiered structure for higher education (i.e. Bachelor—Master—PhD)—followed a particular trajectory in the individual countries that ratified the declaration. The Netherlands, for instance, introduced its terms to national legislature in the 2002-2003 academic year, whereas this happened in Flanders in the 2004-2005 academic year.
For reasons of clarity, the abbreviation used to refer to the Schools of Arts in the remainder of this paper is the same used for international examples: IHAEs [Institutions for Higher Art Education].
Because the system used in Flanders pertains to both the arts (such as visual arts or drama) and design (such as architecture or product design), the paper refers to “arts & design research” in the Flemish context.
This system furthermore disadvantages IHAEs constricted by a numerus fixus system. Seeing that student enrollments and delivered degrees are stable at such institutions, they currently cannot enjoy an increase in research funding induced by rising student numbers even though their research output might be growing.
The consulted stakeholders consist of Flemish IHAE faculty members with coordinating roles—such as research coordinators or directors.
Each Flemish IHAE has a representative in the IW, ensuring that the indicators decided upon reflect the diversity and pluralism of higher art education in Flanders—which covers all artistic disciplines.
This schematic representation of the initial FRIS database segment for arts & design research outcomes only displays the top level of the database architecture, meaning that some entries allow researchers to further differentiate the outcome they are registering. Selecting “Researcher output category” or “Researcher role”, for instance, prompts an added entry field designed to further specify the qualities of the outcome or the role of the researcher. These secondary entry fields include an “If other, specify” option too, moreover, ensuring the autonomy of the researcher in defining the outcome or their own role.
The other expert panels to which artistic research output may be submitted are: Music, Stage Arts, Audiovisual Arts, Visual Arts and an Interdisciplinary panel. As is the case with the output indicators themselves, these panels—and the disciplinary contexts they represent—were decided upon by the stakeholders in the IW.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Barre R. (2010). Towards socially robust S&T indicators: Indicators as debatable devices enabling collective learning. Research Evaluation 19(3) 227–231.

    • Crossref
    • Export Citation
  • Biggs M. & Karlsson H. (2011). Evaluating quality in artistic research. In Biggs M. & Karlsson H. (Eds.) The Routledge companion to research in the arts (pp. 405–424). London UK: Routledge.

  • Borgdorff H. A. H. (2012). The conflict of the faculties: Perspectives on artistic research and academia. Leiden NE: Leiden University Press.

  • Engels T. C. & Guns R. (2018). The Flemish performance-based research funding system: A unique variant of the Norwegian model. Journal of Data and Information Science 3(4) 44–59.

  • Flemish Community. (2014). Besluit van de Vlaamse regering betreffende de codificatie van de decretale bepalingen betreffende het Hoger Onderwijs (Higher Education Codex). Retrieved from http://data-onderwijs.vlaanderen.be/edulex/inhoudstafel.aspx?docid=14650

  • Gielen P. (2013). Artistic praxis and the neoliberalization of the educational space. Journal of Aesthetic Education 47(1) 58–71.

    • Crossref
    • Export Citation
  • Hellström T. (2010). Evaluation of artistic research. Research Evaluation 19(5) 306–316.

    • Crossref
    • Export Citation
  • Jewesbury D. (2009). Some problems with ‘research’ in UK Fine Art institutions. Art & Research 2(2) 1–3.

  • Klein J. (2010). What is artistic research?. Journal for Artistic Research. Retrieved from https://www.jar-online.net/what-artistic-research

  • Laermans R. (2018). Onderzoekskunst? Retrieved from https://www.dewitteraaf.be/artikel/detail/nl/4579

  • Lesage D. (2009). Who’s afraid of artistic research? On measuring artistic research output. Art & Research 2(2) 1–10.

  • Lewandowska K. & Stano P. M. (2018). Evaluation of research in the arts: Evidence from Poland. Research Evaluation 27(4) 323–334.

    • Crossref
    • Export Citation
  • Nelson R. (2013). Practice as research in the arts: Principles protocols pedagogies resistances. London UK: Palgrave MacMillan.

  • O’Riley T. (2011). A discrete continuity: On the relation between research and art practice. Journal of Research Practice 7(1) P1.

  • Paltridge B. Starfield S. Ravelli L. & Nicholson S. (2011). Doctoral writing in the visual and performing arts: Issues and debates. International Journal of Art & Design Education 30(2) 242–255.

    • Crossref
    • Export Citation
  • Rust C. (2007). Unstated contributions: How artistic inquiry can inform inter-disciplinary research. International Journal of Design 1(3) 69–76.

  • Schwarzenbach J. & Hackett P. (2015). Transatlantic reflections on the practice-based PhD in fine art. London UK: Routledge.

  • Sheikh S. (2009). Objects of study or commodification of knowledge? Remarks on artistic research. Art & Research 2(2) 1–8.

  • Sivertsen G. (2018). An attempt to create an indicator for performance-based funding for artistic research in Norway. In Ysebaert W. & Van Kerckhoven B. (Eds.) Evaluating art and design research: Reflections evaluation practices and research presentations (pp. 93–109). Brussels BE: VUB Press.

  • Sullivan G. (2006). Research acts in art practice. Studies in Art Education 48(1) 19–35.

    • Crossref
    • Export Citation
  • Vanhaesebrouck K. (2018). Measuring artistic research or the real agenda behind Bologna in higher arts education. In Ysebaert W. & Van Kerckhoven B. (Eds.) Evaluating art and design research: Reflections evaluation practices and research presentations (pp. 16–22). Brussels BE: VUB Press.

  • Wilson J. (2016). The white cube in the black box: Assessing artistic research quality in multidisciplinary academic panels. Assessment & Evaluation in Higher Education 41(8) 1223–1236.

    • Crossref
    • Export Citation
  • Wissler R. (1997). Artistic practice and research: Towards a community of judgment. In Double Dialogues: The Arts and their Theoretical Paradigms Conference Proceedings (pp. 88–105). Melbourne AU: Theatreworks.

  • Ysebaert W. & Martens B. (2018). The ECOOM-VUB stakeholder-driven evaluation design for art and design research outcomes. In Ysebaert W. & Van Kerckhoven B. (Eds.) Evaluating art and design research: Reflections evaluation practices and research presentations (pp. 93–109). Brussels BE: VUB Press.

Search
Journal information
Impact Factor


CiteScore 2018: 1

Source Normalized Impact per Paper (SNIP) 2018: 0.413

Cited By
Metrics
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 53 53 15
PDF Downloads 17 17 3