Open Access

A Framework for the Assessment of Research and Its Impacts

Cinzia Daraio
Cinzia Daraio
   | Dec 29, 2017
Journal of Data and Information Science's Cover Image
About this article
Previous Article
Next Article
Cite
Article Category: Perspective
Published Online: Dec 29, 2017
Page range: 7 - 42
Received: Jul 30, 2017
Accepted: Sep 06, 2017
DOI: https://doi.org/10.1515/jdis-2017-0018
Keywords
© 2017 Walter de Gruyter GmbH, Berlin/Boston
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Figure 1

An illustration of our framework including its three implementation factors (tailorability, transparency, and openness) and its three enabling conditions: convergence, mixed methods, and knowledge infrastructures.
An illustration of our framework including its three implementation factors (tailorability, transparency, and openness) and its three enabling conditions: convergence, mixed methods, and knowledge infrastructures.

Figure 2

An illustration of the relationship between modeling world, empirical world, and policy world: they are all somewhat overlapping visions or projections of the real worlds.
An illustration of the relationship between modeling world, empirical world, and policy world: they are all somewhat overlapping visions or projections of the real worlds.

Dimensions of methodology: subject and means in our framework.

DimensionType/categoryContent
Subject(of the assessment)Output (baseline)Result of a transformation process which uses inputs to produce products or services
Productivity and efficiencyPartial or total factor productivity with respect to a reference
EffectivenessConsidering inputs and outputs, and accounting for the aims of the activities
ImpactAll contributions of research outside academia
Means(of the assessment)Quantitative approaches
Qualitative approaches
Quali-quantitative approaches

Definitions of education, research, and innovation.

TermDefinition
EducationIn general, education is the process of facilitating the acquisition or assignment of special knowledge or skills, values, beliefs, and habits. The methods applied are varied and may include storytelling, discussion, teaching, training, and direct research. It is often done under the guidance of teachers, but students can also learn by themselves. It can take place in formal or informal settings and can embrace every experience that has a formative effect. Education is commonly organized into stages: preschool, primary school, secondary school, and after that higher education level. See the International Standard Classification of Education (ISCED, 2011) for a more technical presentation.
ResearchAccording to the OECD’s Frascati Manual (2002), research and development (R&D) is the “creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of man, culture, and society, and the use of this stock of knowledge to devise new applications.” The term R&D covers three activities: “basic research, applied research and experimental development. Basic research is experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundation of phenomena and observable facts, without any particular application or use in view. Applied research is also original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily toward a specific practical aim or objective. Experimental development is systematic work, drawing on existing knowledge gained from research and/or practical experience, which is directed to producing new materials, products, or devices, to installing new processes, systems, and services, or to improving substantially those already produced or installed. R&D covers both formal R&D in R&D units and informal or occasional R&D in other units.” See also the more recent Frascati Manual (OECD, 2015b).
InnovationAccording to the OECD (2005), an innovation is “the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new organizational method in business practices, workplace organization or external relations. The minimum requirement for an innovation is that the product, process, marketing method or organizational method must be new (or significantly improved) to the firm. Innovation activities are all scientific, technological, organizational, financial and commercial steps which actually, or are intended to, lead to the implementation of innovations. Innovation activities also include R&D that is not directly related to the development of a specific innovation.”

A characterization of the data dimension in our framework.

DimensionCharacterization
Availabilityusability
sampling vs census
freely, controlled or undisclosed
consumption vs participation
open, institutional provided
commercial vs publicly available
privacy/confidentiality (see Ekbia et al. (2015))
Interoperabilitya very high level is obtained by an OBDM approach (see Daraio et al. (2016b))
Unit-free propertyindependence of the data from the unit of analysis

A non-exhaustive overview of the literature on the theory dimension.

Literature streamReferences
Economics of science and technologyAudretsch et al. (2002), Stephan (2012), Mirowski & Sent (2002)
Theories of growthSolow (1957), Abramovitz (1956), Nelson & Phelps (1966), Romer (1986; 1994), Aghion & Howitt (2009)
Quantitative science and technology researchEgghe & Rousseau (1990), Egghe (2005), Moed, Glänzel, & Schmoch (2004), Ding, Rousseau, & Wolfram (2014), Cronin & Sugimoto (2014; 2015), Glänzel et al. (in press)
Economics of innovation, innovation studiesFagerberg, Martin, & Andersen (2013), Hall & Rosenberg (2010)
Science of science, science and societyFealing et al. (2011, p. 4), Gibbons et al. (1994), Etzkowitz & Leydesdorff (2000), Edquist (2001), Aghion, David, & Foray (2009), Helbing & Carbone (2012), Scharnhorst, Borner, & van den Besselaar (2012), Nowotny, Scott, & Gibbons (2001), Barré (2004)
Economics of knowledgeAntonelli & Link (2014)
Economics of educationBlaug (1966), Johnes & Johnes (2004), Checchi (2006), Hanushek, Machin, & Woessmann (2016)
Education and societyRoper & Hirth (2005), Teixeira et al. (2004), Teixeira & Dill (2011)
Societal impact, science and public policy studiesBornmann (2013), Ebrahim & Rangan (2014), Perkmann et al. (2013), Veugelers & Del Rey (2014), Hill (2016)
History of scienceGodin (2002; 2004), Gingras (2016)

Toward an ethics of research assessment? Some connections of our framework with MacIntyre’ oeuvre.

Enabling conditionPotential connectionMacIntyre work
ConvergenceInvitation to overcome the fragmentation of knowledge and excessive specialization.The end of educationMacIntyre (2006)
Mixed methodsThe need to go beyond a pure quantitative approach (abstract representation of the reality) and include qualitative cases (narratives and storytelling).After virtue, MacIntyre (2007), & Whose justice, which rationality?MacIntyre (1988)
Knowledge infrastructuresRetrieve the values of tradition in communities of practice that regulate themselves by defining their own standards.After virtueMacIntyre (2007)
eISSN:
2543-683X
Language:
English
Publication timeframe:
4 times per year
Journal Subjects:
Computer Sciences, Information Technology, Project Management, Databases and Data Mining
Journal RSS Feed