Measuring Societal Impact Is as Complex as ABC

Ed Noyons 1
  • 1 Centre for Science and Technology Studies, Leiden University, Leiden, The Netherlands
Ed Noyons
  • Corresponding author
  • Centre for Science and Technology Studies, Leiden University, Leiden, The Netherlands
  • Email
  • Search for other articles:
  • degruyter.comGoogle Scholar

Abstract

Purpose

This paper describes an alternative way of assessing journals considering a broader perspective of its impact. The Area-based connectedness (ABC) to society of journals applied here contributes to the assessment of the dissemination task of journals but with more data it may also contribute to the assessment of other missions.

Design/methodology/approach

The ABC approach assesses the performance of research actors, in this case journals, considering the characteristics of the research areas in which they are active. Each paper in a journal inherits the characteristics of its area. These areas are defined by a publication-based classification. The characteristics of areas relate to 5 dimensions of connectedness to society (news, policy, industrial R&D, technology and local interest) and are calculated by bibliometric indicators and social media metrics.

Findings

In the paper, I illustrate the approach by showing the results for a few journals. They illustrate the diverse profiles that journals may have. We are able to provide a profile for each journal in the Web of Science database. The profiles we present show an appropriate view on the journals’ societal connectedness.

Research limitations

The classification I apply to perform the analyses is a CWTS in house classification based on Web of Science data. As such the application depends on the (updates of) that system. The classification is available at www.leidenranking.com

Practical implications

The dimensions of connectedness discussed in this paper relate to the dissemination task of journals but further development of this method may provide more options to monitor the tasks/mission of journals.

Originality/value

The ABC approach is a unique way to assess performance or impact of research actors considering the characteristics of the areas in which output is published and as such less prone to manipulation or gaming.

1 Introduction

Quantitative evaluation studies monitoring the scientific performance of actors (countries, institutes, individuals and also journals) regard primarily scholarly output and impact. Output is usually quantified by counting publications. The impact measurement usually regards scientific impact. Such impact, framed as a proxy for quality, is then measured by citations.

Assessing other impacts, for instance societal impact, is much more complicated and certainly less developed. Traceable effects of research in the societal context varies from one field to the other, from application to application and from case to case. For that reason, such impact is often assessed by case studies or other qualitative approaches.

Research assessments mostly deal with organizational entities, such as universities, research institutes or individual researchers. Another type of assessed entity we can discern is the journal. Its role and quality is evaluated in many studies and reports. The most common assessment of journals is done by using the Journal Impact Factor (JIF). In short and simplified, JIF measures the average number of citations received by papers in a journal within the first two years. This indicator has often been used to assess the quality of individual publications. Being accepted by a journal a paper meets the standard of that journal. Recently, this particular use of JIF has been criticised heavily leading to, for instance, “The San Francisco Declaration on Research Assessment”, DORA (Cagan, 2013).

On top of that, a group of bibliometric and evaluation specialists, scientists, publishers, scientific societies and research-analytics providers recently promoted the need to develop a new suite of journal indicators and other ways to assess the quality of journals (Wouters et al., 2019). The authors state that these indicators and other ways to assess journals are to be developed in the context of the journals’ primary tasks: registering, curating, evaluating, disseminating and archiving.

In this paper, I frame the assessment of journals from a broader perspective. The approach is based on a recent proposal to assess the connectedness of research actors to society (Noyons, 2018). The approach does not intend to cover all aspects of a journal’s primary tasks but it does cover parts of the task of dissemination.

A key point I make in this paper is that the broader impact of journals should not be limited to the output of the publications in the journal itself. We can open a variety of options to assess the impacts of journals and science (other) actors if we consider the content of the entire research areas to which a journal’s output belongs and contributes. In other words, the connectedness and broader impact does not depend on the output and impact of the journal itself alone. The community of research around the journal matters as well.

2 Data and method

2.1 Web of Science (WoS) and publication-based classification

For this study, I use the data from the bibliometric version the Web of Science core collection hosted at CWTS. It is a dedicated version of the Web of Science online for bibliometric studies, with additional features and improvements. The important add-on for this study is the publication-based classification (Waltman & van Eck, 2012) in which publications are clustered on the basis of their citation-based context into coherent groups (Traag, 2015; Traag, Waltman, & van Eck, 2019). Over 4000 clusters (CWTS classification, version 1813, available at www.leidenranking.com) provide a fine-grained structure of all sciences independent of the journals in which research papers are published. This structure enables a topic basic-based analysis and will be used to identify research communities, i.e., clusters of publications on a certain topic.

The VOSviewer map below shows the distribution of the 4,000 clusters. Each cluster is represented by a circle in the map. The position of a circle indicates its relatedness to all others, based on their mutual citation traffic. The denser the citation traffic, the closer two circles are. The size of a circle represents the number of publications included.

Fig. 1 visualizes a virtual landscape of all sciences (as far as covered by WoS) in two dimensions. The actual position of a clusters has no meaning, only the distances between them. Hence, one can rotate or flip the map in any direction. Only the distances matter. I added colors for clarity. These colors represent high level regions (disciplines or main fields) and are labelled as such.

Figure 1
Figure 1

Landscape of science (data: Web of Science 2000–2017). Circles represent clusters of publication (areas), size represents relative volume (numbers of publications), color represents main fields, disciplines.

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

Another add-on to the CWTS database is the direct link of WoS publications to Altmetric data and (PATSTAT) references in patents. For each paper we determine whether it has been mentioned on Twitter, in a news item, in a policy document, in a Wikipedia article etc or whether it was cited in a patent. I consider such mentions or citations as non-academic events or actions related to the research that was mentioned, cited etc. This information I will therefore use as signals of connectedness between research and society.

2.2 ABC method

In the approach to characterize journals I apply a method, coined as area-based connectedness (ABC) to society (Noyons, 2018). This approach assesses the connectedness of the research covered based on the output of an actor, in this case a journal. This connectedness is captured by the communication signals we detect between research output and society. These signals are diverse and represent a variety of dimensions.

To determine connectedness of research areas or communities to society, we measure a set of signals. The individual signals represent different aspects, dimensions of this connectedness. For instance, (co-)authorship of industry in publications indicates technological application and/or commercial use of research. Mentions in policy documents of publications signifies societal or political interest or value. I will demonstrate that each signal has its own specific profile. The profiles relate to their distribution across the science landscape. Different signals connect different research areas.

Until now, such connectedness signals are measured at the level of actors. For instance, the share of output of actor A co-authored with industry would then be a an indication of the societal, economic, technological impact of A. Or the number of twitter mentions of articles by actor B would be an indication of B’s societal impact (Bornmann, 2013; Seppo & Lilles, 2012).

Such approaches have at least two flaws. Firstly, social media metrics are easily manipulated and therefore vulnerable for malpractices (Costas, Zahedi, & Wouters, 2015; Haustein et al., 2016). Moreover, such indicators may be affected by practical limitations. For instance, co-authorship with industry often depends on the willingness of the company involved to co-author. Similarly, mentions of scientific output in non-scholarly output often depends on the accessibility of scholarly literature. If a research paper is not available or too technical, the author of a policy report may decide to cite another publication on the same topic.

The ABC method measures the output of a journal using the character of research communities (areas) to which it belongs. A community is defined as a set of publications on the same topic, i.e. a cluster of publications from the CWTS publication-level classification.

Weighted by the number of publications a journal has within a community or area it “inherits” the character of the community/area. This character involves the connectedness to society as measured by the signals.

The approach may not meet all the criteria for proper journal indicators and/or responsible use thereof, but it does provide a broad range of the indicators to assess the role of journals which are less vulnerable for gaming and manipulation than the existing (mostly) one-dimension journal indicator.

Example

Journal A contains 100 publications in the period 2015–2018. These publications are distributed over 20 classes of the CWTS publication level classification. The 20 classes contain between 200 and 1,000 publications in the period 2015 to 2018. Within that period a certain share of the output in a class is cited by patents, is authored by industry, is mentioned in a policy document, etc. Every class can be characterized along each of these dimensions. Moreover, there is a global average for each dimension. Let’s suppose the global average of publications being cited in patents is 10%. In the 20 classes of Journal A, the share of publications being cited ranges between 0% and 20%. So, some classes have a relatively high share of papers being cited by patents, while other have a low share or even none. The connectedness of Journal A is to be measured by a combination of its output distribution and the connectedness its (20) classes, for instance as follows:

ClassNpubs in classShare pubscited in patentsNpubs in journal A
12001%30
25005%20
330020%20
420010%10
etc

2.3 ABC signals and dimensions of connectedness

The dimensions considered in the ABC method discussed in this paper are based on the signals available to measure societal connectedness. For journal indicators we implement the following dimensions with the according signals between parentheses:

  1. -News (papers being mentioned in news items)
  2. -Policy (papers being mentioned in policy documents)
  3. -Industry R&D (industry authorship)
  4. -Technological or commercial application (papers cited in patents)
  5. -Local scope (papers in local languages, not in English)

In the remainder of this section I discuss the dimensions one by one. For each dimension I discuss the signal being used as well as their distribution across the science landscape.

2.4 News

The News dimension relates to research being relevant in the broadest sense for the general public. It is assessed by the proportion of publications in a research area mentioned in news items. These news items are collected and included in Altmetrics from a large set of sources. We detect around 2,500 news sources. The top 10 most common sources are:

  1. The Conversation
  2. EurekAlert!
  3. Phys.org
  4. MedicalXpress
  5. Health Medicinet
  6. Science Daily
  7. Huffington Post
  8. Medical News Today
  9. Bioportfolio
  10. Yahoo! News.

These approximately 2,500 sources are consulted for mentions of publications. I use these mentions as signal for connectedness in the following way. For each area, the proportion of papers being mention in a news item is calculated. These proportions range from 0 to 50% across the 4,000 areas. Their distribution is visualized in Fig. 2.

Figure 2
Figure 2

Landscape of science (data: Web of Science 2000–2017). Color represents share of publications mentioned in news items

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

In Fig. 2, I use a color coding indicating the proportion in such a way that the Blue areas have a lower proportion or even zero in Dark Blue, while the Yellow areas have a relatively high proportion.

Fig. 2 shows a preferred distribution of the news signal, primarily in the biomedical sciences, social sciences and life & earth sciences. In these fields, the areas are more likely to color Yellow. This does not mean that, for instance, all biomedical research is connected to society in the news dimension, but these fields are more connected to society than others.

2.5 Policy

The policy dimension relates to research being relevant to political issues. For this dimension I use the signal of mentions of research output in policy documents. These documents are retrieved from 56 sources. The top 10 most commonly used organizations are:

  1. National Academies Press
  2. World Health Organization
  3. Centers for Disease Control and Prevention (CDC)
  4. National Institute for Health and Care Excellence
  5. UK Government (GOV.UK)
  6. National Bureau of Economic Research
  7. Analysis & Policy Observatory (APO)
  8. Food and Agriculture Organization of the United Nations
  9. The Publications Office of the European Union
  10. World Bank

The distribution of proportions of papers mentioned in policy documents across the landscape ranges between 0 and 19% and is visualized in Fig. 3.

Figure 3
Figure 3

Landscape of science (data: Web of Science 2000–2017). Color represents share of publications mentioned in policy documents.

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

The policy signal shows a preferred position in the social sciences, life & earth sciences and biomedical sciences. This preference is different from the news dimension. Such differences support the fact that one should look at more than just one dimension when considering connectedness to society.

2.6 Industry R&D

The dimension related to the interest from the private sector is assessed by the authorship of companies in publications. For this dimension I calculate the proportion of output in which the author affiliation is a company in an area, ranging between 0 and 30%. The distribution of this signal is shown in Fig. 4.

Figure 4
Figure 4

Landscape of science (data: Web of Science 2000–2017). Color represents share of publications (co-)authored by Industry (a company).

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

In Fig. 4, we see quite a different picture than in the other dimensions. Here, the preferred areas are in computer sciences, physical sciences, engineering and, to a lesser extent, in biomedical sciences.

2.7 Technology

The technology dimension relates to the support or enabling strength of research. The signal I use to monitor this is citations from patents, operationalized by measuring the proportion of publications being cited in patent as Non-Patent Literature references. The distribution of this signal, ranging between 0 and 40%, is shown in Fig. 5.

Figure 5
Figure 5

Landscape of science (data: Web of Science 2000–2017). Color represents share of publications cited by at least one patent.

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

In this case the signal is primarily detected in biomedical sciences and the chemistry part of the physical sciences. On top of that, we find somewhat stronger signals in engineering.

2.8 Local interest

The dimension of local interest relates to the extent to which research is relevant for local issues. Most likely, research with a local interest is published in a local language. This maybe any language so that I defined it as non-English. English is the most international, i.e. non-local, language. That does not imply, however, that English language papers cannot be aiming at local issues. This in particular is an example why the ABC method is an interesting approach. A journal is usually in one language only. Therefore, characterizing journals by the proportion of non-English papers would neglect journals in English with a local interest. By using the characterization of the areas in which a journal is covered, inheriting their character, we can identify English language journals with a local interest.

The proportion of papers not in English by area ranges between 0 and 83%. The areas with really high proportions are often small. The distribution over the landscape is shown in Fig. 6.

Figure 6
Figure 6

Landscape of science (data: Web of Science 2000–2017). Color represents share of publications not in English language.

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

The preferred areas for non-English publications are in social sciences and clinical and health sciences.

3 Selected results

In this section, I will demonstrate a way to measure the connectedness of journal using the ABC approach. For each journal I count the number of publications and their distribution across the 4,000 research areas. The output of a journal will then be characterized by the connectedness of the areas in each dimension, which means that the connectedness of a journal is the average of the sum of the product of the number of publications in an area and the score on a dimension in an area (weighted average).

The final connectedness on each dimension will then be compared to the overall average of connectedness of that dimension.

As a first example, consider the Journal of Fluency disorder. This journal provides comprehensive coverage of clinical, experimental, and theoretical aspects of stuttering, including the latest remediation techniques (www.journals.elsevier.com) Obviously, this is a specialized journal with a clear clinical focus. Usually, such journals are less well cited than more fundamental journals. Its journal impact factor is 1.73 (JCR 2017/18).

The distribution of publications from this journal across the science landscape is depicted in Fig. 7. The structure is as explained in the previous section. The size of the circle indicates, proportionally, the number of publications from this journal in this cluster.

Figure 7
Figure 7

Landscape of science (data: Web of Science 2000–2017). Size represents number of publications in Journal of Fluency disorder (2014–2017).

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

The cluster to which almost all publications from this journal belong has a relatively high score on the news dimension, while the share of non-English papers is around average (baseline). In all other dimensions the connectedness of this cluster is low. This leads to the ABC profile as depicted in Fig. 8.

Figure 8
Figure 8

Area-based connectedness to society profile of the Journal of Fluency Disorders (2014-2017).

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

This profile tells us that the research published in this journal has a connectedness to society through the news dimension (almost 40% above average) and an average connectedness on the dimension of local interest.

A second specialized journal, Solar Physics, has 95% of its output in one cluster (See Fig. 9) and a modest impact factor (2.6).

The area (ID 43) has a high connectedness through its content on the Industry dimension as shown in Fig. 10 below (20% above average). Hence, this journal may not have the high impact one may expect or want for this journal but the research published in this journal does have its value for application, as indicated by the industry dimension.

Figure 9
Figure 9

Landscape of science (data: Web of Science 2000-2017). Size represents number of publications in journal Solar Physics (2014-2017).

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

Figure 10
Figure 10

Area-based connectedness to society profile of journal Solar Physics (2014–2017).

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

Finally, I illustrate the approach using the Lancet, a journal with an impact factor over 50 and a much broader scope. The broader scope of this journal is demonstrated by the distribution across areas in Fig. 11.

Figure 11
Figure 11

Landscape of science (data: Web of Science 2000–2017). Size represents number of publications in The Lancet (2014–2017).

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

The Lancet is used in biomedical and health sciences in the broadest sense. This yields quite a different profile of connectedness from the other, specialized journals. The “Lancet areas” show particularly a very high connectedness on the policy dimension with almost 4 times the average (see Fig. 12). This is also visible in the news dimension at 75% above average. Maybe a bit more surprising is the high connectedness on the local interest dimension at 50% above the baseline, which most likely relates to clinical research.

Figure 12
Figure 12

Area-based connectedness to society profile of The Lancet (2014–2017).

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0012

4 Conclusion & discussion

Journals are usually assessed by the Journal Impact Factor (JIF). Besides the issues related to this indicator (e.g., Moed, 2002), such an approach is one dimensional and overlooks the other tasks and values a journal has (Wouters et al., 2019). In this study I introduce an alternative and more diverse way to assess journals. The approach is not actor-based but areas-based. This means that the output of a journal is not assessed by the characteristics of publications from that journal but rather by the characteristics of the areas their publications are published in. In other words, journals inherit the characteristics from the areas to which they belong.

The measures I introduce, relate to the connectedness of journals to society or more general, to their dissemination potential. To monitor this quality, the proposed area-based connectedness (ABC) to society approach introduces 5 dimensions. These dimensions are based on signals we can detect between research outputs and society. There are many more possible signals and dimensions to study and the task we take upon us in the near future is to explore as many as possible to cover a broader spectrum of connectedness. The dimensions I introduce in this paper are civil interest, policy interest, industry R&D, technology and local interest. With a few illustrative examples, I demonstrate that the ABC approach provides an interesting new way to assess potential of journals. The approach can be based on the most recent output, and as such the assessment is much more up-to-date than citation-based indicators. Furthermore, by evaluating a journal’s output through the inherited ABC character of the (algorithmically created) areas, the assessment is less vulnerable for gaming and manipulation (Falagas & Alexiou, 2008).

References

If the inline PDF is not rendering correctly, you can download the PDF file here.

OPEN ACCESS

Journal + Issues

Search

  • View in gallery

    Landscape of science (data: Web of Science 2000–2017). Circles represent clusters of publication (areas), size represents relative volume (numbers of publications), color represents main fields, disciplines.

  • View in gallery

    Landscape of science (data: Web of Science 2000–2017). Color represents share of publications mentioned in news items

  • View in gallery

    Landscape of science (data: Web of Science 2000–2017). Color represents share of publications mentioned in policy documents.

  • View in gallery

    Landscape of science (data: Web of Science 2000–2017). Color represents share of publications (co-)authored by Industry (a company).

  • View in gallery

    Landscape of science (data: Web of Science 2000–2017). Color represents share of publications cited by at least one patent.

  • View in gallery

    Landscape of science (data: Web of Science 2000–2017). Color represents share of publications not in English language.

  • View in gallery

    Landscape of science (data: Web of Science 2000–2017). Size represents number of publications in Journal of Fluency disorder (2014–2017).

  • View in gallery

    Area-based connectedness to society profile of the Journal of Fluency Disorders (2014-2017).

  • View in gallery

    Landscape of science (data: Web of Science 2000-2017). Size represents number of publications in journal Solar Physics (2014-2017).

  • View in gallery

    Area-based connectedness to society profile of journal Solar Physics (2014–2017).

  • View in gallery

    Landscape of science (data: Web of Science 2000–2017). Size represents number of publications in The Lancet (2014–2017).

  • View in gallery

    Area-based connectedness to society profile of The Lancet (2014–2017).