A Novel Method for Resolving and Completing Authors’ Country Affiliation Data in Bibliographic Records

In previous IRC studies applying string matching, the task of country identification is done by getting pieces of data from affiliation information and matching against pre-stored lists of common phrases that act as reliable indicators for country disambiguation. The effectiveness of this commonly used method is limited by the fact that many pieces of affiliation data do not include “country” information, as described above. Our proposed approach is, therefore, to resolve and assign affiliation data to particular countries by drawing on Web-based knowledge graphs, which have multiple advantages for that use.

Various knowledge graphs have been developed to describe entities in the real world and the relationships between them. Among the knowledge graphs that are freely accessible, such as DBpedia, OpenCyc, Wikidata, and YAGO, Wikidata is overall ranked the best in data quality (Färber et al., 2018). Wikidata is a well-regarded peer-produced knowledge base that primarily solves issues of data redundancy in different language versions of Wikipedia but by now being utilized for an increasing amount of use cases that benefit from structured background knowledge such as Google search (Pellissier Tanon et al., 2016). Wikidata is built around two main logical constructs: items and properties. While items depict entities in the real world, properties describe the items’ relations (Müller-Birn et al., 2015). For example, the record “Victoria University of Wellington” in Wikidata is an instance of the item “public university” and stores a statement about the location property (i.e. the country is New Zealand). This logical paradigm allows to consider the Wikidata knowledge base as a graph (where items constitute the vertices and properties the edges). Hence it is commonly referred to as a knowledge graph and allows interesting technical use cases that make use of graph traversal through structured queries.

The success and recent widespread adoption of Wikidata have led to many studies about its data quality. There are four categories of quality dimensions addressed for this purpose: intrinsic, contextual, representational, and accessibility data quality. In details, intrinsic dimensions include accuracy, trustworthiness, consistency; contextual dimensions include relevancy, completeness, timeliness; representation dimensions include easy of understanding, interoperability; accessibility dimensions include accessibility, interlinking, and license (Piscopo & Simperl, 2019). Wikidata is highly evaluated in all these dimensions, reaching a 100% quality score with respect to the syntactic validity of RDF documents and syntactic validity of literals, and the 99% with respect to semantic validity of triples reaches (Färber et al., 2018).

It is for the aforementioned features of Wikidata that we decided to use Wikidata via its SPARQL query endpoint to complete missing country values of affiliation information in bibliographic data sets (Nguyen, Dinneen, & Luczak-Roesch, 2019).

We implemented two data enrichment steps in our method to identify the collaborating countries using the authors’ affiliation data in co-authored papers. Figure 1 gives a high-level overview of our approach, which we will describe in detail in the remainder of this section.

Figure 1

First, we implemented algorithmic string matching using standard R packages (e.g. lists of country names and abbreviations). For records with author affiliation data containing countries’ names or abbreviations, these pieces of information were matched to a predefined list of countries. We also took into account some common facts in building the algorithm. First, many affiliation data have the postcodes or the state names or city names of the US states. Therefore, additional predefined lists were also used to identify these cases. Second, some records have the names of the UK components (e.g. Scotland or England). In IRC studies, the UK was usually considered as a whole entity for all its component parts so we mapped all corresponding component names to the United Kingdom. Third, there are different alternative names for China (e.g. “PRC” or “P.R.C”). All these cases mentioned above were incorporated into our method's codes for identifying the responding countries.

There are some limitations of the above string matching step though. For some affiliation data linked to countries that do not exist (e.g. Yugoslavia or Czechoslovakia), we found no ways to map these affiliation data to the corresponding new-formed countries. Therefore, the country identification of these special cases (and others have no information about countries) relies on the success of the second step.

Second, we carried out an additional algorithm using the Wikidata query when it is necessary: for records that could not be identified having no clear country names or state information, we then used the remaining information (e.g. university name) to query the SPARQL endpoint of Wikidata^① executing the following query^②: PREFIX schema: <http://schema.org/>PREFIX wdt: <http://www.wikidata.org/prop/direct/>SELECT ?countryLabel WHERE {    <https://en.wikipedia.org/wiki/[AFFILIATION]> schema:about ?datalink.    ?datalink wdt:P17 ?country.SERVICE wikibase:label    {bd:serviceParam wikibase:language “en”.}}

This query returns English names of countries associated with the location data if there is a matching Wikidata item. For example, querying “McGill University” returns Canada. We implemented these steps in R, and made the source code of both the string matching and the Wikidata query implementation freely available for use in future work^③.

In rare cases where seemingly complete affiliation data does not match with a Wikidata item, each part (i.e. word) of the affiliation data is then queried, assigned a Wikidata item when possible, and those assignments are then checked for consistency. If the assignments match (i.e. are all identified with the same country), the country is regarded as successfully identified. For example, our Wikidata query could not identify the country associated with the exact phrase “Massachusetts Institute of Technology, Cambridge”, so “Massachusetts Institute of Technology” and “Cambridge” were queried. Because “Massachusetts Institute of Technology” is identified as the United States and “Cambridge” as the United Kingdom (i.e. not the same country for the record in question), the record is treated as a non-match and not assigned country data.

The most commonly used data sets in IRC research are SCI/Web of Science and Scopus (Guerrero Bote, Olmeda-Gómez, & de Moya-Anegón, 2013; Luukkonen et al., 1993). Each set entails considerable practical challenges for researchers; for example, only 500 records can be downloaded from WoS at a time, or 2,000 from Scopus. Further, these sets (and Google Scholar) do not have as comprehensive general coverage as, for example, Microsoft Academic Graph (Paszcza 2016; Sinha et al., 2015), and may not have as complete domain-specific coverage as, for example, ACM Digital Library (ACM DL) and IEEE Xplore provide for computer science.

Therefore, we tested our method on Microsoft Academic Graph (MAG), a general scholarly bibliographic data set, and the ACM Digital Library, a scholarly bibliographic data set containing works published by the Association for Computing Machinery and primarily related to computer science. These two data sets are commonly used to study computer science research (Guha et al., 2013; Herrmannova & Knoth, 2016). Importantly, because the fields in these databases that our method evaluates and enriches (i.e. those describing authors’ affiliations) also exist in other bibliographic data sets, the method can also be applied to other bibliographic data sets (e.g. WoS and Scopus).

To make the results of our evaluations comparable across the two sets we filtered out records from MAG that were not relevant to computer science: a list of fields of study was compiled from records present in both ACM DL and MAG, and the 38 top terms (94% of papers in the overlap) were used to filter out irrelevant works. Overlapping papers were also filtered from the ACM DL set to make the sets distinct. Finally, single-author records were filtered out to identify only co-authored papers. Table 1 summarises the data sets used.

After preparing the two data sets, our enrichment method (section 3.2) was applied to both.

Our approach identifies countries for approximately 71%–75% records in each data set (the percentages of different categories of the records processed are shown in Table 2), with the remaining records being either unidentified (~15–17%) or unidentifiable (~8–14%) because of empty or placeholder affiliation values (e.g. “NA” or “None”). Specifically, while the substring matching approach identifies countries for nearly 60–70% of records, the Wikidata querying approach adds an additional roughly 10% in ACM DL and 6% in MAG. In other words, the method provided identifies approximately 82% of the possible records. These results suggest that our approach succeeds in matching the majority of bibliographic records, in both general and domain-specific data sets, to the relevant countries.

We describe here the rationale and procedures for evaluating our method, using the three measurement criteria of utilization, quantitative accuracy, and fairness/justice (Jacobs, Kafry, & Zedeck, 1980), implemented as magnitude, accuracy, and consistency, respectively:

To assess the magnitude of additional information, we examined how the new method improves upon the presumed baseline process for country identification.

4.3.1

Assessment of magnitude of additional information

In this assessment, we had a look at the patterns of the improvement overtime (comparison by years). The comparison was made for the period 1951–2016, in which both data sets tested have whole year data. The charts (a) and (b) in Figure 2 show the ratios by year of improvement made by using Wikidata query in ACM DL data set and MAG data set, correspondingly. In details, these charts compare (1) the numbers of affiliation data unidentified after using string matching (and excluding unidentifiable cases, e.g. “NA” or “None”), and (2) those received after the above step by applying Wikidata query.

Figure 2

The magnitude at which some method of identifying countries improves upon the baseline is an indication of the improvement in the resulting IRC measurement. As presented in the method's outputs section above, the rates of improvement for the total affiliation data are 9.65% and 5.71% for ACM DL and MAG data sets. The rates of identification roughly follow a logarithmic trendline (R²=0.33 for ACM DL and 0.63 for MAG), as contrasted with a linear improvement (R²=0.31 and 0.48). In other words, the rates of improvement by year initially increase quickly, but then stabilize around 40% and 30%. It is thus predictable that rates of improvement will not change considerably as more data (i.e. after 2016) are available and considered.

4.3.3

Assessment of the consistency of additional information

Firstly, we surveyed the difference in the percentages of improvement for different countries (comparison by countries). This was done by calculating the value of (number of countries identified by Wikidata query/number of countries identified by string matching). Figure 3 provides a visual representation of the comparison. The outliers represent countries or territories with relatively few publications (e.g. State of Palestine and Serbia in ACM DL; Cape Verde, Uganda, and Nepal in the MAG), such that just a few additional collaborations resulted in large percentage increases. One such extreme outlier (Cape Verde) was excluded in Figure 3(b) to facilitate visualization.

Figure 3

As the boxplots featured in the above figure reveal, the distributions of data for percentages of improvement are skewed and so the appropriate measures of centrality and dispersion are the median and interquartile range respectively. The summary statistics provided in Table 4 confirm that, for countries having scientific publications, the normal percentages of improvement applying new method of country identification are nearly the same for both data set (the median values are 1.20% and 0.91% for ACM DL and MAG, respectively). The interquartile ranges are 10.18% and 6.30%, respectively, meaning that the improvement does not vary much among the countries, either using ACM DL data set or MAG data set. In other words, our method provides a fairly stable benefit among countries with only a few exceptions (e.g. places producing relatively few research outputs).

Table 4

Summary statistics of the method's results.

	ACM DL	MAG
Mean	5.70%	5.42%
Standard error	0.69%	1.29%
Median	1.20%	0.91%
Mode	0%	0%
Standard deviation	8.10%	17.88%
Interquartile range	6.30%	10.18%
Count	137	192

Secondly, we looked at how the improvement makes a difference to countries. Figure 4 shows the comparison in terms of the percentages of improvement received by our method and the numbers of affiliation records associated with each country (without Wikidata use) in the ACM DL and MAG data sets. As shown in Figure 3, an outlier in the improvement dimension (Cape Verde) was excluded in Figure 4(b) for easier comparison. Overall, we find no significant relationships between the percentages of improvement and the countries’ numbers of affiliation data identified.

Figure 4

To test this more formally, we ran a linear regression analysis between the variables “countries’ size in amount of related records found in the dataset” and “percentual improvement of country identification through the use of Wikidata”, assuming there is a chance that affiliation information of organisations in countries of bigger scientific size may be more well-known than those in smaller ones. The analyses were performed once for all countries and once only for the top countries in terms of the amount of related records found in the ACM DL and MAG datasets. The reason for taking separately top countries into consideration is that IRC comparisons in absolute numbers are significantly affected by both country size (in terms of scientific papers) and collaborative activities (Luukkonen et al., 1993). Therefore, we wanted to compare the relationships for the whole list to those for the top countries only. The R squares observed (0.055, 0.22, 0.001, 0.15) indicate no significant relationship exists, neither in the list of all countries nor the list of top 30 countries in both data sets. In other words, only a small percentage of the variability in the improvement can be explained by the relationship between the percentage of improvement and the country size (in terms of scientific papers), in both ACM DL and MAG data sets.

The method proposed here promises to be useful in both bibliometric research and in practical applications of IRC measurement. To start, use of our method in future studies should lead to increased accuracy in measuring IRC. For example, a map depicting global IRC networks (e.g. Wagner & Leydesdorff, 2005) will be more accurate if generated using our method, as would characterizations of the patterns of research collaboration (e.g. Schubert & Braun, 1990) across groups of countries (e.g. Western Europe as compared with Eastern Europe). Given that our method had similar rates of affiliation identification (over 70%) for both data sets examined, regardless of the initial efficacy of string matching (e.g. low efficacy with ACM DL data), and because it requires only the affiliation data common in most bibliographic data sets, it is likely that uses of our method (like the examples here) will be effective regardless of the data set(s) used (e.g. Scopus or WoS); this is important as many studies are practically limited to one data set.

As our method works well and is shared online, it can also be collaboratively improved and easily adapted for similar uses in research beyond measuring IRC. For example, the improved country identification can be useful for other bibliometric analyses, such as measuring a country's total scientific production or the rates of its scientist emigration and immigration. Further, because our method uses Web-based knowledge graphs, other missing bibliographic data can be completed using our method with minor changes, for example journals’ languages or publishers’ names. While the implementation of our method is publicly shared online, its performance may be further improved by the contributions of other researchers in the future. Given the logarithmic rate of improvement across years observed above, any improvement to the method is likely to improve IRC identification across all publication years.

Our method also may be useful for performing analyses to inform policy decisions. For example, the policy makers for joint research programs (e.g. European Union funding programs) may need to evaluate the technological, societal, and economic impacts of multinational research projects. The more accurately such impacts on each related country can be measured, the more confidently policy makers can be in making decisions for further investment (e.g. extending multi-national research projects). This extends further to strategic decisions about which countries to encourage collaboration with; for example, countries having advanced technical skills (e.g. Korea) may target funding collaborations with countries with complementary knowledge (e.g. theoretical expertise in the U.K.) and thus enable mutually beneficial IRC (e.g. the transmission of know-how in exchange for funding and research resources; Hwang, 2008). Identifying strategic collaborations and monitoring the efficacy of bi-lateral cooperation, however, will benefit from a more accurate and consistent method of measuring research outputs, such as the one provided here.

Our method has been designed and tested to perform well with usual, simple cases of IRC measurement (e.g. countries with singular identities and relatively frequent research outputs), and so it may not apply straightforwardly when measuring the IRC of countries that have, for example, changed identity (e.g. name or composition) or no longer exist as such (e.g. Yugoslavia). Bespoke approaches may be needed for such edge cases.

Although we tested our method with both general and domain-specific bibliographic databases, it has yet to be tested with other popular data sets (e.g. Scopus), and so may perform differently with them. Given the similar performance of our method on ACM DL and MAG, it is reasonable to expect similar performance with other bibliographic data sets, but this should be empirically validated.

Finally, although the evaluation of our method's reliability had promising results (i.e. no false matches), that evaluation was practically limited to manually inspecting the results of only 400 random Wikidata query results (i.e. <1% of the records in our test implementation). Though we have no particular reason to suspect further random samples would indicate considerably different reliability, it is nonetheless true that manually inspecting a larger sample of results may have different implications about the reliability of our method; such inspection should be done in testing future revisions of the method.