The Power-weakness Ratios (PWR) as a Journal Indicator: Testing the “Tournaments” Metaphor in Citation Impact Studies

The numbers of publications and citations are size-dependent: large journals (e.g. PNAS

Proceedings of the National Academy of Sciences of the United States of America (PNAS)

Pinksy and Narin (1976; cf. Narin, 1976) proposed to improve on JIF by normalizing citations not by the number of publications, but by the aggregated number of (“citing”) references in the articles during the publication window of the citation analysis. Yanovski (1981, at p. 229) called this quotient between citations and references the “citation factor.” The citation factor was further elaborated into the “Reference Return Ratio” by Nicolaisen and Frandsen (2008). In the numerator, however, Pinski & Narin (1976) used a recursive algorithm similar to the one used for the numerator and denominator of PWR. This example of an indicator based on a recursively converging algorithm was later followed with modifications by the above-mentioned authors of PageRank, HITS, Eigenfactor, and the Scimago Journal Ranking (SJR; Guerrero-Bote & Moya-Anegón, 2012).

“Eigenfactor,” for example, can as a numerator be divided by the number of articles in the set in order to generate the so-called “article influence score” (West, Bergstrom, & Bergstrom, 2010; cf. Yan & Ding, 2010). Using Ramanujacharyulu’s (1964) PWR algorithm, however, the same recursive algorithm is applied in the cited-direction to the numerator and in the citing-direction to the denominator. “Being cited” is thus considered as contributing to “power” whereas citing is considered as “weakness” in the sense of being influenced. Let us assume that these are cultural metaphors-we return to this in the discussion-and continue first to investigate the properties of the indicator empirically. For a mathematical elaboration, the reader is referred to Todeschini, Grisoni, and Nembri (2015).

In another context, Opthof and Leydesdorff (2010) noted that indicators based on the ratio between two numbers (such as “rates of averages”) are no longer amenable to statistical analysis such as significance testing of differences among the resulting values (Gingras & Larivière, 2011). More recently, other indicators based on comparing observed with expected values have also been introduced (e.g. MNCS by Waltman et al., 2011b; I3 by Leydesdorff et al., 2012; cf. Leydesdorff et al., 2011).

Let Z be the cited-citing journal matrix. If the entries are read row-wise, for a journal in row i, an entry such as Z_ij denotes the citations from journal j in the citation window (2013) to articles published in journal i during the publications window (2011–2012); in social-network analysis these are considered the in-coming links. When the matrix is read column-wise, the entry Z_ij signifies the references from journal j in the citation window (2013) to articles published in journal i during the publications window (2011–2012). In social-network analysis these are considered the out-going links

In social network analysis, the matrix is usually transposed so that action (“citing”) is considered as the row vector.

Using graph theory, Z = [Z_ij] is the notation of the matrix associated with the graph. This matrix can be multiplied with itself. More generally, Z can be raised indefinitely to the k^th power, i.e. Z^k. The Eigenfactor, for example, is a recursive iteration that raises Z to an order where convergence is obtained for what is effectively the weighted value of the total citations (Yan & Ding, 2010). One can find a value p_i(k) for each journal (vector); this can be called the iterated power of order k of the journal i “to be cited”.

For obtaining weakness, the same operations are carried out column-wise by first using the transposed matrix Z^T and then proceeding row-wise among these transposed elements in the same recursive and iterative manner as above. Again, for each journal one can find a value w_t(k), which can be considered the iterated weakness of the order k of the journal i “to be influenced by.” The empirical question remains of whether both p_i(k) and w_i(k) converge for k → ∞. If k → ∞ converges, one obtains the converged power-weakness ratio r_i(k) = p_i(k)/w_i(k).

In more formal terminology: the vector of power indexes is the solution to the equation p = Ap, where Z_ij is the number of times journal j cites journal i and where the matrix A is derived from the matrix Z by normalizing the columns to sum to one. The power p_j of journal j is the sum over all i of the fraction of cites from journal i that go to journal j weighted by the power of journal i. Weakness is defined analogously, mutatis mutandis. As noted, a further elaboration in formal terms is provided by Todeschini, Grisoni, and Nembri (2015). The recursive procedure for formalizing the computation of p_i(k) is given in graph-theoretical terms in Ramanujacharyulu (1964). An algorithmic implementation using the Stodola method of iteration is provided by Dong (1977). In the appendices, we provide routines for calculating PWR from a citation matrix using Pajek (Appendix 1) or Excel (Appendix 2).

Note that a journal is thus considered powerful when it is cited by other powerful journals and is weakened when it cites other weaker journals. This dual logic of PWR is similar to the Hubs and Authorities thesis of the Web Hypertext Induced Topic Search (HITS), a ranking method of Web pages proposed by Kleinberg (1999), but with one major difference. In the HITS paradigm as applied to a bibliometric context, good authorities would be those journals that are cited by good hubs, and good hubs the journals that cite good authorities. Thus, among other things, the elite structure of science can be discussed. Using PWR, however, good authorities are journals that are cited by good authorities and weak hubs are journals that cite weak hubs. Using CheiRank (e.g. Zhirov, Zhirov, & Shepelyansky, 2010), the two dimensions of power and weakness can also be considered as x- and y-axes in the construction of two-dimensional rankings. A review of ranking techniques using PageRank-type recursive procedures is provided by Franceschet (2011).

Among the 83 journals assigned to the journal category LIS by Thomson Reuters, one is not cited within this set and four journals do not cite any of these journals. Annual Review of Information Science and Technology, for example, is no longer published but still cited in this group; but also The Scientist is not providing references as its editorial policy. Seventy-five of the 83 journals are part of a single strong component, so they are mutually reachable directly or indirectly; the remaining eight journals include journals that are only cited by other journals, only cite other journals, or are neither cited nor citing. Note that journals that are cited but not citing obtain (very) high PWR scores because their weakness score in the denominator is minima

The weakness score in this case is determined by the number of self-citations on the main diagonal and otherwise zero.

Table 1 lists ranked PWR values for 15 of the 75 journals in the central component after 20 iterations (after removing the four non-citing journals). JASIST, for example, follows with a much lower PWR value of 1.45 at the 36^th position. All PWR values were stable at k = 20. However, it is difficult at this stage to say whether this ranking provides a meaningful measure of journal impact. Our results can be considered as a test of this hypothesis. In our opinion, PWR failed as an indicator of overall journal standing since we are not able to provide the results in Table 1 with an interpretation. Note that the Pajek macro can handle large network data (e.g. the complete JCR).

Table 1

Fifteen journals ranked highest on PWR among 83 LIS journals.

As noted above, some journals never cited another journal in this set and one journal never received any citations from the other journals in the set. For analytical reasons, PWR would be zero in the latter case and may go to infinity in the former. However, a structural analysis of the LIS set shows that there are two main subgraphs in this set. These can, for example, be visualized by using the cosine values between the citing patterns of 78 (of the 83) journals (Figure 1).

Figure 1

Using the Louvain algorithm for the decomposition of this cosine-normalized matrix, 40 of these journals are assigned to partition 1 (LIS) and 38 to partition 2 (MIS—Management Information Systems; cf. Leydesdorff & Bornmann, 2016). From these two subsets, we further analysed two ecosystems which were selected because they are well-connected homogeneous sets.

Table 2 shows the two homogeneous journal ecosystems chosen for further study (using abbreviated journal names). The JASIST+ set comprises seven journals, all of which have cited JASIST at least 100 times and come from the same LIS partition. The MIS Quart+ set is similarly a set of nine journals strongly connected to one another within the MIS partition

Unlike the JASIST+ set, the MIS Quart+ set is not a completely connected clique, since the International Journal of Information Management was not cited by articles in the Journal of Information Technology during 2013.

Table 2

The two homogeneous journal sub-graphs chosen for further analysis, and their abbreviated journal names.

For each ecosystem, we take the year 2012 as the “citing” year and we use “total cites” to all (preceding) years as the variable on the “cited” side. Since all journals are well connected within the sub-graphs, there are no dangling nodes (where the journals are cited within the ecosystem but hardly cite any other journals in the same system). Using PWR, no damping or normalization (as is used in the PageRank approach) is proposed: one uses the cross-citation matrix without further tuning of parameters. In each case, when k = 1, one obtains the raw or non-recursive value of “impact” (Σ cited/Σ citing) and when the iteration is continued to higher orders of k as k → ∞ convergence of the recursive power-weakness ratios is found in both sets.

Table 3 shows the citation matrix Z for the JASIST+ set of seven journals. The weakness matrix can be obtained by transposing this matrix, and the cases without self-citation are obtained by discarding the entries in the diagonal and replacing them with zeroes.

Table 3

Citation matrix Z for the JASIST+ set of seven journals.

In Table 4 we report the convergence of the size-independent power-weakness ratio r with iteration number k for the JASIST+ journals for the cases with and without self-citations. We see that this indicator can serve as a proxy for the relative qualities or specific impacts of the journals within this set. However, the main effect of the iteration is that the Journal of Documentation and JASIST change ranks after three iterations when self-citations are included. Scientometrics becomes “less powerful” than the Journal of Information Science after a single iteration.

Table 4

Convergence of PWR with iteration k for the JASIST+ journals, with and without self-citations.

Table 4 shows, among other things, that the inclusion of self-citations affects PWR values in this case only in the second decimal.

Figure 2 graphically displays the convergence of PWR with iteration number k for the JASIST+ set without self-citations. As noted, it may be meaningful to proceed with the case where self-citations are not included. Analogously, Figure 3 shows the convergence of PWR for the MIS Quart+ set without self-citations. Again, within this homogeneous set rapid and stable convergence of the PWR values was found.

Figure 2

Figure 3

But can the converged values of PWR also be considered as impact indicators of the journals? In our opinion, one can envisage three different options to interpret, for example, the results in Table 4:

Since the authors of this paper are knowledgeable in information science (or scientometrics), the ranking of LIS journals can be interpreted on the basis of our professional experience. The rank-ordering of LIS journals by PWR could not be provided by us with an interpretation. One does not expect JASIST to be ranked at the 36^th position and Scientometrics at the 48^th among 83 journals in the LIS category.

Table 5

Seven strongly connected journals in LIS (JASIST+) ranked on their PWR within this group. For comparison, the SJR values from 2013 are included (see http://www.journalmetrics.com/values.php).

In sum, the indicator did not perform convincingly for journal ranking even in homogeneous sets.

Let us complete the analysis by combining the JASIST+ and MIS Quart+ sets into a single and arguably non-homogeneous set, since the one is from the LIS partition and the other from the MIS partition. Whereas journals in the former set tend to cite journals in the latter set, citations are not provided equally in the opposite direction.

Figure 4 shows the convergence of PWR for the JASIST+ subgroup of journals. Initial divergence of PWR at iteration number seven was noticed and final convergence was found for the MIS Quart+ journals only after 20 iterations in the case of a non-homogeneous set (Figure 5)

After twenty iterations, the MIS Quart+ set also converged.

Figure 4

Figure 5

In other words, Ramanujacharyulu’s PWR paradigm may offer a diagnostic tool for determining whether a journal set is homogeneous or not, but it may also fail to converge or to provide meaningful results in the case of heterogeneous sets. As noted, the application of PWR may have to be limited to strong components.