I have blogged about journal rankings a couple of times (e.g., here, here and here) but I have not really been explicit about my own view; thus, this post. The main distinction is between rankings by experts (such as the Australian Research Council ranking) and market-based solutions (e.g. a citation based ranking, such as the Washington & Lee one) [similar in an article not limited to law here]. This post comments on expert rankings, a comment on market-based rankings will follow later on.
A frequent criticism of expert rankings is that it is impossible to determine whether a particular journal is really "better" than another one. This point can, however, be challenged as follows: assume that all journals are of equal quality, and that an expert committee (e.g., a research council) randomly picks 10% of the journals and calls them A* journals, and that financial rewards are provided for publications in these journals (by the research council, or by universities e.g. promotions). Then, what happens? These A* get more submissions; thus, they can be more selective, and the quality of pieces published in A* journals will be better than articles of other journals. As a result, the ranking itself would create a more competitive market for publications, and useful signals, which may be regarded as positive.
But, in the real world, there are two problems with this model:
- First, given the human nature, it is not unrealistic to assume that these A* journals do not only decide on merit but that cronyism plays a role as well. Sure, if you only accepted pieces of friends and family, the number of submissions would go down and the expert committee would downgrade you. But in practice things are often more mixed: e.g., you may give half of the slots to your cronies, and let the other half remain competitive. There may also be network effects in place since academics of the same few institutions may be the editors of the A* journals, its authors, and the members of the expert committee.
- Second, one needs to consider that there are different types of journals; in particular we can distinguish between general and specialised journals, and between mainstream and non-mainstream journals. If the expert committee decides on the basis of majority voting, only mainstream general journals will become A* journals. Of course, that may be harmful to innovation since these journals may regard advanced research on particular issues or new approaches as “too exotic”. Thus, to do it properly, the expert committee would need to apply a quota system, giving A*s to a certain proportion of specialist and non-mainstream journals as well. In practice, that may however be quite unlikely (e.g., imagine a committee with ten members, and only one of them supports of a new and controversial method, whereas the others think that it is just nonsense).
So, as a result, I’ve my doubts about expert based journal rankings. I also feel that the two problems outlined here are apparent in the ARC ranking (in the law list almost all of the A* journals are mainstream general journal often affiliated with a small number of institutions).
Posts
