Wednesday, 31 October 2012

Do article length and co-authorship matter? (RAE 2008 law data)

Following up from previous posts using the RAE data (here; here; here; and earlier here; here), now, first of all a really nice positive correlation (0.75) between the page numbers of the submitted law journal articles and the corresponding RAE scores of the law schools/faculties. It can be seen that articles of the top 10 universities have an average of at least 24 pages, while some of the lower ranked universities only have averages of 20 or less (note: it was not possible to consider that journals have different layouts, ie word numbers per page). But there is also some variation: eg the LSE has an average of 29.5 while Oxford has 25, and the university with the highest average is actually Ulster with 31.1 (followed by Queen Mary, the LSE and Nottingham).
Update: checking the Ulster data I saw a few outliers (namely, very long US law review articles). Thus, it may actually be better to use the median instead the mean. Result: correlation almost identical (0.74). The law schools with highest page numbers are: LSE (27), Queen Mary (26), Cardiff, Birmingham and Liverpool (25). Further comment: a tempting suggestion may be that for funding purposes page numbers should be seen as an easy proxy; but, of course, if known, this would set completely wrong incentives; rather, a smart strategy would be if HEFCE used the individual RAE 2008 data (not publicly disclosed) to spot something else that is highly correlated with the results (whatever this may be - average length of words etc?), not disclose this correlation and then use it for the REF 2014 scores…. would save a lot of resources!

Second, co-authorship: I would have expected a positive relationship as well since writing together should improve the quality of your piece (and I would think that working with people from other disciplines in particular is more frequent in the top universities). But then, the figure is quite messy and the relationship even slightly negative (-0.11). So, some of the lower ranked universities have many co-authored pieces – perhaps felt necessary to get a decent number of RAE outputs at all. There is also some interesting variation within the group of top universities: the LSE submitted only 4% co-authored pieces in law, Oxford 11% but UCL, Durham, and Nottingham between 23 and 27%.

Thursday, 25 October 2012

UK Law Journal Ranking (RAE 2008 data)

To explain in advance, I only considered the 30 journals which were submitted most often to the law RAE 2008 (then things get too unreliable since we just have less than 25 submissions). I correlated the number of law journal submissions per university (as % of all law journal submissions of this university) with the law RAE 2008 scores (note: that’s a bit different from what they did at Stirling for the RAE 2001; but similar to my previous post). So, here they are (with correlations in brackets):
  1. Common Market Law Review (0.4287)
  2. European Journal of International Law (0.4118)
  3. Modern Law Review (0.4046)
  4. Law Quarterly Review (0.3612)
  5. Oxford Journal of Legal Studies (0.3460)
  6. International and Comparative Law Quarterly (0.3357)
  7. European Law Journal (0.3152)
  8. Cambridge Law Journal (0.3076)
  9. Legal Studies (0.3008)
  10. Social and Legal Studies (0.2873)
  11. Feminist Legal Studies (0.2751)
  12. Journal of Law and Society (0.2734)
  13. Child and Family Law Quarterly (0.2511)
  14. British Journal of Criminology (0.2409)
  15. Journal of Environmental Law (0.2094)
  16. Intellectual Property Quarterly (0.1860)
  17. Northern Ireland Legal Quarterly (0.1705)
  18. Public Law (0.1267)
  19. Medical Law Review (0.0987)
  20. Law and Critique (0.0880)
  21. European Human Rights Law Review (0.0796)
  22. Criminal Law Review (0.0533)
  23. Industrial Law Journal (0.0155)
  24. Journal of Business Law (-0.0144)
  25. Journal of Social Welfare and Family Law (-0.0203)
  26. Conveyancer and Property Lawyer (-0.0429)
  27. Juridical Review (-0.0547)
  28. European Law Review (-0.0647)
  29. Edinburgh Law Review (-0.1036)
  30. Journal of Criminal Law (-0.2964)
To clarify again: any journal that’s not on this list may or may not be better than those journals (there just isn't enough data for those). Also, of course, the RAE 2008 law panel may have under-/overrated certain papers – I’m just reporting it in a completely unbiased way (actually, some of the journals I published in don’t score too well). And, as always, note that correlation does not imply causation.

Monday, 15 October 2012

Does the RAE/REF suppress international, European and comparative legal research?

Ok, this follows up from the previous post. I don’t want to over-do these correlations (and, yes, I’m aware of their limitations, simplicity etc), but this may still be interesting. One may hypothesise that in the RAE/REF the mainstream UK law journals are seen as the top journals – and international, European and comparative law journal only as a second choice being treated as ‘specialised’ publication outlets. - Now, the data (using the same method as explained in the previous post – searching for journal names with‘international’, ‘European’ and ‘comparative’ in the name):
It shows positive correlations for international (0.17) and European (0.22) and an almost flat line for comparative (-0.04). So, no, apparently international research is not penalised by the RAE/REF – or, of course, given the emphasis on ‘international excellence’ of the star rankings, the positive relationship could also be exactly what one would expect.

Monday, 8 October 2012

Does the RAE/REF suppress interdisciplinary research (in law)?

According to a recent paper by Rafols et al, the answer is ‘yes’, explicitly dealing with the negative impact of journal rankings in business studies and how those mattered for the RAE 2008. How is it in law? Though journal rankings are less common, a similar effect may be possible if publication in a mainstream law journal is seen as an indicator for good quality (even if this happens unconsciously). But, let’s have a look at the law data of the RAE 2008:
I took the 3691 journal articles submitted to the RAE and aimed to identify the propensity of each law school to submit interdisciplinary research. Since I didn’t want to classify all journals, I searched for all journals with “soci*” in the title, e.g., leading me to journals such as Law & Society Review but also sociological ones, in total 253 journal articles (admittedly, this underestimates the number of interdisciplinary papers but since it affected all law schools, I’d still suggest that this is a plausible proxy). In the figure above the x-axis shows these “soci” publications as a percentage of all submitted journal publications of each law school. And the y-axis indicates their RAE 2008 score.

So, now, what can be seen? Well, the correlation is positive (0.26), though fairly flat. In any case, I don’t see evidence that interdisciplinary research did not perform well in the RAE 2008 (and I can’t see why in the REF 2014 this would be different).

PS: the law schools with most absolute “soci” hits are Manchester, Bristol and Kent; and in percentage terms Birkbeck, Strathclyde and Wolverhampton.