Saturday, 31 January 2015

The ‘state of the art’ of legal scholarship? - Reflections on the rhetoric of the REF 2014 report

Many of us have read with great interest how the REF panel report summarised the ‘state of the art’ of legal scholarship in the UK (see pp 70-75 of the Panel C Report). It is the aim of this post to reflect on some of the rhetoric of this report (noting that sometimes the most important things are those left unsaid). So, here we go:
3. The sub-panel received outputs judged to be of world-leading quality from over 85 per cent of submissions and the overwhelming majority of work submitted was of at least internationally recognised quality providing valuable knowledge to the field, confirming that excellent research and scholarship are being conducted in very many law schools across the UK
Ok, great to hear, but, of course, most of the panel members were UK law professors too …
4. The sub-panel firmly shares the view of its predecessors that peer review remains the most reliable method of assessing research quality in law.
Meaning that panel thinks that they did an excellent job, ie trust our judgement!
[id] We found instances of world-leading or internationally excellent quality regardless of the form of the output.
That’s a very diplomatic statement. Yes, it may be fair to say that there were ‘instances’ of world-leading research in books, book chapters, journal articles etc. But the crucial - and unanswered - issue is whether in average one of these output types did not receive higher scores.
6. There was a large volume of outputs covering several core areas of law, including public international law, criminal law, criminal justice and criminology, legal theory (broadly defined and covering a wide range of approaches), public law and human rights, commercial law and EU law. From its assessment of the sub-fields included in submissions, the sub-panel considers that law and religion, intellectual property and IT law, transitional justice, gender, sexuality and the law, and law and finance all appear to be developing fields of interest…. We found numerous examples of world-leading quality across the range of outputs submitted.
Again, a very diplomatic statement. Here too, the panel could have done some number-crunching in order to see whether publications in some areas of law did better than others. PS: interesting how they identify the ‘core’ (eg, international and EU law included but not intellectual property).
8. The sub-panel regards it as an indicator of the strength of the discipline that world-leading work was demonstrated across the wide variety of methodologies employed in legal research and scholarship including doctrinal, theoretical, contextual, historical, socio-legal, critical and empirical approaches. Bodies of knowledge and thought were drawn upon from across the social sciences and the humanities. ….9. The sub-panel views this trend towards methodological diversity and interdisciplinarity as evidence of the growing sophistication of legal scholarship as a body of knowledge and understanding with wide-ranging insights, impacts and implications for the social world…
Ok, methodological diversity is great but is not something missing? Legal scholarship can be thought to operate in a triangle of social sciences, humanities and legal practice. The report highlights the first two elements but it does not seem to regard practical legal scholarship as being of high REF quality.
13. The impact of research was usually demonstrated by how it had informed the development of policy and new legislation and influenced the work of national, EU and international policy makers, judges and legislators… There was less consideration of the importance of research in improving NGO campaigning and input to policy or service provision, and less mention of the impact on end service users.
This issue can be related to the previous comment, ie research which provides practical help may not be rated highly as an output. In addition, it may show that legal scholars like to think in ‘macro categories’ (legislation, case law, law enforcement etc); thus, this can be read as an encouragement to collaborate with disciplines such as psychology and anthropology in order to understand individual behaviour.
22. A number of units set out structures involving the organisation of research into groups, centres or clusters and provided convincing examples of how these have stimulated and fostered vibrant research cultures. However, in some cases it was not always clear how these were providing meaningful support for research and structures sometimes appeared rather over-elaborate and confusing…..
That’s an interesting statement given that the proliferation of those clusters may often be driven by universities’ desire to show to the REF that they promote excellence in a particular area (and not ‘just’ across the board). But it seems that this may sometimes be counter-productive.
[id] The sub-panel saw impressive examples of research environments regardless of the size of the unit, the organisation of its work or its strategic focus and is convinced that there is no one strategic model for a larger or smaller unit which will necessarily deliver research success.
So, this statement seems to challenge the ‘size matters’ trend, eg the biggish Doctoral Training Centres.
26… The trend, noted above, to a more blended, multi- and interdisciplinary approach in research production in some aspects of law was reflected in the description of a number of multi-departmental and multi-institution networks and groupings providing both critical mass and the pooling of infrastructural resources for the production of programmes of research.
That’s an interesting qualification of the previous statements - ie in this respect clusters are indeed useful and size may matter, also saying that form and substance need to go hand in hand.

Monday, 19 January 2015

Law Journal Ranking (based on REF 2014)

Ranking of 40 law journals based on REF data (further explanations below):
1. Current Legal Problems (3.001928)
2. Law Quarterly Review (3.0)
3. Lloyd’s Maritime & Commercial Law Quarterly (2.965556)
4. Oxford Journal of Legal Studies (2.96061)
5. Modern Law Review (2.950283)
6. Cambridge Law Journal (2.948772)
7. International Journal of Constitutional Law (2.945333)
8. Yearbook of European Law (2.9435)
9. Journal of Corporate Law Studies (2.940556)
10. Common Market Law Review (2.934118)
11. Journal of Legal History (2.922667)
12. Medical Law Review (2.922444)
13. International Journal of Law in Context (2.914815)
14. Social and Legal Studies (2.914138)
15. Journal of Private International Law (2.9135)
16. Public Law (2.898929)
17. International and Comparative Law Quarterly (2.898)
18. Journal of International Economic Law (2.89722)
19. European Journal of International Law (2.895862)
20. Intellectual Property Quarterly (2.888421)
21. Human Rights Law Review (2.888065)
22. Child and Family Law Quarterly (2.88619)
23. King’s Law Journal (2.88375)
24. Journal of Law and Society (2.876528)
25. Industrial Law Journal (2.869615)
26. Legal Studies (2.862065)
27. European Law Review (2.861277)
28. Criminology and Criminal Justice (2.853333)
28. Journal of Environmental Law (2.853333)
30. British Journal of Criminology (2.838689)
31. Edinburgh Law Review (2.793125)
32. Leiden Journal of International Law (2.787333)
33. Journal of Social Welfare and Family Law (2.77619)
34. Criminal Law Review (2.764634)
35. Journal of Business Law (2.76375)
36. European Law Journal (2.760938)
37. International Journal of Human Rights (2.711765)
38. Northern Ireland Legal Quarterly (2.71)
39. Howard Journal of Criminal Justice (2.606667)
40. Journal of Criminal Law (2.259231)
Explanations:
  • I used the REF data for the output GPAs and the journal submissions per law school. I included the 66 law schools that took part in the REF 2014 and where output data are available (in addition, London Met took part but the output data show ‘-‘).
  • I considered the 40 journals most frequently submitted to Law’s REF 2014 (see here); for other journals the data would be too unreliable, ie those journals may be higher or lower ranked than the 40 (we just don’t know).
  • I used the following approach (a bit different from what I did previously for the RAE 2008, see here): for each journal (i) I counted how often each law school submitted articles from this journal, (ii) then I multiplied this number with the REF output GPA of this law school (as a proxy for the assessment of these articles given that we don’t know the precise rating of individual outputs), (iii) then I added the scores together and divided them by the total number of articles submitted from this journal à thus, the numbers in brackets indicate the estimated REF evaluation of articles from the respective journals.
  • PS: see here for a corresponding ranking of politics journals.

The Law Journals most frequently submitted to the REF 2014

All submissions to the REF 2014 have just been published. In the unit of assessment for law, 5523 outputs were submitted: 62.54% of those were journal articles, 22.07% book chapters, 13.51% books and 1.88% others. With respect to the journal articles, I just counted which journals were submitted most frequently – and here are the top 40:
1. Modern Law Review (105)
2. Legal Studies (88)
3. Oxford Journal of Legal Studies (81)
4. Journal of Law and Society (69)
5. Cambridge Law Journal (57)
6. Public Law (56)
7. Northern Ireland Legal Quarterly (54)
8. International and Comparative Law Quarterly (53)
9. Current Legal Problems (48)
10. European Law Review (47)
11. Medical Law Review (45)
12. British Journal of Criminology (44)
13. Criminal Law Review (42)
14. Journal of Business Law (40)
14. Child and Family Law Quarterly (40)
16. Law Quarterly Review (36)
17. Common Market Law Review (34)
18. European Law Journal (32)
19. Human Rights Law Review (31)
20. European Journal of International Law (29)
21. International Journal of Law in Context (27)
22. Journal of Criminal Law (26)
23. Industrial Law Journal (26)
24. Criminology and Criminal Justice (23)
25. Social and Legal Studies (22)
26. Journal of Social Welfare and Family Law (21)
27. Yearbook of European Law (20)
27. Journal of Private International Law (20)
29. Intellectual Property Quarterly (19)
29. Lloyd's Maritime and Commercial Law Quarterly (19)
29. Journal of International Economic Law (19)
29. Journal of Corporate Law Studies (19)
33. Journal of Environmental Law (18)
34. International Journal of Human Rights (17)
34. King's Law Journal (17)
36. International Journal of Constitutional Law (16)
36. Howard Journal of Criminal Justice (16)
36. Edinburgh Law Review (16)
39. Leiden Journal of International Law (15)
39. Journal of Legal History (15)
Of course, I’d not suggest that frequency = quality (though there may be some correlation). But, as readers of this blog may suspect, I’ll also do a quality calculation very soon (as I did with the RAE 2008 data). 

Friday, 9 January 2015

REF 2014: the relationship between inclusiveness and quality of submissions

The UK's REF 2014 allowed universities to choose how many of their members of staff to submit for assessment. How could they have made use of it (and what did they actually do)?

According to model (i) the outcome is mainly due to differences in strategy, namely that the universities that only submitted few staff did so to get a high rank in the REF results: thus, one would expect a negative relationship between quality (as measured by the GPA rank) and inclusiveness (in % of staff included). By contrast, model (ii) takes the view that all UK universities pursued a similar strategy (be as high in the ranking as possible, avoid low quality submissions); instead, the differences are due to the fact that less research-active universities just did not have enough good submissions they could make; thus, one would expect a positive relationship between quality and inclusiveness.

Now, the Figure shows the actual result based on the GPA quality ranks and the HESA data on % of staff included. The overall result is that of a fairly strong positive correlation (0.727), thus confirming model (ii). This finding also means that the GPA ranks remain meaningful since the ability to select only reinforced the differences between good and not-so-good universities (while adjusting the ranking for selectiveness is problematic since we just don’t know how non-submitted staff would have performed; cf also here and here).
  From the Figure it is also noticeable that the positive correlation seems mainly due to the difference between two groups of universities: high-ranked universities with high inclusiveness on the one hand (eg, the Russell Group universities), and low-ranked ones with low inclusiveness on the other (mainly the former polytechnics). Therefore, I also split the dataset into the 50% top and bottom ranked universities. Here the correlations between quality and inclusiveness are still positive; it is 0.423 for the universities ranked 65-128, but only 0.056 for the ones ranked 1-64. Thus, unsurprisingly, strategic considerations seem to have played a greater role for the top ranked universities.

Thursday, 1 January 2015

Identities (Knausgaard, Block & comparative law)

ICLMy non-law holiday reading included Knausgaard’s Boyhood Island and Block’s Second Language Identities: very different books but, coincidentally, also with a common theme. In Knausgaard’s book there is a nice section where he reflects on a childhood picture of him:
‘Is this creature the same person as the one sitting in Malmo writing? And will the forty-year-old creature […] be the same as the grey hunched geriatric […] in forty years from now […] Wouldn’t it be more natural to operate with several names since their identities and self-perceptions are so very different? Such that the foetus might be called Jens Ove, for example, and the infant Nils Ove, and the five-to ten-year-old Per Ove [etc…]. Then the first name would represent the distinctness of the age range, the middle name would represent continuity and the last name family affiliation.’
In Block’s book a frequent theme is that identity is not fixed but, to offer a quote as well, that it is about ‘negotiating new subject positions at the crossroads of the past, present and future’ – and this is then further discussed for the situation of second language learners / speakers.
  Reflecting on my research, there may be a link to comparative law as well. The insight that identities are often mixed / subject to change is something that can also be said about legal systems as they shift between legal families. For instance, Knausgaard’s naming suggestion may also work if we say that a country belongs to the traditions of ‘modern French civil law’, ‘the old English common law’ etc. - with ‘civil/common law’ being akin to the last name, ‘French/English’ being akin to the permanent middle name and ‘modern/old’ (or other adjectives) being akin to the altering first name.