Sunday, 26 May 2013

Legal indicators, Goodhart’s law and the risk of impact-relevant legal research

The World Bank's Doing Business Report (DBR) is a prominent example of the use of legal indicators. Many countries care about their performance in the DBR ranking, and some, such as Georgia  and Saudi Arabia, try to implement policies in order to rise in the ranking. But are such attempts useful? I’d suggest not. The DBR indicators are only proxies for ‘good’ law and can only cover a tiny percentage of laws. Thus, a country that changes its law according these indicators can quickly rise in the ranking, but this is fairly useless as far as they the vast rest of their legal system remains unchanged.

Thus, we have a legal variant of ‘Goodhart’s law’ (being akin to Heisenberg’s uncertainty principle). According to Goodhart’s law (dealing with macro-economic policy) ‘once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting social or economic policy, then it will lose the information content that would qualify it to play that role’. Replacing the words ‘social/economic’ with ‘legal’ this becomes ‘once a legal indicator is made a target for the purpose of conducting legal policy, then it will lose the information content that would qualify it to play that role’ and that’s precisely the aforementioned problem of the DBR.

I’m thinking about these topics as something similar may happen with one of my papers. In Schouten & Siems (2010) we compared the rules of significant shareholder ownership disclosure and found that almost all countries – but not Canada – oblige shareholders of listed companies who hold at least 5% of the shares to disclose their ownership. Though we did not suggest that countries should provide such rules, the Canadian Investor Relations Institute reproduced our table and lobbied for the introduction of a 5% threshold (document here). Now, the Canadian law-maker seems to follow suit, as the Canadian Securities Administrators propose such an amendment. Thus, inadvertently, our paper seems to have had an impact – yet, it also weakens the usefulness of our legal indicator … 

Sunday, 19 May 2013

A busy week in ‘Lurbridge’

I don’t usually do these ‘what I’ve done’ blogposts anymore, but this week that’s the only thing I can offer. So what does the title refer to? Well, marking exams (grr), but also:

Saturday, 11 May 2013

UK Law Journal Ranking (RAE 2008 data) - updated

I posted such a ranking a few months ago (here), but now I realised that I can improve it a bit. The previous ranking was based on the general RAE scores, but its better just to use the RAE ‘research outputs’ scores (here). Actually, the changes are fairly small – but the “number 1” has changed and some journals have gone up or done one place. So, here it is:

1. Modern Law Review: 0.4500
2. Common Market Law Review: 0.4252
3. European Journal of International Law: 0.4054
4. Law Quarterly Review: 0.3479
5. Oxford Journal of Legal Studies: 0.3462
6. International and Comparative Law Quarterly: 0.3221
7. European Law Journal: 0.3151
8. Legal Studies: 0.3058
9. Cambridge Law Journal:  0.3049
10. Social and Legal Studies: 0.2941
11. Journal of Law and Society: 0.2878
12. Feminist Legal Studies: 0.2511
13. Child and Family Law Quarterly: 0.2454
14. Journal of Environmental Law: 0.2264
15. British Journal of Criminology: 0.2133
16. Northern Ireland Legal Quarterly: 0.1828
17. Intellectual Property Quarterly: 0.1776
18. Public Law: 0.1352
19. Medical Law Review: 0.1011
20. Law and Critique: 0.0785
21. European Human Rights Law Review: 0.0627
22. Criminal Law Review: 0.0571
23. Industrial Law Journal: 0.0435
24. Journal of Social Welfare and Family Law:  0.0074
25. Journal of Business Law: -0.0027
26. Juridical Review: -0.0266
27. European Law Review:  -0.0680
28. Conveyancer and Property Lawyer: -0.0704
29. Edinburgh Law Review: -0.1048
30. Journal of Criminal Law: -0.2795

Method and limitations: I only considered the 30 journals which were submitted most often to the RAE 2008 (the final one was Law and Critique with 26 submissions; the next one, Civil Justice Quarterly, just had 22 submssions; so this seemed to me a good cutoff point). Then, I correlated the number of submissions per university per journal with the RAE 2008 research output score (note: that’s a bit different from what they did at Stirling for the RAE 2001 here). To clarify: any journal that’s not on this list may or may not be better than those journals (there just isn't enough data for those). Also, of course, the RAE 2008 law panel may have under-/overrated certain papers – I’m just reporting it in a completely unbiased way (actually, some of the journals I published in don’t score too well). And, as always, note that correlation does not imply causation.

Friday, 3 May 2013

What explains the RAE 2008 law output results?

A few months ago I posted some correlations using the RAE 2008 law data (cited below). Now, I thought why not run some basic OLS regressions? – with any suggestions welcome! The dependent variable is the average score for each law unit in the ‘Sub-profile research-outputs’ of the RAE 2008 (available here). Then, I considered a number of variables, all at the level of each law unit (e.g., something like: how many books, in % of all outputs, did a particular law school submit to the RAE?). A   couple of these variables were somehow significant, here in bold (with a decent R2 of 0.783; n = 67):

Explanatory variables


books (in %)
‘proper’ articles (in %)
‘proper’ chapters (in %)
median of journal pages
co-authorship (in %)
comparative journals (in %)
European journals (in %)
international journals (in %)
‘soci’ journal (in %)
Top UK general journals (in %)

I looked at the following possible explanatory variables:
  • As in a previous post (here), I examined different types of research outputs, also considering whether articles or book chapters are “proper” ones, or just shorter notes – with the threshold being 22 pages. It can be seen that having proper journal articles and proper book chapters is highly significant and positive (while we can’t say anything about books; short articles/book chapters were the reference category)
  • As previously discussed (here), I also considered median article length and co-authorship. The former is weakly significant (and positive), even in a regression which also has the variable on ‘proper’ articles, but not the latter.
  • Next, as previously (here, here), I examined whether comparative, European, international and interdisciplinary legal research are treated unfavourable by the RAE. Yet, in none of these cases there is a negative effect. And for articles that have a European or a socio-legal dimension there is even a weakly significant positive one.
  • Finally, publishing in the five established general UK law journals (CLJ, LS, LQR, MLR, OJLS) has a strongly significant positive effect. However, as previously shown, the ‘top 2’ law journals of the RAE 2008 are actually two other journals (here)
I did not consider two further factors which may be seen as important: grant income and types of universities (see the predictions by Chris Hanretty here). Naturally, these matter for the RAE data on research environment and research grants. Yet, the present calculations are specifically and only based on the results for the research outputs: here, these factors may be indirectly relevant - eg, having grants, and having time for research & hiring top researchers (ie differences between research and teaching-focussed universities) may lead to ‘better’ outputs (eg, longer articles etc). So the causality may be something like “grants & good university” à “good outputs” à high RAE output score – while the purpose of my regressions was only to consider the second, not the first, causal relationship.