Research on Open Data and Transparency

Leave a comment

FOI Commission Evidence from Ben Worthy

Indy Commission FoI

Below is my evidence to the Independent FOI Commission-you can see it in pdf here FOI Commission Submission 2015 Ben Worthy. There is also some interesting evidence from the ICO here

Dr Ben Worthy

Birkbeck College, University of London

 FOI Commission Evidence

All Freedom of Information laws are dynamic and uncertain. Over time they are changed by use, legal rulings and political decisions. The effect of any FOI law is both administrative, in terms of changing how the law works, but also political, in terms of having consequences for government. The complexity of the laws mean any reforms can have unpredictable effects.

Q1. Internal Deliberations

1.1. In terms of numbers, most FOI requests seek factual information and few that go either to local or central government concern ‘decision-making’ processes.[1] Those that do are likely to attract disproportionate attention and shape views. Looking at the section 35 and 36 exemptions, there remains, and may always be, uncertainty.[2] However, the Commissioner and Tribunal have sought to protect ‘safe space’, dependent on the time period and sensitivity of the information. As the Information Commissioner recently pointed out, a ‘significant percentage’ of decisions favours withholding.[3]

1.2 Judging the working of exemptions is problematic but two pieces of evidence indicate relatively stable functioning. First, looking at the frequency of exemption use since 2005, section 35 and 36 have been used at a steady rate and, after an early burst of concern, have not risen or obviously ‘spiked’. In 2015 they were the 10th and 11th most used exemption, in 2014 were 8th and 11th with similar levels before going back to 2007. [4]

1.3 Second, the constrained use of the veto may also be an indicator of exemption stability. The Justice Committee recommended s.53 use to ‘protect’ space

…we remind everyone involved in both using and determining that space that the Act was intended to protect high-level policy discussions. We also recognise that the realities of Government mean that the ministerial veto will have to be used from time to time to protect that space[5].

1.4 In terms of the Public Interest Tests built in to section 35 and 36, as Professor Peter Hennessy pointed out in 2012, the two PITs are a symbol of the openness of the UK FOI regime:

The question is, can you so delineate the safe area that the uncertainty goes and everybody knows where they stand? It is very difficult because there has to be a public interest defence in all this-there has got to be-but, also, there has to be a safe house.[6]

1.5 Given the uncertainty of evidence around any ‘chilling’, these parts of the Act should, by default, remain as they are. The alternative would be to reach for an over restrictive change that could leave out any possibility of access.

Chilling Effect

1.6 The discussion of safe space links to that of a chilling effect. This has been a persistent story around FOI since at least the 1980s, when it was used as an argument against the FOI policy being developed in Australia.[7] It is a term for a bundle of claimed effects including:

  • Non-use of formal recording (i.e. minutes) and move to informal modes or those outside the ambit of the Act (particularly the telephone or post-in-note)
  • Reduction in ‘free and frank’ discussion
  • Reduction in free and frank advice
  • Erosion of the anonymity of civil servants


1.7 The Justice Committee ‘was not able to conclude, with any certainty, that a chilling effect has resulted from the FOI Act’.[8] Research highlighted some nervousness and shifts in behaviour at local government level in Scotland and a MORI survey for the MOJ indicated ‘that some people were recording less information and that internal communications had become less detailed and informative than before FOIA’.[9]

However, other research across local government in England pointed to a few cases but no general trend.[10] A 2009 study of central government policy making discovered some negative views and concerns but no ‘change in the substance of government policy making or decisions’.[11]

1.8 Our own studies across central and local government between 2008 and 2011 were able to discover only a few clear examples, which were minor and isolated. There was no systematic or large scale changes to either minutes or free and frank discussion as a result of FOI.[12] Some officials were more concerned over the consequences of not having a record should there be a judicial review.

1.9 Our own studies and others found FOI also had a positive effect, professionalising records as a ‘disciplining’ rather than a ‘chilling’ effect.[13] The MORI survey also identified

…evidence to suggest that FOIA has had benefits for public authorities in encouraging more professional communications, more focused record-keeping and adherence to best practice in decision-making.[14]

Our 2010 study concluded FOI has had no impact on the anonymity or advice of officials. It also found that the Phillips Review of 2000 into BSE revealed identities at a lower level than any FOI request.


1.10 Similarly, evidence from elsewhere points to an occasional ‘chilling’ but one that is not systematic. One study in New Zealand found some changes in politically sensitive cases and there was evidence in Canada.[15]

1.11 However, an early study of Australia, Canada and New Zealand discovered no change to ministerial advice or any ‘post-it-note’ culture while noting it was a persistent myth. In the largest study of transparency laws yet undertaken in the world there was ‘little evidence’ for any alterations to files or records across all levels of government in India.[16]

1.12 There were also signs of a positive, professionalizing effect. The ALRC in Australia found FOI had helped ‘discipline’ communications and ‘focused decision-makers minds’ and in New Zealand 15 years of the Official Information Act had ‘improved the quality’ of policy advice.[17] Even when asked directly, those within public bodies appear uncertain. A survey of Irish local government found 30 per cent of local officials claimed a chilling effect and just fewer than 50 per cent denied it.[18]


Is there an effect?

1.13 There are two problems with offering any firm conclusions. The first is the measurement problem of proving or disproving that a ‘chilling’ is taking place. Anecdote is plentiful but finding hard evidence for such an effect is by its nature very difficult, as it requires proving a ‘negative’ and asking interviewees to admit unprofessional conduct.

1.14 Second, more importantly, the claim of a direct link between FOI and a ‘chilling’ ignores the many other powerful forces acting upon record quality and decision-making. The effects of FOI are frequently conflated, for example, with leaks. Gus O’ Donnell spoke of how for some records ‘we tended to put it in rather plain prosaic language because there could be leaks’.[19] Our studies found emails and electronic communications have had a huge effect on the nature of record keeping, far greater than FOI. Many interviewees were keen to point out the impact of the broader politics of what is recorded, relating to decision-making styles and natural political caution. As the Justice committee pointed out, the Butler report of 2004 raised concerns over informal meetings and sofa government before FOI became operational.

1.15 Consequently, the balance of evidence is that a chilling happens occasionally at the margins but it is not widespread. The difficulty is that the story of a chilling itself may influence behaviour change and becomes self-reinforcing and, as the Information Commissioner called it, a ‘self-confirming myth’.[20]

Q2. Collective Discussion

2.1 Few FOI requests are made for Cabinet documents and very few are released. FOI appears to have had no effect, with one detailed study concluding that, while influencing abstract discussion, it had no impact on how records were kept.[21]Gus O’ Donnell also observed that at ‘a formal meeting like a committee meeting or a Cabinet meeting…we did not reduce the coverage of…minutes. They were accurate’.[22]

2.2 Generally, our 2010 study found that Cabinet discussion was far more likely to be opened up by leaks than any FOI. As Gus O’Donnell pointed out, Cabinet confidentiality

….is affected by all sorts of things: first, the amount of leaking that goes on from different Cabinet members; and, secondly, the propensity of a number of the Cabinet members to write their memoires rather quickly and include things they probably should not.[23]

A further difficulty is that the principle of Cabinet confidentiality and unanimity is not a ‘rigid dogma’ but a flexible instrument, famously summed up James Callaghan’s phrase: ‘You know the difference between leaking and briefing. Leaking is what you do and briefing is what I do’.[24]

2.3 Given the lack of evidence of any decisive change in Cabinet operations, and the move by the appeal systems to mirror the 20 year rule, the protections should remain as they are.

Question 3: Risks

3.1 Given the lack of evidence, it is unclear what effect release has upon risk assessments. It is likely there would be similar uncertainty as there is around proving the ‘chilling’ more generally.

Question 4: Veto

4.1 The veto was the lynchpin of the FOI revisions during the laws development.[25] Looking across the last decade, the UK veto has been rarely used, especially when compared with other FOI regimes. This may be as a result of the successful functioning of exemptions elsewhere lower down the system. Jack Straw argued to the Justice Committee that there would be a political reluctance to use what is the ‘ultimate’ power. The veto has a clear ‘backlash potential’ as it naturally draws attention to the particular topic, generating headlines and making the government appear secretive.

Comparative Veto use in the First Four years of FOI systems[26]

Jurisdiction Veto use in first four years
Australia 48
New Zealand 14
Ireland 2
UK 0


4.3 Vetoes are common, in various forms, across different systems. New Zealand effectively removed its veto power in 1987 and Australia similarly abolished their equivalent in 2009. While former PM Geoffrey Palmer felt the veto change in New Zealand had no effect the consequences of the changes in Australia are still being examined.[27]

4.4 Before the Supreme Court ruling the veto worked in a sparing way. Any future veto power should be kept as close as possible to the precise, limited and ‘exceptional’ model that existed previously.

Q5. The Appeal System

5.1 The UK appeal system had suffered a series of problems common across FOI regimes, especially over delays. However, it appears to have developed into a robust and powerful part of the FOI process. International research has shown how all the varied approaches bring costs and benefits and, given the lack of any obvious better model, the UK appeal system should be kept in its present form.[28]


Q6. Burden

6.1 Measuring the ‘cost’ of FOI is problematic as it involves balancing administrative resources against democratic benefits. Moreover, the exact cost of FOI is very unclear and any figure, high or low, can be challenged.

Attempts to Measure the Cost of FOI in the UK 2004-2011[29]

Study Estimated cost per request
A pre-Act estimate £350


Frontier Economics £293
Cornwall council £150
Bexley council £36 with most requests costing around £19.
Scottish government £193
MOJ estimate £164


6.2 FOI requests are also a moving target as, on the one hand they become more elaborate but, on the other, public authorities deal more efficiently with them – annual surveys by UCL between 2005 and 2010 found a sharp drop in time taken for organisations to process requests, falling by more than 50% in 5 years.[30] Other studies sought to calculate any costs savings emerging from FOI requests through, for example, resources saved from cancellation of policy. There is also the issue of context and, famously, the claims of costs triggered indirectly by US litigation launched by FOI were found to be the equivalent of the amount spent by the US military each year on marching bands.[31]


6.3 The UK Freedom of Information Act is currently, more or less, free to use. Most FOI regimes, from India to the US, have a standard application fee that is charged for most (but not all) requests. However, almost all these charges have been part of the system from the start. Although some regional or state level openness regimes have introduced a fee, only one country, Ireland, went from having no charge to charging in 2003 and then abolishing fees again in 2014.

6.4 The difficulty with fees for governments are both practical and symbolic. One clear practical effect in Ireland after 2003 was, according to the Irish Information Commissioner, a steep fall in [non-personal] requests of 75% in a single year. [32] Certain groups such as MPs and journalists saw a particularly sharp decline. A decade later in 2013, a year before fees were abolished, requests remained at only half of their pre-fee level and represented a ‘tangible barrier’ to ordinary requesters.[33]

6.5 It’s not clear whether a government can claw back any costs. Governments make the case that FOI costs money so the fee goes towards offsetting the resources used but in Ireland, Nat O’Connor concluded that the fees recouped only 1.6 % of the estimated cost.[34]

6.6 There are also questions about implementation. In a number of Australian states public bodies simply didn’t bother to charge if a request was small, as it was cheaper simply to send it out.[35]. Evidence from the UK at local government level, which is the focus of 70-80% of all FOI requests, is that charges for FOI in any form are very rare and it is likely that any ‘fee’ would not be charged. Our research discovered that few local authorities strictly abide with cost limits and most simply process anything reasonable-so changes to cost limits may face a similar non-operability problem.[36] Equally, some requesters may seek to find ways around innovative ways around a fee or even, a seen across the world, crowd-fund requests.[37]

6.7 Politically, as one academic put it, transparency is a ‘contested political issue that masquerades as an administrative tool’.[38] The introduction of up-front fees would be politically difficult and, as occurred in Ireland, would be seen as a ‘signal’ of a government’s negative attitude towards openness. In Ireland fees rapidly became a contentious, party political issue and were reversed when the opposition parties came into power.

6.8 Given the evidence, fees are too blunt an instrument. It would be recommended to keep FOI free, given the practical and political difficulties and, most importantly, the unintended effects on large groups of requesters of limiting public access.


[1] See Hazell, R., Worthy, B., & Glover, M. (2010). The Impact of the Freedom of Information Act on Central Government in the UK. Palgrave: London and Worthy, Ben (2010) ‘More Open But Not More Trusted? The Effect of FOI on the UK Central Government’. Governance 23 (4) 561-582

[2]Justice Committee (2012) q267,

[3] ICO speech Oct 2015

[4] See IFG analysis and, the Ministry of Justice [MOJ] (2011). Memorandum to the Justice Select Committee. TSO: London and Hazell et al (2010).

[5] Justice (2012) para 201

[6] Justice Committee (2012) q255

[7] See Hazell, R. (1989). ‘Freedom of Information in Australia, Canada and New Zealand’. Public Administration, 67(2), 189-210.

[8] Justice Committee (2012) para 180-200

[9]Justice Committee (2012) para 216

[10] See Taylor, J., & Burt, E. (2010). ‘How do public bodies respond to freedom of information legislation? Administration, modernisation and democratisation’. Policy & Politics, 38(1), 119-134 and Shepherd, E., Stevenson, A., & Flinn, A. (2011). ‘Records management in English local government: the effect of freedom of information’. Records Management Journal, 21 (2), 122-134.

[11] See Waller, P., Morris, R.,Simpson, D. and Hazell, R (2009). Understanding the Formulation and Development of Government Policy in the context of FOI. London: Constitution Unit. Waller et al p.60

[12] Hazell, R., Worthy, B., & Glover, M. (2010). The Impact of the Freedom of Information Act on Central Government in the UK. Palgrave: London and Worthy, Ben (2010) ‘More Open But Not More Trusted? The Effect of FOI on the UK Central Government’. Governance 23 (4): 561-582

[13] Richter, P. & Wilson, R., (2013). ‘“It’s the tip of the iceberg’: the hidden tensions between theory, policy and practice in the management of Freedom of Information in English local government bodies—evidence from a regional study’. Public Money & Management, 33(3): 177–184.

[14] MOJ , p48

[15] White, Nicola. (2007). Free and Frank: Making the Official Information Act Work Better. Wellington: Institute of Policy Studies and Roberts, A. S. (2005). ‘Spin control and freedom of information: lessons for the United Kingdom from Canada’. Public Administration, 83(1), 1-23.

[16] Hazell, R. (1989). ‘Freedom of Information in Australia, Canada and New Zealand’. Public Administration, 67(2), 189-210 and in India see the Right to Information Assessment and Analysis Group and National Campaign for People’s Right to Information (Raag/NCPRI) (2009) Safeguarding the Right to Information – Report of the People’s RTI Assessment 2008. New Delhi: NCPRI

[17] See the ALRC/ARC [Australian Law Reform Commission and Administrative Review Council Committee]. 1995. Open Government: A Review of the Federal Freedom of Information Act 1982. Canberra: AGPS and in New Zealand Law Commission (New Zealand). (2010). The Public’s Right to Know: A review of the Official Information Act 1982 and Parts 1–6 of the Local Government Official Information and Meetings Act 1987. Wellington: Law Commission [and later 2012 report].

[18] See McDonagh, M. (2010). ‘Access to Local Government Information in Ireland: Attitudes of Decision-Makers’. Open Government: A Journal on Freedom of Information, 6 (6), 1-20[18]

[19] See Gus O’Donnell’s evidence to the Justice Committee q260

[20] ICO speech Oct 2015

[21] Waller, P., Morris, R.,Simpson, D. and Hazell, R (2009). Understanding the Formulation and Development of Government Policy in the context of FOI. London: Constitution Unit.

[22]See Gus O’Donnell’s evidence to the Justice Committee 260

[23]See Gus O’Donnell’s evidence to the Justice Committee q266

[24] See House of Commons Library (2004). Collective Responsibility of Ministers. London: TSO.


[25] See Worthy, Ben (2016) The Politics of Freedom of Information: How and Why Governments Pass Laws That Threaten Their Power. Manchester: MUP

[26] See Hazell et al 2010 and CFOI paper on the veto as well as House of Commons Library (2014) FoI and Ministerial vetoes. London: TSO.

[27] See on New Zealand Palmer, G. (2007). A Hard Look at the New Zealand experience with the Official Information Act after 25 years. Paper presented at the International Conference of Information Commissioners Wellington. and in Australia see Moon, Danielle (2015). A Matter of Balance? Freedom of Information and Deliberative Documents. Macuire University: MA thesis.

[28] See the debate in Holsen, S., & Pasquier, M. (2012). ‘Insight on Oversight: The Role of Information Commissioners in the Implementation of Access to Information Policies’. Journal of Information Policy, 2, 214-241.

[29] Colquhoun, A. (2010). The Cost of Freedom of Information. London: Constitution Unit.

[30] See the summary of the UCL’s local government surveys 2005-2010

[31] O’Connor, N (2010): An Economic Argument for Stronger Freedom of Information Laws in Ireland, TASC Discussion Paper and Wald, P. M. (1984). ‘Freedom of Information Act: A Short Case Study in the Perils and Paybacks of Legislating Democratic Values’, The. Emory Lj, 33.

[32] See OIC (2004) Review of the Operation of the Freedom of Information (Amendment) Act 2003 and this piece by Martin Rosenbaum . The read across between UK and Irish FOI is limited as Ireland does not have a Data Protection law with subject access rights so FOI is heavily used to get personal records-hence the distinction between personal and non-personal.

[33] O’Connor, N (2010): An Economic Argument for Stronger Freedom of Information Laws in Ireland, TASC Discussion Paper

[34] O’Connor, N (2010): An Economic Argument for Stronger Freedom of Information Laws in Ireland, TASC Discussion Paper

[35] O’Connor, N (2010): An Economic Argument for Stronger Freedom of Information Laws in Ireland, TASC Discussion Paper

[36] See Worthy, Ben, Hazell, Robert, Amos, Jim and Bourke, Gabrielle (2011) ‘Town Hall Transparency? The Impact of FOI on Local Government in England’. Constitution Unit: London

[37] See Worthy, B. (2013). “Some are More Open than Others”: Comparing the Impact of the Freedom of Information Act 2000 on Local and Central Government in the UK. Journal of Comparative Policy Analysis: Research and Practice, 15(5), 395-414 and for more on local government see Worthy, Ben, Hazell, Robert, Amos, Jim and Bourke, Gabrielle (2011) ‘Town Hall Transparency? The Impact of FOI on Local Government in England’. Constitution Unit: London. You can see examples of FOI crowd-funding in this article and this US FOI Machine.

[38] See Fenster, M. (2012). ‘The transparency fix: advocating legal rights and their alternatives in the pursuit of a visible state’. University of Pittsburgh Law Review, 73(3).

Leave a comment

A New Front? FOI and Universities


According to the Guardian, the government is now floating the possibility of excluding universities from FOI. The green paper on Higher Education states on pg. 69:

Public body requirements
17. There are a number of requirements placed on HEFCE-funded providers which do not apply to alternative providers. Many derive from treating HEFCE-funded providers as ‘public bodies’. This is despite the fact that the income of nearly all of these providers is no longer principally from direct grant and tuition fee income is not treated as public funding. Alternative providers are not treated as public bodies. As a result there is an uneven playing field in terms of costs and responsibilities. For example, the cost to providers of being within the scope of the Freedom of Information Act is estimated at around £10m per year.

18. In principle, we want to see all higher education providers subject to the same requirements, and wherever possible we are seeking to reduce burdens and deregulate. However we may wish to consider some exceptions to this general rule if it were in the interest of students and the wider public.

You can see the paper here. FOI controversy has circled around universities for some time now, over climate change, level playing fields and the broader issue of access to research data. In 2014 section 22 of the Act was amended to give universities a research exemption and there has been a constant ‘hum’ of debate about whether they should be completely removed from the Act given that private universities are not covered. Is this a new front in the battle over FOI?

For more on FOI and academia you can see the Constitution Unit’s 2012 survey of university FOI officers here and this short article Worthy_et_al-2012-Political_Insight


Leave a comment

Battle lines Being Drawn?


Last week, Leader of the House of Commons Chris Grayling made a series of comments on FOI use by journalists:

It is, on occasion, misused by those who use it as, effectively, a research tool to generate stories for the media, and that is not acceptable. It is a legitimate and important tool for those who want to understand why and how governments make decisions, and this government does not intend to change that

So Matt Burgess (and others) responded with some examples of how FOI requests by journalists has made a difference-there’s 103 examples here. You can see here for some analysis of the evdience of who uses FOI.

Now the head of Wolverhampton council, Keith Ireland, has waded in at a Scrutiny Board Meeting

The vast majority of requests come from media across the country, be that the BBC, local media, or media in general.They come from people who are out to create trouble for councils and students who are too lazy to do their own research. Others come from big companies who can’t be bothered to look up the data and want to know when contracts are on for re-evaluation. It is a really costly exercise. The original principal of FOI is not what is happening in reality.

He claimed, in addition, that it cost around £500,000 to process requests [see here for the tricky topic of how much FOI costs]. Just to make life even more interesting (and FOI even more political), it appears Labour may be setting up its own FOI Commission