Research on Open Data and Transparency

Leave a comment

Freedom of Information and openness- why bother? The past, present and future of transparency in the UK

 19th April 6.30pm Newspeak House, Bethnal Green

This event looks at why politicians push openness, how they try and back out of it and what happens once the policies are in place. It will look across FOI and Open Data in the UK and offer some thoughts on what may happen to the transparency agenda with Brexit.

The discussion coincides with the publication of Ben’s new book on this topic, ‘The Politics of Freedom of Information: How and Why Governments Pass Laws That Threaten Their Power’. The first chapter is available online here.

To register follow the link here


Leave a comment

Why pass FOI laws? The politics of freedom of information


Why don’t more politicians react to freedom of information (FOI) like Lyndon Johnson? Why don’t more of them run a mile when presented with the possibility of giving the public a legal right to ask for information from the government? When the idea of an FOI law was suggested to Johnson in 1966 by a fellow Democrat Congressman the US President responded, after some swearing, ‘I thought you were on my side?’ As his Press Secretary explained:

LBJ… hated the very idea of the Freedom of Information Act; hated the thought of journalists rummaging in government closets and opening government files; hated them challenging the official view of reality.

For any budding politician, FOI appears to be the ultimate political boomerang. It helps your opponents. It hinders you.

To make FOI laws even less appealing, there are no votes in them. Merlyn Rees, a Home Secretary who fought hard against an FOI law in the 1970s, once exclaimed that ‘the Guardian can go on for as long as it likes about open government… but I can tell you that in my own constituency of 75,000 electors I would be hard pressed to find many who would be interested’. Only in India, where the Right to information Act was part of an anti-corruption campaign, have FOI laws responded to broad public enthusiasm. So how is it that there are now more than 100 FOI laws around the world?

rti_infographic_-_worldwide_en_lang_-_for_website_cover2(Image courtesy of article

The question is really why would a politician support FOI in the first place? Sometimes they believe in openness and sometimes leaders who don’t believe in it have it forced upon them, as Theresa May has discovered over Brexit. Other times it is for pure advantage, because a scandal makes it hard to avoid (as in Ireland), so a politician can ensure that they get information in the future or because it has promised FOI as part of a coalition deal (as in India). It is also about context. Often FOI laws are pushed through when there is lots of other constitutional or legal change going on. Across the world, as Rick Snell points out, organised groups and enthusiastic individuals, often ‘outsiders’, push for an FOI law when other key people are distracted or looking the other way.

There is also the symbolism. Promising an FOI law sends out all sorts of positive messages of radicalism, change and empowerment that new governments find difficult to resist. This is especially the case for an opposition politician coming into power, as with Tony Blair after the ‘sleaze’ and secrecy of John Major. Committing to a law tells voters ‘we are different’ and also offers to give ‘the people’ a new right. FOI also carries a pleasing moral angle for politicians: ‘you can look inside as we’ve got nothing to hide (unlike the last lot) etc.’. All these reasons make FOI laws, at least on paper, hard to resist.

The problem for politicians is that they can overdo it. In 1996 Tony Blair gave a speech where he referred to FOI as ‘not just important in itself. It is part of bringing our politics up to date, of letting politics catch up with the aspirations of people’ and went on to say it would help involve more people in politics and increase trust in government. Blair’s words came back to haunt him, and stop him, when he wanted to water down the law.

The problem is that when the enthusiasm dims FOI is then hard to back out of. Politicians often display public support but have private regrets. Once in power there are fights behind closed doors as regretful enthusiasts and those who weren’t paying attention wake up and fight back: some laws are lost or put on hold and most are watered down. Parliaments, civil society and the media try to keep proposed laws alive. Those FOI laws that survive often emerge as a compromise between the hopes of campaigners and the fears of government.

Once FOI laws are up and running the split between supporters and opponents continues. Lyndon Johnson refused to have a photographer at the signing of the law. Tony Blair quaked at his own imbecility for championing it, devoting a medium sized rant in his autobiography to his own stupidity:

Freedom of Information. Three harmless words. I look at those words as I write them, and feel like shaking my head till it drops off my shoulders. You idiot. You naive, foolish, irresponsible nincompoop.

He went on to claim it was used only by journalists and mainly deployed as a ‘weapon’ (none of which is true). David Cameron, despite talking up Open Data, also felt FOI was a ‘buggeration factor’ and tried to reduce the strength of the law.

Yet outside of government FOI laws are popular. The laws have been used recently to find out, for example, about the hundreds of council tax arrears notices sent to councillors, ward-level Brexit voting figures and the number of patients stranded in hospitals. It has even led to the mass resignation of a parish council.

So FOI laws get here because they are hard to resist in opposition but hard to back out from in power. Even Lyndon Johnson signed the law and, as his Press Secretary pointed out, went and took all the credit for it afterwards.

The Politics of Freedom of Information: How and why governments pass laws that threaten their power, is published by Manchester University Press. You can read chapter one here and order the book here.This post was orginally on the Constitution Unit blog here


Leave a comment

The Politics of Freedom of Information-free sample chapter ‘FOI: hard to resist and hard to escape’


Here’s a sample of chapter 1 from my new book asking why would anyone be reckless enough to decide to pass an FOI Act?

Freedom of Information (FOI) laws are difficult to resist in opposition but hard to escape from once in power. A commitment to an FOI law sends out strong messages of radicalism, change and empowerment that new governments find difficult to resist. However, when politicians regret their promises, as they often quickly do,
the same symbolism makes the reforms difficult to escape from.

To make the picture more complex, FOI laws bring little external advantage and generate internal unhappiness. One of the central paradoxes of FOI laws is that they are symbolically resonant but useless in electoral terms: politicians gain ‘credibility’ but not votes. Within government, FOI laws reach across the whole of government, running against the natural tendency of bureaucracy to be secretive (Weber 1991). Such laws carry the potential to delve deep into bureaucracies’ work, triggering investigation of official decisions and procedures by those hostile to them. So how and why do governments pass them?

FOI laws are, it is argued, frequently passed out of naivety or inattention by inexperienced and new governments responding to reformist impulses from within or without or seeking to create a new ‘open’ approach after a scandal (Berliner 2014, Darch and Underwood 2010). Politicians have many motives for introducing FOI, from the simple politics of wrong footing or neutralising opponents to the longerterm, calculating intention of securing access to information when they are out of power (Berliner 2014). Context is also key, as laws are frequently passed amid wider change or as a response to a particular problem. As well as calculation and context there are a series of symbolic pressures. Politicians can, at least in the short term, earn a form of ‘moral capital’ from supporting openness (Birchall 2014; Michener 2009).

Read the rest here worthy-chapter-1-2

Leave a comment

Freedom of Information — why bother?


‘Freedom of Information. Three harmless words. I look at those words as I write them, and feel like shaking my head till it drops off my shoulders. You idiot. You naive, foolish, irresponsible nincompoop. There is really no description of stupidity, no matter how vivid, that is adequate. I quake at the imbecility of it.”’ – Tony Blair.

Why do governments bother to pass Freedom of information laws? Here’s my attempt to explain why they bother to do something that appears ‘so utterly undermining of sensible government’ (Blair again) foia-why-bother-ben-worthy-freedom-of-information. This is, by the way, a blatant preview for my forthcoming book.



Leave a comment

Does FOI Work?


We tried to find out with this experiment, with a little help from English parish councils. See the full article here.


‘How effective are systems of transparency, such as Freedom of Information (FOI) requests? The ambitious aims of FOI laws hinge on whether requests produce the desired information for the citizens or groups that use them. The question is whether such legally mandated requests work better than more informal mechanisms. Despite the high hopes of advocates, organizational routines, lack of awareness or resistance may limit legal access and public bodies may seek to comply minimally rather than behave in concordance with the spirit of the law. This article reports a field experiment that compared FOI requests and informal nonlegal asks to assess which is more effective in accessing information from English parish councils. The basic premise of statutory access is borne out. FOI requests are more effective than simple asks and the size or preexisting level of openness of a body appears to make little difference to their responsiveness. FOI requests are more effective in encouraging bodies to do more than the law asks (concordance) than encouraging more minimal levels of legal cooperation, when a body simply fulfils its obligations to varying degrees (compliance). This finding indicates high levels of support for FOI once it is embedded within the system.’

parish-council-4wSee our blog summary and also FOI man’s blog post and analysis.

Leave a comment

New Paper: Freedom of Information and the Media


The media are a powerful constituency of users, lobbyists and defenders of Freedom of Information (FOI) laws. Looking at FOI regimes across the world, it argues that the media are important users but also powerful innovators and defenders. This chapter examines how journalists use the laws in the UK and work to protect and extend it. It also looks at how media use is seen to damage trust in the political system and can generate resistance from government. It ends by arguing that FOI must be viewed in context and now fits within a rapidly changing information eco-system and a shifting and hybrid media environment.


Download here

Leave a comment

What’s Going On? More evidence on FOI in the UK

Freedom of Information

Freedom of Information

Labour Party Call for Evidence on the UK Freedom of Information Act 2000 (see the pdf here Ben Worthy Labour Call For Evidence on FOI)

Dr Ben Worthy

Birkbeck College, University of London


  1. What have been the strengths and weaknesses of the Act and the way in which it is currently working? Specific examples pertaining to you or your organisation may be provided.*

1.1 Our research found that, in terms of strengths, FOI has made British government more transparent and more accountable. FOI’s very unpredictability may be a powerful force for by changing behaviour.  Its deeper impact on democracy is less clear. This is not because FOI has failed but simply because changing levels of, for example, participation and public trust are complex- FOI alone is unlikely to make a difference. FOI can be best seen as part of a wider political ecosystem of formal and informal mechanisms designed to scrutinise government.

1.2 Like all FOI regimes, it has suffered delay, uncertainty and unexpected effects. There is less evidence for any of the weaknesses claimed by politicians. Our work concluded that FOI has not had any significant impact on the decision-making process or some of the key constitutional conventions. It does not appear to have led to a chilling effect.

To understand how FOI works, it is necessary to remember:

1.3 FOI is dynamic, and changes over time depending on use, legal rulings and political reforms. It is a constantly changing law that inevitably generates uncertainty.

1.4 The benefits of FOI may be local and hidden. Use is heavily orientated towards local government, with nearly four in every five made to local councils. Most FOI requests are ‘micro-political’, seeking factual information and generally involve matters of private interest, focusing on specialised local or personal issues—waste, street fixing, tax and permits—that don’t attract attention.

1.5 It is about perceptions as well as reality. Understanding how FOI works is not just about what is happening, in terms of statistics or law, but what people claim is happening-so is shaped by perceptions of politicians and the headlines it generates.

  1. Whether Act provides a sensible balance between transparency and “safe space” for the internal deliberations of public bodies. Specific examples may be provided.*

2.1 There is currently a sensible balance. As I pointed out to the Commission, looking at the section 35 and 36 exemptions, there remains, and may always be, uncertainty. However, the Commissioner and Tribunal have sought to protect ‘safe space’, dependent on the time period and sensitivity of the information.

2.2 FOI has not caused a ‘chilling effect’ on frank advice and deliberation, or on the quality of government records. The myth persists, but convincing evidence proved hard to find. Our own studies across central and local government between 2008 and 2011 were able to discover only a few clear examples, which were minor and isolated. There was no systematic or large scale changes to either minutes or free and frank discussion as a result of FOI. Our and other work FOI can also have a positive effect, professionalising records as a ‘disciplining’ rather than a ‘chilling’ effect.

  1. Whether there are any areas in the Act that could be improved, for instance, extending the Act to some private organisations or businesses in receipt of public money or carrying out public functions. Specific examples may be provided.*

3.1 Although it remains a ‘complex’ legal grey area, section 5 allows government to extend the law to cover companies within the scope of the Act. This change has been previously discussed at length in the UK between 2007 and 2009 and under the separate Scottish FOI Act on two occasions—resulting in limited extension to leisure trusts in 2012 with possible further coverage of housing associations and private prisons mooted for 2015-2016.

3.2 The Coalition and now the new Conservative government took a different approach of contractual enforcement. Rather than extending the Act under section 5, they have championed the use of new FOI clauses in public sector contracts. It’s not exactly clear how far this is working.

3.3 There has been some gradual natural ‘creeping’ outwards of FOI. Network Rail became subject to the Act in March 2015 due to a change in accountancy designation. Other new bodies covered since 2005 include exam boards, what was formerly ACPO, free schools (once they are open) and Police and Crime Commissioners (though this report was ‘deeply’ worried about how transparent they were-see page 11-12). The Police Federation is now set to follow.

3.4 More significant than this ‘creep’ is the influence of legal decisions from appeal bodies and the courts. A legal ruling in 2015 Fish Legal v Information Commissioner and others over FOI’s sister Environmental Information Regulations appeared to extend the law to water companies-and this may potentially include other utilities too.

3.5 The issue of extension remains a political one. Public sector contracts to private providers are currently worth around £93 billion per year according to the ICO. A UK tracker found 75% of respondents seeing extension as an ‘important’ issue . Polling by the Scottish Information Commissioner showed that extension is supported by the public. A full 76% of Scots asked felt private prisons should be covered with 79% believing that housing associations should be as well.

3.6 Any politician pushing for any large scale opening up, such as using section 5, faces three main problems

  • First, there may be a potential reluctance to cooperate or publish and an enforcement difficulty in making companies do so. Although our study of FOI and local government found that most companies cooperate with FOI requests, any sceptical business can argue it is (i) unnecessary as so much information is published anyway (ii) a costly burden-see this analysis here.
  • Second, added to this may be the complexity any change involves, that will take time and en The devil, as someone warned of extractives, is in the detail and how it interacts with other systems. So, for example, the UK’s push to open up Beneficial Ownership is slightly stymied by the fact that the EU equivalent will only be partially open.
  • Third, given these problems there needs to be a lot of political will, energy and attention to follow through. Any politician or party pushing large scale openness needs either a very good reason or very strong principles. Most likely it will only happen when there is a very obvious problem to solve or a very obvious political benefit (or both).

3.7 FOI extension is not the only way forward and the Information Commissioner has recently offered a range of options to fill the ‘transparency gap’.

  1. How does the cost of operating the Act compare to other types of public expenditure; and should the threshold for refusals on cost grounds by public authorities be raised or not.*

4.1 There is an ongoing debate about how much FOI ‘costs’, also part of the FOI Commission’s investigation. Costs are often discussed within the context of whether the Act is being ‘abused’ by the media or troublemakers. A number of submissions to the Independent Commission complained of the resource burden. There appears to be some support for a form of charging from some, but not all, officials.

4.2 Measuring the ‘cost’ of FOI is problematic as it involves balancing administrative resources against democratic benefits. Moreover, the exact cost of FOI is very unclear and any figure, high or low, can be challenged. Methodologically it is almost impossible to obtain a precise figure on the cost of FOI. Estimates range from an average of £350 to £36 per request. The Daily Telegraph calculated FOI used up 0.0016% of the overall central government budget and 0.018% of local government.

4.3 FOI requests become more elaborate but public authorities deal more efficiently with them over time – annual surveys by UCL between 2005 and 2010 found a sharp drop in time taken for organisations to process requests, falling by more than 50% in 5 years.

  1. Whether the Act has achieved its objectives and made public authorities more open and accountable. Specific examples may be provided.*

5.1 Our studies of the six objectives of FOI concluded that the core objectives of greater transparency and accountability were achieved.

5.2 FOI has increased transparency. All the evidence showed that the amount of information released has increased across a vast range of subjects from nuclear convoys to Ministerial gifts and from parking fines to councillors’ expenses. It has also led to increasingly open cultures within organisations and to the now regular pro-active release of a variety of information. The exact impact varied as local government was already fairly open and parts of central government less so. Some Whitehall departments still struggle due to senior attitudes or simply the nature of the information they deal with.

5.4 FOI has increased accountability across central and local government and all the way to the police, NHS and Monarchy. It has been used by the media, MPs and campaigners to make government more accountable. Eye catching examples include the role of FOI in kick starting the 2009 MPs’ expenses scandal, as part of a wider chain of accountability, and the mass resignation of an entire parish council in Walberwick in Sussex in 2012. But the extent to which FOI can be used to increase accountability on a more day-to-day level is dependent on whether other actors (the media, NGOs, etc.) are willing and able to make use of it.

5.5 FOI is frequently used alongside other tools of accountability often as part of a process of building a larger picture, or putting together pieces of a jigsaw as in the case of extraordinary rendition or a nationwide campaign against library closures. Below are a selection of high profile or influential requests:

  1. Extraordinary rendition-the UK’s involvement in extraordinary was revealed by FOIs from the All-Party Group on extraordinary rendition.
  2. Details of the Universal Credit welfare reforms
  3. The Libor banking scandal and knowledge of it
  4. Lists of visitors to the Prime Ministerial residence at Chequers (and Ministerial meetings and diaries now proactively released)
  5. Creation of the famous ‘Weapons of Mass Destruction’ dossier
  6. The Monarch’s involvement in vetoing legislation
  7. The results of local restaurant hygiene inspections
  8. The planned closure of local libraries up and down the country
  1. Whether the Ministerial veto has been used appropriately and whether the Supreme Court ruling has undermined that.*

6.1 Before the Supreme Court ruling, the veto worked well, limited by the ‘exceptional’ nature and political reluctance. Consequently, it was rarely used, especially when compared with other FOI regimes. Any future veto power should be kept as close as possible to the precise, limited and ‘exceptional’ model that existed previously.

Further Reading see: