Research on Open Data and Transparency

Leave a comment

Write Me A Letter: 10 thoughts on the Prime Minister’s letter on Openness and Data


Theresa May is a keeper of secrets, by inclination, style and force of habit. So the publication of a letter urging her government to open up may come as a bit of a surprise, along with a seeming openness push. This is especially the case this week, when the government is making the DUP’s money secrets more, not less, opaque. But is the letter less than the sum of its parts? What does it all mean? Here’s 10 quick thoughts…

  1. It’s all very David Cameron-a letter not a speech, an article or launch. My suspicion is a letter is designed to make it look like you’ve done something (‘I’ve written them a letter! What more do you want?’) (Cameron wrote one in 2010 then another to tax havens in 2013).


  1. What standing does a letter have? Do you have to do it? Should you take notice of an (undated) private minute from the Prime Minister? Can it be safely ignored? I’d guess ‘not much’, ‘no’, ‘no’ and ‘yes’.


  1. The ‘next stage’ actually sounds very 2010-2011.


  1. It’s released on a Friday, one week before Christmas on a heavy EU news week (though aren’t they all now?).


  1. There’s a reprimand that ‘a small number’ of departments have fallen behind. It seems more than that as ‘Departments have become less transparent since 2010 and have not consistently fulfilled their requirements’. According the IFG the rot set in, ironically, in 2010 but has got worse recently. I think May should focus on whoever was in charge of the Home Office 2010-2016, as they seem to have been one of the worst performers in terms of FOI.


  1. It’s oddly worded part 1. It has a pretty tepid tone. It reads a bit more like a forced Christmas thank you letter than a ringing call to arms for openness.


  1. It’s oddly worded part 2. It talks of a strange, vague thing called ‘online transparency’ that seems somewhere between open data and openness. And, as many people have pointed out, it’s a pdf. That’s neither transparent or useful, in ‘online’ terms.


  1. It’s oddly worded part 3. No mention of FOI despite an ongoing FOI consultation (launched in late November).


  1. It’s not re-launching or pushing any new policy. One would expect a Prime Minister to perhaps use the letter to push, for example, gender pay transparency (not going too well) or even anti-corruption.


  1. Highlighting civil service sickness and absence data first seems slightly out of place-is this designed for ‘lazy sickie (Remoaner) civil servants’ headlines?


Leave a comment

Freedom of Information in the UK: Past, Present and Future

With government getting more secretive and Brexit looming, where do we stand with FOI?

Listen in to this talk by myself and the BBC’s Martin Rosenbaum at Newspeak House to find out where FOI came from and where it might go next.

FOI-Why Bother? (19 April 2017)

We ask why do governments pass FOI laws when they have no votes in them? What happens once they are passed? And how will the 2017 General Election and Brexit shape the future of openness?

You can see the slides hereFOI Newspeak and chapter one of my book here.

Leave a comment

New Paper: Brexit and Open Government in the UK


How will Brexit influence the UK’s transparency regime and how, in turn, will openness shape the UK’s Brexit process? There are three ways of looking at Brexit and open government: through possible changes to old policies and the pushing of new ones, through the new Prime Minister championing transparency or supporting secrecy, and the openness of the Brexit process itself, which so far has seen a struggle between the executive’s secretive prerogative powers and the legislature’s rights to know.


Download the paper here


Leave a comment

The Open Government Partnership in the UK: How Far Have We Come?


The UK government is now nearing the end of its Second National Action Plan created as part of the Open Government Partnership process. The plan is a series of 21 open government commitments drawn up in 2013 with the co-operation of civil society groups (see a history here). The government itself published a series of updates and self-assessments on how it has done as it went along.

The new IRM is a separate independent assessment based on a series of interviews and desk research. One thing to note is that it looks at implementation (whether policies have been put in place) not impact or effect (what has happened as a result of the policies). You can read the new report in full here.

What Went Well?

There were some eye-catching commitments that have been strongly pushed by the government. Probably the most high profile, unsurprisingly given David Cameron’s support, is the move to publish data on Beneficial Ownership. As of April 2016 there will be a public register of who owns or exercises ‘significant control’ over all UK companies. Another similar move has been to open up UK companies engaged in natural resource activity (e.g. oil and gas) by implementing two new EU Extractives Transparency laws and joining the Extractive Industries Transparency Initiative. Finally, Aid Transparency is being extended with data publication offering new ways of understanding who is giving what aid to whom. Less about data, but still significant, is the government’s new Anti-Corruption Strategy.

A little further below the radar, but equally as important, are the initiatives around local community data such as tax, homelessness or help to buy. All politics is, of course, local and so is a lot of data. It is locally that data is often of most interest to the public and where it can kick-start innovations.

Behind the scenes, there’s been important work started on the ‘building blocks’ of openness such as strengthening records management, with work by TNA turning UK legislation into open data and on digital records, and creating frameworks, with the Cabinet Office led National Information Infrastructure.

What Went Not so Well?

The commitments involving public participation in decision-making were a disappointment to many civil society bodies, as they did not go far enough. Some of the areas of work offered some interesting ideas but faced difficult problem of co-ordination and bumped up against the all-important issue of privacy (as, for example, showed). On a more practical level, some of the commitments were rather vague.

Looking across the plan as a whole, while there were big, eye-catching moves forward on transparency, there has been a lot less progress on two of the other OGP’s aims of accountability and participation. Data alone does not, of course, automatically bring these things. Across many of the commitments there is a need to ‘link up’ data to means of engaging the public. All this new data needs to work and fit with either old or new tools of accountability and participation to make it truly effective.



And Now?

One important recommendation I made is to follow up on the commitments after the plan ends. It is very important that someone (such as a Parliamentary select committee or an expert) gathers evidence on what happens to all these different policies over the coming months and years. The devil is, after all, in the detail, especially for the complex or wide-ranging reforms.

It’s also important to look at the ‘gaps’ and what is not covered. I highlighted government surveillance and lobbying as two important missing pieces. Other external issues, such as changes to the Freedom of Information Act, may influence the future direction of openness. Perhaps most importantly, as the government has recognised, the devolved bodies and local government, some of who have been developing their own policies, need to play a far greater part in developing future ideas.

As attention now moves to putting together the third NAP, there is a bigger question lurking underneath. What are these reforms around Open Data and open government all about? This is not only about how all these disparate ideas and approaches fit together but what the ‘vision’ of state and society, politics and government is that guides it all. Perhaps asking it a different way, what would all these commitments mean for how government will look 20 years down the line? More politically, who wins and who loses?


The talk is available on itunes here

You can see the full IRM report here

Open Government Civil Society Network

More on the OGP UK


Leave a comment

RTI vs. FOI: New Paper Comparing FOI in the UK and RTI in India

RTI image

My draft article comparing the UK and India is hereBen Worthy Access to Information in India and the UK 2015.

It examines the impact of two pieces of transparency legislation: the UK Freedom of Information Act 2005 and the Indian Right to Information Act 2005. It looks at the origins and composition of the laws before examining how the two pieces of legislation function. Both laws have led to transparency and accountability by exposure and raising of ‘fire alarms’, with information disclosure used to bring about accountability as well as, to a more limited extent, reform and behaviour change. Of the two laws, the Indian RTI has proved more ‘politicised’ and more capable of initiating political participation. Yet the ‘transformative’ powers of such reforms are limited by poor implementation and resistance. Moreover, the effectiveness of such laws is shaped by context, with India in particular facing deep and complex socio-political obstacles that may prevent the laws having the ‘revolutionary’ effects advocates hoped.

Worthy, Ben, Access to Information in the UK and India (Jan 16, 2014). Available at SSRN:


10 Years of Freedom of Information in the UK: Tony, Tension and Turbulence

Operation Unthinkable

The truth is that the FOI Act isn’t used, for the most part, by ‘the people’. It’s used by journalists. For political leaders, it’s like saying to someone who is hitting you over the head with a stick, ‘Hey, try this instead’, and handing them a mallet.

Tony Blair 2010

The Freedom of Information Act has enhanced the UK’s democratic system and made our public bodies more open, accountable and transparent. It has been a success and we do not wish to diminish its intended scope, or its effectiveness.

House of Commons Justice Select Committee 2012 Post-Legislative Scrutiny of FOI

These two comments sum up the difficulties of measuring how successful the UK Freedom of Information Act has been. It isn’t just about statistics on numbers of requests, users or refusals (though there are some here if you are interested). What people think also shapes how it works and how others then behave. So a former Prime Minister sees it as one of his biggest mistakes while a Parliamentary committee see it as a vital part of democracy. Which is it?

What Does FOI look like?

One way to think about FOI is as an iceberg (as Nicola White said in her great book). iceberg

We can see only a small part of the overall process, those high level, often national FOI requests that attract controversy or attention-the MPs’ expenses scandal in 2009 was the big one that set off a chain reaction of resignations and reform.[i] There has also been FOI exposure of all sorts of subjects, from the Iraq war to health reform. In recent days, for example, we have had stories about where the Environment Agency invests its money, the number of pupils at new Free schools or the cost of green policies–see David Higgerson’s FOI Friday round-up for more.

But underneath this, there are a large number of requests, probably 90 % or more, that we don’t see. Our research found most requests go to local government, somewhere between 70 to 80 % or 3.5 to 4 in every five requests (see what the Justice Committee said here). These are about local or personal issues-waste, street fixing, tax and permits- that are often ‘under the radar’. That’s not to say they can’t be spectacular-Tony Blair probably never expected his new law would lead to us knowing that 3 people were banned from Birmingham’s new library for being too smelly or to the resignation en masse of Walberswick Parish council in Suffolk. But real FOI is local, focused and probably bringing hidden benefits we don’t easily see.

Who Uses FOI?

A key problem underlining all FOI analysis is the lack of knowledge about requesters and their motivations. The table below is based on estimates of requester types to central and local government from FOI officers.

Requester Local Government (%) Central Government (%)
Public    37 39
Journalist    33 8
Business    22 8
Academic    1–2 13


Contrary to the views of Tony Blair, FOI requesters are not just journalists. The largest group across central and local government appears to be members of the public. We felt that the public consists of a small group of politically engaged with a larger group pursuing issues of “micro-politics” or of private importance.

There is a clear rise and fall of public interest with the news agenda-snow leads to requests about snow plows, spying about government snooping levels. However, many requests were focused, “quite niche” or on “specialised” issues- a planning dispute or parking fine at local level or access to benefits at central government (or even who was paid what to switch on the town Christmas lights).

Requesters’ motivations were also diverse. Even the small sample of requesters we found and spoke to in our studies gave a huge variety of reasons for using FOI, from “concern about wasted money” to “curiosity”, “general interest” and personal campaigns against “corrupt” local government. So, the sheer variability of requester motivations and use underscores the variability of impact of the Act upon different public bodies.

Who’s Against It? Fighting on the Border

FOI has been subject to clear ‘battle’ over how the Act is working, with a public divide between sceptical (mainly) senior politicians and officials and supporters of the Act in the media, NGOs and the appeal system.

The extent to which FOI is supported varies across departments and local government bodies, dependent on the individual leadership, culture and environment of different public bodies. On the whole, government officials support the Act and work with it.

While welcoming it as an idea, senior politicians have been less keen on the loss of control or unexpected issues or scandals sprung on them. As well as claiming it is vaguely ‘abused’, a number of senior officials and politicians have argued that FOI negatively affects decision-making processes, though we found there was no real evidence for this (which didn’t stop some rather interesting anecdotes to the Select Committee). While Tony Blair was clear in his views that FOI was an all-round disaster, David Cameron’s more subtle approach has been to argue that ‘real freedom of Information’ was about ‘spending’ while other requests ‘furred up the arteries’ of government-a comment that revealed a very particular view of what information rights ‘should’ be used for.

Numerous politicians have also highlighted the ‘cost’ of FOI, though, like many economic arguments, this is actually smokescreen for a political debate. And when different studies have concluded that requests costs either £200, £36 or £19 each, the discussion becomes a little confusing (see this post here and a longer report here). The danger is that all this combined negativity may encourage poor behaviour and lead to a small ‘anti- FOI’ group at the very top of government. While, for example, Blair’s claim that FOI is used only by journalists is demonstrably untrue, it adds to a distorted view of FOI.

On the other side of the divide, FOI has a clear constituency of supporters in the media, Parliament and across various NGOs as well as in the courts and the appeal system. Supported by high profile cases such as MPs’ expenses, the symbolic importance of FOI legislation offers the reform a robust protection, backed up by a powerful and vocal constituency. Supporters of the Act have constantly innovated, pushed key cases and also sought to persuade successive governments to extend the Act private bodies working on behalf of public authorities.

Since 2005, but gathering pace since Tony Blair’s comments in 2010, there remains a continuous ‘fighting on the borders’ over where the Act begins and ends and whether it is ‘good’ or ‘bad’. In 2009, supporters scored some success by persuading Gordon Brown to shorten the period of disclosure of historical records from 30 to 20 years. David Cameron’s Transparency Agenda has undoubtedly helped push further openness, as have events like the Hillsborough inquiry.

However, at the same time the scepticism from the top of government has encouraged a series of attempts to restrict the Act. This included an attempt to change the costing regime in 2006, to remove Parliament from the ambit of the Act in 2006-2007 and introduce greater protections for Cabinet documents in 2010. Only the removal of the Monarch and heir from the Act was successful, probably because it went largely unnoticed (though the Monarchy is not out of the woods yet). As of writing now, the government has hinted that it may seek to limit what it ambiguously describes as ‘industrial’ users, though this close to a (close) General Election it’s unlikely. The fact that all but one attempt was seen off shows how strong FOI’s support base is. For now…

And so?

FOI appears to be a success and is (probably) here to stay. This is not just about numbers- it is supported, used and co-operated with by most officials. It can, and does, bring very public benefits and may also be locally bringing positive outcomes we don’t see.

It has not only led to new issues on the agenda (not least the UK’s role both in extraordinary rendition and covering it up) but also helped in the creation of a new watchdog to regulate MPs’ expenses and a change in the law over the tax status of members of the House of Lords. It has also kicked off developments like mySociety’s WhatDoTheyKnow and has popped up in all sorts of interesting areas, such as the app that lets you know if politicians are editing Wikipedia pages.

It is not without its difficulties or problems-it can be abused, misused or misunderstood. All openness brings problems of one kind or another. But the disruption and uncertainty may be an essential part of any openness law. One way of thinking of FOI is as a form of turbulence-an instrument of unpredictability, like e-petitions. So Tony Blair’s complaints may be (in part) a sign of the Act doing its job well. It is its very unpredictability and annoyance that makes it powerful. As I’ve said elsewhere, it also enshrines an important principle-but not one that lets politicians sleep soundly in their beds.[ii]

Further reading

Justice Committee (2012) Post-legislative Review of Freedom of Information here

Worthy, Ben and Hazell, Robert, The Impact of the Freedom of Information Act in the UK (August 26, 2013). Nigel Bowles, James T Hamilton, David Levy (eds) Transparency in Politics and the Media: Accountability and Open Government, London: L.B. Tauris, 31-45, 2013. Available at SSRN: and work by the Constitution Unit here.


[i] I’m obliged to point out the scandal was a combination of an FOI request by Heather Brooke, four years of appeals, a court case and (finally) a good old fashioned paid for leak.

[ii] I shamelessly borrowed this from Orwell’s definition of liberty ‘If liberty means anything at all it means the right to tell people what they do not want to hear’ (from his unpublished preface to Animal Farm) and tried to rework this to fit transparency: ‘Transparency is the right to ask questions those in power don’t want asked and look for information they don’t want us to see’ see this post.

P.S. for anyone interested, the pictured document above comes from Churchill’s planning for the very aptly named ‘Operation Unthinkable’, an appraisal of the consequences of a war between the US/UK and the USSR in 1945.