Research on Open Data and Transparency

Leave a comment

Michael Gove, FOI and Lyndon Johnson


Michael Gove on a school visit

The last week or so has not been kind to FOI. Last Thursday it appeared that the government was pursuing retention and deletion policies which involved deleting emails after 90 days. On Tuesday we saw Iain Duncan Smith’s apparent refusal to publish statistics relating to those dying after having had their benefits cut. It now seems that new Justice Secretary Michael Gove would like to look into reform of FOI. He said during questions in Parliament yesterday that

 …we do need to revisit the Freedom of Information Act. It is absolutely vital that we ensure that the advice that civil servants give to Ministers of whatever Government is protected so that civil servants can speak candidly and offer advice in order to ensure that Ministers do not make mistakes. There has been a worrying tendency in our courts and elsewhere to erode the protections for that safe space for policy advice, and I think it absolutely needs to be asserted.

 The effect of FOI on policy discussions generates lots of heat but very little evidence. Tony Blair claimed FOI had led to more caution over recording decisions. Former Cabinet Sectary Gus O’Donnell also claimed it has ‘hamstrung’ government, though when pressed he could only offer isolated examples-one hypothetical and one based on the coalition negotiations, one of the most unique and unusual political events in recent decades.

There are two problems with the idea that FOI has cramped decision-making or led to changes in records. One, it’s very difficult to prove or disprove. The Justice Committee ‘was not able to conclude, with any certainty, that a chilling effect has resulted from the FOI Act’ and also felt the protections for policy were more than sufficient. Our own studies found a few examples but no systematic changes-and also many officials more concerned about the dangers of not having a record if a judge came knocking.

Two, it’s also very difficult to prove that any changes are a result of FOI. Many, many things cause records to change-lack of resources, leaks, the arrival of iphones, emails, even changing decision-making styles (‘sofa government’ anyone?). Holding just FOI responsible seems a little skewed.

But the claim won’t go away. This is partly psychological. Politicians believe it happens and keep repeating it, so it then becomes true to them. It is a rather wonderful example of a self-confirming myth, especially as the myth itself may then make people wary. It is, of course, more politically, a convenient and half acceptable way of attacking FOI, for those politicians who don’t like the disruption FOI brings.

Michael Gove is by no means the first to suggest ‘amending FOI’. Here’s a quick list of the many attempts by governments of all shades to change the law, either coming from government or Parliament since 2005:

  • Introduce fees or change the cost limits (2006)
  • Remove Parliament (2007)
  • Removal of Monarch and Heir (2010)
  • Clampdown on ‘industrial users’ (2012-2013)
  • Amend the veto (2015)

Interestingly, only one has succeeded so far, removing the Monarch and Heir (done by the Labour government but, because it commenced in 2011, blamed on the coalition) . This change, probably not incidentally, was the one that got least press coverage.

So why have all but one of these attempts failed? FOI is very powerful as a symbol. It also has lots of supporters in Parliament, the media and among NGOs. This is especially the case in the UK where ‘FOI’ is linked to the magic words ‘MPs’ Expenses’. Trying to cut back on FOI instantly galvanises this opposition and, almost by default, makes you look secretive-see the Daily Mail from yesterday. It won’t necessarily lose you votes but it provokes political pain for little gain.

Which takes us to Gove’s idol, President Lyndon Johnson (see this speech). Many politicians support FOI through gritted teeth (see Daniel Berliner’s great paper here). LBJ was the ultimate reluctant supporter, who fought the original US FOI bill all the way and had to be persuaded out of vetoing it by his press secretary, who whispered gently in his ear and later recalled:

 LBJ had to be dragged kicking and screaming to the signing… He hated the very idea of the Freedom of Information Act; hated the thought of journalists rummaging in government closets and opening government files; hated them challenging the official view of reality.

 He then went out, as Moyers points, to take all the credit for it.


LBJ may also offer FOI skeptics another lesson. Johnson was the ultimate backroom politician. His career, such as his spectacular time as Majority Leader of the Senate or passing civil rights legislation in the 1960s, was based on secrecy, information control and holding back on telling anyone more than they need to know, which was often far from the truth. And this, ultimately, was his undoing. His credibility gap on the Great Society and Vietnam stemmed from his being too secretive as reality exposed his untruths and many distortions.

Politicians naturally fear exposure or openness that may damage them. But hiding things or trying to back track brings pain too and Cameron’s decision to publish the benefits figures is a wise one, given the attention not disclosing them attracted. Can politicians like Michael Gove, who cast themselves in the mould of LBJ as populist warriors, go a step further and embrace the radicalism of FOI?

Leave a comment

Elysee Eavesdropping and Radical Transparency


A fascinating reminder of a very powerful source of transparency that’s often ignored-leaks. Today it seems the French President has called an emergency meeting to look into claims that the US has bugged the last three French Presidents Jacques Chirac, Nicolas Sarkozy and Francoise Hollande-if true, a hugely controversial action. You can read the Wikileaks press release here. Imagine if the French intelligence services had listened in on Obama, Bush and Clinton? At the same time the government and GCHQ is under pressure about drones outside of recognised war zones, specifically in Yemen.

Leaks of course sit on the grey zone between legality and illegality and are rarely part of mainstream openness debates. It perhaps offers, as Claire Birchall argues, a new, more radical form of transparency that challenges some of the cosy ideas about where openness and secrecy begin and end. Leaks are also something that governments ignore at their peril-just see the US Freedom Act.


Leave a comment

Open data on council spending is largely unread by voters



Here’s my Guardian article on Open Data (you can see it here)

Open data is sweeping the world as government after government seeks to publish data and reap the hoped-for benefits: increased transparency, accountability and efficiency.

One of the former coalition government’s flagship policies was to request all English local authorities publish all their spending over £500. This, the government argued, would bring all sorts of good things. Councils would experience a revolution in openness and accountability, and citizens would become auditors from the comfort of their own armchairs. As well as unleashing a new roving public eye over council accounts, it would also encourage developers to create all sorts of clever spending apps.

However, there has been less interest in the data than many hoped. Council finance data is viewed around 200 times each month. The person on the street has not been desperate for data. No army of armchair auditors has sprung up. There are some enthusiasts here and there; a concerted campaign in Barnet used open data to build its case against the Conservative council. But few people have the time and, most importantly, the motivation to scroll through complicated pdf documents of raw council spending data. Data needs a narrative, and pdf documents and spreadsheets don’t yet tell a good enough story.

This links to a further problem. If you do find a smoking gun among the spreadsheets, who do you send it to? The council itself, the opposition or the local press? It’s notclear what the next step would be and how it would fit in with the processes by which councils are accountable to citizens.

The better news is that something is being done with the data. Users include businesses, pressure groups and journalists as well as a handful of members of the public. While they haven’t unleashed a wave of accountability, there have been sudden bursts of data-driven questioning of local authorities.

Most interestingly, new apps and websites have actually appeared to help us make sense of it all. There are comparison sites where you can track spending on contracts, such as Openly Local or, and the Local Government Association’s LG Inform tool, where you can generate benchmarks to measure your own council’s spending or performance against the national average. The data could become even more meaningful when linked up, for example, to newly released council tax details.

So the precise effects of all this spending data are complicated. It also raises the bigger question of how citizens take in information and whether they really behave as rationally as they should with it. Studies have found that voters at the local level have a heavy negative bias (remember the bad and ignore the good) and rarely get involved, even when it’s made very easy for them to do so.

Some local councils saw all the new data as a pointless distraction. Others saw it as a method for curbing freedom of information requests, giving out data while cutting back on pesky questions from the public. A few saw it as a Machiavellian scheme from national government to hand potentially lucrative financial information to private companies . In times of sharp austerity, others saw the new spending transparency as a way to make local councils look like reckless wasters of public money.

The real impact is more complex and political than it looks, which is important to bear in mind as the new government rolls out local devolution. There has been no data revolution yet, and few armchair auditors, but as new apps appear, things may start to get interesting very soon.



Leave a comment

Who Is Using Local Spending Data

Here’s some of the results of my survey of English Local Authorities on Open Data (£500 spending). You can also see it as a pdf Survey of English Local Authorities on Open Data

Conducted: April-October 2013 via online survey (BOS)

Population: 102 (2 x excluded)

1.     What type of transparency information is of most interest to users?

user interest

2. Which group or groups are the primary users of the data?


primary user pie chart



3. How would you characterise the level of use of the spending data?

q3 use level


4. What type of transparency will this data create?

q4 type of rans

 5. What type of transparency will this data create?


 q5 barriers


Pageviews of Spending Data on Selected Local Authority Websites January-December 2012[1]

Council Type Page Views per month (average)
District 35
District 87
District 99
District 341
Borough 355
District 437
City 450
District 528
District 610
District 636
City 2021


Comparative Visits for January 2012 on government and non-government Open Data sites

Site Visits per month
Local Council Average 200 540, 609 161,101
TheyWorkForYou 200-300,000
WhatDoTheyKnow 100-200,000

( Esche 2011)

For more details

Worthy, Ben, ‘David Cameron’s Transparency Revolution? The Impact of Open Data in the UK’ (November 29, 2013). Available at SSRN:

Worthy, Ben ‘Evidence to PASC on the Aims of Open Data’

Worthy, Ben ‘Evidence to PASC on the Impact of Open Data’




[1] Based on FOI requests to English local authorities in 2013 asking for hits/views on spending data pages of their website.