Research on Open Data and Transparency

Leave a comment

Revolution or No Change? The PASC report on Open Data


The Public Administration Select Committee has examined in detail the progress of the Government’s Open Data agenda (you can see the evidence here). Its final report has made some important points about how the new agenda is developing:

First, the report argues that it’s hard to tell what impact the new Open Data reforms are having: ‘It is very difficult to assess the performance of Government in enhancing accountability through opening up its data […] measuring progress on this important agenda is difficult if not impossible’. Consequently ‘this allows supporters of open data to claim the revolution is well under way and the sceptics to say nothing has changed’.

Second, the government itself is still unclear on what it all means: ‘The concept of open data is poorly defined’. Something is happening here but no one, so to speak, knows what it is. The government seems to be making several assumptions about what will happen. It expects economic growth without recognising that innovation comes from unexpected areas. It wishes to increase accountability but, as the report argues ‘simply putting data “out there” is not enough to keep Government accountable’. It is trying to encourage more public participation but ‘Government…make too much use of jargon’ which can ‘alienate and confuse people who do not have expert knowledge of the technical terms’.

Third, as a result of the above, implementation and impact is uneven across government. Some departments and bodies may be fully behind it but others may be simply republishing old data (as happened in the US with Without clarity or standard measures, it’s very hard to know.

So what should be done? The report offers a number of suggestions (see also the report’s conclusions)-

  • There should be a central champion for Open Data in the Cabinet Office, helped by a streamlined panel of experts to replace the advisory board. At present there are simply too many competing groups and bodies.
  •  Government should ‘clarify’ what it means by Open Data and explain the benefits to a sceptical (and sometimes concerned) public.
  •  To ensure consistency the Government should adopt a ‘star-rating system for engagement…for measuring, and reporting to Parliament on, Departments’ progress on increasing accountability through open data. The Government should expect Departments to set out plans to move towards Five Star Engagement for all their data releases’. This can be used to create a definitive list of what data departments should publish.
  • There needs to be awareness of the pitfalls and dangers around privacy but ‘the recent controversy over demonstrates the danger that concerns about privacy will unduly undermine the case for open data’.

Hidden away inside are further interesting recommendations:

  •  Open Companies? In terms of who should be open the report says ‘Open data principles should be applied not only to government departments but also to the private companies with which they make contracts’. The Committee goes on to recommend ‘that companies that…provide contracted or outsourced goods and services should be required to make all data open on the same terms as the sponsoring department.’ This is interesting in the light of discussion around extending FOI.
  •  What should be opened up? The report argues that ‘core data needs to be released fast and, above all, free’ including Postcode Address File (PAF) that has been the subject of intense debate for a number of years. This core data provides the building blocks for innovation.

Overall, the report offers a kind of ‘optimistic concern’, speaking of the huge opportunity Open Data presents but also its concern that it is not being fully exploited:

‘Today there are unparalleled opportunities to harvest unused knowledge that otherwise goes to waste, which can be used to empower citizens, to improve public services and to benefit the economy and society as a whole’

We will await the government’s response…

You can see my written evidence here, oral evidence from Helen Margetts, Tom Steinberg, Rufus Pollock and me here and Nigel Shadbolt here.

Leave a comment

Who is using all this Open Data?


As politicians promise more and more openness, there is one big gap in our understanding: Who is using all this information?

Work on Freedom of Information showed that nailing down users is tricky. Our best guess is FOI users are a diverse mix of public, businesses, NGOs and (a few) journalists (see this attempt at a typology of FOI users written with Greg Michener). But because we don’t know, all sorts of myths and claims fill this gap in our knowledge. Tony Blair believes FOI is ‘mostly’ used by journalists and David Cameron probably thinks similarly, given that he believes those using FOI are ‘furring up the arteries of government’-see this table from our evidence  to the Justice Select Committee for our best guess.

Estimated Requester Groups to Local and Central UK government compared with EU Access to Documents legislation (%)

Requester group Local Government Central Government EU Access to documents Legislation
Public 37 39 32
Journalist 33 8 3
Business 22 8 -8
Academic 1-2 13 23

So what about Open Data? My own research and other work in to the impact of publishing all spending over £500 points to a few interesting things. First, local use seems low. Most authorities appear to registering a few hundred views a month of their online spending information, though a few are far higher (see also this research from the Finnish Institute and this transparency survey from the LGA).

Estimated Breakdown of Requester Groups-FOI and Spending Data (%)

Requester group Local Government FOI Local Government Spending Data
Public 37 23
Journalist 33 31
Business 22 39
Academic 1-2

Second, local users are diverse. We seem to be seeing a mixture of businesses, NGOs, members of the public and a sprinkling of journalists. Who the ‘public’ are remains a bit of a mystery. Like FOI, they may be a hard core group of activists with a wider, looser group made up of the curious or those seeking very specific information. Research into the government’s data portal by Tim Davies found a similar mixed group of activists and ‘techy’ people.

So what does this mean? It means that any impact on local government is likely to be uneven and unpredictable. It could create odd or perverse incentives. A journalist highlighting spend data on hotel bills will have a very different reaction from an authority than a business looking for spending on stationary or a member of the public seeking allotment costs.

It also quite fragile. Like participation in politics generally, all this Open Data it depends on a few people getting active and involved. Whether such uneven and fragile work can create an army of auditors or bring big changes to how local authorities work is another matter.

The low use also tells us that this spending information isn’t precisely what is wanted. As the Local Government Association has pointed out, and ten years of FOI tell us, people want information of value to them-‘real time’ bin collection activity and costs not spreadsheets.  But that’s not to say they cannot be merged, as I argue in my research.

Finally, if you are an Open Data user can you help with my survey?