Research on Open Data and Transparency

Leave a comment

Reflections on UK End-of-Term Report for 2016-18 NAP


The End of Term report for the UK’s last Open Government National Action Plan (NAP) has been published. The report summarizes the results of the 3rd NAP, covering October 2017 to May 2018, and includes some relevant developments up to October 2018.

So here’s a few things the NAP tells us about where we are in the UK with openness:

-Openness is not just about FOI. There were, as you can see, 27 different commitments in all sorts of areas, some public and some slightly more ‘backstage’. They ranged across everything from FOI to contract data and from data-driven academic studies to ethical supply chains.

-Openness is a mixture of old and also new. Some of the commitments carried on work that was already ongoing from previous plans. The issue of Beneficial Ownership and foreign companies was actually part of the last NAP, very much David Cameron’s ‘thing’, as were many of the anti-corruption reforms and open contracting. Others, such as commitment seven on a common standard for voting data, are newer and also tie in to ongoing debates about electronic voting in Scotland and Wales.

-Openness is UK wide. The NAP was actually made up of separate parts this time, reflecting the make-up of Britain and its devolved governments. It included UK-wide commitments, as well as separate commitments for Wales, Northern Ireland and Scotland. So Wales had nine commitments, Northern Ireland had four and Scotland had one (though it has a separate process for its pioneer programme).  It also led to the first UK wide meeting on openness with representatives from every nation of the UK.

So how did it go?

This action plan was launched, seemingly long ago, by David Cameron in June 2016. The Brexit referendum on the UK leaving the EU, subsequent change of government and General Election all led to delays. In the first 16 months of the 2016-18 NAP the UK had two Prime Ministers, two governments and four different lead Ministers. In the space of less than a year, Britain also had a referendum on EU membership and a General Election-both of which yielded, let’s say, unexpected results. Northern Ireland meanwhile, as the OGN pointed out, hasn’t had a government for some time, since January 2017. So the fact that so many commitments progressed so far, is impressive.

There have been a few flashing warning signs for openness since 2016. The Institute for Government has shown FOI responses and Open Data publication are slowing at central government level year on year. In Scotland the Scottish Information Commissioner found a twin system for processing FOIs while concerns have been raised in Northern Ireland over record keeping.

It’s been shown time and time again that a combination of distraction and disinterest can seriously hinder openness-benign neglect can be as fatal as outright resistance. For openness to happen need energy and support, and if policies run out of these then momentum can slow and enthusiasm wane.  The danger is that the ‘Brexit effect’ becomes a brake on anything else happening. So far, progress has been made but the price of openness seems to be eternal watchfulness and pressure.

Meanwhile, other transparency pushes are still happening, as the Treasury consults on anti-corruptionand the Scottish parliament examines its FOISA (my evidence to it is here).

To read more on openness in the UK, see my free overview here Worthy, B. (2018). ‘How transparent and free from corruption is UK government?’ in Dunleavy, P et al. (2018). The UK’s Changing Democracy: The 2018 Democratic Audit. LSE Press. Download the chapter here.

Table 1: At a Glance


End of term

Number of Commitments

27 27

Level of Completion

Completed 2 12
Substantial 12 10
Limited 13 5
Not Started 0 0

Number of Commitments with…

Clear Relevance to OGP Values 26 26
Transformative Potential Impact 3 3
Substantial or Complete Implementation 14 22
All Three (✪) 2 2

Did It Open government?

Major 4
Outstanding 0

Moving Forward

Number of Commitments Carried Over to Next Action Plan N/A

Leave a comment

My Evidence on the Freedom of Information (Scotland) Act 2002


(image courtesy of Scottish Information Commissioner)

Here’s my evidence on FOISA-you can also download it as a pdf Ben Worthy Post-legislative scrutiny of the Freedom of Information (Scotland) Act 2002 Evidence


Dr. Ben Worthy, Birkbeck College, University of London

The Scottish FOISA has high levels of use, good rates of disclosure and strong public support. It has been undermined by game-playing at senior political levels and there are signs of patchy compliance, especially with Arm’s Length Bodies, and less pro-active disclosure than hoped.

  1. In your view, what effects has the Freedom of Information (Scotland) Act 2002 (FOISA) had, both positive and negative?[1]


1.1 There are a series of positive effects from the FOISA. It is often argued that FOI laws bring greater transparency and accountability to public bodies, as well as improved public trust. Research around the world has shown that FOI laws do make for greater openness, through both direct and indirect effects on the behaviour of officials. However, exactly how well a law functions very much depends on if or how it is used and the context it is placed in.

1.2 FOI laws are intended to bring about greater openness reactively and proactively. They are reactive in making bodies respond to requests and proactive in encouraging the publishing of information in anticipation of interest. On the reactive side, one positive sign of a well-functioning regime is if it is being used and requests are being made. The volume of requests can vary immensely, even when weighted to different populations, from almost none in Switzerland to large numbers in the UK. Scotland has relatively high levels.

Table 1: Snapshots of FOI request numbers in single years[2]

Country Numbers
Scotland 77, 528
UK (central government) 49,961
UK  (local government) 467,000
Germany 2000
Switzerland 249
Kosovo 1999
Montenegro 1782


1.3 Greater openness: In terms of whether information is forthcoming, around 75% of all requests made in Scotland, or 3 in every 4 requests, are fully or partially released[3]. As a comparison, just under 50 % of FOI requests to UK central government in 2018 were fully or partially released, a trend that is causing concern[4]. Other positive signs are that public bodies in Scotland appear mostly compliant and supportive of the law, and a study by the SIC in 2014 found that failure to respond to requests was limited to a small group of five public authorities.[5]

In terms of what sort of information is opened up, a sample of news stories based on FOISA requests gives a sense of the wide variety of areas and institutions opened up by the law, from health policy to graduation fees, across many bodies, from the Scottish NHS to universities.

Table 2: Selected news stories based on FOI

·         ‘HMP Perth one of the most targeted prisons in Scotland for drone contraband’ (Courier  March 6 2019)

·         ‘Lack of consistency’ warning over funding for new Carers Act (BBC, 12 March 2019)

·         ‘Edinburgh homeless forced into B&Bs due to social housing shortage’ (Common Space 5th March 2019)

·         ‘Scottish Government received just two complaints about Gaelic road signs’ (Press and Journal January 21, 2019)

·         ‘Scottish Students paying for graduation ceremonies’ (BBC Scotland 8 January 2019)


1.4. The proactive aspect of any law is harder to measure, and is one area that is often neglected by users and public bodies. In Scotland, all public bodies have a publication scheme listing the information that is proactively published. Research into the UK FOI found that publication schemes had been neglected because they had been superseded by search engines-users don’t consult them but just Google what they are looking for. The SIC’s model publication scheme made it easier, in many cases, to secure internal approval for the publication of information[6]. However, a mystery shop exercise in 2018 found that, although the vast majority of authorities publish some information, there was unevenness in terms of how much and what areas were covered. There was a fear that pro-active disclosure was motivated by a ‘box ticking’ mentality rather than whole-hearted commitment[7].

1.5 Open cultures: Experiments using requests in England, replicated in the Netherlands and elsewhere since, found that FOI works better than informal routes to open up bodies, and can be a force for encouraging publication beyond what the law asks[8]. There was a sense that FOISA has made for more open ‘organisational cultures’ within Scottish public bodies and has also improved records management, driving a ‘more professional’ approach to the recording of information[9].

1.6 As with other FOI laws, Scottish FOISA use is also local and most requests go to local bodies, as the table below shows[10]. The real value of FOI, as one Scottish Information Commissioner put it, is to be found in the pages of local newspapers[11]. Although it is often national scandals that grab the headlines, most of the positive impact of FOI use is at the local level, which means it benefits issues of importance to people’s everyday lives. However, it also means some of the benefits are ‘micro-political’ and hard to trace.

Table 3: Top 10 self-reported authorities by number of requests[12]

Public authority                                                        Requests
City of Edinburgh Council 2,762


Glasgow City Council 2,692


Police Service of Scotland 2,475


Scottish Ministers (including most Government Agencies and Non-Ministerial Officeholders) 2002
Scottish Fire and Rescue Service


Fife Council and Licensing Board 1,750


Aberdeen City Council


South Lanarkshire Council




Perth and Kinross Council and Licensing Board 1,300


Highland Council and Licensing Board




1.7 The law has strong public support. As of 2017 85% of respondents had heard of the Freedom of Information Scotland Act. More importantly 94% agreed (“strongly” or “tend to”) that it is important for the public to be able to access information and 77% would be more likely to trust an authority that publishes a lot of information about its work[13]. Public support has a series of spin off benefits-it is vital in creating compliance and preventing any reversal or negative changes to the law and may has some positive impact on perceptions (in terms of, for example, public trust).

1.8 This again compares well with elsewhere. A global survey in 2015 found fewer than 40% of those surveyed in various countries were aware of their information rights[14]. There appears to be low to very low levels of awareness of laws across many European countries, from Albania to Switzerland, which has a detrimental effect on use and support.


1.9 As with all FOI regimes there have been be delays, inconsistency and problems in Scotland. However, one problem in understanding FOI laws is that the operation of any FOI law is as much about perceptions as realities. Nicola White described the practical operation of FOI is an ‘iceberg effect’, with a minority of high-profile requests attracting attention, making headlines and shaping attitudes, while a hidden stream of more everyday ‘micro-political’ actions are processed without a problem.[15]

1.10 This has an effect at different levels. One Scottish study spoke of how FOI officers and those in daily contact support the principles but take an approach that ‘stresses the letter rather than the spirit of the law’[16]. At more senior levels, officials and politicians are often notified of or copied into the 1 or 2 % of particularly troublesome requests, sensitive cases or, worst of all, those involving them. This can give a distorted view and prompt a series of complaints and, sometimes, resistance that can undermine the law (see below).

1.11 Resources: One Scottish study concluded that ‘resource limitations pose great difficulties for delivering the integrated, concerted and timely responses needed for successful FOI’[17]. While in 2013/14 only 20 Scottish bodies had more than a 1000 requests per year, by 2016/17 it was estimated that 32 bodies would have more than 1000.[18] The combination of growing numbers and a general lack of resources for local government (the main recipients of requests) can undermine a law, and there should be greater help for FOI officers on the frontline in times of cuts.

1.12 However, the idea that FOISA requests are a ‘burden’ is questionable. First, because it is difficult to calculate with any accuracy how much a request ‘costs’, and studies have come to wildly varying estimates[19].

Table 4: estimated cost of individual requests FOI in Pounds


U.K 293
Scotland 189
Ireland 364
Canada 637
Australia 748
U.S. 248

1.13 To illustrate from English local government, Cornwall Council calculated that the average cost of responding to an FOI request was £150 while Bexley Council found it to be around £36 with most requests costing around £19[20]. Secondly, and perhaps more importantly, the supposed ‘burden’ or ‘cost’ must be matched against the vital-but hard to measure-democratic benefits that openness laws create. Putting an imprecise ‘cost’ on the democratic right of transparency is the wrong way of looking at the issue, and is sometimes used as a way to de-legitimise the law.

1.14 Perceptions of use and abuse. Another perceived negative is supposed ‘abuse of the Act’, particularly, it is argued, by journalists. Tony Blair famously claimed that ‘the truth is that the FOI Act isn’t used, for the most part, by ‘the people’. It’s used by journalists’. Journalists do use FOI laws and their requests can make headlines, attract attention and cause nervousness. However, research into the UK FOI law found only a small percentage use it regularly and do, on the whole, behave responsibly with it[21]. There is little evidence of any wholesale abuse.

1.15 In 2019 MSP Alex Neil claimed FOISA was ‘abused by a handful of people’ and that ‘less than 50 per cent of FOI requests made in 2017/18 came from individual members of the public’[22].  From what information exists across Europe, this is the general trend, as the public are normally the largest group but not often the majority[23]. It is not known who is using the law in Scotland. Probably, as with regimes across Europe, they comprise a mixture of members of the public, businesses, NGOs and journalists, some of whom attract more attention than others.

1.16 However one clear problem is uneven use. FOI laws tends to be used by those already engaged in politics, reflecting the broader biases in political participation in terms of gender, education and background[24]. In Scotland, again reflecting trends elsewhere, there is a significant difference across gender with males being twice as likely as females to have made a request. Surveys found two areas of concern amongst young and old: only 25% of secondary school pupils in Scotland know they have freedom of information (FOI) rights. Another poll found uncertainty over FOI rights was highest amongst over 65 year olds, and overall awareness of FOISA was much lower amongst the disabled population[25].

1.17 Behaviour change and the ‘Chilling effect’ In the early 1980s, claims had abounded that FOI had a negative effect on decision-making, creating a ‘chilling effect’ whereby decisions were either not recorded and kept off paper or obscured in some way. Establishing that such an effect is real is very difficult, as it involves proving a negative, asking officials or politicians to admit unprofessional conduct, and disentangling the effects of FOI from all the influences that make or don’t make a record.

1.18 In UK central government there was concern about it, with isolated instances but no general trend and at local government level similarly there appear to be a few exceptional cases but no systematic effect[26]. At the level of officials in both Scotland and England there was some concern at informal recording but also some evidence of a positive professionalising effect on records[27]. However, events in 2018 in Scotland and Northern Ireland point to avoidance taking place at higher levels of government[28].

1.19 Tracing specific cases in Scotland is hard. There was some recent evidence of non-recording in Scotland when, in November 2017, the Ferret unearthed evidence that the Scottish Environment Protection Agency asked for ‘verbal’ updates instead of written documents, mentioning FOI as a reason[29]. The SIC’s investigation of the Scottish government in 2018 also flagged up concerns over the quality of the paper trail in certain areas.

1.20 Political management, manipulation and avoidance. A number of FOI regimes developed techniques to manage certain requests, often involving media and communications strategies designed to mitigate fallout. While this can be sometimes legitimate, it can also mean bending rules, overt political manipulation or crossing the key principle that laws ignore requester identity.

1.21 In Scotland, there appears to be growing resistance and avoidance at the top. In 2018, a report by the Scottish Information Commissioner concluded that the government had sought to create a ‘two-tier’ system delaying journalists or politically sensitive requests[30]. Though the system didn’t change what was eventually disclosed, there was a ‘noticeable difference in time taken and exemptions’ were ‘pushed to their limit’. The SIC report concluded that this was not only against the spirit of FOI legislation but impacted on the trust between the Scottish Government and users. Other signs of unhappiness were seen in successive government’s reluctance to extend the Act (see below), despite repeated warnings that not doing so would weaken the law. [31]

1.22 Again, this is a common theme across other countries. Other avoidance techniques in FOI regimes involve using private email, something done by Michael Gove and, allegedly, key members of the Trump team in the US[32]. Authorities elsewhere have gone further and used systematic ‘silent noncompliance and blunt adversarialism’ to deter requests or even use it as an ‘early warning system’ for trouble[33]. Not only is this against the principles of FOI, it de-legitimises it and sends out poor signals to others about whether to comply with the law.

  1. Have the policy intentions of FOISA been met and are they being delivered?

2.1 FOISA has clearly establish a legal right of access to information, anchored by use and wider public support for the law. The SIC has been recognised as a strong, powerful commissioner, having made a series of important decisions, such as over patient mortality data. One important practical achievement has been few delays[34].

2.2 The law covers ‘more than 10,000 public authorities’ and has extended over time, as its creators intended. The Act has been extended to leisure trusts (2014), private prisons and grant aided and special schools (2016) and registered social landlords (ongoing)[35]. The extensions clearly have public support with ‘at least two thirds of the Scottish population favour extending Scotland’s FOI laws to cover bodies such as housing associations, leisure trusts, PPP/PFI projects and private prisons’[36].

2.3 However, one 2015 study found that ‘some of the newer [Arm’s Length Bodies]’ covered by the law were ‘failing to comply with their newly assumed statutory obligations’ with poor take up of pro-active obligations and publication schemes. This has created various ‘quagmires’ and a ‘postcode lottery’ of compliance.[37]

2.4 There have been fewer attempts to change the law in a retrograde way in Scotland than elsewhere. It was estimated that there was an attempt to weaken the UK law once every 18 months since 2005, including mooted fees, excluding parliament from the ambit of the Act or removing the Monarch and Heir from the law. Countries as diverse as Denmark and Japan FOI laws have re-drawn legislation to weaken access rights. In Scotland public support for the law has helped protect it.

  1. Are there any issues in relation to the implementation of and practice in relation to FOISA?

3.1. FOISA is local. It is at the local, micro-political level where FOISA has become a valuable tool but is also vulnerable. As the data below shows, requests to local councils and other bodies are rising each year, and is the place where officers need most support.

Table 5: Requests for all local government bodies (Councils, Licensing Boards, Assessors, IJBs and VJBs)[38]


FOISA requests EIR requests Total requests received
2013/14 32021 5321 37342
2014/15 36235 5555 41790
2015/16 37102 5546 42648
2016/17 40526 5871 46397
2017/18 42750 6453 49203
Q1 of 2018/19 10843 1466 12309


3.2 Leadership is key. Senior politicians help by speaking positively about the law and promoting FOI and openness generally[39]. This means helping practically with resources and facilitation for support networks and, as one report recommended ‘Scottish local authorities FOI Officers’ Network [should] be more strongly developed, particularly in their capabilities to support learning and communication of good practice’[40]. Senior FOI champions should also be created within individual organisations to assist with compliance. Such promotion and support also means encouraging diversity and, where possible, shifting the imbalance in use around gender and age.

3.3 FOI is part of a wider, changing, landscape. FOI shouldn’t be viewed in isolation but as part of a whole range of mechanisms and laws that now promote openness, from open data to procurement legislation and the Equality Act of 2010. All of these are evolving and changing, functioning together as a system of openness and scrutiny. The different areas could be made to work more closely. The link between Records Management practices and FOI is one vital area, and Rosemary Agnew’s urge that information be created to be released, discoverable and uniform is an important principle.[41] Similarly, open data publication can promote proactive disclosure and may, in some cases, even reduce request numbers[42].

You can also see the Scottish Information Commissioner’s evidence here.

[1] This evidence is based on a series of studies of FOISA in Scotland (Burt and Taylor 2007: 2010: Mc Cullagh 2017; Dunion 2011: Johns 2009) as well as data and research from the Office of the Scottish Information Commissioner. It also incorporates broader research on FOI, including work by myself and others on the UK and other regimes across Europe.

[2] See Worthy (forthcoming) ‘FOI in Europe’, National Statistics (2018) Freedom of Information statistics and Parsons, A and Rumbul, R (2019) Freedom of Information in Local Government Note these numbers are a simply an illustrative snapshot not accurate cross comparisons-only the Scottish and UK statistics cover 2017-2018

[3] SIC (2018) Annual Report and Accounts 2017/18. Edinburgh: SIC.

[4] See IFG (2019) Whitehall Monitor ‘Communicating Transparently’

[5] Burt and Taylor (2007) The Freedom of Information (Scotland) Act 2002: New Modes of Information Management in Scottish Public Bodies?: Report to the Scottish Information Commissioner. Scottish Information Commissioner. Cherry 2013) SIC (2014) Commissioner’s Special Report – Failure to Respond to FOI Requests: extent, impact and remedy

[6] SIC (2018) Model Publication Scheme Monitoring Report 2018.Edinburgh: SIC.

[7] SIC (2017) Commissioner’s Special Report – Proactive Publication: time for a rethink? Edinburgh: SIC.

[8] See Worthy, B., John, P., & Vannoni, M. (2016). ‘Transparency at the parish pump: A field experiment to measure the effectiveness of freedom of information requests in England’. Journal of Public Administration Research and Theory, 27(3), 485-500. See

[9] See Burt and Taylor (2007), 5

[10] This matches the UK experience see the recent mySociety report Parsons, A and Rumbul, R (2019) Freedom of Information in Local Government

Dunion, K. (2011). Freedom of information in Scotland in practice. Dundee, Scotland: Dundee University Press.

[12] SIC (2017) Commissioner’s Special Report – Proactive Publication: time for a rethink? Edinburg: SIC.

[13] See SIC (2017) Ipsos MORI public awareness research 2017 and other polling here

[14] World Justice Project (2015) WJP Rule of Law Index 2015.

rule-of-law-index (last accessed 28 June 2015).

[15] See White, N. (2007) Free and Frank: Making the Official Information Act Work Better. Wellington:

Institute of Policy Studies.

[16] Burt, E., & Taylor, J. (2007). The Freedom of Information (Scotland) Act 2002: New Modes of Information Management in Scottish Public Bodies. Scottish Information Commissioner.

[17] Burt and Taylor Burt, E., & Taylor, J. (2007). The Freedom of Information (Scotland) Act 2002: New Modes of Information Management in Scottish Public Bodies?: Report to the Scottish Information Commissioner. Scottish Information Commissioner.

[18] SIC (2017) Commissioner’s Special Report – Proactive Publication: time for a rethink? Edinburgh: SIC.

[19] See Colquhoun, A. (2010) The Cost of Freedom of Information. London: Constitution Unit.

[20] See Worthy, B., Amos, J., & Bourke, G.  (2011). Town Hall Transparency. The Impact of Freedom of Information on Local Government in England. London: Constitution Unit.

[21] See Hazell, Robert, Worthy, Ben and Glover, Mark (2010) Does FOI Work? The Impact of the Freedom of Information Act 2000 on British Central Government. London: Palgrave Macmillan

[22] See Scotsman ‘Freedom of Information laws being abused by ‘handful of Scots’’ 10 Jan 2019

[23] See Worthy (forthcoming) ‘FOI in Europe’

[24] See Hazell et al (2010) Does FOI Work? The Impact of the Freedom of Information Act 2000 on British Central Government. London: Palgrave Macmillan.

[25] SIC (2018) Ipsos MORI Young People in Scotland Research: SIC (2017) Ipsos MORI public awareness research 2017

[26] See Worthy, B. (2013) ‘“Some are More Open than Others”: Comparing the Impact of the

Freedom of Information Act 2000 on Local and Central Government in the UK’. Journal

of Comparative Policy Analysis, 15 (5): 395–414 and Shepherd, E., Stevenson, A., and Flinn, A. (2011) ‘Records Management in English Local Government: The Effect of Freedom of Information’. Records Management Journal, 21 (2): 122–134.

[27] See Richter, P., and Wilson, R. (2013) ‘“It’s the Tip of the Iceberg”: The Hidden Tensions between Theory, Policy and Practice in the Management of Freedom of Information in English Local Government Bodies – Evidence from a Regional Study’. Public Money & Management, 33 (3): 177–184.

[28] See Worthy, Ben (2018) ‘Fight, Hide, Avoid: Resistance to Freedom of Information Laws’

[29] The Ferret (2017) ‘Pesticide report suppressed after freedom of information warning’ 8 November 2017

[30] Scottish Information Commissioner (2018). Intervention Report. Edinburgh: SIC.

[31] McCullagh, K. (2017). ‘Information access rights in FOIA and FOISA–fit for purpose?’ Edinburgh Law Review21(1), 55-87.

[32] See Worthy, Ben (2018) ‘Fight, Hide, Avoid: Resistance to Freedom of Information Laws’

[33] See Walby et al 2017 Luscombe, A., & Walby, K. (2017). Theorizing freedom of information: The live archive, obfuscation, and actor-network theory. Government Information Quarterly34(3), 379-387: Camaj, L. (2016). From ‘window dressing ’to ‘door openers’? Freedom of Information legislation, public demand, and state compliance in South East Europe. Government Information Quarterly33(2), 346-357.

[34] Holsen, S., & Pasquier, M. (2012). ‘Insight on oversight: The role of information commissioners in the implementation of access to information policies’. Journal of Information Policy2, 214-241.

[35] Cherry, M., & McMenemy, D. (2013). ‘Freedom of information and ‘vexatious’ requests—The case of Scottish local government’. Government information quarterly, 30(3), 257-266.

[36] See for example SIC (2009) Public Awareness Research Report. Edinburgh: SIC.

[37] Liddle, C., & McMenemy, D. (2015). ‘A Scottish freedom of information regime for a denationalised environment: rhetorical or authentically practical?’ Information & Communications Technology Law24(3), 225-241.

[38] Data courtesy of the SIC

[39] Gibbons (2019) The FOI Officer’s Handbook. London: Facet Publishing.

[40] Burt and Taylor (2007). The Freedom of Information (Scotland) Act 2002: New Modes of Information Management in Scottish Public Bodies? Scottish Information Commissioner.

[41] SIC (2017) Commissioner’s Special Report – Proactive Publication: time for a rethink? Edinburg: SIC.

[42]Sunlight Foundation (2018) ‘Siblings or Silos?’