Research on Open Data and Transparency

Leave a comment

Fight, Hide, Avoid: Resistance to Freedom of Information Laws

trump-oval-office-advisorsAlthough there is no clear evidence of any government-wide, systematic resistance to freedom of information laws in the UK or elsewhere, there are examples of avoidance and resistance, often at very high levels and in sensitive situations. There appear to be numerous strategies to avoid or mitigate FOIs. Some clearly contravene the law while others run against the spirit of openness. Examples include:

Use of private email systems. In the UK, Michael Gove was found to be using a private system when he was Education Secretary, leading to an ICO ruling in 2012 clarifying the distinction between public and private communications (see below). Allegations have been made about other politicians trying to circumvent FOI through private emails, from Sarah Palin to various members of Donald Trump’s team, including, according to CNN, Steve Bannon, Jared Kushner and Ivanka Trump. Most famously, Hilary Clinton’s use of a private server and email may have violated the Federal Records Act and later archiving regulations, though whether she did this simply for convenience of access (as Clinton claimed) or to hide from FOI requests and potential Congressional investigation (as her opponents alleged) is unclear.

Non recording of information (known as the ‘Chilling Effect’). This type of non-minuted, informal mode of decision-making can take many forms, stretching from using phone calls, verbal decisions or disposable post-it-notes to having informal meetings entirely without records. The Swedish government, with the world’s oldest openness law, calls this the ‘empty archives’ phenomenon and mentions of ‘post-it-note’ approaches date back to the US in the 1960s and Australia in the 1980s. There was some recent evidence of non-recording in Scotland when, in November 2017, the Ferret unearthed evidence that the Scottish Environment Protection Agency asked for ‘verbal’ updates instead of written documents, mentioning FOI as a reason (see the emails here on page 11-12). This chain from 2008 shows a UK official apparently urging others to delete emails because of FOI.

Informal meetings, made famous by Tony Blair’s (pre-FOI) ‘sofa government’ style, are more difficult to prove. However, in March 2018, giving evidence to the RHI ‘Cash for Ash’ Inquiry, the Head of Northern Ireland’s Civil Service admitted that ‘the practice of taking minutes had “lapsed” after devolution’. The BBC quoted him as saying “Ministers liked to have a safe space where they could think the unthinkable and not necessarily have it all recorded” and the ‘DUP and Sinn Féin were sensitive to criticism and in that context, senior civil servants had “got into the habit” of not recording all meetings’. He said this was done on the basis that it was sometimes “safer” not to have a record which might be released under Freedom of Information.’ He did, however, qualify this and ‘agreed with the inquiry panel that when it came to ministerial decisions on matters of public money it should be recorded.’ Former Head of the UK Civil Service Gus O’Donnell had made similar claims for the UK Cabinet being ‘hamstrung’ by FOI, though relied on anecdote and hypotheticals to illustrate his point (see this analysis).

Involvement of press advisors and media officials. The Scottish Information Commissioner spoke of claims of ‘deliberate delaying tactics and requests being blocked or refused for tenuous reasons’. One study of more than 2000 ATIA requests in Canada by Professor Alasdair Roberts found exactly this happening, with ‘requests that were identified as sensitive, or which come from the media or political parties, found to have longer processing time, even after other considerations are accounted for’. Procedures within government focused on ‘giving special attention to politically sensitive requests for information’, ‘flagging’ difficult requests electronically and using communication officers and Ministerial offices as gatekeepers for releases. Similar claims were made in the UK for the ‘Clearing house’ that was intended to co-ordinate responses across government.

Timing, requesting and release. One approach seen in Ireland and elsewhere is for government departments to deprive journalists or others of scoops by publishing request responses online simultaneously. Alternately, releases can be buried by publication on a busy news day or at the end of a week. Even more creative has been the approach of the FBI, who recently asked that FOIA requests only be sent by fax (though the FBI FOI email appears to still exist).

One important point to make is that while high profile examples exist, mainly from politicians, we found officials were generally as fearful of not having a record as having one, with the pressure from bosses or potential judicial review to keep a record outweighing the danger from FOI. Research at local government level in Scotland and the UK found there to be a positive, professionalising effect on records.

Two problems: Is it happening? Is it FOI?

Any research in this area hits two major difficulties.

How do we know it is happening? It is unclear how widespread such activity is. Are the examples above are the tip of the iceberg or isolated misbehaviour? By its nature, it is very difficult to prove a negative (i.e. something didn’t happen) in terms of records or avoidance and is often reliant on anecdote. One study found many comments to ‘not record’ were often jokes or light hearted quips. However, the problem is that the myth could become reality. One previous UK Information Commissioner warned that constantly ‘talking about a chilling effect’ can make it a ‘self-fulfilling prophecy’, so when Tony Blair claims FOI laws make politicians ‘very cautious’ or David Cameron argues it is ‘furring up the arteries of government’ they could make it happen.

How do we know it is FOI? Isolating and disentangling FOI as the cause of the problem is almost impossible. Fear of leaks, the arrival of new technology, new decision-making styles and the key question of resources all influence how and if records are kept. Ex-Downing Street Communications Director Craig Oliver complained that accusations made during the Brexit referendum campaign that Cameron was avoiding FOI by using WhatsApp misunderstood that it was simply quicker and more secure than email. Concerns over the non-recording of information go back far into the past. As the Justice committee pointed out, the 2004 Butler report raised serious concerns over Tony Blair’s use informal meetings and ‘sofa government’ a year before FOI came into force. James Callaghan’s famous quip captures the nuances and grey area between official and unofficial disclosure: ‘I brief, you leak’.

What can be done?

There exists rules on records and destruction in almost all FOI laws (and Scottish FOISA has them in section 60 and 61). In March 2018 the first charges were bought for a Councillor in Thanet ‘blocking, concealing or destroying records held by Thanet council “with the intention of preventing the disclosure by TDC of information”. This also touches on rules around whether emails are archived or deleted and more generally what records are kept and for how long, issues that proved controversial recently in relation to Scottish care homes and a panel looking into paramilitary disbandment in Northern Ireland.

Some laws also cover private emails being used for public business. In 2012 the UK ICO, for example, created new guidance in wake of Gove email controversy, making clear that texts or emails are covered if ‘information “amounts to” public authority business’ or “generated in the course of conducting the business of the public authority”.

In 2017, former Scottish Information Commissioner Rosemary Agnew called for a ‘radical re-think’ around how FOI works in terms of disclosure, recording keeping and release, so bodies ‘invest in dissemination of information as a way of doing business’ rather than through requests.

The letter from Scottish journalists in 2017 called for a specific legal duty to record decisions. The UK information Commissioner has also championed the creation of a ‘duty to create records or “duty to document”…a positive duty in law to create records of significant decisions, actions and events’. That means public bodies must keep ‘records explaining and providing context to why a specific course of action was taken [including] Minutes of important meetings, decisions, that led to policy change and new initiatives’. Such a law exists in British Columbia following an email deletion scandal, though campaigners claim the law lacked teeth as it did not mandate the creation of records.

Further Reading

  • Worthy, Ben and Hazell, Robert, Disruptive, Dynamic and Democratic? Ten Years of Freedom of Information in the UK (December 28, 2015). Parliamentary Affairs, Forthcoming. Available at SSRN:
  • Roberts, Alasdair S., Spin Control and Freedom of Information: Lessons for the United Kingdom from Canada (December 20, 2003). Public Administration, Vol. 83, No. 1, pp. 1-25, Spring 2005. Available at SSRN:




Leave a comment

Gender Pay Gap Transparency: Will It Work (updated)?

Gender pay gap two

In July this year the BBC, with a bang and probably a muffled whimper, released details of its highest earners. It predictably provoked outrage at the overpaid but also, less predictably, re-ignited the debate on the gender pay gap. Political leaders were quick off the mark to condemn the stark gap between male and female presenters. Theresa May criticised the BBC for paying women less for doing the same job as men and Jeremy Corbyn suggested a pay cap.

How Big is the Gender Pay Gap in the UK?

Measuring the gap is tricky. Here’s a summary from the ONS of some of the key  figures for the UK in 2016:

  • Average pay for full-time female employees was 9.4% lower than for full-time male employees (down from 17.4% in 1997).
  • The gap for all employees (full-time and part-time) has reduced from 19.3% in 2015 to 18.1% in 2016 (down from 27.5% in 1997).

So the gap is nearly 10% or 18% depending how you measure it. This FOI request shows how the gap has altered in the past decade or so in the UK. The pay gap is high, and higher than the UK, in many other parts of the EU, where the UK sits about seventh from the top: ‘across Member States, the gender pay gap varied by 21 percentage points, ranging from 5.5 % in Italy and Luxembourg to 26.9 % in Estonia’.  To get some sense of the scale of the problem, in 2015 ‘women’s gross hourly earnings were on average 16.3 % below those of men in the European Union (EU-28) and 16.8% in the euro area (EA-19)’.

Gender pay

So what’s being done?

Something, finally. Successive governments have been determined to open up gender pay. Gender pay transparency is actually a Labour policy from long ago in 2010. Theresa May’s sound and fury has been heard before. Back in 2010 a certain Theresa May, writing in the Guardian no less, already claimed she was ‘clearing a path towards equal pay’ in 2010.What she forgot to say was that the Conservative-Liberal coalition she was part of didn’t actually engage the requirement to publish gender pay, contained in section 78 of (Labour’s) Equality Act of 2010. They wished to pursue a ‘voluntary scheme.’ Alas, few volunteered. Four years into the scheme only 4 companies had reported.

David Cameron, in a second wind of revolutionary ardour, committed to engage mandatory reporting (5 years after not doing so). This would ‘eradicate gender pay inequality’. All companies over 250 employees would have to publish the data. As of April 2017 companies have a year to produce the data and a written statement explaining, if there is a gap, what action will be taken. After 2018 organisations not publishing will be contacted by the Equalities and Human Rights Commission. The light of transparency will, it is hoped, end pay inequality.

How’s it going so far?

Although a number of companies have been voluntarily publishing the data, as of May 2017 only 7 companies had reported. An email from the GEO from July informed me there were now 26 and, according to a spreadsheet on, there are now 40 (see update below).

That’s from an estimated 7,000 companies with 250 or more employees. On a very generous rounding up, that means only 0.57% companies have reported. At this rate, if the Equalities and Human Rights Commission must send out notices next April, they’d better fire up the old email wizard or buy plenty of stamps.

There is also concern over the coverage of the policy, as this paper argued:

Only around 6000…of the 4.7 million businesses in the UK have more than 250 employees. Thus, around 59% of employees would be unaffected by the provisions if reintroduced in their current form.

The government calculated that the pay gap reporting would cover 34% of businesses with a further 12% covered by regulations for public bodies, meaning ‘approximately 8,500 employers, with over 15 million employees’ would be opened up.

The Women and Equalities Select Committee argued that the data needed to be broken down by age and status, and applied to companies with less than 100 employees-moving to 50 in the next two years (the government argued smaller businesses may find it ‘difficult to comply due to system constraints’). May appeared to promise further action on gender pay before the General Election and there was a mention of more data in the manifesto but, like much in that doomed document, we’ll probably never know what, if anything, was intended.

What will publication do?

On a practical level much may depend on how the data is published and who accesses or uses it. Underneath this is a serious question for all transparency policies: what exactly will publication do? While opening up such data is useful, measuring gender inequality is highly complex and a ‘moving target’ and is caught within wider issues of female representation in public life, professions and boardrooms. There is a long way between publishing data on a problem and ‘eradicating’ it.

In the case of the BBC, the controversy has led to a letter and high profile lobbying but will it lead to real change? Tony Hall has set a deadline for action (2020) and promised representation and consultation. Now FT journalists may strike over it and Sheryl Sandburg has weighed in.

The former Secretary of State for Equalities spoke of how publication of gender pay gaps would have benefits in terms of ‘transparency, concentrating the mind and helping people make employment decisions’, all of which are either a bit tautological (transparency will make everything more transparent) or vague. More worryingly, a survey for the Young Women’s Trust found that many business were unconvinced ‘44 per cent of those making hiring decisions say the measure introduced last April will not lead to any change in pay levels’. In the 2016 the Women and Equalities Select Committee concluded that pay publication focuses attention on the issue but is not a solution: ‘It will be a useful stimulus to action but it is not a silver bullet’ and recommended that ‘the government should produce a strategy for ensuring employers use gender pay gap reporting’.

As the committee put it, openness is ‘a first step for taking action rather than an end in itself’. It is hoped that publication could drive up pay and standards-though the evidence of what publishing pay generally does is rather mixed (publishing executive pay appears to push overall pay up not down). Companies could be embarrassed into action but could, equally, ignore it, wait for the storm to blow over or kick it to the long grass with a consultation.

As with all sorts of openness, mandating publicity is only the start. Gender pay data must not sit on a spreadsheet but needs to wielded, repeated and find a place as a staple, symbolic benchmark-and become, like the ‘scores on doors’ restaurant star rating, a mark of quality or reason to avoid.

8th March 2018 Update

As of today there are 1,691 companies reporting (or 24% of estimated companies if the stats are right though the FT puts it lower at 1,442 with a five out of every six not reporting). Though the UK government is improving its own gap, it seems the Cabinet Office has the largest. A series of FOI requests by the Ferret in Scotland has shown some improvement in public bodies since Nicola Sturgeon’s equality drive but they have published the names of 28 Scottish bodies with a gender pay gap larger than the Scottish national average. This month even Parliament was found to have a 10% pay gap due to the outside earnings of MPs. This is higher, ironically, than at the BBC. As the final reporting deadline approaches, the Labour party has also called for fines to punish those not publishing.

Images from UK government equality report and EU gender pay gap pages

Leave a comment

Freedom of Information and Universities UK: Could it? Should it? Why Should It Worry UKIP?


Could Universities UK be covered by the FOI Act, as this e-petition asks? I’ll try to answer this, using the old EU joke, to see if it works in theory as well as in practice.

In the UK, universities are covered by FOI (as Vice Chancellors have found out) but UUK is, as of this moment, not. You can’t really ask a body not covered (though this hasn’t stopped people trying). So could and should UUK be covered?

Doubt it…

Most FOI Acts cover ‘public bodies’ and, to very different degrees, private bodies doing work for them. This means legally ‘bodies that appear to be exercising functions of a public nature or who are providing, under contract with a public authority, any service whose provision is a function of that authority.’ Only two Access to Information Acts in the world directly cover private bodies as a whole (South Africa and Nigeria seeing as you asked/need to know in case of some unimaginably obscure pub quiz).

The legal status of UUK looks pretty far from being a public authority. UUK is the ‘voice’ of UK universities made up of the ‘vice-chancellors or principals (executive heads) of universities in the UK’. Legally speaking it is a ‘Company Limited by Guarantee’. This means, according to my brief foray on Wikipedia, it is like a private company but, as pointed out here, ‘a major difference is that it does not have a share capital or any shareholders, but members who control it’.

This status is often used for charities and others to prevent liabilities. According to Wikipedia[1], common types of companies limited by guarantee are:

So, a big fat no in theory?

On a practical level, it takes ages. Gordon Brown (that clever PM who didn’t call snap elections) took two years just to think about who to extend it to. By the time he’d decided and extension was begun, David Cameron (the one who called snap referendums) was in power. A full five years.

Or Could It?

The government can extend the FOI to cover broadly who they want. Section 5 of the Act allows government to re-designate bodies (sorry to get a bit legal), roughly as long as it has a ‘number of definable relationships with public authorities’.

The first actual section 5 extension (in November 2011) designated the Universities and Colleges Admissions Service (UCAS), the Association of Chief Police Officers (ACPO) now known as the National Police Chiefs’ Council (NPCC) and the Financial Ombudsman Service (FOS).

As of May 2018, you’ll be pleased to know, another 15 bodies will be covered including my favourite Independent Monitoring Board The Independent Monitoring Board for the Military Corrective Training Centre and my second favourite development Corporation the Ebbsfleet Development Corporation. The separate Scottish FOI Act has been extended to cover ‘secure accommodation for children and young people, grant-aided schools, independent special schools and Scottish Health Innovations Limited’.

But even if the government want to do it, can they justify it or, to quote George Galloway (rather out of context), are they ‘mad enough to do it?’ A broad moral case to open them up could be made. Many other university bodies in the same ‘regulatory ecosystem’, admittedly of very different legal status, such as UCAS or HEFCE are covered by FOI, as is the Student Loans Company (go on-what are you waiting for?). As the ‘voice of universities’ it also does seem to at least appear to be exercising functions of a public nature’ and has a ‘number of definable relationships with public authorities’. The Public Accounts Committee, among others, has also repeatedly called for FOI to ‘follow public money’ and post-Grenfell there was a further attempt by Private Members’’ Bill to extend FOI.

Even the fact it is a ‘Company Limited by Guarantee’ may not totally exclude it. What’s interesting is that almost all the wikipedia list of examples have been the subject of discussion about extension. In 2009 and again in 2016 (post Kids Company) there was some serious discussion of making charities subject to FOI. Though various investigations have concluded they shouldn’t be, the possibility is still there. Even more fun, Network Rail came to be covered by FOI almost by mistake, when its legal status changed for accounting purposes (you can see what people have been asking here-I now know how many platforms every station in the UK has and the length of all of Reading station’s platforms).

Political parties aren’t covered by FOI in the UK but in some countries, such as Indonesia according to an ongoing discussion, it seems they are. There was a hellish row in India about trying to get Indian political parties covered by their Right to information Act and, seeing as we are talking cricket, pressure to make the Indian Board of Control for Cricket a public body for the purposes of the law.

And so?

Technically, UUK is not covered. For the present, you can ask for ‘material held by a private company “on behalf of” a public authority’. This is very confusing and no one really knows what’s going on but it may mean some UUK material is covered. Moreover, many of the public authorities UUK interacts with, meets and emails-such as all the universities where all the VCs are based, the government departments they work with etc-are covered by FOI.

Even if Theresa May decided, in yet another flush of pointless and futile Brexit distracting incompetent zeal, to get out of the chimney she was in and extend the law (thus robbing yet another Ed Miliband policy), it would take time. One thing to reflect on. The more such bodies are covered, the closer we could get to getting UKIP covered by FOI. Oh, the emails, as America’s new racist in chief would say.

See my piece on VC expenses here and on opening up the private sector. There’s some better legal analysis by FOI man here and a post on wider legal openness developments from Act Now.

[1] Note to my students: don’t ever use or cite Wikipedia for essays. I am doing it ironically.