Was it just a simple database query?
So the sensitive personal details of 25m people has been lost and there is a huge political furore over it. Whose fault is it? As far as I can see, (and this is my personal opinion,) blame must lie with IT, specifically the IT contractor and either the contract they work with or the perception of that contract.
The National Audit Office asked HM Customs and Excise for child benefit in “desensitized form”. Sensitive details were specifically asked to be removed, ostensibly to make the file size smaller. This would require a bespoke query to be run. It was deemed too costly so it was assumed that a full extract of the data would do. The fact that this was then burned to a CD, posted unregistered mail and lost is not the point (that is stupidity). What is the point is the IT contract prohibits the business (in this case the governmental offices) to do their job properly.
What sort of contract demands extra payment for a simple database query for “NI numbers, child benefit numbers and children’s names in order to select a risk-based sample of cases to audit as part of anti-fraud work“?
Surely this is an extra request that an experienced database analyst could easily run in the course of a day? If not you must ask why not – is it because the database is badly designed with nested tables and stored procedures and stuff that would make a decent DBA go eugchhh (I’ve seen that happen). If this is the case, the IT contractor has done a bad job; if an electrician worked in your house and left a mess of an electrical installation, would you keep employing them, even if they were cheap?
Maybe however it would not have incurred a cost and this was just the perception; “we must not… run additional scans/filters that may incur a cost to the department”. If this was case it suggests a breakdown in the relationship between the business and IT, with tendency towards the confrontational and transactional rather than co-operation and partnership.
Organisations that outsource their IT often fail to realise what the true costs are. Anything outside the terms of the contract is a change request. It is not unusual for the request itself to incur a cost (someone has to write the documentation, specifiy the design, estimate the effort) before a line of code is written. (At one organisation I worked with that had outsourced their IT function, I was told that to add some basic client-side field validation to a single field on an application form on their website was likely to cost in the region of £60k). The business starts to believe that everything costs and IT becomes a hindrance and a vicious cycle commences.
How could things have worked differently? Let’s say the HMRC IT department was run on more lean/ agile lines. With agile it embraces change. The request comes in (let’s assume such requests are not regular occurances) and in the morning stand-up the BA describes the request and asks the developers for its feasibility in a word. Someone says “yes, I ran a simiar querry last month, it’ll take me ten minutes”. (In reality double or treble that estimate), but it will not have an imact on the developers ability to get thier prioritised work completed. Alternatively the developers say “given the database structure we have inherited that’s a lot of effort” or the project manager says “another request?! pritoritise it like the others!” and it is prioritised in the weekly iteration planning meeting (pushing something else out) and then it gets done.
My hope is that when the inevitable investigation takes place, they don’t just look at the policies and procedures, but also at the underlying structure of the way that IT is managed.
>> Must a database query be expensive?
It is when it returns the same article 5 times for your front page 😉
Kidding aside, you have to wonder if incompetence is a requirement for winning government IT contracts.
The email trail is found here – http://www.bbc.co.uk/blogs/nickrobinson/Informationrelatingtochildbenefitdata.pdf
it makes interesting reading
Good post, Marc. Fail to realise what the true costs are, indeed.
In today’s paper it said the cost of producing the sample requested (hopefully involving nothing more than writing “select field1, field2, field3… limit X” instead of “select *”) was £5000! Five thousand pounds? Just to replace a few characters in an SQL statement? Is that a lot, or what?
Is anybody else irked that the data keeps on being described as “lost”? It’s not lost. They’ve just open sourced it. They still have a copy of the data. If they had deleted the records from the database so they don’t know who’s claiming child benefit any more then fair enough, that’s lost.
Has anybody else noticed that quite a lot of this data could do with being open sourced anyway? How many of the 25m names that could now be in the hands of criminals are criminals themselves? Are my neighbours upstairs claiming child benefit? For how many children? One, which they have? Fifteen?
The underlying cause? It could be because of this: competent civil servants don’t like being made subordinate to incompetent outsiders from private corporations. The civil servants retaliate by not being exactly, uh, helpful. When asked, “So how do things work around here?” the civil servants aren’t inclined to give particularly good answers. Or they could go beyond that, and give the wrong information.
Could that be how the enormous cost overruns get generated? In HMRC’s case they had a big contract with EDS which got scrapped, and then another contract with someone else that promptly tripled.
Did anyone else think that merging Inland Revenue and Customs was a dodgy idea?
The public reaction reminds me of the Michelangelo computer virus in 1991. People were scared their computer was going to be destroyed by Michelangelo. So they went round checking their computers for the first time and finding every virus except Michelangelo.
Today, people reading their bank statements for a change, and finding all kinds of unauthorised direct debits that have been there for ages. Changing their PINs for the first time ever. Overall, security boon. The Government should do this more often.
By the way, check this if you haven’t seen it already, “Life inside the beleaguered HMRC”:
http://news.bbc.co.uk/1/hi/uk/7104395.stm
Gives an insight into what happens inside a government department when it is forced to merge with another while having its IT outsourced.