Email from 1999 reveals Post Office ECCO+ system crash problems

[TrueBlueTerrier]

https://www.computerweekly.com/news/366 ... h-problems

An email from 1999 reveals a system used in hundreds of Post Office branches in the 1990s, known as ECCO+, experienced freezing which could have caused lost transactions.

A former Post Office manager told Computer Weekly that when these freezes occurred, the system lost vital transaction information when it was rebooted due to its inability to revert back to its previous state.

Information about ECCO+, its errors and effects on users is hard to come by, but the events of the Post Office scandal make it essential that the ECCO+ system issues and the effects on branch staff and subpostmasters are examined.

The email, published by the Post Office Horizon public inquiry and titled “system freezes”, was sent between Post Office staff in 1999 during the roll-out of the ill-fated Horizon system in thousands of branches.

At the time, system freezes were being experienced with the Fujitsu/ICL Horizon software as it was being rolled out across the Post Office branch network. According to the email, staff at the system’s supplier wanted to “prepare a line” to play down these issues by describing them as being at a level to be expected with a computer-based system. The email included a request for data on system freezes experienced by users of ECCO+ to allow a comparison with the “potential high-severity acceptance incidents” being experienced with Horizon.

In the email, a Post Office executive wrote: “As you probably know, one of the potential high-severity acceptance incidents still outstanding [with Horizon] relates to the number of times that the system freezes and has to be re-booted before it can be used again. [Fujitsu/ICL] seem to be preparing a line on this that the problem is only at the level that we should have expected from a computer-based system and is in line with industry norms. We need to be in a position to refute this.

“In parallel, it would help our case if we could show that we do not have this level of incident with our existing systems,” it said. The email also requested “performance stats or help desk analyses” for other systems including ECCO+.

ECCO+ was used in Crown branches in the 1990, with some subpostmasters who took over crown branches having also used it. There could have been more than 600 branches converted from Crown branches to sub–Post Offices, which used ECCO+ at some point, according to the National Federation of Subpostmasters (NFSP)

The ECCO+ data revealed information for three months in 1999. From April to June that year, there were 56 calls to the helpdesk regarding ECCO+ system freezes.

Given there were 6,305 terminals using ECCO+ in total, the number is small, but former Post Office manager Rupert Lloyd Thomas told Computer Weekly that the numbers “are probably just the tip of the iceberg”.

He said that when branch staff called for support, they were probably told to reboot, which meant they probably rebooted again without calling the helpline the next time it happened. With system freezes, account balancing problems would occur for ECCO+ users because the system had no way of automatically reverting to the last known good configuration, resulting in lost transactions, he added.

Lloyd Thomas spent nearly three decades at the Post Office from the early 1970s and has detailed knowledge of ECCO+.

“ECCO+ was chronically unreliable, it was a cheap and nasty system,” he said. “I had a lot to do with work trying to get that system fixed. There was a crisis with it in 1996 and an investigation was commissioned. It was carried out by the Post Office IT department and a report was produced. The gist of that report was that the system was defective.”

Lloyd Thomas said the investigation culminated in what was known as the Kirk report in 1996: “They didn’t want to do anything, and I made such a fuss about it and said, ‘We can’t have these people in the front line with a system that’s crashing all over the place’.”

He said an IT executive at the Post Office produced the report which found there was a condition known as sector slip, which occurred when the system went down and attempted to recover.

Following the new evidence and in light of the Horizon scandal, the NFSP is calling for a “full investigation into ECCO+”.

Unheard stories
The struggles of users of the Post Office’s Horizon system are well documented, and those that suffered at the hands of faults in its Capture software are being heard, but there is a cohort of subpostmasters yet to be properly examined.

Calum Greenhow, CEO at the National Federation of Subpostmasters (NFSP), said: “The concern the NFSP has is it now appears that not only did Horizon and Capture have errors, bugs and defects, the evidence is emerging that a precursor to Horizon, ECCO+, which could have been installed in around 640 Modified Sub-Offices run by subpostmasters, had similar problems that resulted in them having to refund Post Office for losses that may have not been user error. As such, we are calling on government to carry out a full investigation into ECCO+ in the same fashion as they have with Horizon and Capture.”

Last year, when concerns were raised about the ECCO+ system, the government said it would look into them, with a DBT spokesperson confirming: “We are looking into the ECCO+ system used in Post Office branches and its potential impact on subpostmasters and branch workers.”

Stories of financial struggles experienced by subpostmasters, who inherited the system when they took over Post Office-owned Crown branches, are coming to light.

Unlike the Post Office’s Horizon and Capture systems, which were used in sub-Post Office branches, ECCO+ was mainly used in Post Office-owned Crown branches. But subpostmasters who took over former Crown branches suffered financial difficulties after inheriting the Post Office’s ECCO+ system.

Janette Armour was a subpostmaster in Scotland. From 1977, she worked in Crown branches and was promoted to the Post Office headquarters in Glasgow, before buying a sub-Post Office branch near Glasgow, with her husband.

They were then asked if they would be interested in taking over a Crown Office in East Kilbride, which they did in 1994. ECCO+ was used in the branch, which had six counters.

“The system didn’t work properly the first day it was installed,” said Armour.

She had worked manually in her previous Post Office branch roles and received a day of training on ECCO+, which she described as “fine”.

Problems from the beginning
But shortfalls began almost immediately. “Right away, we began having losses every week,” Armour said, adding that she had not experienced problems like this before. “We’d be there till midnight sometimes, going through everything, double checking and running the reports.”

They were losing around £350 a week, said Armour. “I told my area manager, and she just seemed unconcerned about it.” But things took a turn for the worse when there was a £500 loss, leading Armour to contact the Post Office special investigation branch.

“They came out and interviewed all the staff, said that two of the staff were stealing and they were accused and they walked out,” Armour said, adding that she is now trying to contact the former staff to tell them what has happened. “I know that being accused of stealing ruined one of their lives.”

She estimates that her husband and herself covered at least £16,000 worth of shortfalls a year, for two and a half years: “We lost everything we had made selling a previous business.”

This took its toll, and Armour and her husband were eventually advised to sell their smaller Post Office branch. But, after losing more money, they had to sell their former Crown branch as well. On the last day at the branch, it was revealed to Armour that even Post Office staff knew ECCO+ was problematic.

“On the transfer day, the audit team came out and had balanced £350 short and the auditor said to me ‘You’re balanced’ and I said, ‘Well, no, we’re £350 short’, and she said, ‘Did no one tell you that you can’t use the reports from ECCO+ because they don’t add up properly?’”

Armour and her husband eventually bought another branch and used the Horizon system where they also experienced problems. It took the couple more than 10 years to pay off debts they had built up because of problems at their branch that used ECCO+. They are part of the Horizon Shortfall Compensation scheme but have not been permitted to make claims for losses experienced when they were using ECCO+.

Unanswered questions
Computer Weekly asked the Post Office how many former Crown branches were taken over by subpostmasters (franchised) between 1980 and 2000, and how many of these branches continued to use ECCO+ software after they were taken over.

The Post Office was also asked how many prosecutions of subpostmasters who used ECCO+ it has instigated and how may Crown branch staff were punished as a result of unexplained losses in branches using ECCO+.

The Post Office did not answer the questions but suggested a freedom of information (FOI) request be submitted. Computer Weekly submitted an FOI request and the Post Office said it was unable to answer due the request exceeding cost limits.

The NFSP submitted a freedom of information request for the Kirk report and the issue of sector slip, but the Post Office said it could find no evidence of either.

“We have carried out reasonable and appropriate searches to our email repositories, SharePoint sites and Team Chats for any mention of the ‘Kirk report’, ‘Jack Kirk’ and ‘sector slip’ or ‘ECCO+’, between 01 January 1990 to 31 December 1999. However, these searches returned nil results. We therefore confirm we do not hold the information you have requested.”

A Post Office spokesperson told Computer Weekly: “It is concerning that ECCO/ECCO+, a system which was in use in some branches in the 1990s, may have caused accounting issues. We have been in contact with the Department for Business and Trade, and there is still very limited information on ECCO/ECCO+, so it is important that any issues are raised so they can be properly reviewed.

“We encourage anyone who believes they may have been affected by accounting problems linked to ECCO/ECCO+ to come forward to the Department for Business and Trade, or, if they prefer, the NFSP.”