Contents 
Summary
The
failure
Effects
on service
Findings
of the Oftel investigation
Compliance
Action
Results
of Compliance work
Conclusion
Summary
This
report sets out the background to the failure at a BT site in Southampton
on 25th April 2002 and summarises the result of the Oftel
investigation into the causes of the failure, details the corrective
actions performed by BT and presents Oftel’s conclusions.
The
failure
1.
A sequence of events was triggered by the failure of rectifiers at a
BT site in Southampton on the afternoon of 25th April 2002.
The failure of a power supply alarm resulted in this event not being
detected. Batteries carried the load for approximately one hour but
when they were exhausted, at approximately 18.30, the transmission equipment
failed. This resulted in the isolation of a number of local processors.
2.
As a result substantial loss of service occurred, including access to
999 fixed line service, in the Hampshire area.
3.
After the rectifiers were repaired, service was gradually restored,
and 80% of the affected customers had service by midnight. BT declared
the incident over by 03.55 in the following morning.
Effects
on service
4.
The effects on service were as follows:
- Approximately
430,000 customers in the Hampshire area lost service (including access
to 999) for varying periods between 18.30, 25th April 2002
and 03.55 on the 26th April. Calls to 999 from mobiles
continued to operate and all were answered during this period
The Hampshire police
emergency control room was isolated for a period for fixed line calls,
though calls were re-routed via VHF radio from a neighbouring control
room.
Findings
of Oftel investigation
5.
BT fully co-operated with Oftel’s investigation into this incident.
The findings of Oftel’s investigation are as follows:
- The failure of
an alarm to report the original rectifier failure critically delayed
deployment of staff to carry out repairs to the power supply;
- Subsequently
the widespread isolation hindered efforts to mobilise appropriate
staff to repair the failure;
- Once the scale
of the failure was recognised, restoration was carried out in a professional
manner with appropriate management co-ordination;
- The isolation
of a large number of customers, including the police control room,
over an extensive area was caused by a single point of failure; and
- BT immediately
launched an internal investigation into the incident.
Compliance action
6.
Following the initial investigation into the incident and review of
BT’s proposed corrective action plan, the following measures were taken
by BT:
- To proactively
seek to improve power alarm strategy;
- To instigate
a programme to check power alarms at major sites;
- To establish
a Disaster Recovery Plan for the Hampshire Police;
- To investigate
improvement to staff call out procedures for large geographic incidents;
and
- To investigate
the resilience of all emergency control rooms and discuss findings
with individual emergency organisations.
Results of compliance
work
7.
Oftel has reviewed progress with the items listed above and all are
proceeding in accordance with BT’s timetable. In some cases, the actions
taken will form the basis of ongoing and improved practice within BT.
Conclusion
8.
Oftel’s investigation has established that there were unacceptable failures
of emergency power equipment monitoring at a BT Southampton site on
25th April 2002 primarily due to the mode of operation of
BT’s power alarm. Oftel is now satisfied that BT has made adequate provisions
to reduce the risk of another occurrence of this type. Oftel will continue
to monitor the performance of the BT network with particular regard
to the provision of the emergency 999 service.
|
