Optus has released the findings of an independent review of the September 18 Triple Zero outage which includes 21 recommendations including a change to the company’s siloed way of working.
The report was also clear in pointing out poor execution and a focus on getting things done rather than getting things right in times of crisis.
The review was conducted by experienced executive and director Dr Kerry Schott and took a comprehensive look at the failure of the Tripe Zero services in South Australia, Western Australia, The Northern Territory and parts of New South Wales.
The review found gaps in process, accountability and a failure in information and escalation protocols.
The report also said there needs to be work to address emergency call services and how devices operate at critical times on the network.
Dr Schott also called for industry-wide collaboration to avoid this happening again.
“The Schott Review makes an important contribution to support the work underway to reform Optus,” says Optus CEO Stephen Rue.
“When I started as CEO last year, we launched a company-wide program of change, underpinned by a long-term strategic plan which would transform the company over the coming years.
“The Schott Review underlines the critical importance of this work and the need to accelerate and embed key reforms.
“Many actions and improvements are already underway, and we have a clear program to complete the remaining priorities.
“In line with our commitment to be open about the events of 18 September, we will regularly report our progress to restore confidence in Optus and our network.
“Our commitment is not just about compliance – it’s about setting a new standard for our performance as a critical infrastructure provider of essential services and serving the community. At the heart of this is putting our customers first in everything we do.
“Australia deserves world-class emergency call services. We are working closely with government, regulators, and the wider telecommunications sector to enhance the reliability of the Triple Zero service for our customers.”
The review forensically examines the incident and points out that at least 10 mistakes were made with the usually routine network upgrade carried out by Optus and Nokia.
Optus’s instructions given to Nokia’s network engineers were incorrect, and Nokia went ahead and implemented the incorrect request.
The result was a locked gateway at an exchange in Regents Park, SA which cut off regular voice and emergency calls.
Regular voice calls were shortly restored via alternative paths on the network, but emergency calls remained blocked and it would take Optus and Nokia 13 hours to realise the problem.
There were five calls to the Optus call centre who tried to find a technical problem either with the network or the caller’s device.
And because the call centre had not been advised of any outage it was not escalated any further.
The other puzzling part of the outage was why the majority of callers at the time trying to reach Triple Zero could not “camp on” to another network.
The report stated that usually devices can take up to 60 seconds to camp on and in an emergency people are unlikely to wait long.
It was also found there were calls from the same types of devices that were successful and unsuccessful when they needed to camp on.
The closure of 2G and more recently 3G where the Triple Zero calls are usually routed was also a factor.
The review also criticised Optus response which wasn’t activated until 2.30pm on September 18 and limited to the fact there was an issue.
And it wasn’t until 8pm that 100 failed Triple Zero calls were identified.
The report stated if there was an escalation in classifying the severity of the situation at time the CEO would have been informed sooner and the crisis management and communication team would have swung into action.
There were two executives waiting for more accurate information, but they did not have the same information the network team had.
The review said the notifications to the State Premiers were done far too late – they were not informed until after the press conference at 5.30pm on September 19.
The situation was made worse by the time of day and by the lack of familiarity with Optus contacts, the report stated.
“There are several matters to highlight for improvement in Optus’ response and following notifications,” Dr Kerry Schott said in the summary of the extensive report.
“The first is the necessity to attend to the siloed way of working so that information can flow easily around the company.
“This will take time as there is a significant cultural dimension to address. The Call Centre has already taken steps to ensure that Triple Zero matters are escalated, which is important – but across the whole company there needs to be a recognition that silos must communicate.
“Second, Optus must learn from its mistakes so that this type of event does not ever happen again.
“Networks must pay attention to processes and the controls applied to them. To have
a standard firewall upgrade go so badly is inexcusable.
“Execution was poor and seemed more focussed on getting things done than on being right.
“Supervision of both network staff and Nokia must be more disciplined to get things right.
“While changes to incident and crisis management have recently been approved, further training, rehearsals and exercises are needed. This applies especially within Networks.
“They need to know when to move from managing a network incident locally to central
management of that incident. Routine incidents can unfold into incidents that need central management and communication skills.
“Networks are not equipped to manage an incident that needs coordination across the company and good internal and external communication.
“The system being put in place to enable more efficient and faster tracking of normal voice calls and Triple Zero outages should be activated as soon as possible.”
The 21 recommendations from Dr Kerry Schott’s review
For Networks
1.Ensure that there are controls to support the robust execution of the correct processes and procedures when making routine changes within the Networks area.
- Encourage Network staff to escalate any issues outside their immediate group if they have doubts and would benefit from more experienced oversight.
- Expedite the move of the entire Operations Centre onshore as much as is prudently possible.
- Conduct incident management exercises at Networks to improve their judgement about when to escalate an incident to a more severe level. A routine incident can develop a higher level of severity and Networks must recognize what role they can play and what is best delegated elsewhere. Where central management is needed, the management control must shift from Networks to the Incident and Crisis Management Team and its related Communications Team.
- Expedite the implementation of the system to enable faster tracking of caller details in the event of an outage for voice calls and/or Triple Zero calls.
- Monitor the recent change to the ‘emergency time-out’ mechanism to ensure that the 600-second timing now set remains the most appropriate.
For the Call Centre
- Continue the improvement processes underway at the Call Centre, including the deployment of visible senior staff with whom matters can be raised and the movement of further Call Centre operations onshore (including some 24/7 operations).
For the Triple Zero System Generally
- Recognising that this incident has provided useful information about device behaviour, assist the government, the Department, the new Triple Zero Custodian, UTS, ACMA and the wider telecommunications industry with the further work needed on device behaviour.
- Encourage investigation of whether making the Triple Zero system data-enabled is worth doing.
- Inform all customers that their devices may take 40-60 seconds to connect to Triple Zero and encourage them to test their devices to ensure they work for a Triple Zero call. To ensure that the Emergency Call Person does not get unnecessary calls, set up a system where devices can be tested. This is an industry-wide matter and best done as an industry-wide initiative.
For General Optus Management
- Continue with the significant transition to reform underway and use this incident to learn and hasten change where possible.
- As noted, an effective first line of risk management was not evident within Networks through this incident. This must be addressed to avoid further incidents.
- The second line of defence, the internal risk management team that reviews and challenges risk management practice in the business, is in the process of improving and must gain the seniority and capability it needs.
- The third line of defence, Internal Audit, needs to build and strengthen its operations in line with the broader uplift underway in local governance and risk management.
- Consideration should be given to contract management at Optus, especially for complex and essential services, and whether or not service quality could be improved by a less adversarial approach.
- Address the siloed nature of work at present at Optus and facilitate a shift to a more cooperative, company-wide way of working. This has a cultural dimension and will take some time.
For Notifications
- The team with responsibility for external notifications should consider a closer relationship with State and Territory governments above the operational level of emergency services. As telecommunications is now an essential service, this level of government is increasingly important.
- This communications team should update their list of contact details frequently.
For the Optus Board
- To strengthen the recent move to more local responsibility, the Board should consider the adequacy of its skill base and depth and, if appropriate, make the changes needed.
- Oversight the efforts already commenced to upgrade risk management in all three lines of defence and assess whether these are sufficient by the standards of a large and complex organisation in a critical industry.
- Ensure that the CEO and executive team have sufficient authority and delegations to manage the approved transition to reform.
