Rogers Communications Inc. says an upgrade to its network caused a malfunction that “very quickly” knocked out telecommunications services earlier this month, according to a letter to Canada’s broadcasting regulator that was released Friday.
The outage, which began early July 8 and lasted for days for some customers, left millions without cell phone and Internet service, prompting questions from the federal government and the Commission on Telecommunications and Broadcasting Canada (CRTC).
“An update to our core IP [internet protocol] network … caused our IP routing network to malfunction,” the letter said.
The letter, posted on the CRTC’s website, met the regulator’s deadline for Rogers to answer questions about the outage. However, it has many redactions where Rogers is believed to have offered more specific details about the problem and his plans to prevent something similar from happening again.
LOOK | CRTC Requests Responses:
CRTC orders Rogers to answer for massive network outage
Canada’s telecommunications regulator has ordered Rogers to explain what caused its massive network outage, while Rogers announced new compensation plans for customers.
The CRTC said Rogers submitted two versions of the letter, one in full and one with redactions, and that it released the latter to protect “highly sensitive information” about Rogers’ operations.
Among other things, the CRTC had required Rogers to explain why 911 services were down in some areas and how it plans to fulfill CEO Tony Staffieri’s promise to proactively credit customers’ accounts.
“In order to regain the trust of Canadians, it is important that we provide open answers to the questions they have about the outage,” Rogers’ letter said. “That’s why when responding to the CRTC … Rogers is being as transparent as possible.”
Rogers also said he has hired a third party to review and provide information about what happened.
LOOK | How does Canada avoid another national disruption?:
How does Canada avoid another nationwide telecommunications outage?
Panelists debate what should happen after Friday’s Rogers Network outage to help protect consumers and businesses from being left in the dark again.
Rogers officials and a host of other stakeholders will appear before a parliamentary committee in Ottawa on Monday to explain more about the cause of the outage and outline the steps they are taking to make sure it doesn’t happen again.
In his letter, Rogers said the update’s coding removed a routing filter that “allowed every possible route to the Internet to go through the routers,” which flooded and overwhelmed the main network, causing cause it to stop processing Internet traffic.
“As a result, the Rogers network lost Internet connectivity.”
A sign at a Vancouver store on Friday warns customers that credit and debit services are down. (Justine Boulin/CBC)
He said many Rogers employees who wanted to fix the problem were affected and unable to connect to the company’s computer and network systems. As a result, only those “equipped with emergency SIMs on alternative carriers” could initially resolve the outage.
“While every effort was made to prevent and limit disruption, the fallout from the encoding change affected the network very quickly,” Rogers said.