There is still no official explanation for the freak breaks in two separate undersea African Internet cables that happened 11 days ago. However, as reported in the Sunday Times yesterday, the most common causes of such breaks are damage by ship anchors and earthquakes at sea.
The freak occurrence of two separate cables being cut simultaneously far out at sea, as happened on the morning of 16 January, can only be explained by sea-bed activity. One of the cables was cut in two places, and it is widely believed that a third major cable was also cut.
The cable damage mostly occurred in or near an area called the Congo Canyon, which starts inland and extends 220km into the sea. It is known for having the world’s strongest “turbidity currents”, underwater sediment avalanches over hundreds of kilometers, which are known to destroy undersea cables.
The most likely culprit is a 5.6 magnitude earthquake that struck the Atlantic Ocean near Ascension Island shortly before the cables were cut on the morning of 16 January. The earthquake occurred just before 8am South African time, and local ISPs reported losing international access from just before 10am. The epicentre of the earthquake was more than a thousand kilometres off the coast of Africa, but disturbances caused by seismic activity at sea become more powerful as they approach the coast. Combined with turbidity currents, this could well have taken out all cables in the area.
The West Africa Cable System (WACS) was cut in two places, and the South Atlantic 3 (SAT3) cable in one location. Industry insiders believe that the Africa Coast to Europe (ACE) cable was also cut, but it has not been publicly confirmed.
South Africa is connected to the global Internet via seven such cables, with a total capacity of 42.3 terabits per second (tbps). These cables, in turn, connect to additional cables connecting the West and East coasts of Africa, with a single cable running from Angola to Brazil providing another 40 tbps.
However, it emerged in the past week that smaller ISPs in South Africa had bought capacity on only one or two cables. In a freak occurrence, two of the most commonly used cables, the WACS and SAT 3 cables, were cut simultaneously, plunging millions of Internet users into data darkness.
Customers of the major mobile network operators – Vodacom and MTN – were largely unaffected, as these tend to have both part-ownership and access to most of the cables running up both the East and West coasts of Africa.
Visit the next page to read about how ISPs have battled to reroute access, how massive resources are needed to deal with these kinds of outages, and when the ship will reach the breakage points.
ISPs battle to reroute access
However, many ISPs use one of the two cables for their main access, and the other as a back-up. It is unprecedented for both to go down, and ISPs like Afrihost, Webafrica, Axxess, and Mind The Speed were caught flat-footed. Internet users have been scathing in their criticism of their ISPs, but the reality is that access to each additional backup cable adds significantly to the cost of the service.
“We had complete redundancy on SAT3, should our WACS capacity fail,” said Afrihost CEO Gian Visser. “It was unprecedented that two completely geographically separate cables run by completely separate companies would fail within hours of each other. We also had emergency capacity on a third undersea cable, but this was not sufficient due to the total traffic routed to that cable by other ISPs in the same situation.
“When the severity and the time it would take to repair the cable breaks became apparent we decided we needed to procure and set up more capacity on a completely different cable as soon as possible.”
The cost implications are massive, he said.
“Having dedicated capacity, whether you use it or not, on any cable is extremely costly, so ensuring available capacity on multiple cables significantly increases the cost of providing connectivity to clients. On top of this, having a single route makes networking much easier to manage, in terms of the best client experience, so this means that a large portion of your cost is spent on unused capacity.
“In every situation in the past, having full redundancy on a completely separate cable was always sufficient. The incredibly unlikely scenario of two separate line breaks on two separate cables on the same day led to both redundant paths being compromised.”
Afrihost uses WACS as its primary route and SAT3 as secondary route. At present it is routing traffic over the SEACOM and EASSy cables that run up the East coast of Africa.
Rain, the independent ISP that was the first to roll out 5G wireless broadband in South Africa, was marginally affected initially. However, said CEO Willem Roos, it buys capacity from multiple cable providers so that it has redundancy available to handle such situations.
“If the agreements are in place, failover to other cables is a fast and simple process,” he said. “We have also designed our network to take maximum advantage of local peering and caching in order to minimise international traffic. We are pleased that our redundancy strategy worked well. However, as you would appreciate, it is costly to reserve this type of capacity.”
Visit the next page to read about how massive resources are needed to deal with these kinds of outages, and when the ship will reach the breakage points.
Massive resources needed
Vodacom spokesperson Byron Kennedy sympathised with the plight of smaller ISPs, pointing out that massive resources were needed for a guaranteed service.
“Success factors for a resilient network include elimination of single points of failure, equipment and infrastructure redundancy, sufficiently dimensioned, geographically diverse routes, able to withstand multiple simultaneous failure scenarios,” he said. “It takes weeks to implement fixed telecommunications links involving various multinational operators to be implemented. All of this needs to be in place, together with efficient traffic engineering to ensure all routes are available at all times to carry traffic.
“Restoration requiring human intervention and coordination amongst operators in different countries can be very time-consuming and inefficient. It requires significant resources and prior investment to ensure a carrier-grade service at all times.”
It was essential for carriers like Vodacom to invest in such resources, he said, since more than 80% of the Internet content in South Africa is typically sourced from Europe and North America.
MTN, the single biggest investor in the WACS cable, also has a stake in the ACE and EASSy cables. It was briefly affected by the outage, said MTN SA’s executive for corporate affairs, Jacqui O’Sullivan.
“MTN SA currently uses the EASSy cable system which has sufficient capacity to accommodate the international bandwidth demand,” she said. “The shift in traffic onto the EASSy cable system resulted in an increase in international latency and a degradation in throughput for fixed-line subscribers, which was mitigated shortly after the WACS outage. No mobile subscribers were affected.”
Visit the next page to read about when the ship will reach the breakage points.
Ship should reach cable break tomorrow
Meanwhile, the repair ship deployed to fix the cable breaks, the Leon Teverin, is due to arrive at the site of the first break tomorrow.
The ship, which spends most of its time docked at Cape Town, is operated by Orange Marine, a subsidiary of French telecommunications giant Orange. It is specifically tasked with repairs to the WACS and SAT3 cables. However, it was confined to the harbour for several days as storms lashed Cape Town in the wake of the Internet outage.
Telkom’s wholesale division Openserve issued regular updates on the status of the mission, finally announcing on Wednesday that the ship had set sail at 9.30pm the night before.
However, it would take up to a week to reach the site of the break on the SAT3 cable, between Angola and Gabon. From there is will head to the first break on the WACS cable, further north near Libreville, Gabon.
The cables are operated by two separate consortiums and Openserve said it “has made its resources available to both consortiums to assist wherever possible”.
Once the ship reaches each site, it must lift the cable, attach bouys, examine the cable, splice new lengths of cables to repair the break, conduct testing, and then re-lay the cables on the sea bed. The process can take another week. This means final repairs may take more than a week from today to be completed.
However, all Internet access has been routed round the breaks, typically up the East coast of Africa, and across the Atlantic from Angola to Brazil. The headache, right now, has been transferred from users to the service providers.