According to the FCC, fully 25 percent of the cell sites in the affected area were out of operation because of the super storm that hit the East Coast. People are asking why those who use only wireless phones were unable to call for assistance or talk to their families in the affected areas, and why it took such a long time to get most of the sites back on the air. Now the government and others want answers as to why this happened, and why the network operators were not better prepared. I have talked to a number of operators, vendors, and others involved in having these sites turned back on and have determined that, in fact, the East Coast was lucky in that only 25 percent of the sites went down.
There are many reasons a cell site stops working because there are multiple points of failure. It is virtually impossible to build cellular networks that will withstand the ravages of a Sandy or a Katrina. The network operators were prepared. They sent in crews ahead of time, they staged extra fuel for the generators, and they moved in Cells on Wheels (COWs) and other equipment. When the storm hit they were as prepared as they could have been. What, then, went wrong?
By multiple points of failure, I am referring to a number of things that will cause a cell site to go off line. If the connection between the network and the cell site is destroyed, the cell site may in fact be working but the information to and from that site has nowhere to go. If the power to that site is disrupted, the site will switch to back-up battery and/or generator power, and if there is damage to the antennas or the structure that holds the antennas or equipment, this will also cause the site to appear to be off the air. The point(s) of failure has to be determined for each site that is off the air and people have to be able to access the site to make the repairs or wait until the affected power company restores power for the provider of the interconnection between the site and the main network.
Making things worse, many cell sites house multiple networks. In many places two, three, or even four networks share the same cell site and if the site goes down, all of the networks are affected.
Let’s look at each of these points of failure separately. First, if the fiber or wired connection to the cell site runs above ground on the same telephone and electric poles that carry power and cable TV, and trees fall on them and take out both the connection and the power, this will cause a site to fail. If the site is fed with underground utility service it can still fail if it becomes flooded, and at some point it probably transitions from underground to above ground. That is why sites have back-up power. Some have both batteries and generators, but some only have battery backup that will last for about 12 to 18 hours. Sites that have generators normally have enough fuel in the tank to last three to four days, beyond that the power either needs to be restored or someone has to access the site and provide additional fuel.
Where the site is built has a great deal to do with how well it will endure a storm. In the United States, especially in cities, many of the sites are located on buildings, not towers. It is not possible to place a generator and fuel on the roof of a building so the generators, if there are any, are usually located on the ground or even in the basement. One thing we learned from Katrina is that generators do not work underwater, so if there is flooding, as in this case, the generator will cease to run. Once the water has dissipated, the generator will require service in order to bring it back on line and this requires people at the site. Interestingly, most of the sites that were built on towers with weather resistant equipment shelters withstood the storm better than the sites on the buildings. This is because when a tower is erected it is designed to withstand winds of up to 120 MPH and more, and the antennas are designed to withstand the same type of winds. These sites generally weather the storm better than building-located sites. However, if the power fails, or the connectivity to the tower is lost, the site will still be down.
As you can see, there are many failure points and to bring a site back into operation requires people: people from the electric company and the connectivity provider; people to re-fuel or repair the generators; people to replace damaged antennas or cell site radios. All of these people were available during Sandy, but there is one more factor that plays into this.
If people are not permitted into an area because of storm damage and danger to themselves and others, the personnel needed to put the cell site back on line won’t be able to go to the site. The electric company was overwhelmed, and many sites ran out of generator fuel. No one was permitted to travel to the site, so the delay was because the power company and/or the connectivity company had not been able to reach the area to repair the damage.
Can we build cell networks that will always withstand these types of natural disasters? The short answer is no, we cannot. There are just too many variables. I have heard people complain that there was no generator at a specific site, but we don’t know if there was connectivity at that site either. It is a pretty safe assumption that if the power is out, connectivity is also not available so even a generator would not have done any good. The bottom line is that when all of the utility companies can build infrastructure to withstand the worst possible scenario, wireless networks, along with power, landline phones, cable systems, and other utilities will continue to fail.
As a final note, there were some public safety radio systems that also failed but unlike cellular phones, public safety radios can continue to operate and communicate even when the network is down. Public safety systems are built to a higher standard and there are not as many towers serving a given area. However, when the network does fail, personnel can still talk unit to unit. That’s something that cannot, today, be done on a commercial wireless network. When the commercial network is down, your cellular device is a paperweight.
If we insist on more redundancy and more ways of delivering power and connectivity to the cell sites, we will have to pay the higher cost. The bottom line is that there will still be power outages, wired and cable outages, and of course, cell phone outages.