Experience Notes for Emergencies

While most electronic engineers appreciate the value of uninterruptible power supplies (UPSs) and generators in coping with emergencies, the reality of the World Trade Center disaster has brought some overlooked points to the forefront. Below is a list of these points based on my background in the seismic arena.

War can create a lot of dust. To deal with this situation, remember the following:

1. Cooling towers are not recommended. Dry coolers are preferred because water from evaporator cooling towers will become contaminated with dust and can cause HVAC systems to fail.

2. Dry coolers can get clogged with dust, as ours did, and a means of cleaning them should be available. At such times, HVAC repair staffs can be overloaded and unable to provide quality timely service.

3. Keep a supply of air filters for all sizes of HVAC equipment in stock. You should be able to replace them yourself. While you're at it, have a stock of belts as well.

4. Have an electrostatic safe vacuum cleaner and antistatic workspaces because equipment can be contaminated. 3M makes a suitable unit. I spent three days just cleaning two racks of critical equipment. To clean the power supplies, I had to disassemble them, vacuum them, then use compressed air from our compressor to clean them. However, don't use compressed air on analog or digital processing circuit boards because it can cause electrostatic discharge failures.

5. Keep a supply of face dust masks on hand. These became heavily used in Manhattan and difficult to obtain even 30 miles away. The concrete dust is tough on the eyes, so goggles are also helpful. At my workplace, management had obtained a self-contained breathing apparatus when a halon extinguisher system was replaced. We didn't need to use it, so we donated it to the New York Fire Department. While it was hardly a replacement for the equipment they lost, some redundancy to assist the emergency services can be helpful.

6. Our satellite uplink was located on the 21st floor and remote enough to escape significant dust, but it could have coated the dish and caused attenuation.

7. A generator running for a long time may require more than one change of intake air filters, so have these available.

8. Dust can trigger smoke detectors and shut down the air conditioner. The system alarms we have seem to have recovered satisfactorily, but we had them tested to make sure.

9. Have a regular vacuum cleaner and spare bags available. I had to vacuum the floor over part of the facility so we would not be tracking dust into the relatively clean areas. Then we called a computer facility cleaning company who could clean under raised floors.

Power problems can be expected.

1. We had no power for eight days. Had our generator ran for that long, its tanks would have needed refilling. The oil tanker had to make three attempts at delivery and was getting turned back by security (police and national guard) the third time when I persuaded them to let him deliver.

2. When the tank runs dry, and there is no power, the diesel will need priming. Be familiar with how to do that. In our case, the main tanks were in a sub basement and difficult to manually transfer up to the day tank in the generator shelter above the loading dock. So I purchased kerosene/oil containers and a hand pump. Gasoline (petrol) containers are not approved to carry diesel as confusion could be dangerous.

3. A generator running for a long time can fail. Ours developed a leak in a radiator hose and stopped as the water level became low. Because we had an old transfer switch which had proved to be unreliable, we had not run our generator on load for a while.

4. We were in the process of replacing our UPS and transfer switch at the time, so the street power problems did damage some of our equipment. Avoiding such exposure is recommended. We and our electricians were fortunate to escape the WTC alive.

5. When the building emergency generator was connected, it did not have the capacity to run the full load. We had to keep our load light. Also, the phase rotation was reversed from the previous so all the HVAC units and the UPS had to have their phase reversed. When the street power returned, this rotation was at least maintained.

6. The emergency light and fire alarm system batteries went flat. When they are flat for a long time, they can sulpate up and lose capacity. This does not presently appear a problem.

7. If the CMOS battery goes flat, you can lose the contents of that memory. This can happen over a long power outage.

8. As the building initially had no power, I had to use a flashlight on stairways and elsewhere.

Communication problems can be expected.

1. The lines to our PBX were all on a T1 and lost, the main number still does not operate for incoming calls and it is now two months we have been waiting for repair.

2. We had two copper lines to different COs, and while they worked, we could not call long distance for a couple of weeks on them.

3. We lost four fiber circuits. It took about a month to replace them. The original vendor was quoting at least three months. I believe this was because the other vendor had fiber into the building, but the original vendor would have to relay it in the street and redesign the connectivity and reterminate.

4. This area now has a number of temporary cellphone towers around, but that takes time to implement. On the 11th, many people complained that their cell phones were not working. However, payphones were not much different. I expect that congestion was part of the problem.

5. Alternative means of communication are desirable. As we are a television facility, we had an intercom system that was part of our TV transmission system from Manhattan to Littleton, Colorado. This was the only communication system that was working that day. While voice over IP (VOIP) may not be the greatest, if your company has a T1 or other connectivity to other locations, it is an alternative.

6. I have observed hits on the fiber, probably as fibers are in proximity to WTC excavation work.

Other problems can be expected.

1. In an earthquake, the secondary problems of power hits or failure—water from broken dams or pipes, or fire—can be more of a problem than the shaking and structural damage of the quake itself (e.g., one small water pipe break can seem insignificant, but when it contaminates the sterile supplies for a hospital, that could be serious).

2. Building codes vary from area to area. However, it has been determined that if the federal building in Oklahoma City had been constructed according to California seismic building codes, there would have been much less damage and lives lost. New York City has in recent years adopted seismic codes, but that was after WTC was built. Whether this would have made any difference is not currently determined, but it is certainly worth knowing. New York, however, has a large number of brick or other unreinforced masonry structures which are not tall (less than 5 stories). Such structures should be avoided for companies' critical operations.

3. When security comes into effect, you will need personal and company ID. Also, your commute or travel can be significantly impacted. I used to travel by train to Manhattan, now I come by ferry, and my commute can be an hour longer. The train is not expected to be in operation for at least a year. Take an interest in your city's disaster contingency plans. Manhattan enacted its plan even though the disaster center was destroyed. I travelled by tugboat home to New Jersey, which I perceived as planned, not something organized during the chaos of the day.

4. I know someone who went to do repairs at the Pentagon. He asked me how he could get spares FedExed when the planes were not flying. I said that he was fortunate, because if it was critical to the military, they would see to its delivery. We civilians cannot depend on delivery of spares.

5. Can preparation be done? Well, a capital city of a country elected a mayor who was by profession an architect. He managed to pass legislation that any commercial building which did not meet seismic building codes had ten years to be corrected or removed for replacement. This was accomplished even though it affected 40% of the buildings in the downtown business district. He was Michael Fowler of Wellington, New Zealand. So the answer is yes.

6. Wire nuts used to be used in New Zealand many years ago. They are no longer used, apparently because they can come untwisted and fail under tension, which can happen during an earthquake. Screw clamp-type fittings are now used, and additional ones are built into wall switches and light sockets to provide the additional connections.

7. All power sockets are wired with an isolated ground. No conduit ground is used for ground pins because the sections can come apart during a quake but leave the outlet alive.

8. Copper water pipes are not soldered, but brazed using a lower temperature brazing alloy. This may be stronger.

9. The WTC disaster illustrates the wisdom of offsite backup of data. Include items like floor plans as well as operational data.

10. Assuming that we had none of the above problems, we would still have had to shut down operations relating to a number of television channels because civilians were excluded from the area for a week.

11. There are two established methods of protecting racks in seismic areas. The telecommunications racks are fastened to a steel grid which is fastened to the building steel. Electronics racks on a raised floor are fastened on pedestals which are fastened to the building floor, and the raised floor is fastened at the edges to the building steel. Another approach in lower risk areas is to use large pipes (e.g., 6 inches at the top of racks) in a truss configuration, which is fastened to the building steel. This can be retrofitted and may be suitable for areas like New York. I would like to see a structural engineers review of this approach.

Frank W. Bell, Lead Technician, AT&T Broadband, 395617 Battery Pl. #232, Manhattan, NY 10004-1101; E-mail: [email protected]; Phone: (212) 824-4400; Fax: (212) 824-4403; Pager: (888) 602-5128