A High Availability Electrical Design for Today's Data Center

Today's large data centers represent an evolutionary change from their predecessors. Continuous availability remains the paramount objective, however, contemporary data centers require substantially higher load capabilities due to new computer technologies. In recent years, server-based equipment has become capable of performing many functions previously requiring mainframe computers.

Manufacturers and end users usually install this server-based equipment in compact packages that consume a relatively large amount of energy for the floor area occupied. For example, some large mainframe-based data centers operate at a load density of 30W per sq ft while other server-based data centers operate at 70W per sq ft. Further, computer equipment manufacturers have announced new server-based computers with substantially higher load densities than previous models.

Accordingly, new data center electrical designs must accommodate substantially higher levels of computer loads for the same area of computer room. These higher load levels have led to new approaches to electrical design that maintain high availability.

One electrical system design incorporates these new approaches with a minimum level of redundant equipment. The premise of this design is to provide redundant paths of power flow, coupled with redundant distribution equipment to and including the computer equipment.

In effect, redundancy is provided throughout the distribution system from beginning to end. Let's refer to the one-line diagram on the facing page as we detail the design's operation features.

Utility Services

The redundant paths for power flow start with the utility service. Two primary utility feeders serve the facility from separate substations and follow independent routes to the site. These feeders terminate in MV Main Distribution Switchgear A and B. The utility services are sized so either one is capable of supporting the entire load.

Normal operation

In this mode, Breakers 1 and 6 in Main Distribution Switchgear A and B remain closed while Breakers 2, 3, 4, and 5 are open. As such, the facility load splits between the two utility feeds, with Utility Feeder A supporting Unit Substation A and Utility Feeder B supporting Unit Substations B and C.

One utility feeder fails

If Utility Feeder A fails and the other feeder (Utility Feeder B) is still available, automatic controls will open Breaker 1 and close Breakers 2 and 5. Now, Utility Feeder B is supporting all facility loads. On the other hand, if Utility Feeder B fails, automatic controls will open Breaker 6 and close Breakers 2 and 5. Now, Utility Feeder A is supporting all facility loads.

This utility service arrangement offers superior reliability since a high degree of independence exists between Utility Feeders A and B. Further, under normal conditions, both sources are active and operating at partial load. This assures their readiness to assume the entire critical load should one source fail.

Both utility feeders fail simultaneously

In this situation, automatic controls start all four generators and synchronize them to the generator bus by closing Breakers 7, 8, 9, and 10. Next, the controls open Breaker 1 and close Breaker 3 to transfer Main Distribution Switchgear A to generators. Then, the controls open Breaker 6 and close Breaker 4 to transfer Main Distribution Switchgear B to generators.

The generators are sized so that only three are needed to support the entire load (N+1 redundancy). This ensures full standby generator backup, even if one generator fails or is down for maintenance.

Utility power returns

Automatic controls then initiate a closed transition (make-before-break) transfer of loads from generators to utility power. This is done by synchronizing the generator plant to Utility Feeder A; then closing Breaker 1 to parallel the generator plant to Utility Feeder A; then opening Breaker 3. Next, the controls synchronize the generator plant to Utility Feeder B; close Breaker 6 to parallel the generator plant to Utility Feeder B; then open Breaker 4. Breakers 7, 8, 9, and 10 open, and standby generators are shut down in preparation for the next operation.

Another attribute of the standby power system is the capability for operators to perform a closed transition (make-before-break) transfer of facility loads from utility power to generators. This feature is useful for conducting standby generator system load tests without disturbing critical loads, and to allow operators to assume facility loads on generators when unstable utility conditions are expected. This automated feature operates in similar fashion to utility failure conditions. When manually initiated by operators, the automatic controls start all four generators and synchronize them to the generator bus by closing Breakers 7, 8, 9, and 10. Then, the controls synchronize the generator plant to Utility Feeder A; close Breaker 3 to parallel the generator plant to Utility Feeder A; then open Breaker 1. Next, the controls synchronize the generator plant to Utility Feeder B; close Breaker 4 to parallel the generator plant to Utility Feeder B; then open Breaker 6. When operators exit this mode, they manually initiate the process. The automatic controls respond by following the same sequence described above for the return of utility power after an outage.

Unit Substations

The redundant paths for power flow can also be found at this level of the system (i.e. 13.2kV). Many economic factors come into play when supplying power to large loads, so MV distribution was selected. Two dedicated radial feeders serve each unit substation. They are redundant with one originating at Main Distribution Switchgear A and the other at Main Distribution Switchgear B. Each unit substation is arranged in a double-ended configuration. That is, two unit substation transformers reduce distribution voltage from 13.2 kV to 480V, and each has the capacity to support the entire load of the unit substation.

It is critical to maintain redundant paths of power flow to the output bus of Unit Substations A, B, and C. To illustrate this point, let's look at a typical unit substation, such as Unit Substation B. As you can see, Transformer TBB normally supports the load, with Transformer TBA off-line (Breaker 14 closed and Breaker 13 open). Should power from Transformer TBB fail, the load supported by Unit Substation B can be transferred to Transformer TBA by opening Breaker 14 and closing Breaker 13.

UPS Systems

This design has Unit Substations A, B, and C each supplying input power to UPS Systems UPS-A, UPS-B, and UPS-C. Each consists of three paralleled modules and associated system switchgear. The modules are sized so only two are needed to support the load connected to the entire system (N+1 redundancy) under normal conditions. In addition, each UPS system contains a static bypass circuit and a continuous duty static switch to automatically bypass the system in the event of a failure and to assist in clearing faults.

The outputs from UPS systems UPS-A, UPS-B, and UPS-C are connected to Tie Switchgear AC, BC, and AB. The Tie Switchgear interconnects the UPS outputs and includes automated controls to allow transfer of loads between UPS systems. Tie Switchgear AC allows transfer of loads between UPS-A and UPS-C. Tie Switchgear BC allows transfer loads between UPS-B and UPS-C. Tie Switchgear AB allows transfer of loads between UPS-A and UPS-B.

In normal operation, all three UPS systems operate independently. In this mode, Breakers 17, 19, 20, 22, 23, and 25 are closed; while Breakers 18, 21, and 24 are open. Each UPS system is on-line and operating at a maximum of two-thirds load.

Should any one UPS system require maintenance or fail and transfer to static bypass, the automated controls will allow a closed transition (make-before-break) transfer of loads normally served by that UPS system to two alternate UPS systems.

To illustrate this scheme, let's assume you need scheduled maintenance on UPS-A, requiring you to take it off-line. To do so, the operator first commands Tie Switchgear AC to transfer load AC from UPS-A to UPS-C. This initiates an automatic sequence:

1. Synchronize the outputs of UPS-A and UPS-C.

2. Close Breaker 18 to parallel UPS-C to UPS-A.

3. Open Breaker 17.

The operator then commands Tie Switchgear AB to transfer Load AB from UPS-A to UPS-B. This initiates another automatic sequence:

1. Synchronize the outputs of UPS-A and UPS-B.

2. Close Breaker 24 to parallel UPS-B to UPS-A.

3. Open Breaker 23.

Now UPS-C supports Load AC, UPS-B supports Load AB, and UPS-A is off-line. The loads normally supported by UPS-A are distributed to UPS-B and UPS-C, increasing the load level on each system from two-thirds to full load. When maintenance is completed on UPS-A, the process reverses in a similar manner.

This UPS system architecture maintains redundant paths of power flow to each of the six UPS loads (AC, CA, BC, CB, AB, and BA) while minimizing redundant equipment.

Power Distribution Units

UPS output power from Tie Switchgear AC, BC, and AB is distributed to computer equipment via computer power distribution units (PDUs). A different type of PDU is applied depending on the characteristics of the computer equipment.

One cord is connected to each of two conventional PDUs for dual-cord computers. Static-switch PDUs are used for single-cord computers. The computer manufacturer configures the dual power cord computers to operate when either or both cords are energized. Typically, both cords are active and supply a portion of the dual-cord computers' load. If either cord fails, the computer internally transfers all load to the remaining cord. The single-cord computers require that their single cord be continuously energized to operate.

The PDU configuration for dual-cord computers maintains redundant paths of power flow to the computer input terminals. This is accomplished by connecting each cord to a different PDU, which is supplied by a different UPS, and is connected to a different Tie Switchgear. Thus, if the input source to either cord of any dual cord computer unexpectedly fails, the computer will automatically switch to its second cord.

For this approach to work, the distribution system must be sized so either source can support the entire load of all connected dual-cord computers. Thus, all PDUs, PDU feeders, and Tie Switchgear must be rated at double their normal load. In addition, you must consider the possibility that an entire UPS system (UPS-A, UPS-B, or UPS-C) could unexpectedly fail, and fail to transfer to static bypass. To account for this possibility, it is best to install the PDUs in pairs, with the input sources to each PDU of a pair alternated between Tie Switchgear AC, BC, and AB. Using three different pairs leads to a desired result.

The selected pairs are connected to Load AC and BC, Load AB and CB, and Load CA and BA, respectively.

The PDU configuration for single-cord computers maintains redundant paths of power flow by application of static switch PDUs. These devices essentially are electronic transfer switches. Two input sources are provided; one is the preferred source and the other the alternate source. The load is normally supplied by the preferred source. In the one-line diagram on page 15, Load CB is the preferred source while Load AB is the alternate source. In this condition, SCRA is conducting and SCRB is turned off. If the preferred source (Load CB) unexpectedly fails, the load is automatically transferred to the alternate source (Load AB). These transfers are open transition (break-before-make), but are very fast. The transfers occur in less than one-quarter cycle (4 milliseconds), and the single-cord computers can sustain this brief outage and remain in operation.

For the static switch PDU to complete a successful transfer (computers to remain in operation), the two input sources must be within acceptable voltage limits and phase tolerances. To maintain the two sources within these limits, you must specify specialized controls in the UPS systems. These controls maintain all UPS systems (UPS-A, UPS-B, and UPS-C) within acceptable limits when any or all systems are operating on batteries, generators, or on utility power.

Similar to the approach for dual power cord computers, it is necessary to maintain redundant paths of power flow for single-cord computers by connecting each input source of the static switch PDU to a different UPS and different Tie Switchgear.

The static switch PDUs also offer maintenance capabilities. You can isolate the switching elements (SCRA and SCRB) for maintenance by operating an interlocked breaker scheme. This scheme allows a closed transition transfer (make-before-break) to either the preferred or alternate source.

Robert Yester is chairman and design principal at Swanson Rink Consulting Engineers, a firm specializing in data processing center reliability design. You can reach him at [email protected].