Powering Change in the Data Center

These challenges have created the need for a power infrastructure capable of adjusting to changes in the number of devices, the density of those devices, and where those devices are located. Such an infrastructure must encompass critical power management from the utility input to the point of use, including:

• Power availability as determined by the UPS type and configuration,

• Power distribution from the UPS to the rack, and

• In-rack power management.

Failing to design flexibility into these systems can significantly shorten data center life and threaten equipment availability.

Power availability

Using redundant UPS modules helps ensure you can achieve appropriate levels of availability. Depending on the configuration type, redundancy may also enable scalability (see Types of UPS Systems below).

This redundancy also allows servicing of the UPS modules without affecting the quality of power to connected equipment. Single-bus systems with redundant UPS modules can support availability of 99.999% or higher, while dual-bus systems are designed to support continuous availability by eliminating single-points of failure between the UPS and the connected equipment.

The simplest approach to redundancy is a 1+1 system, in which each UPS module has the capacity to support all connected equipment. This configuration provides redundancy with the minimum number of UPS modules, keeping parts count and system complexity low. New software-based approaches to scalability allow 1+1 systems to be sized to current requirements while enabling capacity growth up to 100% without adding UPS modules. The N+1 configuration attempts to balance scalability and availability and can effectively accomplish that if system complexity is controlled. This requires proper sizing of the UPS modules. If the initial facility load is light compared to future requirements, it can be tempting to size UPS modules to initial requirements. However, this can create reliability problems in the future if the UPS module count gets too high.

We have analyzed power system reliability and determined that single-bus, multi-module UPS system reliability is acceptable up to a 3+1 configuration (click here to see Fig. 1). Beyond this point, reliability starts to diminish quickly because the increase in the number of modules increases the system parts count, which, in turn, increases the probability of failure and the risk of maintenance-related failures. Unlike server arrays, which can operate at reduced performance if multiple failures occur, an N+1 UPS system will shut down if UPS module failures put the system below the load capacity point.

If reliability is important, you should size UPS modules to no less than one-third of the projected total facility load. If there's a good chance growth will occur, start with modules sized at one-half the initial load. This provides room for growth while ensuring adequate availability throughout the life of the facility. Batteries, which are one of the most expensive components of the power system, can be sized to initial load with additional capacity added as required.

In addition to reliability considerations, you should factor system cost into decisions regarding UPS module size. The cost-per-kilowatt for a UPS system drops as module size increases. This makes it more expensive, for example, to implement 10 modules of 10kW each, rather than one 100kW module. Figure 2 (click here to see Fig. 2) shows how cost-per-kilowatt drops as unit size increases, based on an analysis of actual costs of one manufacturer's UPS products.

Compartmentalized distribution

Traditional power distribution designs use an approach in which the UPS feeds a required number of power distribution units (PDUs), which then distribute power directly to equipment in the rack. This was adequate when the number of servers and racks was relatively low. With today's equipment, however, it presents scalability and flexibility challenges. Too often breaker space is expended long before system capacity is reached.

Two-stage power distribution represents an emerging alternative that compartmentalizes distribution between the UPS and the server to enable greater flexibility and scalability (click here to see Fig. 3). The first stage of the two-stage system provides mid-level distribution. Similar to the traditional PDU, the mid-level distribution unit receives 480V or 600V power from the UPS. It includes most of the components that exist in a traditional PDU, but with a mix of circuit and branch-level distribution breakers. Instead of doing direct load-level distribution, this device feeds floor-mounted distribution cabinets via a panelboard distribution section, which provides the flexibility to add output breakers of different ratings as needed. The load-level distribution unit distributes power directly to the rack-mounted equipment and can be tailored to the level of availability the application requires, including single-sourced, dual-sourced (for dual-bus applications), or fed from four different inputs or sources. The use of two stages separates deliverable capacity and physical distribution capability into separate systems to eliminate the breaker space limitations of the traditional PDU.

The UPS-sizing philosophy covered previously extends to the mid-level PDU. If the mid-level PDUs are sized properly, they can support a series of growth cycles through the addition of breakers to the mid-level PDU and additional load-level units as required, creating the ability to scale the power distribution system without disrupting operations.

Another advantage of this approach is its effect on airflow. With two-stage distribution, under-floor cabling is significantly reduced. Cross-aisle cabling is limited to the feeds that run from the mid-level distribution units to the load-level units. Distribution from the load-level units, which are typically placed alongside IT equipment racks, runs vertically below or through the racks, keeping cold aisles free from power cables, as well as reducing the lengths of cable needed to feed the rack-level equipment.

Rack-level power management

Changes in rack density create new challenges in rack-level power management. With growth in the overall number of servers and the numbers of servers per rack, racks have become cable management nightmares. That isn't just a headache when changing out equipment — thick bundles of cables can severely limit the ability to move air through the rack, increasing the chance of overheating.

In addition, higher densities increase the likelihood that new devices will overload circuits. New compact servers may leave rack space and outlets unused while consuming all available power to the rack. This can present problems when the individuals responsible for deploying new servers are not aware of power capacity constraints within specific racks. Consequently, servers may be added to existing racks with insufficient capacity, thus overloading the power circuit and tripping the branch circuit breaker, causing the entire rack to go down.

Intelligent power strips present a cost-effective, easy-to-implement solution to in-rack power management (Photo above). These strips can mount either vertically or horizontally to simplify equipment changes and reduce cable clutter while providing increased visibility and control of rack power consumption. They monitor electrical attributes of an individual power strip, including real-time remote management via simple network management protocol (SNMP) and local LED display of volts, amps, watts, and phase loading of the power circuit.

More advanced strips can also provide receptacle-level control of power ON/OFF, enabling receptacles to be monitored and controlled locally or remotely to prevent the addition of new devices that could create an overload condition.

For some applications, power monitoring may be more effective when implemented at the branch circuit level. This provides less granular visibility than strip-level monitoring, but allows monitoring of more variables. In some high-density environments, a circuit may support only one high-density rack; therefore, branch circuit monitoring becomes equivalent to rack-level monitoring. Power distribution monitoring may be applied to mid-level distribution as well.

Most power distribution units provide power monitoring of the main input breaker that provides even more data than branch circuit monitoring. Additionally, monitoring can be extended to sub-feed breakers feeding load-level distribution units. This provides increased visibility and management capabilities you can use to identify capacity constraints and potential overload situations.

With power requirements continuing to rise, the need for greater downstream visibility into power consumption becomes more pressing. Applying monitoring technology at various points in the critical power infrastructure can provide a comprehensive view of the entire data center power infrastructure that supports the data center. Power monitoring at the branch-circuit or rack level should be considered a critical part of any power infrastructure that is expected to support rapidly changing loads.

Increasing capacity requirements and the need for increased availability have forced data center operations to focus their attention on infrastructure design that allows greater flexibility, higher availability, and the lowest total cost of ownership. Best practices in UPS systems configuration are being combined with more robust, flexible power distribution configurations that enable users to better manage their critical infrastructure by deploying the newest monitoring technologies. These strategies can enable both existing and new facilities to support higher densities and capacities while ensuring faster, more cost-effective response to future changes in information technology.

Shetler is the director of engineering for the Liebert AC Power Business of Emerson Network Power located in Columbus, Ohio. He can be reached at [email protected].

Changes in Power Requirements

Consider how power requirements for IT equipment racks have changed in the last 10 years. In 1996, a fully populated server rack could house 14 single-corded servers operating at 120V. This rack consumed approximately 4kW of power. By 2001, a fully populated rack could house 42 servers, which were likely to be dual-corded and operating at 208V, single-phase. In a matter of five years, the number of power receptacles required to support a fully populated rack grew from 14 to 84, and total power consumption increased from 4kW to almost 20kW.

The emergence of blade servers is driving even more change. Today, a standard rack can house six dual-corded blade chassis operating at 208V, single-phase, with a power consumption of 24kW.

Types of UPS Systems

The internal design of a UPS system determines the relationship between the UPS and incoming utility power and, ultimately, the effectiveness of the UPS at protecting against certain types of power disturbances. There are three types of UPS in use today:

• Passive standby,
• Line interactive, and
• Online double conversion.

The online double conversion reverse transfer topology protects against the full range of power disturbances and is recommended for applications that are currently mission-critical or expected to become mission-critical. Selecting an online double-conversion UPS ensures that availability requirements will not outgrow UPS topology.

The Table (click here to see Table) shows a summary of the most common system configurations in use today. Single-module systems are typically able to support availability levels of 99.999% (or less than 1 hour of unplanned downtime annually) and leave critical systems unprotected during UPS maintenance.

Summary of most common UPS system configurations.