Nearly 35 years after its first report on the NASA facility, EC&M returns for a look at an extensive electrical redesign.
As Dave Hopkins recently made the nearly 200-mile trip from the office of engineering design firm Fred Wilson & Associates (FW&A) in Jacksonville, Fla., down I-95 to ferret out the gremlins in a new 15MVA transformer that had tripped at a client's facility, the gravity of the situation sunk in. An idled transformer is never a good thing, but the day this particular transformer decided to wig out was no ordinary day: The transformer supplied one-third of the power to the industrial complex at NASA's Kennedy Space Center (KSC), Merritt Island, Fla. And the day it crashed was the launch date of space shuttle Atlantis plus one.
Although the outage lasted only 15 min, it knocked out power to the central instrumentation facility (CIF), which provides critical instrumentation and data processing for the facility's two main areas — Launch Complex 39 (LC39) and the Orsino industrial complex. Not exactly a comforting thought when only 24 hr earlier the shuttle sat on the pad with more than two million lbs of highly combustible solid fuel in its two solid rocket boosters.
It's no surprise, then, that NASA reacts to power outages — both scheduled and unscheduled — much the same way the CIA approaches breaches in national security. And when Hopkins, FW&A's electrical project manager, recalls the transformer trip, it's with the sort of nervous laugh born of the knowledge you've stepped out from in front of a wrecking ball one second before impact. “Had that happened [the previous day], they might have scrubbed the launch, and everyone of us would have probably been escorted out the gate,” he says.
Hopkins and the rest of FW&A can afford to enjoy a self-effacing chuckle from time to time after the decade they've had. Since 1992, the engineering firm has developed a solid relationship with the electrical team at the Merritt Island complex and become their longstanding on-call electrical engineer. And that relationship has grown as the firm has set to the task of updating the nearly 40-yr-old electrical system at the space center. More than 10 yr and four on-call contract awards later, Hopkins estimates the work is only half finished.
Work began on the restoration and renovation of the space center's electrical system in 1992 with the expansion of the 13.8kV substation at LC39. At the time of its construction in the mid-'60s, the vertical assembly building (VAB), located in the LC39 area, housed several offices for NASA employees in addition to serving as the storage facility for the Apollo spacecraft prior to launch. Unlike its predecessors, the modern shuttle, which includes the orbiter, external tank, and solid rocket boosters, is assembled in the VAB, making it too dangerous to maintain the office space. As more buildings were constructed to support the displaced staff, the demands on the LC39 area transformer station increased, making the substation expansion inevitable. Enter FW&A.
After applying for the job and competing against dozens of other firms in a selection process unique to federal work (engineering firms are chosen based strictly on technical qualifications and experience — fees are negotiated only after the firm is selected), the firm set to work designing the substation expansion, which supplies power to the LC39 area.
Since then, the firm has completed more than 100 projects for NASA, ranging from the replacement of perimeter lights at the launch complex to the replacement of transformer arrays at the Orsino substation. In fact, according to William Wilson, principal of FW&A, several of its design projects came about as a result of analysis the firm conducted on-site. “They said, ‘OK, based on some of these studies, we need to do some work,’” he says. “They didn't really ask us to develop those projects, they instead used the studies to tell them the condition of things and develop the projects on their own.”
Perhaps most surprising is the fact that it took nearly 30 yr before much of the system designed and installed at the space center's inception would be replaced. To be fair, the equipment chosen at the time of the installation was advanced well beyond anything used in the private sector, giving the space center a technological head start on most public and private facilities. However, as time went on and advances like vacuum circuit breakers became the norm in even the most standard electrical installations — public or private — the space center, which still employed oil circuit breakers, found itself in desperate need of a facelift; despite their continued reliability, some pieces of equipment were so old it was no longer possible to find replacement parts.
Predictably enough, the biggest hurdle for KSC's electrical team to overcome in getting the renovation and restoration project underway was money. But it wasn't for a lack of trying. Ernie Camacho, branch chief of the electrical design group, acknowledges the system's shortcomings. But as it became more difficult for the space program to secure funds of any kind in recent years, things like system improvements were sidelined in favor of projects that actually involved putting the shuttle into space. “We're all competing for dollars out here, and our main mission is the exploration of the planets and launching space shuttles,” Camacho says. “It's time to replace [the electrical system], but there's just so much of it and there's limited dollars.”
The other factor contributing to all the foot dragging was the robust nature and low failure rate of the existing system. When funds are tight, you have to make due with what you have. “In the 9 yr that I've been here in this group and the previous 20 yr, I have never heard of a failure in the electrical infrastructure, the power delivery system to the launch structure, or to the control rooms that has caused the scrub of a launch countdown,” Camacho says. “We may have a 30-yr-old infrastructure in a lot of areas, but it's redundant.”
Although Congress opened its wallet and the decision was finally made in the late '80s and early '90s to begin work on updating the system, continuing to acquire the funds necessary for the work has not been easy. Projects are regularly pushed back from one fiscal year to the next as costs exceed budgets, and getting Capitol Hill to approve adjustments to those budgets is next to impossible, but the work has gone on. “We help NASA determine of what they want to do what's the most critical to get done this fiscal year and push the remaining items out to future fiscal years,” Hopkins says. “Once they've gotten those dollars approved — they have some contingencies built in to handle minor changes — if it's a major change, then they have to go back to Congress to get a higher budget approved. And NASA probably isn't one of their favorite groups right now.”
No longer a technological symbol of America's race against communism, NASA's stock has fallen with both the government and the public; the Red Army has collapsed, but the bills keep adding up. Yet for the space program's stumbles, the size and scope of the electrical system by which the space center is supported is no less impressive.
Occupied by NASA and shared by Cape Canaveral Air Force Station, the Kennedy Space Center spreads out over a 140,000 acre plot of land on the eastern coast of Florida. The area is served by a 115kV closed-loop transmission system owned by electric utility Florida Power & Light that feeds the two main transformer substations in the LC39 and Orsino areas. The redundant system provides for seamless power in the event of the loss of an FP&L transmission line during launch periods. Although they enter the system through separate substations, both transmission lines circle the entire complex, allowing one to pick up the slack if the other should fail.
Yet even with such redundancy, Camacho and his team are never satisfied. With such high-priced equipment on the line — Space Shuttle Endeavor cost tax payers nearly $1.7 billion — even the slightest production errors caused by a power failure can cost millions of dollars, or worse, endanger the lives of the shuttle crew and support staff. Not only that, NASA maintains an aggressive launch schedule that can't afford to be altered by power reliability concerns. The need for uninterrupted power is so great, in fact, that NASA places severe restrictions on the kind of work that can go on near the date of a scheduled launch: A no-dig policy prevents anyone from even putting a shovel into the ground anywhere on the complex within 72 hr of a launch to avoid severing utility lines.
Although situations like the Space Shuttle Challenger disaster in 1986 increased concern for safety and proper process management, the strict demands for redundant, reliable power date back to the construction of the facility in 1965. Because the majority of the system infrastructure had yet to be updated when FW&A arrived on-site, the designers had a chance to see the lengths to which the original design team had gone to provide redundancy — and in some cases they found themselves scratching their heads.
“They've got things on top of things,” says Dan Field, senior electrical designer. “By trying to make the existing system work without having to take a full-fledged outage, they had configurations that sometimes were really stressful to look at.”
Although much of the confusion in the original design has been cleared up in the past 10 yr, the fact remains that NASA will simply not accept outages. Because the work that goes on in the various processing facilities is so important, they even consider it an outage to take down a redundant feed, making it difficult to do the renovations necessary to make sure power won't fail in the future. No one wants their system to fail, but such stringent guidelines for what constitutes acceptable power loss has not only given the FW&A team pause when designing the system upgrades, but also made scheduling of outages for analysis or construction like “pulling hens' teeth.”
“Even with backup generation, they get real antsy if we have to pull down one of their backup feeds for maintenance,” Hopkins says. “They have to have meetings of all kinds, and it takes a while. They don't volunteer to have their redundant feeds taken away. A lot of the contracts get extended for months because of the fact that they may be processing a payload, and they're not willing to give up that redundant feed until that processing has finished.”
Even Camacho agrees the space center's demands for system redundancy can cause a problem and that sooner or later, something has to give. “It's a tough sell, but you have to inform them that when they come out of this configuration they'll be in a much better situation because this is a planned outage,” he says. “If we keep on delaying this project, eventually some of that equipment is going to fail, and then you're going to be in what we call an emergency outage. There your options are very limited.”
The problems with scheduling outages have manifested themselves most recently in the current project to replace the transformers at the Orsino substation in the industrial complex. Constructed in the '60s with six transformers supplying 45MVA — four 10MVA units and two 2.5MVAs — the substation was due for an overhaul because the original configuration, which placed the four larger transformers on two buses and the remaining two smaller transformers on a third bus, was incapable of supplying an acceptable level of power in the event of a bus falling off-line. FW&A's new design replaces those six transformers with three 15MVA base rating units (25MVA with fan cooling), one on each bus, for the same base capacity. This arrangement provides greater flexibility should one transformer fail or need to be taken out of service for maintenance.
Camacho says that even during peak loading times in the summer the industrial complex needs little more than 25MVA to 26MVA of capacity, meaning the loss of one 15MVA transformer for any reason would still leave them with enough power for normal processes to continue; even if a second unit crashed, the remaining transformer has as much as 25MVA of power with fan cooling, which is adequate for the worst-case load.
Aside from NASA's rules regarding outages, FW&A has had to contend with other roadblocks beyond their control, be they the predictable bureaucracy of a federal agency or the unforeseen effects of a national tragedy.
The engineering firm has experience navigating the red tape-laced chains of command in the government, having completed work for military installations along the East Coast (including projects at Naval stations in Norfolk, Va., and Pensacola, Fla.), but that experience could not have prepared them for the multi-tiered approval process at KSC. Whereas two or three military representatives would sit in on planning meetings when the firm did work for the Navy, upwards of 30 KSC employees could be present at similar meetings.
“It's so interlaced,” says Field. “Everybody is relying on everybody else at NASA.”
Hopkins attended several of those meetings, and the number of people in the room never ceased to amaze him. “A lot of times you end up having to coordinate with six or eight different groups to get a project done,” he says. “And each of these entities has a shot at reviewing your design and making comments on it.”
Although having to please so many different groups was intimidating at first, Hopkins says it rarely became a factor after the plans had been drawn up. “For the most part, they just say ‘We need power here. We need this replaced. Give us the best you've got,’” he says. “They pretty well give us the latitude to do what we need to do.”
And then came Sept. 11. Just one month after construction began on the Orsino substation, the terrorist attacks on New York City and Washington, D.C., brought work at the facility to a stop. The space center entered Threatcon Charlie, the terrorist force protection condition level that applies when an incident occurs or intelligence is received indicating that some form of terrorist action is imminent. For several days members of the firm — along with any other contractors — were not allowed on the base. Work resumed in a week, but security had heightened to such a degree that it took hours just to get into the complex through security checkpoints. Contractors or engineers without full security clearance had to be escorted around the complex at all times.
KSC has stepped down security in recent months, but the measures created somewhat of a military state for the first few months. Both Hopkins and Field had full security clearance at the time, but that didn't mean they weren't subjected to full searches when entering the facility. “They had guys out there walking around with machine guns,” Hopkins says. “And I had my truck inspected just about every other time I went in the gate.”
Despite the conflicts along the way, the work at the Orsino substation is progressing and scheduled to be finished this December. Transformer trips aside, Hopkins and Camacho both believe the renovations at the industrial complex — as with the projects throughout the space center — are much needed changes that will bring immeasurable improvements to KSC. It's not every electrical system that can, or should, go 30 yr to 40 yr without an upgrade, and even though it was state-of-the-art when first designed, the time was right for an upgrade. “[The equipment] was reaching the manufacturer's recommended life of 30 yr to 35 yr, so it was time to put together a replacement plan for it. That's the main reason for the project,” Camacho says.
But more than just the satisfaction of knowing they've helped to keep running the country's on-ramp to space, Hopkins, Wilson, Field, and the rest of the firm's electrical design group are satisfied with the relationship they've built — one that could last for years to come. “This is the longest we've ever worked a continuous contract with a client,” Hopkins says. “They have basically just considered us as an extension of their engineering staff for years now. They're happy, and we're happy.”