So You Think Your Critical Cooling System Is Reliable?

An HVAC system with 3 components in series

System Reliability Versus System Complexity

 

I just read an article in Engineered Systems magazine that reminded me of one of my own blog postings from a few months ago.  The difference is that the author in the ES magazine article went all mathematical on us and showed the formulas for calculating HVAC system reliability given the reliability of the individual components in the system.  Although it was a long time ago I remember going through the mathematical exercise in one of my engineering courses back at the University of Texas...so we have all known about this procedure for a very long time.

During this same week I have been asked to do a competitive analysis on a "new" system concept compared to one of our systems.  While I could name names that is not necessarily the important point of this posting.  What struck me about the competitor's "new" system concept was just how many parts were required to accomplish the task of providing "free cooling".  Many of those component parts had dependencies that meant that the proper reliability analysis was the "series" analysis.  You can refer to the latest issue of Engineered Systems if you don't remember what that means but in keeping with my simple approach to my postings the bottom line is that the reliability of a "series" of components is the compounded product of each item's reliability multiplied together.  In other words if the "new" system concept required 4 compressors that are staged in series, a direct drive exhaust fan array, a direct drive supply fan array, a sensible heat wheel motor, sensible heat wheel belt, sensible heat wheel bearings, digital control module, etc...and we gave each of those items a reliability of 98% (which sounds pretty good and is generous for some of the items in the chain)...then the "system" reliability would be:

 

.98 x .98 x .98 x .98 (compressor section) x .98 (exhaust fan) x .98 (supply fan) x .98 (sensible heat wheel motor) x .98 (sensible heat wheel belt) x .98 (sensible heat wheel bearings) x .98 (digital control module) = .817

So, the "new" system concept under this scenario would actually have a reliability of only 81.7%...not quite so good I think you would agree. 

The information from this example is actually directly from the competitor's product literature...and I left out some components for simplicity sake.  The reliability of the components at .98 was an estimate and you can plug in whatever numbers you think are accurate.  The important thing is to recognize that the more complex the system is the lower the reliability will be.

In a previous blog posting I quoted Albert Einstein who said ..."make everything as simple as possible, but not simpler.".  I still think Albert was a pretty smart guy and when I look at some of the design solutions being proposed today for data centers, pharmaceutical warehouses, or cooling in general I just have to wonder why we sometimes design such complex solutions. 

Remember..."it is not sustainable if it is not maintainable"...and, as a corollary to that statement, "it is not maintainable if it is too complex and has too many parts".

Dusting Off Your Data

CONTAMINANTS IN THE DATA CENTER

Time to get back on my soapbox again…this time it is about “contaminants” in data centers as an excuse to avoid using fresh air cooling or having outside air enter the white space.  The bottom line is that unless your data center is located in “an emerging country” then the odds of a contaminant-created hardware failure in anything like a short time frame are about the same as winning the lottery…assuming you take some pretty basic steps in the design.

Contaminant control, or more correctly, concern over contaminant control has been around for decades.  I remember doing some research over 25 years ago on the impact of ozone on telecommunications equipment.  Bell Labs, as it was known long ago, had performed some pretty interesting tests to document what could be a very real problem under the right circumstances.  The results of those tests indicated that, with the exception of certain locations, the air in the equipment room was worse than the air outside so it made more sense to flush the room with outside air than to avoid bringing outside air into the space.

Particle and gaseous contaminants CAN be a problem if ignored.  However, the extent of the problem and how quickly it manifests itself needs to be considered. 

Phenomenon like copper creep and circuit bridging do occur…but only when the conditions at the server are right to support those failure modes.  Two things generally need to be in place for the failure mode to even begin.  First, there needs to be a fine coating of dust particles on the circuit boards.  Second, the relative humidity at the board needs to be at the deliquescent RH…or the point where the dust starts to absorb moisture and become “wet”.  If the RH is too low then dust might affect localized temperatures on the board but the mechanism to cause bridging simply does not exist.  The converse is also true…no dust…then no mechanism even with a relatively high humidity level.

Dust can come from anywhere.  Every time someone enters the data center they bring in some amount of dust particles.  Every time a box is opened in the data center particles are created.  And, yes, every time outside air is brought into the data center it is possible that dust can enter.  In fact, a data center with no outside air is actually vulnerable to the worst kind of dust intrusion…uncontrolled infiltration through doors, cracks, pipe openings, or wind pressure.  Maintaining a positive pressure in the white space helps to prevent infiltration and keeps the worst dust (and gases) out of the data center. 

ASHRAE, through the TC 9.9 committee, has set a target for data center “cleanliness”.  It is ISO Class 8.  ASHRAE has also noted that ISO Class 8 conditions can be met with a MERV 8 filter…a common and inexpensive filter available at virtually any HVAC parts house.  If the air being filtered is coming from the outside then ASHRAE recommends a MERV 11 or MERV 13 filter.  These might not be quite as common as the MERV 8 but they are also readily available and can fit in a standard 2” filter rack.

The interesting side note about the ASHRAE recommendations is just how extremely conservative they are.  ASHRAE recommends no more than 15mg/m³ of “fine” particles…defined as particles less than 2.5mm in size.  However, IBM (who should know something about computers) has a limit of 150mg/m³ and a “fine” particle definition of particles less than 5mm in size.

Once again, owners are being led down a path to purchase cooling systems and equipment that fail to optimize their energy savings through an inflated fear of something that happens very rarely in the developed world and is easily controlled with proper filtration.  Products such as our Aztec ASC indirect evaporative cooling systems are designed with MERV 14 filters in mind and can actually accept MERV 16 filters…the highest MERV rating point…that is suitable for operating rooms and can remove all bacteria and most tobacco smoke.  This allows the Aztec system to optimize the use of fresh air cooling and use a more efficient heat transfer system than air-to-air heat exchanger systems…and still exceed the extremely conservative ASHRAE recommendations for particulate control.

Shading and Make-Up for Building Designers

I just returned from a meeting in Florida and I was reminded of a couple of basic concepts that apply to virtually all building designs.

Our meeting room faced an outside wall with a couple of French doors to a nice patio area.  The weather was unusually cool for Florida and everyone sitting on that side of the table was able to experience that coolness first hand...even though the doors were closed.  During breaks the smokers in the group would gather on the patio and, again, in spite of closed doors the meeting room started to smell of cigarette smoke.

The problem, of course, was a lower pressure in the meeting room compared to the outdoors.  Somewhere in the conference center there was an exhaust system churning away without a counterbalancing make-up air system.  The only way that the exhaust system could satisfy its demand for air was to pull that air from outside the building, through the conference room, to its final point of exit.  Cold, smoke-laden air was drawn into the meeting room and occupant comfort was compromised.  Simply adding a make-up air system similar to the Applied Air DFL-series would have improved the indoor environment and cost very little extra to operate.  Remember that all of that cold air that was being sucked into the building caused the occupants to raise the thermostat set-point in order to compensate for being cold and forced the large main air handlers to operate for more hours than necessary.  Maintaining a positive pressure in buildings controls infiltration of smoke, dust, and un-tempered air and it is relatively simple to achieve.

The other basic concept that popped into my head is how important the building envelope is to controlling operating costs.  This particular resort was built many years ago but employed some pretty effective passive shading for the guest rooms.  My room had a wall of windows for natural light and a view but had a deep setback that prevented direct solar radiation.  This deep setback meant that the air conditioning system would see far fewer operating hours than an unprotected glass exposure would allow.  Since solar radiation is also a significant portion of the building cooling load the setbacks allowed a reduction in HVAC equipment size as well.

Building design has changed since the days when this hotel was built and deep setbacks are much less common.  But effective solar shading is still feasible through the use of external shades and louvers.  External shade technology has advanced to the point where it is possible for the shades to track the location of the sun and automatically provide continuous reduction in solar radiation.  Some external shading systems such as those developed by Colt Group actually contain photovoltaic cells that can reduce the building electrical demand by more than providing shading alone.

So two basic concepts for sustainable building design:  maintaining a positive indoor pressure to eliminate unwanted and untreated outside air from entering the occupied areas; and using modern external shading technology to reduce the solar load in the occupied areas which, in turn, reduces operating and capital costs.

Equilibrium

Equilibrium…we all try to achieve it in our lives.  An argument can go on forever if both sides maintain a high energy level and refuse to cool things down.  An argument can end when both sides take it down a notch and each reaches a happy place that they can both accept…a state of emotional equilibrium.

Odd as it may seem your air conditioning system is trying to do the same thing…reach a happy state of equilibrium…a balance between the high energy state and the low energy state.  Fortunately the system won’t get there under most circumstances because when the high and low energy states are equal then the unit stops providing cooling.

To simplify our thinking about this, substitute the word “temperature” for the word “energy”.  Now remember back to your days in physics class and remember that energy flows from a high state to a low state until the two states match and then the flow stops.  An air conditioning system takes advantage of that basic law of physics by absorbing the warm energy in a room and sending it to a lower energy place where the warmth is released and the cycle can start all over again.

One problem in this description of an air conditioning unit is that we usually don’t want the high energy released back into our rooms so we have to send that energy outside the room, or building, to get rid of it.  We do that by using refrigerants or water to transport the heat energy.  We also have to be sure that when we send it outside that it is at a higher energy level than the outdoors.  That is why we have compressors (and chillers which are just really big compressors) in our systems.  The compressors act as both a pump and as a device to actually add energy to the fluid that is pumped through the cooling coil in the room.  If you grab the side of a pipe entering a cooling coil it will feel relatively cold.  If you grab the pipe between the coil and the compressor it will feel a bit warmer.  If you grab the pipe on the leaving side of the compressor you might burn your hand.  The system has added enough energy to make sure that when the refrigerant or water reaches the outdoors it is at a higher energy state than the air outside the building.  That can be quite a challenge in a place like Phoenix or Dubai.

Most manufacturers realize that their equipment might be installed in those climates so they pick components in their systems that can operate under those circumstances.  But there are limits to what can be done.

The most widely available commercial cooling systems on the market are DX packaged units.  In order to satisfy the largest market (and sell the most equipment) these units are intended to be used for comfort cooling of people.  Since most people are “comfortable” when their office is around 75 degrees these units are designed around that operating point.  That is their happy point and that is the temperature of the air that is being returned to the cooling coil where heat energy can be absorbed into refrigerant or water and then sent outside to be removed.  The units will continue to operate at higher temperatures but remember that we need to be sure we send the heat outside at a higher level than the air outside. 

If you read the technical manuals for virtually every rooftop unit on the market you will see that, for a lot of really esoteric reasons, that rooftop unit is designed to operate at no more than 90 degrees returning to the cooling coil.  At that point the combination of components in the rooftop unit will be “maxed out” if the outside air temperature is in the 120 to 130 degree range. When the outside air temperature is 135 degrees on a roof in Phoenix then the high energy state and low energy state are so close together that almost no work is done and the temperature of the air coming out of the air conditioning unit starts to go up because the system is no longer rejecting much heat.  The system has quit working as intended and the situation usually spirals out of control as more heat builds up in the refrigerant or water.  Eventually so much heat has built up, and is compounded by the compressor, that the system shuts itself down to protect itself.

So, what does all this rambling have to do with anything?  Most of my blogs lately have been about mission critical/data center energy issues.  It is a big deal, and lots of folks are working on solutions, but economics sometimes trumps clear thinking or limits what can be achieved.

We are seeing more and more data centers specified with DX rooftop packaged units.  While these are normally high quality products they were originally designed to be at a happy place with, at most, 90 degree air being returned to the coil.  In a data center that is being designed to the latest ASHRAE standards the cold aisle can be anywhere between 80.6 and 113 degrees F.  When you allow a 20 degree F temperature rise across the servers before you return the air to the unit then I think you can see the problem.  The rooftop unit is being asked to operate well above its built in safety circuit limits.  Thus you end up with a self-limiting factor on how effective you can be in reducing the operating expense in the data center.  Even if you believe that your servers will be fine at 80.6 degrees F your HVAC unit probably will not be so fine.  And, to be honest, this same logic applies to CRAC units as they are nothing more than split DX systems.  So you have self-limited your options to cold aisle temperatures of no more than about 70 degrees F and your data center costs more to run than it could.

There is a class of rooftop unit that is better equipped to handle these situations and that class of equipment is commonly known as a DOAS, or Dedicated Outdoor Air System.  These systems, like our Applied Air FAP product, have been designed to expect Phoenix type temperatures across the cooling coil.  Returning hot aisle air at 105 or 110 degrees F is well within their “normal” operating ranges.  These systems are more expensive than a conventional rooftop packaged unit because of the components that are selected but they also provide the operating range that will allow the designer and operator to take advantage of the elevated temperatures that ASHRAE and the IT equipment people recommend for reducing data center operating expenses.

Mestex Hosts Independent Representatives at ASHRAE

Mestex Representatives Attend ASHRAE

With the annual ASHRAE/AHR meetings and exhibits in Dallas for the first time in 6 years, Mestex took advantage of the opportunity to host over 100 independent Mestex reps at the Mestex facility. We were also joined by a number of Mestek corporate employees including Stewart Reed, Mestek CEO.




Mestex DDC Dashboard

Test Area Demonstration

The reps were provided with guided factory tours that included presentations at four key areas in the plant...the gas-fired products test area, the hydronic products test area, the "Mestex Mall" show unit area, and the top secret Mestex R&D area. In addition to highlighting the extensive final test processes that every Mestex product endures the tour also highlighted the latest version of the Mestex DDC control system with full web-enabled interface and user "information dashboard".



"Dallas" Based Theme

Following the tours the reps gathered in the stage area that was set up in the plant for formal presentations on new software technologies that Mestex is introducing in 2013, a more detailed look at the DDC "dashboard", and a glimpse into a huge new sales opportunity. The presentations wrapped up with the introduction of the 2013 Sales Incentive program. The overall formal presentations were introduced by Mestex personnel who played the parts of characters from the TV series "Dallas".

Mestek Booth at ASHRAE/AHR Show

Over the following three days, Mestex personnel hosted a number of engineer and customer visits to the facility and also attended the ASHRAE/AHR show as part of the large Mestek contingent.

 

The MEP World Collides

Well, this weekend the MEP world (Mechanical, Electrical, and Plumbing) will converge on Dallas for the AHRI/ASHRAE annual meeting and trade show.  This is the one place where many engineers and contractors can view the vast assortment of products and services that are used in making the buildings we work and live in functional. 

The exhibit company says there are 3,500 booths set up in the Dallas Convention Center.  That is a lot of different companies that are invested in the construction industry...and many companies in the industry are not even showing!

Of course, our parent company Mestek will be on the show floor with a wide variety of products ranging from residential baseboard to machinery for producing ductwork for commercial buildings.  Look for the most diverse company in the HVAC industry in booths 2632 and 2845.

As for Mestex itself since we are based in Dallas we have a unique opportunity.  On Sunday, the 27th, we have over 100 sales representatives visiting our factory to learn more about what Mestex has planned for 2013 and to tour the facilities and see some of the hardware.  Then Monday through Wednesday we will be hosting several engineer and contractor tours of our operations. 

This is an exciting way to kick off the new year for Mestex and for the entire industry.  It also shows just how diverse our industry is and just how important to building occupants our industry can be.

The Changing Face of Real Estate

One of my more enjoyable activities that I have is to act as chairman for a developers forum as part of the NAIOP organization.  This activity provides insights into the thinking, planning, and expectations of commercial and industrial property developers and owners across North America.  At our annual meeting a couple of months ago there were many presentations and discussions that focused on 2013 and beyond.  I thought I would share just a few of the points from that meeting.

The NAIOP Research directors provided some interesting factors to consider going forward that tended to revolve around the way technology is changing the office and industrial markets.  E-commerce is projected to have a negative impact on mom-and-pop retail and small start-up retailers until the housing market makes a big recovery, according to Cassidy Turley-Terranomics.  They went on to say that while middle market retailers will continue to struggle, the luxury and discount retailers will continue to expand and open new retail and distribution facilities. 

Speaking of distribution facilities, Jones Lang LasSalle indicated that they believe that distribution center users will continue to push for higher bays...up to at least 36 clear feet...in order to increase efficiencies in handling e-commerce transactions.  Another interesting impact of e-commerce that was highlighted by IMS Worldwide and by Liberty Property Trust is that changing real estate requirement for an e-commerce focused distribution center.  The number of transactions per day in an e-commerce site can be 10 times greater than for a traditional distribution center.  Each of those transactions must be touched by someone so the number of employees in an e-commerce center is much higher.  Parking for up to 1,000 cars in addition to trucks means the land required for these centers can be 40 or 50 acres greater than a comparable "traditional" distribution center.  Implied in this scenario is also the need for a temperature controlled work environment for those 1,000 workers instead of the old "just keep the pipes from freezing" distribution or cross-dock environment.

Another impact of technology and e-commerce is that a DC ("distribution center") for e-commerce has an element of "mission critical" to it in order to process all of the transactions.  Developers and users of these new types of distribution centers look for locations that have reliable fiber optic and cable network access, as well as dual primary power substations in order to minimize downtime in the event of a power disruption.  Other location related decision criteria include being in a right-to-work state and in a state that does not charge sales tax on e-commerce transactions.

Shifting back to the office market, CBRE-Canada, noted that employees are changing how they work and the traditional office with walls is going away.  They also noted that employees, especially the younger ones, communicate with each other by text message versus phone reducing the "noise level" in the office down to the clicking of small touchscreens...reducing the need for walls to control cross conversations.

PPR/CoStar commented that the average lease that they see in the office market has decreased from 5,000 square feet to 3,600 square feet.  This statistic is reinforced by the results of a CoreNet survey of 500 corporate real estate executives who have changed their office plan metric from 225 square feet per employee down to 175 square feet in 2012 with a projection of only 150 square feet by 2017.  This change means that development of new buildings will continue to be pressured as it will take longer to absorb space in overbuilt markets. 

The final point from the annual meeting is that while there is abundant capital available for the right deal all of these other factors are driving developers to spend that capital on remodeling and repurposing of existing space. 

How I Spent My Summer Vacation

It has been a while since I have posted anything to this blog...no, I was not on sabbatical on some desert island...I have been traveling around North America talking to consulting engineers, contractors, and data center owners and operators.  This posting just provides a few insights that I garnered over the last 2 months on the road.

First, the data center/mission critical market continues to occupy the minds and the design resources of many, many companies in the design community.  It is clear that this is a market segment that is vibrant and all indications are that it will continue to be for quite some time to come.  The latest issue of Datacenter Dynamics FOCUS indicated that the world is now consuming over 300 Tkwh annually to drive data centers, with the US consuming over 25 Tkwh alone.  The consumption in the US is projected to grow over 9% in 2013.  While this information points to a growing market it also points to the urgent need for improved operating efficiency in data centers.

Second, and related to the first item, is the lack of knowledge about new "best practices" in data center design.  I have talked to dozens of engineers, contractors, and data center people who are not aware of the latest design guidelines from ASHRAE.  In fairness, those guidelines were only officially announced a few weeks ago...but they have been rumored and discussed for about a year now.  I mentioned in one of my earlier posts that education of the design community is an important, and ongoing, task.  This has been reinforced to me over the last 2 months.

Third, for those engineers and contractors who understand and embrace the new standards, is the challenge of convincing the data center people to adopt those standards.  This is less of a problem at the top levels of the data center company than it is on the floor of the data center.  The IT equipment operators who live in "the white space" seem not to understand the allowable operating temperatures of the equipment that they manage every day.  I have heard many different reasons for their reluctance to adopt the new best practices but I think it comes down to fear.  Because of stringent SLAs the operators worry about losing any equipment for any period of time...even though there is mounting research that this fear is unfounded.

Fourth, I have heard of several cases where the local electric utility has started to put limits on the available service capacity for planned centers.  In the US we are so comfortable with the idea that our electric grid can provide unlimited power that we forget that is not true.  We have a fixed number of powerplants with only so much generating capacity.  With the tremendous growth of data centers, and data centers with 300 to 500 watt per square foot electrical demands, there is a limit to what a utility can do.  And timing is another element of the equation.  A data center can be built in a matter of months...a powerplant takes years.  So even when a utility sees the demand coming they cannot add capacity as quickly as the demand can be added.

So, these are a few observations from the last couple of months.  Of course there is more to the story and feel free to comment on this post with any questions you might have.  I will try to respond as quickly as possible.

How to Save Almost $100,000 Per Year In Your 1 Megawatt Data Center

Over the last few weeks while I have been traveling there have been some interesting bits of information released in the mission critical world.

For example, Dell introduced their 12^th generation PowerEdge servers.  This generation of servers is warranted to handle temperature excursions up to 45 degrees C, or 113 degrees F, for up to 90 hours per year.  One of Dell’s rationales behind marketing the server at those conditions was to allow fresh air cooling in virtually the entire continental US.  Other research by Dell has indicated that their servers can operate 87% of the year in Washington, DC using fresh air cooling alone.

The energy saving potential of raising the inlet temperatures that high can be enormous.  Instead of running chillers or compressors 8,760 hours a year they are only operating 1,138 hours per year. 

To put that into numbers is difficult but let’s try a little example.

If the PowerEdge server power consumption is 300 watts then the cooling system must remove 300 watts times 8,760 hours per year or 2,628 kwh of heat (8,961,480 btu).  That can either be accomplished using mechanical cooling or fresh air cooling or a combination of the two.

A pretty efficient HVAC system will remove about 4.5 watts of heat per watt of electrical energy used.  So to cool that PowerEdge server using mechanical cooling will require 2,628,000 watts of heat divided by 4.5, or 584 kwh of compressor power.

To cool that same server using fresh air for 87% of the year will only require 75.8 kwh of compressor power.  Of course, the fan energy stays the same in both cases but the compressor savings of 508.2 kwh PER SERVER can really start to add up.  At an aggressive electric rate of 4.5 cents/kwh that amounts to $22.87 PER SERVER PER YEAR.

At modest densities of, say, 40 servers per rack the savings amounts to $915 PER RACK PER YEAR.  Now consider how many racks are in the typical server room or data center.  If the data center has a server load of 1 megawatt then a density of forty, 300 watt, servers per rack will translate into 83 racks.  So the annual savings would be almost $76,000 in this example.

To make the savings even greater the same HVAC unit that provides the fresh air could also provide indirect evaporative cooling and completely eliminate the compressor-based cooling…adding another $3.50 PER SERVER PER YEAR of savings.  That would add another $11,620 PER YEAR in savings.

Too Hot to Handle? A Simple Reminder

Well, this is embarrasing.  I have been in the HVAC industry for over 40 years now and have helped design and manufacture some of the more sophisticated products that have been introduced.  But, in spite of that I have to admit that I messed up.  And the lesson that I was reminded of can help you too if your residential, commercial, or mission critical system is struggling to keep up with the heat.

Over the past couple of weeks the temperature here in Texas has been over 100 degrees F every day...sometimes up around 105 to 110.  That is nothing unusual for Texas in the summer and not as bad as last year.  But I started to notice that my residential HVAC unit was no longer able to maintain my thermostat setpoint of 77 to 79 degrees F.  The system was consistently running 3 degrees behind and running non-stop...and was only installed a year ago.

Refrigerant leak?  Undersized?  Dog left the door open?

No...it was one of the most common problems in any HVAC system that is not running correctly...the condenser coil was coated with a fine film of dirt.  Let me repeat that...a FINE film of dirt.  Not clogged...not even very obvious at a quick glance...a FINE film.  In my case it was actually a fine film of dryer lint since the clothes dryer outlet was located behind the condensing unit...but the point is that had a service tech not looked at the coil with a flashlight I never would have noticed the dirt.  Running water over the coil from a garden house to wash off the film dropped the system head pressure and restored the system's ability to maintain the thermostat setpoint without running non-stop.

Many years ago Louisiana State University conducted some tests on residential HVAC systems to determine the impact of dirty condensing coils.  The results were eye-opening.  A fine film of dirt, similar to what I had on my system, would reduce system capacity by up to 20%.  If your home, business, or server room is too hot then imagine what giving it an extra 20% of capacity could do...and it would only cost you a bit of water and time to wash off the coils...with no service tech assistance required.