blog

The great cooling debate and how Alcatel-Lucent changed the game.

Fri, 23 Jul 2010 14:13:48 +0100

I am scheduled to visit the Alcatel-Lucent facility in Plano . Texas next month to take a first hand look at their new modular server rack cooling system. Your first question is probably "Alacatel-Lucent makes a cooling system?!?" This longtime staple of the IT world and particularily the telecom industry decided to design and build their own cooling sytem. The primary reason behind this was frustration with the current state of computer room cooling. Alcatel has made a big investment in IP video since their primary customers the AT&Ts and Verizons of the world have made that their next big play. IP video on a commercial scale requires the highest density computing environment you can imagine.....squared. In their Plano testing facilitythey were routinely seeing inlet temperatures exceeding 95 to 100 degrees with hot spots in the 120s. Needless to say, the hot aisles in places were positively solar. The traditional cooling guys solution was "more capacity". The problem was that Alcatel had been stacking on capacity for years and they had basically run out of room to put cooling equipment. Instead of building new facilities to deal with this problem, the guys that brought you push button phones, callerid and digital telephone systems, decided that they would take a shot a doing it a different way...and they did it and it is amazing.
The really interesting part is the obvious dissatisfaction of one of the CRAC industry's biggest customers with the status quo. The technology at the server switch level has changed dramatically in just a few years and the advent of technologies like virtualization and ip video has created densities the likes of which we have never seen in the data center environment. The juxtaposition of this is the way we reject the heat from those spaces, this hasn't changes much in many years. I can't tell you at the times I have stood in big data rooms where the AC was still cooling this big space that actually held 6 racks. Everytime I see this I always think "What a waste!". The shrinking server footprint is the other side of the cooling coin but it comes from the same basic source...we do cooling pretty much the same way that we have done it for years.

What's Your C.O.D.

Sat, 26 Jun 2010 11:39:24 +0100

Most companies have not looked at their particular cost of IT downtime. What does down time cost YOU? You know it costs something, right? Unfortunately most companies don't really take a hard look at what downtime costs them until they experience an extended outage of some kind. Even then, quite often they don't really do the math they just realize that it was painful and they do not wish to repeat the experience.

The catch phrase Mission-Critical gets used a good bit but understanding what is mission-critical for YOU is the real issue. This can differ greatly from organization to organization. Is the internet mission-critical for you or is some internal application the real lynch pin? Determining this is essential to the next step in the process, assigning a cost.

One company I worked with had experienced several significant phone/data outages so they could easily calculate their cost. They just looked at an average normal day's revenue against the averaged revenue of the days they had outages and subtracted...voila....they had a pretty good idea of what their downtime cost. If you don't have those numbers another way is to analyze where your revenue comes from by percentage. For instance, if 20% of your revenue comes from phone/fax and 80% comes through the internet in some fashion then you take your average daily revenue....say $10,000.00 and divide accordingly, in this case $8000.00 and $2,000.00. In this example, an internet outage would cost this company $1,000.00 per hour, based on a normal 8 hour day. Now you may be saying that this is really simplistic math, and it is, so why do I need to do this? Simple, until you do this, you are making critical business decisions based on hazy speculative information. Not having this information is just like saying, "If we add ABC widgets to our line of products, we are pretty sure that we can increase our sales some." Sadly, that's probably the way some decisions get made but even the people making them would probably tell you that this is not the best way to do it. Armed with the right information, you now have the ability to make really smart decisions about your infrastructure. For instance, ABC Company had 16 hours of downtime last month costing them $16,000.00 (using our earlier example). If they invest $2000.00 per month in maintenance, they should be able to decrease downtime by at least 8 hours netting the company an extra $6,000.00 per month ($16,000.00 less $8,000.00 for the downtime they didn't have and minus the $2,000.00 for the maintenance). Now you are talking about something that everybody can understand and support. In a business situation, is this not the logical way to think about most everything since the main objective of every for-profit entity is to make profit? This is just pure cause and effect reasoning, because we don't do X, we are experiencing Y and Y costs us.......fill in the blank.

In order to move initiatives forward, you have to have the proper tools. Which would you rather do, go into a meeting with your boss and say, "I need to spend $200,000.00 to beef up our infrastructure and maintenance." Or "Last year we lost $350,000 due to IT downtime, if we beef up our infrastructure and maintenance, we can save over $100,000 per year." Hmmmmm, which one of those would get the more positive reception?

The Reasons Why You Need to know what your C.O.D. is:

1. It's like PUE and all the other things that you need to know, you just need to
have that information

2. Downtime is a cost just like electricity and salaries....
it must be measured and controlled

3. C.O.D. is the easiet way to get everyone in the organization
to understand the concept and buy in!

So......what's YOUR C.O.D.?

Disaster Avoidance instead of Disaster Recovery

Fri, 21 May 2010 14:26:30 +0100

A recent white paper from a company called Zero Nines talks about "The Disaster of Disaster Recovery" The basic premise here is that with even the latest Disaster Recovery solutions there will be a certain amount of downtime and a certain amount of data loss. The words "acceptable amount" come to mind here. Zero Nines operates in the international banking sector where the acceptable amount is NONE but most of the rest of us live in another reality where the cost of a "one to many" multiple replicated system would be a little out of our league. So we have set up our vitualization and the hot sites. We monitor the "heartbeats" and our heads rest easily on the pillow at night. I want to pose two questions here : 1) What if you don't have a "state-of-the art disaster recovery solution? Some folks don't you know. 2) What if your disaster recovery solution fails? Please don't say "That can't happen" because you darn well know it can.

Let's look at some recent real world disaster scenarios. These things actually happened and were chronicled in IT Management magazines "Top 8 Datacenter Disasters of 2007":

Hosting firm Rackspace US Inc. suffered back-to-back outages in just 36 hours. The first outage was caused by a "mechanical failure" in the company's Dallas datacenter on Sunday, November 11, 2007. Customers experienced "intermittent service interruptions" and a team of more than 100 techs was deployed to find and fix the problem. Then, on the following Monday evening, a pickup truck struck a utility pole and brought down the transformer feeding the datacenter. Emergency generators kicked in and operated as intended, and Rackspace

transferred its power to its secondary utility power system and brought its chilling units back online. However, the utility shut down power in order to give emergency workers safe access to the downed transformer. Temperatures rose within the datacenter. Rackspace shut down selected servers in order to avoid overheating all of them.
A power outage hit downtown San Francisco on July 24, knocking out 365 Main Inc. - a 227,000 square-foot facility and datacenter development company. At least three of 365 Main's eight co-location centers were knocked out. Among the Web sites that went down for a few hours were giants like Craigslist, GameSpot, Yelp, Technorati, Typepad and Netflix. Power was restored after 45 minutes. 365 Main later estimated that between 20 and 40 percent of its customers were affected.The company ultimately attributed the disaster to backup generators made by the Dutch firm Hitec, which failed to kick in. It seems that an incorrect setting in one tiny generator component prevented the component's memory from resetting properly.
In the Rackspace scenario, in the first outage, mechanical failures happen but past experience tells us that it becomes more likely when maintenance is less than perfect. There is really no excuse for the second outage.Oviously there was only enough emergency power (generators) to run the IT equipment and not the chillers or some other means of keeping the data center cool. I would imagine that the phrase "But that would never happen." was uttered many times prior to this incident. ( side note, "But that would never happen." and variations there of are the most dangerous words ever uttered.)

In the 365 Main instance (I hope the irony in their name isn't lost on anyone) , the generators failed to start. In the life safety/medical area, failure testing is required annually. Failure testing means simulating a real failure.....cutting the commercial power for instance. It might not have uncovered this particular issue but you could certainly say with conviction that "We checked and tested everything possible".

The point here is simple , while spending millions upon millions of dollars on disaster recovery doesn't it make sense to spend as much on trying to avoid having the disaster? How many have seen the Ford commercial where the young lady squeals with glee as her SUV parallel parks itself? That is undoubtedly the scariest thing on TV. Not insisting on rigorous preventative maintenance and not testing emergency systems under real world conditions is kind of like that. Just take your hands off the wheel and trust that the technology will work! WHAT IF IT DOESN'T? Let's think about what happens if that thing that will never happen...HAPPENS! Like the secondary power grid going down or the generator not starting.

Here's another scenario of a disaster where everything worked:

No customers were affected by a fire that broke out in an electrical room at a Terremark data center in Culpeper, Va., early on April 30. Terremark spokesman Xavier Gonzalez said electrical failover systems were activated and there were no interruptions in power supply to the computer rooms. "Our redundant power systems worked and kicked in properly," he said, explaining that one of the data centers on the company's NAP of the Capital Region campus was switched to generator power. "Everything continued to work properly there." The fire was caused by an electrical fault in a medium-voltage room, something Gonzales said was a relatively common occurrence in data centers. The fire damaged one electrical-gear cabinet before it was extinguished by members of the Terremark team together with the local fire department. Firemen were dispatched on site after a fire alarm was activated around 12:30 a.m. EDT and employees at the data center confirmed that there was a fire. According to Gonzalez, it was extinguished fairly quickly. "At the end of the day, the design of the facility and the redundant power systems worked properly," the spokesman said. "That's a good sign for us."
There are currently two active 50,000 square foot data centers at NAP of the Capital Region, a campus that has room for a total of five such facilities. Terremark began construction of the third facility in April.- This is an excerpt of an article from TechFlash

Disaster Recovery and Business Continuation are kind of like parachutes or those floating cushions under your seat on an airplane, you certainly want them, won't fly without them but you really really don't want to have to use them. Instead lets make sure the plane has all it's PMs , that the tires have plenty of air in them and that the pilot doesn't even get to have strong mouthwash. Granted this cannot completely eliminate the need for the safety devices but it certainly lengthens the odds on their use. After all , can you be 100% certain that the chute will open?

If you would like to read ZeroNines Whitepaper, go here to download it

The Case AGAINST Scheduled Maintenance

Mon, 12 Apr 2010 16:53:03 +0100

The case for Scheduled Maintenance has been made over and over, ad nauseum for the last 50 or 60 years since Air Conditioning became popular in the 40s. Many who are far more intelligent and learned have recited all the reasons why this is the way to do things. Instead of trying to re-invent the wheel , I would like to recall some of the reasons that I have heard out in the field for NOT doing maintenance. I promise to try my best to defend these positions.

“We just don’t do maintenance contracts.” – Obviously this organization sees the folly of getting trapped in a contract. Now they don’t say they don’t do maintenance they just don’t do contracts. We all know that the best way to shop for something is on the spur of the moment, right! Why would you want a relationship with a Vendor anyway? Everybody knows that Vendors are all out to get you.

“The unit is still in warranty.” – This gets at the heart of how a warranty should work. If you offer a 10 year 100,000 mile power train warranty, you need to manufacture a product that will last for 10 years and 100,000 miles regardless of what the end user does or doesn’t do!

“We just don’t have the time/resources to do maintenance.” – How can you schedule preventative maintenance when you are fixing all that broken equipment? Besides, there are a lot more hours and jobs generated this way. Let’s keep America working!

“The energy savings that you receive from maintenance just aren’t worth the recurring cost.” – A 10, 20 or 30 percent savings on a $5000.00 a month electric bill is piddly compared to the staggering cost of a maintenance contract. (Let’s see 10% of $5000 is $500 a month vs. $800 a month for maintenance…… see there’s no way to come out on it. You are $300 in the hole every month!)

The next time someone wants to talk about PREVENTATIVE MAINTENANCE, just throw some of these tried and true reasons at them. I am sure that they will make haste to leave and never darken your door again.

Six Energy Saving Ideas For Your Data Center

Wed, 07 Apr 2010 15:01:34 +0100

Monitoring,Benchmarking and Data Center Metrics - This is probably the most important part of any energy saving strategy. You cannot fix what you don't know about or understand.

Here's a good real world example: Company A decided to monitor its power usage and see how it was trending. They found several spikes and anomalies one of which they determined was the CRAC equipment fighting amongst themselves - one dehumidifing while another was humidifing while another was reheating. A simple networking strategy for their units saved them hundreds per month.

Variable Speed Drives - The idea here is to vary the air flow on chilled water based cooling equipment based on the actual need for cooling. Also Electronic Contolled fans

on the output side can do similar functions on equipment that uses freon or glycol.

Air or Water Side Economizers -this approach is more popular in colder climates

but some Southern based data centers have used this along with some other technologies

to tack on extra savings,

Ultrasonic Humidification - this one doesn't seem like much but ultrasonic humidification

uses as much as 70% less energy than steam based units and are virtually maintenance free.

Plus this is one of those hot button technologies that the feds love and is often eligible for rebates.

Virtualization/Consolidation - Yes I know this seems obvious but what we see sometimes are data centers virtualizing and getting smaller...in the same big inefficient room! Take a look at the physical size of your room and see if a smaller room can be created or if not, look at doing a cold aisle containment so that you are cooling the places where the equipment is instead of where it is not .

DC Power instead of AC - This can be kind of tricky but bringing in a good EE can get you where you need to be. Telcoms have been doing this for years and the savings here can be multifold. Reduce the cost of individual servers by going to central power supplies. DC takes a conversion out of the process (DC back to AC) thus it's more efficient.

To Contain or Not To Contain.....What's The Answer?

Sat, 06 Mar 2010 15:55:02 +0100

The use of hot aisle/cold aisle configurations had been around for several years and the physics of it make sense - put the cold air in the front where the equipment breathes and put all the hot air in a separate aisle - but in this traditional design in a high-density data center, up to 40% of the cool air is wasted. Hot and cold air mix over the top of racks and around the rows. So doesn't this seem like only part of the solution?

The issue here is air mixing, if you mix hot and cold air together, you are expending energy to cool the hot air that you have already cooled and run through your equipment. Basically its a self defeating loop. Containment further separates the hot and cold air , thus making the process exponentially more efficient . But which is the best to contain - hot or cold?

Liebert/Emerson sides with Cold Aisle Containment while APC/Schneider seems to favor Hot Aisle Containment. The reasoning behind this is both sound and self-serving. Since Liebert's customer base is primarily the big traditional "blow the air under the floor" type data center, cold aisle containment makes sense for them. APC's focus is on putting the cooling right on the rack and they have several items in their line that take the heat from the back of the rack and duct it into a hot air plenum.

With either method the idea is to limit air mixing to as close to zero as you can.The critical piece for either method is blanking panels to create a physical separation inside the cabinet at about the mid-point so that the cold air stays on the cold side and the hot air stays on its side.

I participated in a seminar recently on data center efficiency and one of the panelists was from HPs data center support group. I asked him which one was more energy efficient, hot aisle or cold aisle containment. He indicated that the inate energy efficiency for either was similar, however in a hot aisle scenario , the practical considerations in a production environment negated some of the energy savings. For instance , the temperature of the hot aisle containment would typically be over 100 degrees, so that if someone had to work on that side, you would need to open up the containment and maybe even blow some cold air in to make it habitable thus negating some of the benefits.

The fact remains that a aisle containment strategy can make your cooling more efficient and more economical.Dean Nelson, the senior director of global data center strategy at eBay Inc,said in a recent interview with SearchDataCenter.com that one of eBay's data centers had a power usage effectiveness rating of more than 2, which is close to average. After installing containment in his data center, eBay got the number down to 1.78."It was an overall 20% reduction in cooling costs, and it paid for itself well within a year," he said. "It is really the lowest-hanging fruit that anyone with a data center should be looking at."

The Difference Between Remote Access , Alert Notification and Monitoring.

Thu, 19 Nov 2009 17:33:54 +0100

I am couching this discussion in terms of Mission-Critical Infrastructure but this really applies to several different disciplines especially IT.

First let's define and qualify each of these items.

Remote Access is the ability to access a piece of equipment from a remote location. A remote location can be another part of the building on the same LAN or another part of the world across the internet. The basic premise here is that you can access the device interface without having to connect directly (plug in to) that device.

Alert notification is the ability to set parameters like "if the the temperature is above 80 degrees" send a message.The message can be an email , a text message , a blinking light or a phone call.

Monitoring is when someone is constantly watching a displayed set of parameters from a device.

About now , some of you are saying "why the Ned's First Reader definition ?" You may not be confused about which one is which but trust me there is some confusion out there. There seems to be special confusion regarding Alert Notification and monitoring.

The best analogy is to the way voice mail works. If you just have a very simple voice mail on your phone, then while you are out of the office , you need to periodically call in and access the voice mail to see if you have a message.

If you have outcalling or voice mail to email , then you get a notification pushed out to you that you have a message. This is how alert notifications work.

If you are sitting at your desk and the phone rings , you answer it.....this is monitoring.

Lets say you have a problem with your cable tv , would you rather leave a message in the voice mail that that person will get the next time they check it or one where that person will be notified of the message? Obviously notification is better but what is that person turns off their cell or doesn't check their email? What you really want is to call and have a person answer and take care of your problem. Monitoring provides the fastest response to problems and unlike the other two , monitoring can be PRO-active instead of RE-active. With monitoring , problems can be detected and dealt with BEFORE they cause a failure.

Monitoring offers a PRO-active approach which will head many issues off at the pass. In the event that an emergency response is needed, monitoring gives the fastest reponse to critical issues.

Hardy offers all three of these options to our mission-critical clients depending on their equipment and needs. Our Premium Care Plan includes monitoring as a standard part of the package.

The Value in Value

Wed, 21 Oct 2009 10:21:35 +0100

OK the title sounds at the least repetitious and maybe even silly,but I have been thinking a lot lately about value. Gitomer and all my other guru's talk a lot about "building value for the customer". What is valuable to your customer?

In my world, I build value on three levels. One with the knowledge and expertise that they gain access to by doing business with me and my company. This knowledge base is not just what's in my head but the cumulative knowledge and experience of all the people that I work with and the ones that we interact with that don't necesarily belong to the inner circle of the company.

The second value builder is how we can leverage technology to help them accomplish the things that they need to do or want to do, effectively, reliably and efficiently.

The third is the way we do business. We apply the principles of trust based selling to every project. If you always have the best interests of the customer in mind,even if that doesn't always parallel what makes you the most money, you will prevail and prosper in the end.

Why your Mission Critical Infrastructure needs to be monitored.

Sat, 17 Oct 2009 13:29:12 +0100

Using the same rationale as we do about our information systems, wouldn’t the best people to watch the power and cooling, the fire suppression system and the physical security be the ones that built and installed the systems and that work on them every day?

Familiar Territory - Unfamiliar Setting

Sat, 17 Oct 2009 13:27:16 +0100

Having spent many years around server rooms and telephone offices , the magnitude of the systems that support information technology doesn't surprise me.

What does occasionally give me a start is that all of these diverse processes have "a guy" attached to them who is the sole master of that process. The cooling equipment has "an AC guy" . The power equipment usually has two "guys" , the electrician and sometimes the vendor who sold it to you. The fire system has a "guy" and so on. I don't know about you but I don't really like having a bunch of "guys" . Mostly because, in closely integrated systems, if the "guys" don't agree , you are the one who loses out.

We all have our stories about finger pointing and while occasionally amusing , they are almost always not pleasant. So how do you avoid the finger pointing "The UPS guy said that the electrician had to......" scenarios?

One way would be to have one "guy" who deals with all the other "guys", a General Contractor "guy" who deals with all the subcontractor "guys". This works pretty well in the construction industry so why not adapt that concept.

Here's a real life conversation I had a few days ago, Facilities Manager:" I need to move the cooling units in my data center, can you help me?' Me: "Absolutely , let's talk about what we need to accomplish." Facilities Manager: " We need to move this unit over to this wall so that the airflow is better and so that we can access the inside of the unit easier," (This part also adds to the One Guy argument, but we'll get into that later) Me: " Okay , that makes sense, let me get one of my HVAC people in here to line out what all we will need to do." Facilities Manager: I guess we need to get someone in to handle the changes in the raised floor?" Me:" We can take care of that part." Facilities Manager: " and the electrical wiring......." Me:" I have that covered as well."

The reason for highlighting my conversation is not because I want to tell you how great I am but because I don't think this is so very remarkable, its the way this particular conversation is supposed to go. However, in the real world , we all know how this type pf project usuallly proceeds, 3 or 4 different "guys" from that many different companies and nobody takes ownership of the whole project , except you of course.

I would be really interested in thoughts and opinions on the "One Guy" concept. What systems in the data center (or anypart of the building) would benefit from Single Point of Contact (SPOC from my telephone days).

Thanks

Gary Dunlap