Azure and Google and Twitter, Oh My!

The flying monkeys attacked in force on Thursday July 26, 2012, taking down cloud leaders Microsoft, Twitter, and Google Talk. It seems the cloud is no Yellow Brick Road after all, guiding merry executives to some imagined Oz where the sun always shines and outages never happen.

The more I talk to IT planners, the more I find they are looking at reinvesting their cloud savings into business continuity. They rightly hope to compete based upon reliability, and to protect their businesses by exchanging the extremely high and unpredictable costs of outages for the predictable and low costs of the cloud and business continuity. They've clearly got the right idea, especially when you consider the noise outages like yesterday's can make.

Microsoft Azure

• Area affected: Europe, via the Dublin datacenter and Amsterdam facility.
• Duration: About two and a half hours.
• Cause: Unspecified, but one expert suspects infrastructure troubles.
• Effects: Loss of cloud service throughout Western Europe. Businesses like SoundGecko were unavailable.
• Source: WebTechInfo.com

"In Azure’s case on Thursday, the constant availability of power and lack of a software culprit, such as the Feb. 29 one that downed several services, points to 'more of an infrastructure issue' where some undetected single point of failure in a network switch or other device temporarily disrupted availability, said Chris Leigh-Currill, CTO of Ospero, a supplier of private cloud services in Europe." (source)

Google Talk

• Area affected: Worldwide (source).
• Duration: About five hours.
• Cause: Unspecified, but one expert suspects a bad hardware or software upgrade.
• Effects: System unusable, granting access but providing only error messages.
• Source: TechNewsWorld.com

"Outages like this 'often happen as the result of a hardware or software upgrade that wasn't properly tested before installation,' Rob Enderle, principal analyst at the Enderle Group, told TechNewsWorld" (source).

Twitter

• Area affected: Worldwide.
• Duration: About an hour and possibly more in some areas.
• Cause: Datacenter failure and a failed failover.
• Effects: "Users around the world got zilch from us."
• Source: CNN

"Twitter's vice president of engineering, Mazan Rawashdeh… blamed the outage on the concurrent failure of two data centers. When one fails, a parallel system is designed to take over -- but in this case, the second system also failed, he said" (source).

Am I Repeating Myself? Am I Repeating Myself?

My last blog post was titled "A Flurry of July Outages – And All of them Preventable". On Thursday we had another flurry, and all of them in the cloud. And I have to say again, with caution, that these were preventable. I am cautious because we don't know for sure if Mr. Leigh-Currill and Mr. Enderle are correct in their assumptions about the Azure and Google service interruptions, but if they are correct then these outages were almost certainly preventable.

And the Twitter outage was clearly a case of failover failing us once again. That's what it is when "a parallel system that is designed to take over" fails to do so when the primary goes down. It's the same story we see over and over again.

After all these years in the industry I am still amazed that the biggest tech names in the world continue to rely on the ancient failover paradigm. The leaders of these organizations are trusting their reputations, their revenue, their shareholders' profits, their customers' businesses, and potentially peoples' lives to a disaster recovery process that quite likely won't work.

What Some Companies are Already Doing About It

Earlier in this post I mentioned the companies that are reinvesting cloud savings into reliability. These are typically smaller, more agile companies. They'll be able to take on the tech behemoths based on reliability, because they are thinking beyond mere cost savings and efficiency. They see the cloud as a means of creating their own "failsafe" hosting paradigms.

As most readers of this blog know, Always Available™ technology from ZeroNines replaces failover and backup-based recovery. It is already cloud-friendly. The Twitter outage is exactly the kind of thing we prevent. Companies hosting an Always Available array on Azure would have had virtually no risk of downtime, because all network transactions would have continued processing on other nodes. Always Available prevents data disasters before they happen, whether a power supply fails, or software gets corrupted, or a tornado picks up your Kansas datacenter and relocates it to Munchkinland, or someone melts your servers with a bucket of water, or the flying monkeys carry off the last of your support staff. Why clean up after an expensive disaster if you can prevent it in the first place?

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

RIM co-CEOs Resign: Is This the Cost of Downtime?

Back in October I commented in this blog about the enormous RIM BlackBerry outage [source]. I wrote that "even a massive outage like this is unlikely to cause the demise of a large and important firm, but combined with other woes like a less-than-competitive product and poor business model it could well be the deciding factor."

And now for the fallout. RIM is still in business, but its beleaguered co-CEOs/co-Chairmen Jim Balsillie and Mike Lazaridis have resigned and taken other positions within the company [source]. I'm sure it was not the outage alone (or all RIM outages put together) that caused this leadership shakeup. But it could well have been the deciding factor.

Outages and CEO Job Security

RIM's product problems are certainly serious. But I see a fundamental difference between 1) the prescience needed to get the right product to market at the right time, and 2) the technical ability to keep an existing product up and running. Customers might to some degree forgive a company whose product is reliable but behind the times. They will abandon if it doesn't work when they need it even if it is the newest, slickest thing around.

The October outage has RIM "facing a possible class action lawsuit in Canada" [source]. Add the cost of that in addition to the costs of recovery, customer abandonment, shareholder value and so forth. (Stay tuned; I will be commenting on the legal issues around cloud outages in the next few days.)

To put RIM's decline in perspective, the company was worth $70 billion a few years ago but today has a market value of about $8.9 billion [source]. Their stock dropped about 75% last year and was down to $16.28 before the market opened on Monday January 23, 2012 [source].

So according to the rules of modern business, someone has to pay and in this case it is the CEOs.

Now Imagine This at a Smaller Company

Can you imagine a three-day outage at a smaller software company? Or even a one-day outage? Imagine a typical e-commerce technology provider with 50 retail customers, 100 employees, and an SaaS application. If the core application, image server, database server, customer care system, inventory system, orders & fulfillment system, or other key element goes down that could be the end of them. Many smaller companies do not survive a significant downtime event. And many smaller retailers do not survive if they are unable to do business on a key shopping day such as Black Friday or Cyber Monday.

Or even if the email system goes down for a couple hours. It happens all the time. Email is a key element of workflow and productivity and what company can afford to sit still for even a couple hours?

It's more than the CEO whose job is at risk. Here's where an ounce of prevention is worth far more than a pound of cure.

That Rickety Old Failover

Remember my earlier comment about outdated yet reliable products, versus outdated and unreliable products? Ironically, the failover disaster recovery model that failed RIM back in October is one of those old and unreliable products. It was designed for systems and architectures that no longer bear any resemblance to what businesses are actually using. If failover worked I would not be writing this because there would be no need for its replacement.

But if you want to find out about real business continuity and getting away from failover, take a look at ZeroNines. Our Always Available™ architecture processes in multiple cloud locations, on multiple servers, and in multiple nodes. There is no hierarchy so if one goes down the others continue processing all network transactions. ZeroNines can bring application uptime to virtually 100%. It is a complete departure from the failover that RIM is using, and that small businesses everywhere stake their futures upon.

Time Will Tell

"RIM earned its reputation by focusing relentlessly on the customer and delivering unique mobile communications solutions… We intend to build on this heritage to expand BlackBerry's leadership position," RIM's new CEO Thorsten Heins is quoted as saying [source].

Let's hope this "focus on the customer" also includes a strategic initiative to build genuine uptime and availability, or maybe we'll be reading about another new RIM CEO next January.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

What Did One BlackBerry User Say to the Other BlackBerry User?

Nothing, according to twitter user @giselewaymes (source).

In what has to be every large enterprise IT manager's worst nightmare, a big high profile outage grew into a monster, expanded to global proportions, made headlines everywhere, and after three days seemed to have no end in sight. The cause was a failed failover that could have been avoided.

Background: RIM BlackBerry 

BlackBerry is produced by Canadian firm Research In Motion (RIM). It is one of the leading smart phones among business users. Its real forte is encrypted mobile email and instant messaging. BlackBerry has about 70 million users worldwide (source). Several high-profile outages and many smaller ones have tarnished its reputation, and this week's seems to be pushing the company to the breaking point if all the buzz on the Internet is to be believed.

The Problem: Failed failover 

On Monday morning October 10 2011, millions of BlackBerry users in Europe, the Middle east, and Africa lost access to messenger, email, and Internet. The outage spread to every continent and may eventually have effected half of all BlackBerry users (source).

RIM explained things to some degree on their website on Tuesday October 11: "The messaging and browsing delays that some of you are still experiencing were caused by a core switch failure within RIM’s infrastructure. Although the system is designed to failover to a back-up switch, the failover did not function as previously tested (source).

In other words, their failover-based disaster recovery system failed. It can be inferred that this led to cascading failures that knocked out other systems in other regions, leading to this worldwide problem. As of Wednesday evening the 12th it was still not fully resolved, with an interesting update posted on their site outlining the status in various parts of the world (source). By Thursday morning it looked like things were finally under control, with service almost back to normal in most areas.

The Cost: Paid compensation and a blow to the business 

I don't doubt that RIM will compensate users in one way or another, perhaps in the form of free service (which seems to be the industry's de-facto compensation currency). RIM Co-CEO Jim Balsillie said that such a step would be considered but that their immediate focus was fixing the problem (source).

More damaging is the additional blow to RIM's reputation. Lots of users are claiming on Facebook, Twitter, and other online forums that this is the last straw and that they will quit BlackBerry. For many this may be a hollow threat but there is genuine peril here. "This outage… comes at a particularly bad time for RIM, since it faces increasing competition in the smarpthone market… Apple's iPhone and phones on the Google Android operating system have been gaining ground, and the new iPhone 4S goes on sale Friday (October 14)" (source).

The cost can be high outside of RIM as well. "The outage caught much of D.C. off guard Wednesday and underscored the region’s reliance on the BlackBerry — which is still the only federally approved smartphone for employees in some government agencies (source).

As for RIM itself, back in June there was a flurry of articles suggesting RIM was potentially facing bankruptcy (source). And this week there have been a number of stories about growing momentum for a RIM breakup or merger (source). Even a massive outage like this is unlikely to cause the demise of a large and important firm, but combined with other woes like a less-than-competitive product and poor business model it could well be the deciding factor.

The Solution: Eliminate failover systems

RIM is in trouble for a number of reasons but downtime like this does not need to be one of them. I contend that the core problem was not a failed switch but a failed failover. Switches will fail and there is no avoiding that. If you can architect the perfect switch, I invite you to do so and you'll be richer than Bill Gates.  It's what happens after the inevitable switch malfunction (or other disaster) that matters most. Failover systems will fail too. RIM's apparently worked fine during a test but the strain and chaos of a real-world crisis was too much for it. At ZeroNines, we propose eliminating the failover systems in favor of something that will turn failures into virtual non-events.

ZeroNines' Always Available™ technology eliminates the need for failover, processing the same applications and data simultaneously on multiple servers, clouds, and virtual servers separated by thousands of miles. All servers are hot, and all applications are active. So if a switch fails in one network instance there is no need for a risky failover to another. Other instances are already processing the same transactions in parallel and simply continue processing as if nothing had happened. Once the problem with the switch is rectified, that instance is brought back into the Always Available array, is automatically updated, and resumes processing along with the others.

The Numbers

RIM says that its service "has been operational for 99.7% of the time over the last 18 months" (source). That equates to about 1,576.8 minutes of downtime, or 26.28 hours per year.

A good industry standard for uptime is 99.9% or three nines. That is 525.6 minutes of downtime, or 8.76 hours per year.

ZeroNines can provide in excess of five nines of uptime, or 99.999%. That is less than 5.3 minutes of downtime per year.

I do not know if planned downtime was included in RIM's 99.7% calculation. Companies often do not include planned downtime in their business continuity projections, counting only unplanned outages. But downtime is downtime from a user's perspective, whether caused by an accident or a planned maintenance cycle. ZeroNines protects against both.

In the last 12 months since ZenVault Medical went live on an Always Available cloud-based architecture it has experienced true 100% uptime, with no downtime whatsoever for any reason. That includes planned maintenance, upgrades, and other events that would have taken an ordinary network offline. 

Visit the ZeroNines.com website to find out more about how our disaster-proof architecture can protect businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

TD AMERITRADE Outage and How Failover Fails Finance

Online brokerage TD AMERITRADE was offline for 80 minutes on Thursday May 20, 2010 [source]. Because of the outage, some of their clients could not log in to their accounts to place trades during the powerful market downdraft that occurred that day [source]. Outages among financial firms have gotten a lot of coverage in the last couple years, no doubt because of the universally amped-up sensitivity to any kind of news with the word “financial” attached to it. Here’s a brief look at this outage, and some commentary on outages in general among financial companies.

Background: About TD AMERITRADE

Online discount broker TD AMERITRADE has millions of U.S. customers (Wikipedia reports over six million), and many more internationally. The company has grown rapidly through acquisition and was the 746th-largest US firm in 2008 [source]. It acquired thinkorswim Group, Inc., another popular online brokerage, in January 2009. Lots of average Americans use TD AMERITRADE to generate income and manage retirement accounts. I use them myself and really like their system, but did not notice the outage because I was doing other things at the time.

The Problem: An Outage of Some Kind

At about 11:40 AM Eastern time, clients found that they could not log on to the TD AMERITRADE retail website. The outage ended at about 1:00 PM. No disruption was reported on their mobile site or at their subsidiary thinkorswim [source]. Clients already logged in experienced no trouble, urging one writer to speculate that it was a web authorization issue of some kind. [source]. If TD AMERITRADE has made a formal announcement of the cause, a half hour of Googling on my part failed to find it.

Was This a Failed Failover?

Posted on the TD AMERITRADE site [source] is the “TD AMERITRADE Business Continuity Plan Statement” [source]. One of the statements in this brief public document reads “Disruption of service at any of our service centers will result in calls, orders and electronic communications being re-routed to an alternative service center located in a different region of the country with a separate power grid and transportation system.”

Let me state clearly that I am entering the realm of speculation here. The statement quoted above implies that TD AMERITRADE is relying on a business continuity plan based on failover architecture. Failover or cutover has been the de-facto choice for business continuity and until recently it has been the only real game in town. But it is by nature unreliable and even the best systems are subject to downtime. If their backup plan is indeed based on failover, then failover obviously failed them.

The Cost: As Always, it’s the Intangibles

As in so many outages of this kind, the real costs are difficult to estimate. Easiest to ponder are the lost commissions from trades that could not occur during an extremely busy trading day. Less tangible are the effects on reputation and customer satisfaction. No one wants a broker that is unavailable when they need them most. One customer claimed to have lost about $2,000 from being unable to log in [source]. TD AMERITRADE stock fell about 3.7% that day but this may not mean much because markets overall were down about 3%.

According to a May 2007 article from Financial Services Technology, a study from the Meta Group revealed that “the cost per hour for downtime – ranging from simple network outages to major emergencies – in the financial services sector is, on average, $1.4 million” [source]

An Ugly Thought: Downtime among High Frequency Traders

For many, the cost will be far higher. Some banks, hedge funds, and other high-power financial firms engaged in High Frequency Trading (HFT) make billions of trades a day over ultra-high speed connections [source]. Many trades live for only a few seconds. Enormous transactions are conceived and executed in half a second, with computers evaluating the latest news and acting on it well before human traders even know what the news is. HFT is having a significant effect on markets; there is evidence that the history-making “Flash Crash” of May 6 2010 was caused and then largely corrected by High Frequency Trading [source]. What would happen if one of these HFT systems was down for an hour and a half? Or even just a minute? Whatever your stance on the ethics of HFT, I think it fair to say that those engaged in it need to avoid downtime at all costs.

Failover Can’t Handle It

Even a successful failover event may cause some glitches and lost trades among the average retail trading populace. But if a High Frequency Trading system experiences such a glitch, billions of dollars could be lost in the blink of an eye. The trades themselves may fail, and by the time the system comes back up the conditions that made those trades possible are a thing of the past. And that’s for a successful failover. A failed failover can leave businesses out of the race for minutes, hours, and even days.

The Alternative: Active/Active Architecture

High profile financial systems clearly need something better than failover. The typical outage is caused by failures of server hardware, server software, upgrades, maintenance, and sometimes more dramatic stuff like fires and floods. The best protection in these cases is to eliminate failover entirely, and switch to an “active/active” or “hot/hot” architecture that eliminates the chance of a failed cutover and the resultant downtime. Always Available™ business continuity architecture from ZeroNines is one such system. Always Available processes all network transactions continually, simultaneously, and equally in multiple locations on multiple servers, all of which are hot and all of which are active. Always Available can offer virtually 100% uptime, because instead of relying on failover Always Available simply continues running the same apps and data at two or three additional locations, with no interruption to the user. So if a web server or database goes down somewhere, the other nodes of the system continue processing without missing a beat. Visit the ZeroNines website to find out more.

Alan Gin – Founder & CEO, ZeroNines

Most Businesses Don’t Know what Downtime Costs Them

I just discovered the results of a survey about the need for application availability among businesses [source]. The survey was conducted by ITIC and Stratus Technologies. Results were released in April 2009. It basically sought to find out how much application uptime businesses think they need, and what they intend to do about it.

The survey found that overall, IT executives are aware that the need has grown for high-availability applications and the infrastructure to support them. But budgets are too low to support them, and most companies do not know what their downtime is costing them. This makes it difficult for these same executives to make a budgetary case for implementing high uptime solutions.

Downtime is a business killer. As an example, consider that of the 350 companies in the World Trade Center before the 1993 truck bombing, 150 were out of business a year later because of the disruption. [source: Gartner/RagingWire report cited in “Without the wires,” Fabio Campagna, Disaster Recovery Journal, Winter 2002].

The big lesson here: there is a significant competitive advantage for investing in uptime.

Here are some key facts from the survey, and my thoughts about them.

1) “Two out of five businesses – 40% – report that their major business applications require higher availability rates than they did two or three years ago. However an overwhelming 81% are unable to quantify the cost of downtime and only a small 5% minority of businesses are willing to spend whatever it takes to guarantee the highest levels of application availability 99.99% and above.”

Clearly, the field is wide open for companies to pull ahead if they go for four or five nines of uptime (or more), particularly those who serve vital and highly regulated sectors such as financial, healthcare, defense, data hosting, and so forth. A company that falls out of compliance with strict regulations like Sarbanes-Oxley or HIPAA can be driven to the brink by fines, the costs of regaining compliance, and lost business.

2) “The survey results uncovered many “disconnects” between the levels of application reliability that corporate enterprises profess to need and the availability rates their systems and applications actually deliver.” In other words, businesses are not getting the uptime they require, whether it is to meet SLAs or simply conduct everyday business.

In reality, the uptime that company leaders “profess to need” is probably insufficient. Considering that a downtime event of only a few seconds can cause a cascading failure in applications and databases, they probably need uptime of practically 100% in order to avoid a bigger disaster. Once that first domino falls, maybe you can grab it and stand it back up but all the rest are already falling. The damage is done.

3) “Some 41% said they would be satisfied with conventional 99% to 99.9% (the equivalent of two or three nines) availability for their most critical applications.”

I can’t imagine a company being without its “most critical application” for between 8+ hours (for 99% uptime) and four full days (99.9%). Companies have gone out of business after downtime of less than that. I can’t help but believe that the executives who answered this question like that are somehow out of touch with the realities of their environment. Maybe they are in industries where expectations are really low. But can you think of a bank or stock brokerage or hospital where one- or two-day outages a couple times a year are the norm? I can’t. And that is probably because such companies cease to exist.

Contrast that with this: “An overwhelming 81% of survey respondents said the number of applications that demand high availability has increased in the past two-to-three years.” High availability is typically considered to be four nines (99.99% availability and above) or less than 53 minutes of downtime per year. Yet 41% of respondents say they would be satisfied with only two or three nines? Astounding.

The Disaster of Disaster Recovery

IT executives typically prepare for downtime by implementing some variation of the backup/failover paradigm, even though most are aware it is unlikely to work. I invite you to read the ZeroNines whitepaper “The Disaster of Disaster Recovery” (available on the ZeroNines.com website) which looks at the causes of downtime and explores the shortcomings of the predominant failover disaster recovery technique. It also discusses the ZeroNines alternative, which can bring uptime beyond any measure of “nines” to virtually 100%.

ZeroNines Technology, Inc. is not affiliated with ITIC, the Information Technology Intelligence Corp. or with Stratus Technologies.

Visit the ZeroNines.com website to find out more about how our disaster-proof architecture can protect businesses (and government agencies) of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Fixing The FAA’s Single Point of Failure

“The difficulties started when a single circuit board in a piece of networking equipment at a computer center in Salt Lake City failed around 5 a.m…” [source]

All too often it seems that the biggest problems are caused by the smallest failures. This blog is full of posts about how generator transfer switches, router programming changes, and problematic network hardware can bring businesses to their knees. Now a single circuit board failure causes havoc among airlines, airports, and air travelers.

I stand by my earlier assertion: Trying to guarantee application and data uptime by eliminating all possible sources of failure is not possible. The more complex a system gets, the more likely some part is going to fail, and it is impossible to identify them all. But there is a way to prevent these little disasters from becoming big ones.

Background: The Flight Plan Management System

The failed FAA computer system was the National Airspace Data Interchange Network [source], which manages flight plans and ground traffic. This is one of two nationwide computer centers that collects flight plans. The other is in Atlanta. This was the third time since June 2007 the system has failed [source].

The Problem: Hardware Failure Blocks Access

When the circuit board failed on November 19, 2009, access to data and communications was blocked, making flight plans filed by airlines inaccessible [source]. Air traffic controllers had to enter flight plans manually in several parts of the U.S. The problem was fixed about five hours later.

The Cost: Mostly to the Airlines

The FAA being a governmental agency, no direct fiscal impact can be readily estimated. However, the cost to airlines has to be considerable, since many flights were canceled or delayed. Airline stocks were down that day – whether the computer failure was the cause or not – and our poor beleaguered airlines can’t help but suffer when something like this happens. They were still down even after the problem had been fixed [source]. And of course individual travelers, such as myself, will bear the brunt too in the form of delays, costlier alternative travel, and unplanned hotel stays. Not to mention missed business meetings which can cost a business a lot more than a replacement airline ticket. The domino effect of airline delays is a disaster unto itself.

The Solution: Sidestep the Single Point of Failure

Doug Church, a spokesman for the National Air Traffic Controllers Association, said…"We think it's a single-point failure that occurred somewhere in the system," he said. "One single glitch was able to shut down the entire system." [source]

This is perhaps the scariest statement about the whole affair. The simple fact that they went dark shows that their backup systems also failed. This is not surprising; most disaster recovery systems use the “failover” or “cutover” technique which is outdated, unreliable and can lead to cascading failures and increased downtime. Such occurrences are frighteningly common.

At ZeroNines we propose a different approach. Instead of trying to catch a downtime event with a failover recovery, like a ninja trying to catch an arrow, we simply double up all the processing in multiple data centers around the country or around the world. Each processes the same thing at the same time so if “a single circuit board in a piece of networking equipment at a computer center in Salt Lake City” fails, the additional networking equipment in Atlanta or Omaha or Dusseldorf or wherever keeps on processing.

The likelihood of application or data downtime – where users lose access to the tools and information they need to do their jobs – drops to virtually zero because the chances of all data centers, or clouds, or virtual environments failing simultaneously is statistically negligible. In this instance, had the National Airspace Data Interchange Network been protected by our Always Available™ technology, the Atlanta network node would simply have continued processing while Salt Lake City was repaired and brought back online. Then the system would have automatically updated Salt Lake with all the transactions that had occurred in its absence.

Visit the ZeroNines website to find out more about how our disaster-proof architecture can protect businesses (and government agencies) of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Enabling Cloud Confidence

A week ago I wrote about the Sidekick disaster and how events like that just keep doubts growing, pushing the wholesale adoption of the Cloud further away. This doubt has made it into the mainstream media, where it will taint the opinions of potential cloud users, both consumer and commercial. We at ZeroNines think we have the solution that will enable the cloud to perform as it needs to.

The core problem with outages is not the existence of hazards that can damage servers and knock elements of a network (cloud or otherwise) offline. Storms, fires, and equipment failure will always happen and there is no way to eliminate them. The real problem is the reliance of cloud providers on obsolete failover-based recovery paradigms that simply can’t maintain continuity when disaster does strike.

L.A. Times columnist David Sarno perfectly sums up the cloud’s tenuous situation in his October 18 article “Still hazy on cloud computers' security” [source]. “A series of incidents involving cloud computing over the last several months has poked holes in the hype bubble, raising questions about the cloud's dependability -- and whether it's ready for use by a broader group of workers and businesses.” He is right on target.

Meeting the Need to Fortify

As Sarno puts it, “As e-mail, word processing and data storage continue to move from users' computers to the Web, companies must fortify their servers from a variety of potential disasters -- natural and man-made -- to help ensure that the data and the applications are accessible at all times.” He quotes Google’s SEC filing:

"(Google’s) systems are vulnerable to damage or interruption from earthquakes, terrorist attacks, floods, fires, power loss, telecommunications failures, computer viruses, computer denial of service attacks" as well as sabotage and vandalism...

The good news is that today, ZeroNines' Always Available™ CloudNines™ technology can fortify servers from damage or interruption from earthquakes, terrorist attacks, floods, fires, power loss, telecommunications failures, computer denial of service attacks, as well as sabotage and vandalism. We leave the viruses to others to deal with, but we can add most types of routine maintenance, unplanned maintenance, data migrations, equipment upgrades, software upgrades, and a number of other potential causes of downtime.

Forget Failover

The IT world fatalistically believes that downtime is inevitable, and is something to be lived with and minimized if you’re fortunate. This view predominates because until now the only disaster recovery solution available has been the flawed failover paradigm, which everyone in IT knows can be a disaster unto itself. During a crisis or failover event, cutover can cause additional problems, downtime, and cascading application failures as computing switches from primary to backup systems.

But the IT world has it wrong. Disasters will happen and must be dealt with, but the downtime they cause can be prevented.

Always Available™ Means Virtually 100% Uptime

ZeroNines’ Always Available™ solution eliminates failover and backups, instead providing synchronous identical processing on multiple cloud nodes geographically separated by thousands of miles. If a storm wipes out your East Coast cloud, CloudNines enables processing to continue on clouds in other parts of the country and around the world. If you need to upgrade server software, you can isolate one cloud node, do your upgrade, and bring it back online once it is stable. Our technology has journaling and updating features to assure that all transactions are completed and that any cloud node that goes offline is brought up to the most accurate logical state once it comes back online.

CloudNines can push application availability beyond the industry-accepted standard of 99.999% (five nines) to virtually 100%. In our ongoing test case, the ZeroNines MyFailSafe environment has never experienced any downtime at all, for any reason. It went live in July 2004, and had individual network nodes knocked offline a number of times due to hurricanes, power outages, server migrations, and other causes. All applications experienced full 100% availability throughout.

Will ZeroNines eventually be recognized as a vital cloud-enabling technology? That remains to be seen but you can bet that is how we see ourselves. If you want to find out how we can make the cloud a viable option for you, let me know.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Gmail Maintenance Leads to Router Overload

It is often the mundane problems that cause the most trouble. The two-hour Gmail outage on Tuesday, September 1, 2009 had a fairly unspectacular cause, but its effects are shaking a tech giant’s plans and causing some commentators to wring their hands over the acceptance of SaaS offerings in general.

Whatever its effects on the industry, this is one of several outages in the past year which are harming Google’s efforts to sell its email services as a corporate tool. At the very least it cost them a lot of money. Fortunately, such outages are avoidable.

Background: Google Hopes for Significant Gmail Revenue

Gmail is Google’s free email app, and is used worldwide by millions of people. Gmail also has paid services, and Google is trying to build it up into a corporate app that can generate significant revenue. Analysts and customers alike have been watching it closely over the years to see if it really can grow into a reliable corporate power tool, but have been disappointed by a number of recent outages.

The Problem: A Classic Cascade Failure

Last Tuesday’s problem “was caused by a classic cascade in which servers became overwhelmed with traffic in rapid succession” [source]. Google had taken several Gmail servers offline for maintenance. Recent changes to routers were intended to increase routing efficiency, but instead caused some routers to become overloaded. Traffic got shunted to an increasingly small pool of available routers until the system collapsed.

The Cost to Google

Google wants to get more customers onto its paid Gmail service. The outage adds to the image of Gmail as being not stable enough for business use and makes it harder to persuade corporate users to actually pay for it.

By way of compensation, Google “…added three days to year-long subscriptions to its corporate Google Apps email service, which costs $50 per-user-per-year.” [source] This equates to approximately $50 million. Unfortunately, users would rather have uptime than compensation, and Google got a lot of bad publicity which will make it harder to get business users to switch from other offerings. [source]

The Solution: A Network that Can Absorb Failures

“Google said it would focus on making sure that the request routers have sufficient headroom to handle future spikes in demand, as well as figuring out a way to make sure that problems in one sector can be isolated without bringing down the entire service.” [source]

Isolation of problem servers or nodes is a core function of ZeroNines’ Always Available™ technology. If they had been using Always Available, Gmail could have tested their new router/server configuration in isolation while the rest of the network was left to operate in the usual way. The new configuration could have been rolled out one server at a time without interrupting service. If one or more of the newly configured routers became unstable, that failure would have been confined to just that sector and the rest of the Gmail network could have continued processing in its usual way.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Seattle Database Fire Unnecessarily Shuts Down Businesses and Online Services

Cascade failure. If you’re in IT, that’s a particularly frightening term. In the case of last week’s Seattle data center fire, the term is especially appropriate since it was literally a cascade of water that wrecked everything and sent a number of businesses and online services offline. Here’s a look at this disaster and a way it could have been prevented.

Background: Fisher Plaza, a Major Hosting Facility

Fisher Plaza is “a self-styled carrier hotel in Seattle, and home to multiple datacenter and colocation providers.” [source] A partial list of organizations hosted there includes: payment service provider Authorize.net (which itself has 238,000 merchant customers), Port of Seattle email system, Swedish Hospital’s internal IT systems, Pacific Science Center website, geocaching.com website, major TV and radio station KOMO, online Facebook game Bejeweled Blitz and dozens of other businesses [source].

The Problem: Fire Leads to Cascade Failure

Early on Friday morning, July 3, 2009, Fisher Plaza’s main generator/transfer switch failed. This caused an overload. This caused a fire. This triggered the fire suppression system and brought firefighters to the scene, both of which shot water into the generator room. The generators stopped, and we deduce that power from the grid was shut off too. The UPS and the cooling system also failed. Temperatures in the facility rose high enough to wreck some servers and destroy data [source].

Think about the downstream effects. 238,000 merchants potentially have their transactions interrupted or lost because Authorize.net’s servers are forced offline. One can only hope they had their own functioning backup plan. A hospital’s IT system became unavailable; I have no information on what impact this had on patient care. And apparently KOMO had to transmit from a mobile unit in their parking lot [source]. It is not hard to imagine the impact to these and other organizations.

The Solution: Fire- and Flood-Proof Hosting

No, ZeroNines does not wrap servers in asbestos. There is no way to know what bizarre little accident will happen next, so prevention is unlikely. Some will trigger chain reactions that become major IT disasters.

What we do is to prevent a catastrophe in one place from knocking out a business everyplace. In this case, if any of the clients or tenants at Fisher Plaza had been using our technology, their data, transactions, apps, and other assets would have all been processing simultaneously and in perfect replication in other data centers hundreds or thousands of miles away.

This is not a cutover scenario. Processing would not have “switched” from Seattle to elsewhere. It simply would have stopped in Seattle and continued in real time in San Jose, or Denver, or Singapore, or wherever else they placed their data centers. There would be no loss of business continuity. Their businesses would not have gone down, and the real disaster – lost connectivity, productivity, and revenue – would not have taken place.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Uptime and the Cloud Crowd at CSIA

A few days ago, Jake Smith of Intel and I presented at the Colorado Software Industry Association (CSIA) monthly meeting in Denver (source). We talked about cloud computing and the elements that will determine its rate of adoption: the needs of businesses, their expectations of cloud performance, and the real-world limitations of the cloud that are currently stalling its adoption. The biggest issue is reliability, and I introduced ZeroNines’ technology as a potential solution. It was a great crowd, and their hunger for a reliable cloud was obvious.

Businesses need their applications and data to be available all the time. So far, clouds and cloud providers have not succeeded in proving that they can actually offer that. The industry needs to overcome the cloud’s downtime problems before serious business can be done on it. I believe the Big Three (Amazon, Azure, and Google) will refocus their efforts on providing highly available cloud infrastructures and market this capability accordingly.

The Cause is Academic

Of course every network is subject to threats and failures that can cause downtime, and there’s no getting away from that. It doesn’t take an earthquake to knock vital networked apps offline; some recent high-profile cloud provider outages have shown that all it takes is a failed OS upgrade. New and unexpected problems crop up every day. But the cause of an outage is really only academic for the business relying on the cloud. Service should simply continue because the business needs it to.

The scary thing is that the current disaster recovery paradigm (failover) is insufficient for protecting businesses when these things happen, and can’t be relied upon to prevent downtime or even a speedy recovery. In addition, there is an increase in catastrophic risk from poorly architected virtualized environments, and most notably in server consolidation, which is a core technology of the cloud.

The Solution is Continuity

At the CSIA meeting, we introduced the crowd to our Always Available™ technology, which maintains cloud continuity by synchronizing and protecting multiple private, public or hybrid clouds. It can mix cloud computing and physical hosting via datacenters hundreds or thousands of miles apart. The distance prevents any single regional disaster from damaging more than one data center. There is no server hierarchy, so all transactions run simultaneously and equally on all cloud and server nodes. Best of all, they update each other constantly in real time so if one goes down the others simply continue processing with no interruption to service.

To protect against an outage during an upgrade, I would postulate the following solution: Isolate one cloud or network node in an Always Available configuration and do your upgrade there, while the other nodes manage the clients’ transactions. Test the upgrade and slowly roll it out to the other nodes. If things start to go haywire, isolate the misbehaving node, solve your problems, and start the rollout again. There would be no need to risk the entire service on an untested upgrade.

Always Available works for cloud customers as well as service providers. It is provider- and platform-agnostic, so you can mix and match all you need to.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Hurricane Charley Couldn’t Stop the Email

In this first Disaster Litany Posting, I look at a sequence of events that is near and dear to ZeroNines. Our own real-world experience with a hurricane, power outages, and an email system will show just how our downtime-preventing technology works, and serve as a pattern for the solutions we suggest for other disasters.

Background: MyFailSafe™ Email System

ZeroNines offers Always Available technology that can virtually eliminate downtime among networked applications, data, and other assets. To test our technology, we created the MyFailSafe Email Service and launched it on our Always Available network in July of 2004. This was specifically intended to test Always Available in the real world, by running MyFailSafe just like any other email service is run, with real customers and real traffic, and subject to the same threats that any other network or email system is vulnerable to.

The Problem: A Hurricane

All readers who remember Hurricane Charley please raise your hands… For those of you who don’t, Charley hit Florida on August 13, 2004. According to Wikipedia, it killed about thirty people and caused $15 billion in damage. Widespread flooding, wind damage, power outages, and other problems crippled much of the state for several days. I don’t have statistics on downtime among private business networks or service providers, but it’s a safe bet that it was serious.

Charley hit about a month after we launched MyFailSafe. It caused electrical grid fluctuations that drained the Orlando local exchange carrier battery backup systems, isolating the Orlando node of the ZeroNines Always Available infrastructure. Our own battery system prevailed and still had a 75% charge when commercial power was reliably restored, but the site could not communicate for 16 hours because of LEC downtime.

The Solution: Hurricane-Proof Architecture

During this 16 hours, when our Orlando node was effectively offline, the MyFailSafe email service did not experience any downtime at all. Any user whose power was still on and whose desk was not under water experienced true 100% uptime throughout, whether they were in Florida, Colorado, Canada, Asia, or anywhere else.

How? Our Always Available deployment has additional nodes and data centers in Colorado and California. All applications, transactions, data exchanges, and other network activities run equally and simultaneously on these multiple secure application servers, geographically separated by hundreds of miles. In IT parlance, all servers are hot, and all instances of all applications are active. There is no server hierarchy, and consequently no single point of failure. When the Orlando node fell silent, all MyFailSafe processing continued uninterrupted on the others. There was no need for failover or recovery because these other nodes were far from the storm, they never went down, and continuity was maintained.

Since activation on July 15, 2004, the MyFailSafe network has never experienced any downtime for any reason, including this and other hurricanes, two migrations from server collocation providers to clouds, a data center move, and an email worm attack that interrupted email service from AOL and other major providers. These potential disasters, which forced our servers offline, had no power to bring our applications down. All applications and information retained 100% availability throughout.

Contact ZeroNines to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

A Litany of Disasters: Downtime Events and How to Avoid Them

“Aviation in itself is not inherently dangerous. But to an even greater degree than the sea, it is terribly unforgiving of any carelessness, incapacity, or neglect."
-- Anonymous

Years ago, I saw those words on a poster of a World War One aircraft stuck about 20 feet off the ground in the limbs of a tree. If we were to update this and adapt it to the business user’s desktop, it would lose its poetic charm but strike home with a whole new audience:

“Networked assets in themselves are not inherently dangerous. But to an even greater degree than stuff on your hard drive, they are terribly unforgiving of any carelessness, incapacity, or neglect."

The warning is clear: Disaster may be only inches away, particularly for the unprepared. It’s a lot harder to recover after some accident knocks out a hundred or a thousand users than it is to re-boot your own machine.

In this blog, we will be looking at some actual disasters that have struck organizations when their networks have taken a hit from storms, fires, attacks, and far more mundane threats like human error and equipment failure.

For a business, there may be little correlation between the physical effects of a disaster and its financial impact. Imagine a business dependent upon a distant data center in the U.S. Tornado Belt. One good storm could leave their personnel and property untouched, yet destroy their ability to do business by wiping out their data, applications, and transactions. Elsewhere, an earthquake could cause deplorable loss of life and property damage, yet leave a business relatively unharmed if its networked computing capabilities remain intact. And an otherwise strong corporation could suffer irreparable damage by something as quiet as a software failure or equipment malfunction, which to the outside world does not qualify as a “disaster” at all.

I’ll be describing some instances where ZeroNines’ solutions for networks, virtualized environments, and clouds could have prevented disastrous downtime, and helped avoid unwanted headlines and losses to productivity, reputation, and revenue. Our approach does not use any kind of failover or cutover, since those occur after the downtime event and are not true disaster prevention. After all, it’s far better to avoid the downtime in the first place than to try to recover from it afterward.

Next week: How MyFailSafe really did provide fail-safe email during Hurricane Charley.

Contact ZeroNines to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – ZeroNines, Founder & CEO