How Short Outages Become Long Outages

Early in the morning of Friday July 27, 2012, Hosting.com experienced an 11-minute outage. Although service was restored very quickly, many customers weren't prepared and experienced hours of downtime as a result (source).

The key story here is that even though a few minutes of hosting provider downtime is probably well within the parameters of the service level agreement (SLA), the customer's actual downtime far exceeds that. I'm going to quote my own blog from just a couple weeks ago because it accurately sums up the situation:

Your cloud (or other hosting) provider no doubt promises a certain amount of uptime in their service level agreement. Let's imagine that allows one hour of downtime per year. If they have one minor problem it could cause downtime of just a few minutes. But if your systems are not prepared, that interruption could corrupt databases, lose transactions in flight, crash applications, and wreak all manner of havoc. Their downtime glitch will become your costly business disaster unless you are prepared in advance to control it on your end (source).

Hosting.com's Service Level Agreement is posted publicly on their website (source). A quick read reveals that it does NOT promise 100% uptime. Datacenters fail, and that's a fact of life. When signing with any hosting or cloud provider, it is vital that you understand exactly who is responsible for what, and whether total downtime is measured according to the unavailability of their infrastructure or the amount of time it takes you to recover.

Recently, the Paris-based International Working Group on Cloud Computing Resiliency (IWGCR) found that costs for outages between 2007 and 2011 among the 13 providers they reviewed exceeded $70 million (source). No SLA from any provider is going to compensate for those kinds of losses. If the industry demanded this of them, no hosting provider would be able to stay in business. It will be far better for their customers to invest in reliability than to expect dollar-for-dollar restitution after a disaster.

Background: Hosting.com and its Customers

According to the company website, "Hosting.com is a next generation cloud hosting and recovery services company focused on ensuring your mission-critical applications are AlwaysOn™" (source). They are a leading provider of other enterprise hosting solutions and services as well, with datacenters in Dallas, Denver, Irvine, Louisville, Newark, and San Francisco. One source says they host over 65,000 websites (source). This includes financial services, healthcare, media, retail, software as a service (SaaS) providers, and content distribution networks (CDN) (source).

In contrast with other recent outages and other providers whose explanations were late or non-existent, Hosting.com CEO Art Zeile stepped up very quickly during this crisis and alerted his customers of the problem, its cause, and its effects. Though they won't be thrilled with news like this, customers need clear communication and honesty from their providers. That way they know what to tell their own customers and management, and their overworked internal IT teams will have a better chance of taming the chaos. I applaud Mr. Zeile and his actions. We need this level of leadership to benefit the cloud industry at large.

The Problem: Human Error, a Power Outage, and a Chain Reaction

Mr. Zeile explained that "An incorrect breaker operation sequence executed by the servicing vendor caused a shutdown of the UPS plant resulting in loss of critical power to one data center suite within the [Newark, Delaware] facility" (source). The power was back on within 11 minutes, but "customer web sites were offline for between one and five hours as their equipment and databases required more time to recover from the sudden loss of power."

I wasn't there but I can surmise what happened. When the power went out, an unspecified number of servers were shut off without proper shutdown procedures. Applications and databases were abruptly terminated. Other applications and databases that depended upon them suddenly lost transactions in flight. They crashed too, taking down other apps and databases in turn. And so on down the line in a classic cascading failure scenario.

Recovery of the customers' crashed apps and databases required hours. Each customer needed its own data and apps restored, and those that were still running probably had to be shut down and then re-started in proper sequence. Servers had to be checked for damage after their "crash" shut-downs. Apps and data that successfully cut over or failed-over to secondaries had to be cut over again, from the secondaries back to the primaries, and I'll bet there were further failures as that happened.

The Solution: Make Your Datacenters Expendable

Many of the apps and data on the system were undoubtedly protected by failover and backup recovery architecture, or by one of the Hosting.com business continuity solutions. Many of these certainly continued running as they successfully failed over to their secondaries. But equally clear is that apps and data for about 1,100 customers (1.7% of the total Hosting.com customer base) did not continue running. Either they were not equipped with adequate business continuity systems, or the failovers failed. One writer quotes Zeile as saying that although Hosting.com offers a backup option "few customers, at the affected location, had elected to purchase it" (source).

I am unaware of any hosting or cloud provider who publicly promises 100% uptime. So the customer must expect to have some amount of downtime, if only for maintenance. Logically, customers need to provide adequate business continuity systems to protect themselves.

Datacenters go offline all the time for any number of reasons. Thus, your business needs to be able to continue talking to customers, sending billing statements, shipping goods, and paying creditors despite untoward events like power outages, fires, human error, hardware failure, and so forth.

ZeroNines does not recommend or use a failover- or backup-based recovery paradigm. We take a different approach aimed at preventing downtime in the first place, rather than recovering from it afterward. In the case of the Hosting.com outage, Always Available™ architecture from ZeroNines offers two solution scenarios:
1)  Hosting.com already operates multiple geographically separated datacenters. Always Available architecture would allow processing to continue on any or all of the remaining five when any one of them goes down.
2)  Hosting.com customers could deploy their own Always Available array that would simultaneously replicate all transactions and data on other Hosting.com datacenters, or in other clouds or with other providers.

In either case, the end user experiences no downtime because the remaining nodes continue processing as usual. The offending datacenter simply drops out of the array until power is restored or until your staff can repair it. The other nodes of the Always Available array will update the damaged node once it is functioning again, and bring it to an identical logical state.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Building Outage Resistance into Network Operations

An article I read the other day in MIT's Technology Review [source] nicely sums up what I've been hearing about cloud operations from dozens of clients, partners, and other colleagues around the country. The cloud is great for development, prototyping, and special projects for enterprises, but don't rely on it for anything serious. As that article says, "For all the unprecedented scalability and convenience of cloud computing, there's one way it falls short: reliability."

But the truth is that the tried-and-true models of network operations aren't all that reliable themselves, and neither are the disaster recovery systems that are supposed to protect them. Granted, they are probably more reliable than the cloud at this point, but downtime is downtime whether it's in the cloud or in a colocation facility. The effect is the same.

What is really needed is outage resistance that is built into network operations, whatever the model

Why downtime happens

I recently read an interesting whitepaper from Emerson Network Power [source] that describes the seven most common causes of downtime as revealed by a 2010 survey by the Ponemon institute (http://www.ponemon.org/index.php). The causes are all pretty mundane: UPS problems such as battery failure or exceeded capacity, power distribution unit and circuit breaker failures, cooling problems, human error, and similar things. All of them apply to any data center, whether in-house or in the cloud. None of the exciting stuff like fires, terrorism, or hurricanes made it into the top seven, though of course they could lead to a failure of a battery, circuit breaker, or cooling unit.

The Emerson whitepaper describes best practices that can reduce the likelihood of downtime induced by each of the top seven causes. That is all well and good, but some are very costly, such as remodeling server rooms "to optimize air flow within the data center by adopting a cold-aisle containment strategy." Other recommendations include regular and frequent inspection and testing of backup batteries, installation of circuit breaker monitoring systems, and increased training for staff.

These are good ideas but costly, if not in capital for server room reconfiguration then in staff hours and other recurring costs. The paper contends that problems caused by human error are "wholly preventable" but I believe this is a mistake. No matter how stringent the rules or how well-documented the procedures, someone will take short cuts, overlook a vital step in the midst of a crisis, or sneak their donut and coffee into the control room. Applications fail under stress, databases fail to restart properly, and any number of other things can and do go wrong. There is no way to write contingencies for each, particularly when the initial failure leads to an unpredictable cascade effect.

And what of the cloud?

I believe the cloud brings tremendous value to developers, SMBs, and other institutions that need low cost and great flexibility. Where else can an online store launch with a configuration that is not only affordable but also ready for both super-slow sales and a drastic ramp-up if sales shoot into the stratosphere? But like most “better, cheaper, faster” initiatives, the cloud has genuine reliability problems. A company running their own data center could choose to incur the expense and work of instituting all of Emerson's best practices since they are in control of the environment. But all they have from their cloud provider (or colocation provider for that matter) is their Service Level Agreement (SLA). They can't go in themselves and swap out aged batteries or fire the guy who persists in smuggling cinnamon rolls into the NOC.

The Technology Review article tells us that some companies are looking for ways to make their cloud deployments far more disaster resistant to start with, rather than just relying on their cloud provider's promises [source]. Seattle-based software developer BigDoor experienced service interruptions as a result of the Amazon cloud's big outage in April 2011. Co-founder Jeff Malek said "For me, [service agreements] are created by bureaucrats and lawyers… What I care about is how dependable the cloud service is, and what a provider has done to prepare for outages" [source].

The same article describes the Amazon SLA and its implications:

Even though outages put businesses at immense risk, public cloud providers still don't offer ironclad guarantees. In its so-called "service-level agreement," Amazon says that if its services are unavailable for more than 0.05 percent of a year (around four hours) it will give the clients a credit "equal to 10% of their bill." Some in the industry believe public clouds like Amazon should aim for 99.999 percent availability, or downtime of only around five minutes a year.

The outage resistant cloud

ZeroNines can give you that 99.999% (five nines) or better, whether you are running a cloud or just running in the cloud. Cloud service providers could install an Always Available™ configuration on their publicly-offered services, providing a highly competitive edge when attracting new customers.

Individual businesses could install an Always Available array on their own networks, synchronizing any combination of cloud deployments, colocation, and in-house network nodes. It also facilitates cloud migration, because you can deploy to the cloud while keeping your existing network up and running as it always has. There is no monumental cloud migration that could take the whole network down and leave the business stranded if there's a glitch in starting an application. Instead, Always Available runs all servers hot and all applications active, enabling entire nodes to fall in and out of the configuration as needed without affecting service. The remaining nodes can update a new or re-started node once it rejoins the system.

ZeroNines client ZenVault Medical (www.zenvault.com/medical) developed and launched their live site in the cloud using an Always Available configuration. Since the day of its launch in September 2010 it has run in the cloud with true 100% uptime, with no downtime at all. That includes maintenance and upgrades. When a problem or maintenance cycle requires a node to be taken offline, ZenVault staffers remove it from the configuration, modify it as necessary, and seamlessly add it back to into the mix once it is ready. ZenVault users don't experience any interruptions.

Visit the ZeroNines.com website to find out more about how our disaster-proof architecture can protect businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Amazon EC2 Outage: Déjà Vu All Over Again

It seems we can always rely on cloud outages to spice up the news feeds. Today, it's another Amazon EC2 Cloud outage, which is a nice departure from the wildly gyrating stock market and the U.S. debt downgrade.

I didn't write about Amazon's big April 2011 EC2 outage simply because I was overwhelmed with other work (along with texts, tweets and emails about the outage). That outage affected big-name customers like Netflix, Foursquare, HootSuite, and Reddit (source). Some EC2 customers' websites were down for as much as two days.

Then just this past weekend an electrical storm over Dublin Ireland led to a lightning strike on a transformer and a subsequent explosion, fire, and loss of power at an Amazon data center. Backup generators could not be started. Amazon's European EC2 Service was affected for as long as twelve hours. Some Microsoft cloud services were knocked out as well (source).

I am a huge proponent of the cloud; however, I believe reliability can and should improve. As a frequent speaker and panelist at cloud-related events, I find that many in the audience are not convinced that the cloud is reliable enough to meet the needs of mission-critical applications. Outages like this don’t help. However, I am aware of several successful implementations of robust, outage-resistant cloud deployments that simply have not gotten any attention because the clients are not motivated to share how they did it with their competitors. Some of these early adopters took risks and made large investments when the mainstream would not, and they feel they deserve some advantage while they can get it. Naturally enough I think ZeroNines has the right solution, but read on for now.

Background: Amazon as a major cloud provider

Amazon EC2 is the Amazon Elastic Compute Cloud (source). It provides thousands of online service providers and software developers easy access to cloud computing capacity that is variable in size. Customers pay only for what they use. Their customers include Netflix (streaming movies and TV shows), Instagram (photo sharing), Reddit (social networking for sharing news), and Foursquare (location-based social networking).

The Problem: Something's rotten in the state of Virginia

I have not found a clear statement yet that describes the exact cause of the August 8 outage, but PCMag.com says that it "closely mirrors a similar cloud outage Amazon suffered in April" (source). It also happened in the same Virginia data center. The April 2011 outage "happened after Amazon network traffic was 'executed incorrectly.' Instead of shifting to another router, traffic went to a lower-capacity network, taking down servers in Northern Virginia." (source). So Amazon loses points for allowing the same problem to happen twice in the same place, but wins a few back for apparently being ready this time and containing the August 8 outage to minutes rather than days.

The Cost: Revenue and reputation

As always with these outages there is talk of the provider compensating its customers through waived fees and such. Mark that against Amazon's balance sheet. Customers no doubt lost business, and you can mark that against their balance sheets. Reliability issues will chase away customers who don't want to risk their own revenue with a service notorious for crashing. But if the cloud nonetheless offers the best business model, what do these customers do? Press for lower fees and more favorable service level agreements for one.

The Solution: Prevention, not recovery

If you're an actual or potential cloud user (with any provider), Always Available™ from ZeroNines can protect your existing systems without changing providers, hardware, operating systems, or applications. If there's a disaster in any part of your system, all your networked transactions and applications continue functioning as normal on the other network nodes. Our CloudNines™ application can protect your cloud-based infrastructure, VirtualNines™ can protect virtualized environments on your own machines, and EnterpriseNines™ can add Always Available protection to any other network infrastructure. You can mix and match so all these can interoperate seamlessly. For businesses of any size, the result is uptime of virtually 100% regardless of the disasters that may strike any individual node in the Always Available array.

The cloud providers themselves could use the same CloudNines product to protect their systems, virtually eliminating downtime and avoiding headlines like Amazon's. We are currently developing and monitoring on Amazon and other cloud platforms. Our technology is certified for Windows Server^® 2008, compatible with Windows Server^® 2008 Hyper-V™ and Hyper-V™ Server, and certified as VMWare^® ready.

Visit the ZeroNines.com website to find out more about how our disaster-proof architecture can protect businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Most Businesses Don’t Know what Downtime Costs Them

I just discovered the results of a survey about the need for application availability among businesses [source]. The survey was conducted by ITIC and Stratus Technologies. Results were released in April 2009. It basically sought to find out how much application uptime businesses think they need, and what they intend to do about it.

The survey found that overall, IT executives are aware that the need has grown for high-availability applications and the infrastructure to support them. But budgets are too low to support them, and most companies do not know what their downtime is costing them. This makes it difficult for these same executives to make a budgetary case for implementing high uptime solutions.

Downtime is a business killer. As an example, consider that of the 350 companies in the World Trade Center before the 1993 truck bombing, 150 were out of business a year later because of the disruption. [source: Gartner/RagingWire report cited in “Without the wires,” Fabio Campagna, Disaster Recovery Journal, Winter 2002].

The big lesson here: there is a significant competitive advantage for investing in uptime.

Here are some key facts from the survey, and my thoughts about them.

1) “Two out of five businesses – 40% – report that their major business applications require higher availability rates than they did two or three years ago. However an overwhelming 81% are unable to quantify the cost of downtime and only a small 5% minority of businesses are willing to spend whatever it takes to guarantee the highest levels of application availability 99.99% and above.”

Clearly, the field is wide open for companies to pull ahead if they go for four or five nines of uptime (or more), particularly those who serve vital and highly regulated sectors such as financial, healthcare, defense, data hosting, and so forth. A company that falls out of compliance with strict regulations like Sarbanes-Oxley or HIPAA can be driven to the brink by fines, the costs of regaining compliance, and lost business.

2) “The survey results uncovered many “disconnects” between the levels of application reliability that corporate enterprises profess to need and the availability rates their systems and applications actually deliver.” In other words, businesses are not getting the uptime they require, whether it is to meet SLAs or simply conduct everyday business.

In reality, the uptime that company leaders “profess to need” is probably insufficient. Considering that a downtime event of only a few seconds can cause a cascading failure in applications and databases, they probably need uptime of practically 100% in order to avoid a bigger disaster. Once that first domino falls, maybe you can grab it and stand it back up but all the rest are already falling. The damage is done.

3) “Some 41% said they would be satisfied with conventional 99% to 99.9% (the equivalent of two or three nines) availability for their most critical applications.”

I can’t imagine a company being without its “most critical application” for between 8+ hours (for 99% uptime) and four full days (99.9%). Companies have gone out of business after downtime of less than that. I can’t help but believe that the executives who answered this question like that are somehow out of touch with the realities of their environment. Maybe they are in industries where expectations are really low. But can you think of a bank or stock brokerage or hospital where one- or two-day outages a couple times a year are the norm? I can’t. And that is probably because such companies cease to exist.

Contrast that with this: “An overwhelming 81% of survey respondents said the number of applications that demand high availability has increased in the past two-to-three years.” High availability is typically considered to be four nines (99.99% availability and above) or less than 53 minutes of downtime per year. Yet 41% of respondents say they would be satisfied with only two or three nines? Astounding.

The Disaster of Disaster Recovery

IT executives typically prepare for downtime by implementing some variation of the backup/failover paradigm, even though most are aware it is unlikely to work. I invite you to read the ZeroNines whitepaper “The Disaster of Disaster Recovery” (available on the ZeroNines.com website) which looks at the causes of downtime and explores the shortcomings of the predominant failover disaster recovery technique. It also discusses the ZeroNines alternative, which can bring uptime beyond any measure of “nines” to virtually 100%.

ZeroNines Technology, Inc. is not affiliated with ITIC, the Information Technology Intelligence Corp. or with Stratus Technologies.

Visit the ZeroNines.com website to find out more about how our disaster-proof architecture can protect businesses (and government agencies) of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Enabling Cloud Confidence

A week ago I wrote about the Sidekick disaster and how events like that just keep doubts growing, pushing the wholesale adoption of the Cloud further away. This doubt has made it into the mainstream media, where it will taint the opinions of potential cloud users, both consumer and commercial. We at ZeroNines think we have the solution that will enable the cloud to perform as it needs to.

The core problem with outages is not the existence of hazards that can damage servers and knock elements of a network (cloud or otherwise) offline. Storms, fires, and equipment failure will always happen and there is no way to eliminate them. The real problem is the reliance of cloud providers on obsolete failover-based recovery paradigms that simply can’t maintain continuity when disaster does strike.

L.A. Times columnist David Sarno perfectly sums up the cloud’s tenuous situation in his October 18 article “Still hazy on cloud computers' security” [source]. “A series of incidents involving cloud computing over the last several months has poked holes in the hype bubble, raising questions about the cloud's dependability -- and whether it's ready for use by a broader group of workers and businesses.” He is right on target.

Meeting the Need to Fortify

As Sarno puts it, “As e-mail, word processing and data storage continue to move from users' computers to the Web, companies must fortify their servers from a variety of potential disasters -- natural and man-made -- to help ensure that the data and the applications are accessible at all times.” He quotes Google’s SEC filing:

"(Google’s) systems are vulnerable to damage or interruption from earthquakes, terrorist attacks, floods, fires, power loss, telecommunications failures, computer viruses, computer denial of service attacks" as well as sabotage and vandalism...

The good news is that today, ZeroNines' Always Available™ CloudNines™ technology can fortify servers from damage or interruption from earthquakes, terrorist attacks, floods, fires, power loss, telecommunications failures, computer denial of service attacks, as well as sabotage and vandalism. We leave the viruses to others to deal with, but we can add most types of routine maintenance, unplanned maintenance, data migrations, equipment upgrades, software upgrades, and a number of other potential causes of downtime.

Forget Failover

The IT world fatalistically believes that downtime is inevitable, and is something to be lived with and minimized if you’re fortunate. This view predominates because until now the only disaster recovery solution available has been the flawed failover paradigm, which everyone in IT knows can be a disaster unto itself. During a crisis or failover event, cutover can cause additional problems, downtime, and cascading application failures as computing switches from primary to backup systems.

But the IT world has it wrong. Disasters will happen and must be dealt with, but the downtime they cause can be prevented.

Always Available™ Means Virtually 100% Uptime

ZeroNines’ Always Available™ solution eliminates failover and backups, instead providing synchronous identical processing on multiple cloud nodes geographically separated by thousands of miles. If a storm wipes out your East Coast cloud, CloudNines enables processing to continue on clouds in other parts of the country and around the world. If you need to upgrade server software, you can isolate one cloud node, do your upgrade, and bring it back online once it is stable. Our technology has journaling and updating features to assure that all transactions are completed and that any cloud node that goes offline is brought up to the most accurate logical state once it comes back online.

CloudNines can push application availability beyond the industry-accepted standard of 99.999% (five nines) to virtually 100%. In our ongoing test case, the ZeroNines MyFailSafe environment has never experienced any downtime at all, for any reason. It went live in July 2004, and had individual network nodes knocked offline a number of times due to hurricanes, power outages, server migrations, and other causes. All applications experienced full 100% availability throughout.

Will ZeroNines eventually be recognized as a vital cloud-enabling technology? That remains to be seen but you can bet that is how we see ourselves. If you want to find out how we can make the cloud a viable option for you, let me know.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Sidekick: Two Disasters for the Price of One Really Big One

Already being called one of the largest data failures in recent memory, October’s Sidekick disaster was actually two disasters rolled into one. First, the cloud-based service suffered an outage which stranded thousands of users. Second, the backup/storage system failed and erased the personal data of thousands of users. Every failure like this leads to a round of hand-wringing over the cloud, and this one is no different. It underscores the need for a far more robust cloud architecture, where a failure in one area is truly isolated from the rest of the system and can’t cause an outage.

Background: Sidekick and the Cloud

The Sidekick mobile device is developed by Microsoft subsidiary Danger and is sold and serviced by T-Mobile. It holds a special place in the hearts and hands of a select group of users because its QWERTY keyboard promises ease of use and its cloud-based data storage gives it the appearance of real go-anywhere, do-it-anytime utility. Unlike other hand-helds like the iPhone and Blackberry, the Sidekick backs up personal data to cloud-based storage at Microsoft and not to your computer’s hard drive. And there’s the seed of the trouble.

The problem: Hardware Failure Leads to Database Failure

It seems that beginning at about 1:30 AM on Friday October 2 [source], a “hardware failure… took out both the primary and backup copies of the database that contained Sidekick users' information.” [source] This apparently occurred during an upgrade to the Danger/Microsoft Storage Area Network [source]. When they discovered their Sidekicks weren’t working, many users re-set their Sidekicks (some under instructions from T-Mobile customer service) which wiped the devices’ hard drives. Combined with the back-end server failure, this led to apparent permanent data loss for anyone who tried to re-set their Sidekicks.

The cost to T-Mobile and Microsoft

This is going to cost millions. At least. T-Mobile halted sales of all Sidekicks shortly after the event and is compensating its affected users with a period of free data service [source]. There were the usual rants about users refusing to continue paying on their contracts, and news that T-Mobile was voluntarily letting anyone out of their contract who wanted out [source]. Lawsuits were filed [source]. Sarcasm and criticism runs thick online. Whatever the actual facts, this is a marketing disaster of the greatest degree for T-Mobile and Microsoft. There is no way to calculate how many of the approximately 800,000 existing sidekick customers [source] will jump ship, how many potential new customers will be lost, and what this means for Microsoft’s “Pink” project, intended follow-on to the Sidekick [source].

The Solution: A Robust Cloud

ZeroNines’ CloudNines™ product enables the cloud to function as it is supposed to, by processing every transaction simultaneously and equally on multiple cloud-based network nodes in an Always Available™ configuration. In the Sidekick disaster, CloudNines would simply have cut off the node with the hardware failure. All processing would have continued on other geographically separated nodes that were running identical active instances of the affected applications and databases. The failure would have been contained. There would have been no service downtime, and no need for ill-advised attempts to re-boot individual Sidekicks.

Not only would the Sidekick applications have continued operation, but the databases would too. There would have been no apparent loss of customer data. After the event, one author bitingly asked “But the question remains, why wasn't there a true independent backup of the data?” [source]. ZeroNines and Always Available technology would have made this a moot point.

As of this writing, T-Mobile and Microsoft have announced that they “have recovered most, if not all, customer data” [source]. I can’t help but breathe a sigh of relief for them even though I am not a Sidekicker myself. But wouldn’t it have been far better to have avoided the problem in the first place?

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Uptime and the Cloud Crowd at CSIA

A few days ago, Jake Smith of Intel and I presented at the Colorado Software Industry Association (CSIA) monthly meeting in Denver (source). We talked about cloud computing and the elements that will determine its rate of adoption: the needs of businesses, their expectations of cloud performance, and the real-world limitations of the cloud that are currently stalling its adoption. The biggest issue is reliability, and I introduced ZeroNines’ technology as a potential solution. It was a great crowd, and their hunger for a reliable cloud was obvious.

Businesses need their applications and data to be available all the time. So far, clouds and cloud providers have not succeeded in proving that they can actually offer that. The industry needs to overcome the cloud’s downtime problems before serious business can be done on it. I believe the Big Three (Amazon, Azure, and Google) will refocus their efforts on providing highly available cloud infrastructures and market this capability accordingly.

The Cause is Academic

Of course every network is subject to threats and failures that can cause downtime, and there’s no getting away from that. It doesn’t take an earthquake to knock vital networked apps offline; some recent high-profile cloud provider outages have shown that all it takes is a failed OS upgrade. New and unexpected problems crop up every day. But the cause of an outage is really only academic for the business relying on the cloud. Service should simply continue because the business needs it to.

The scary thing is that the current disaster recovery paradigm (failover) is insufficient for protecting businesses when these things happen, and can’t be relied upon to prevent downtime or even a speedy recovery. In addition, there is an increase in catastrophic risk from poorly architected virtualized environments, and most notably in server consolidation, which is a core technology of the cloud.

The Solution is Continuity

At the CSIA meeting, we introduced the crowd to our Always Available™ technology, which maintains cloud continuity by synchronizing and protecting multiple private, public or hybrid clouds. It can mix cloud computing and physical hosting via datacenters hundreds or thousands of miles apart. The distance prevents any single regional disaster from damaging more than one data center. There is no server hierarchy, so all transactions run simultaneously and equally on all cloud and server nodes. Best of all, they update each other constantly in real time so if one goes down the others simply continue processing with no interruption to service.

To protect against an outage during an upgrade, I would postulate the following solution: Isolate one cloud or network node in an Always Available configuration and do your upgrade there, while the other nodes manage the clients’ transactions. Test the upgrade and slowly roll it out to the other nodes. If things start to go haywire, isolate the misbehaving node, solve your problems, and start the rollout again. There would be no need to risk the entire service on an untested upgrade.

Always Available works for cloud customers as well as service providers. It is provider- and platform-agnostic, so you can mix and match all you need to.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Cloud Failure: The Myth of Nines

Visit Reuven Cohen's blog

For about as long as there have been computer networks, administrators have attempted to keep these networks up and running. It seems to be a continuous battle between faulty hardware, poorly written software, unreliable connectivity and random acts of God. With the emergence of cloud computing we are now for the first time close to realizing a computing environment where we are able to focus less on keeping our applications up and more on making them run more efficiently and effectively.

In the era of cloud computing uptime guarantees and service level agreements (SLA) have started to become standard requirements for most cloud providers. Google, Amazon, and Microsoft have all started to implement some kind of SLA. They do this in an attempt to give their cloud users the confidence to utilize these systems in place of more common in house alternatives. The common goal for most of these cloud platform is to build for what I consider the myth of five nines. (Five nines meaning 99.999% availability, which translates to a total downtime of approximately five minutes and fifteen seconds per year.) The problem with five nines is it's a meaningless goal which can be manipulated to meet what ever you need it to mean.

In the case of a physical failure such as Flexiscales recent one, the hardware downtime might be small, but the time to restore from a backup might be considerably longer. A minor cloud failure could cause a cascading series of software failures causing further application outage of hours or even days for those who depended on the availability of the given cloud. Meaning your cloud may achive five nines, but your application hosted on it doesn't.

Lately it seems there are a number of people in the cloud computing community who are starting to discuss alternatives to the dreaded five nines concept and looking at ways that cloud based infrastructures could be configured / deployed in a mannor that is more proactive than reactive to disasters. There is a growing consensus that cloud based disaster recovery may very well be the "killer app" for cloud computing. To achieve this, we need to start creating reference architectures and models that assume for failure. One that doesn't need to worry when the next disaster will happen next, just that it will happen and when it does, it's going to be business as usual.

In a recent conversation with Alan Gin founder of a super secret stealth firm called Zeronines, Alan described an interesting philosophy. He said the problem with most disaster recovery plans is the recovery is reactive, it is what happens after a disaster has already harmed your business. He said on its face, this is an unsound strategy. He went on to say; That current disaster recovery architectures, which uses the synonym “failover,” is based on the cutover archetype: a system’s primary component fails, damaging operations; then failover to a secondary component is attempted to resume operations. The problem with current cutover approaches is that it views unplanned downtime as inevitable, acceptable, and so requires that business halt.

I really liked this quote from an executive from EMC, a leading computer storage equipment firm, “current failover infrastructures are failures waiting to happen.”

To be competitive in today's always connected, always available world. We need to reinvent the fundamental idea of disaster recovery. One of the major benefits to using cloud computing is that you can make these types of failover assumptions well before they happen using an emerging global toolset of cloud components. It's not a matter of if, but a matter of when, when you take into consideration that application components will fail then you can build an application that features "failure as service". One that is always available, one with Zero Nines.

Reuven CohenFounder & chief technologist for Toronto based Enomaly.