Azure and Google and Twitter, Oh My!

The flying monkeys attacked in force on Thursday July 26, 2012, taking down cloud leaders Microsoft, Twitter, and Google Talk. It seems the cloud is no Yellow Brick Road after all, guiding merry executives to some imagined Oz where the sun always shines and outages never happen.

The more I talk to IT planners, the more I find they are looking at reinvesting their cloud savings into business continuity. They rightly hope to compete based upon reliability, and to protect their businesses by exchanging the extremely high and unpredictable costs of outages for the predictable and low costs of the cloud and business continuity. They've clearly got the right idea, especially when you consider the noise outages like yesterday's can make.

Microsoft Azure

• Area affected: Europe, via the Dublin datacenter and Amsterdam facility.
• Duration: About two and a half hours.
• Cause: Unspecified, but one expert suspects infrastructure troubles.
• Effects: Loss of cloud service throughout Western Europe. Businesses like SoundGecko were unavailable.
• Source: WebTechInfo.com

"In Azure’s case on Thursday, the constant availability of power and lack of a software culprit, such as the Feb. 29 one that downed several services, points to 'more of an infrastructure issue' where some undetected single point of failure in a network switch or other device temporarily disrupted availability, said Chris Leigh-Currill, CTO of Ospero, a supplier of private cloud services in Europe." (source)

Google Talk

• Area affected: Worldwide (source).
• Duration: About five hours.
• Cause: Unspecified, but one expert suspects a bad hardware or software upgrade.
• Effects: System unusable, granting access but providing only error messages.
• Source: TechNewsWorld.com

"Outages like this 'often happen as the result of a hardware or software upgrade that wasn't properly tested before installation,' Rob Enderle, principal analyst at the Enderle Group, told TechNewsWorld" (source).

Twitter

• Area affected: Worldwide.
• Duration: About an hour and possibly more in some areas.
• Cause: Datacenter failure and a failed failover.
• Effects: "Users around the world got zilch from us."
• Source: CNN

"Twitter's vice president of engineering, Mazan Rawashdeh… blamed the outage on the concurrent failure of two data centers. When one fails, a parallel system is designed to take over -- but in this case, the second system also failed, he said" (source).

Am I Repeating Myself? Am I Repeating Myself?

My last blog post was titled "A Flurry of July Outages – And All of them Preventable". On Thursday we had another flurry, and all of them in the cloud. And I have to say again, with caution, that these were preventable. I am cautious because we don't know for sure if Mr. Leigh-Currill and Mr. Enderle are correct in their assumptions about the Azure and Google service interruptions, but if they are correct then these outages were almost certainly preventable.

And the Twitter outage was clearly a case of failover failing us once again. That's what it is when "a parallel system that is designed to take over" fails to do so when the primary goes down. It's the same story we see over and over again.

After all these years in the industry I am still amazed that the biggest tech names in the world continue to rely on the ancient failover paradigm. The leaders of these organizations are trusting their reputations, their revenue, their shareholders' profits, their customers' businesses, and potentially peoples' lives to a disaster recovery process that quite likely won't work.

What Some Companies are Already Doing About It

Earlier in this post I mentioned the companies that are reinvesting cloud savings into reliability. These are typically smaller, more agile companies. They'll be able to take on the tech behemoths based on reliability, because they are thinking beyond mere cost savings and efficiency. They see the cloud as a means of creating their own "failsafe" hosting paradigms.

As most readers of this blog know, Always Available™ technology from ZeroNines replaces failover and backup-based recovery. It is already cloud-friendly. The Twitter outage is exactly the kind of thing we prevent. Companies hosting an Always Available array on Azure would have had virtually no risk of downtime, because all network transactions would have continued processing on other nodes. Always Available prevents data disasters before they happen, whether a power supply fails, or software gets corrupted, or a tornado picks up your Kansas datacenter and relocates it to Munchkinland, or someone melts your servers with a bucket of water, or the flying monkeys carry off the last of your support staff. Why clean up after an expensive disaster if you can prevent it in the first place?

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

BAE, Microsoft, the Cloud, and Planning for When it All Goes Horribly Wrong

"If it fails in Ireland, it goes to Holland. But what if it fails in Holland as well?"
Paraphrase of Charles Newhouse, BAE [source]

Cloud news circuits have been abuzz the last few days over BAE rejecting Microsoft's Office 365 cloud solution because of the Patriot Act. This is the highest-profile rejection of a cloud offering I have seen. I am shocked and dismayed that after all the advancements that have improved continuity in the cloud, the network architectures our cloud service providers are offering are still in the stone age. They're still trying to use failover and pass it off as advanced and reliable. I can only assume that if given a 787 they would try to fly it off a dirt landing strip.

When you read the articles closely, it is clear that the big issue for BAE was data sovereignty. How does one retain control of data during a network disaster, and where does it go when your service provider has to failover from the primary network node to the backup? To quote Charles Newhouse, head of strategy and design at British defense contractor BAE,

"We had these wonderful conversations with Microsoft where we were going to adopt Office 365 for some of our unrestricted stuff, and it was all going to be brilliant. I went back and spoke to the lawyers and said, '[The data center is in] Ireland and then if it fails in Ireland go to Holland.' And the lawyers said 'What happen[s] if they lose Holland as well?'" [source]

And earlier in the same article he described the user experience during a cloud outage:

"A number of high profile outages that users have suffered recently demonstrated just how little control you actually have. When it all goes horribly wrong, you just sit there and hope it is going to get better. There's nothing tangibly you can do to assist" [source].

It's About More than Just the Patriot Act

The big focus in these articles is the Patriot Act. BAE lawyers forbade the use of Office 365 and the Microsoft public cloud because as a U.S. company, Microsoft could be required to turn BAE data over to the U.S. government under terms of the Patriot Act [source].

It is true that the Patriot Act can require cloud service providers like Microsoft (and Amazon, Google, and others) to give the U.S. government the data on their servers, even if those servers are housed outside the United States [source]. Newhouse also said that "the geo-location of that data and who has access to that data is the number one killer for adopting to the public cloud at the moment" [source].

But European governments are already moving to eliminate this loophole. As explained in November on ZDNet.com, a new European directive "will not only modernize the data protection laws, but will also counteract the effects of the Patriot Act in Europe" [source]. Sounds to me like Microsoft's jurisdictional problems will be solved for them. And failing that there is probably some creative and legal business restructuring that would do the trick.

It's Really about Failover and its Shortcomings

So if European law will provide data sovereignty from a legal standpoint, why reject the Microsoft cloud? It all comes back to "when things go horribly wrong."

When Newhouse describes the Ireland-to-Holland scenario, he is clearly talking about Microsoft failing-over from their Ireland datacenter to their Holland datacenter. I find it hard to believe that Microsoft thinks the outdated and flawed failover model is suitable for a leading cloud offering. Office 365 and their customers deserve better.

Apparently BAE agrees. It put its foot down and refused to play because the reality does not match the promise.

Failovers often fail, causing the downtime they were supposed to prevent. If the secondary site fails to start up properly (which is very common) or suffers an outage of its own, the business is either a) still offline or b) failed over to yet another location. The customer quickly loses control, network transactions get lost, and their data goes… where? Another server in Europe? Part of an American cloud? How many locations is Microsoft prepared to failover to, and where are they? And with the cloud these issues loom even larger because there is no particular machine that houses the data.

The Solution: Cloud and Data Reliability without Failover

ZeroNines offers two potential scenarios that will solve this problem:

1) Prevent downtime on Protect the cloud provider's systems from downtime, offering a far more reliable cloud.

2) Protect the business' systems from a cloud provider's downtime.

Our Always Available technology is designed to provide data and application uptime well in excess of five nines. ZenVault Medical has been running in the cloud on Always Available for about 14 months with true 100% uptime. Always Available runs multiple network and cloud nodes in distant geographical areas. All servers and nodes are hot, and all applications are active. If one fails, the others continue processing as before, with no interruption to the business or the user experience. There is no failover, and thus no chance for outages caused by a failed failover.

So if Microsoft were to adopt our Always Available technology, a storm like the one that knocked out their data center in Ireland this past August would not affect service. The Ireland node might go down, but all network activities would proceed as usual on other cloud data centers in Holland, Italy, or wherever they have set them up. Users would never know it.

If BAE adopted Always Available, they could bring their Microsoft cloud node into an Always Available array with other cloud nodes or data centers of their own choosing. A failure in one simply means that business proceeds on the others.

The business or the service provider can determine which nodes are brought into the array. BAE could choose to use only European cloud nodes to maintain data sovereignty.

ZeroNines' Always Available technology is built precisely for the moment "when it all goes horribly wrong." The difference is that with ZeroNines, it won't mean downtime.

Visit the ZeroNines website to find out more about how our disaster-proof architecture protects businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines

Announcing ZenVault Medical: Your Cloud-Based, Secure, Encrypted Personal Health Record

I had a heart attack back in 2008. I was lucky. My local emergency room facility and the intensive care unit hospital that I was transferred to happened to share my medical records in electronic format. But only about 10% of U.S. hospitals use electronic records so if this had happened away from home I probably would have died because no other doctor or hospital would have known about my pre-existing medical conditions.

It was suddenly very easy for me to see the need for a system that would allow consumers to take their medical records with them wherever they go. Not only for emergencies but for everyday reference. Some quick Googling revealed Personal Health Record (PHR) solutions from Microsoft (HealthVault), Google (Google Health) and a large number of others, but consumer adoption was low. I also discovered that the Electronic Medical Records (EMRs) used by hospitals and doctors were no solution because they are inaccessible to consumers and practitioners outside the system.

I enlisted the help of my personal doctors, friends and classmates who work in the healthcare field as well as other technologists who are consulting to large medical organizations around the country. All told, we have consulted with 36 experts who freely gave us their opinions about the issues surrounding EMRs and how a comprehensive PHR should be designed in order to deliver high value to consumers while potentially saving lives. I summarize the issues in BOLD and describe how we address them.

So today we at ZeroNines introduced ZenVault Medical (www.zenvault.com/medical), a Cloud-based, private, encrypted, online PHR for consumers that you can access through a computer or mobile device. In addition to helping people with their medical care, it’s a great example of how the Cloud and other cutting-edge technologies can come together to create a unique and valuable consumer product.

Background: The Need for Digital Medical Records

If you’re like most people, your medical records are scattered among a number of doctors and they are hard to get to. The Obama administration wants the country to convert to Electronic Medical Records. The goal is to improve healthcare and cut costs by making an individual’s collection of medical records available electronically at any hospital or doctor’s office, cutting down on paper volume, saving time, and increasing accessibility particularly in emergencies. This truly needs to happen – my own experience proves that – but the issue is how.

The Problem: Security, Privacy, and Reliability

Questions surrounding security and privacy make many citizens and consumer advocates reluctant to jump on board. Will such a system be run by the government or by business? Who will have access? Will sensitive personal information about illnesses, prescriptions, and treatments be turned over to insurance companies? To marketers? To employers? Can any body of law successfully regulate how such highly personal information is handled and protected, enabling it to benefit the individual yet keeping it out of the hands of those who would profit by violating privacy? Is it even the government’s place to get involved with personal medical records? And what technology is secure enough to handle all this?

Security: Any medical records system needs to keep hackers at bay. Well-publicized data breaches with Microsoft and Google call into question their ability to protect medical privacy. Frankly, I decided to subscribe to one of these systems before we came up with ZenVault, but was concerned with who might be accessing my records and selling it to insurance companies and marketing firms.

Privacy: Many companies offering free digital medical records turn around and sell customer data to pharmaceutical and insurance companies. And a September 16 2010 article in the Wall Street Journal described a data breach wherein a Google engineer broke the company’s privacy policies by accessing private customer information.

Reliability:If anything needs 100% uptime, it’s medical applications. Take a look at some of the high-profile downtime events discussed in the rest of this blog and then imagine the cost in lives and well-being if they had affected hospital emergency rooms.

The Solution: Customer Control of a Safe, Secure, and Always Available™ Personal Health Record

Simply putting control of the health record in the hands of the individual consumer or patient addresses the bulk of these concerns. If no one can read the record but the customer, that’s most of the battle won. So what is the difference between ZenVault Medical and other consumer-facing PHRs like Google Health and HealthVault?

Security: ZenVault encrypts stored records with a patent-pending variant of the NSA-approved encryption protocols that protect top-secret information. ZenVault does not employ a “key ring” that stores customer encryption keys which means there is no copy available for anyone to find and rummage through your data. The customer creates his or her own unique encryption key so only they can access and edit their private medical records. SSL-secured sessions protect data in transit from computers, smartphones, and tablets.

Privacy: ZenVault never shares information. Period. We don’t sell it, rent it, or give it away, not even in a “sanitized” format like some admit to doing. We charge consumers for our service and our business model is based on customer trust. If they don’t trust us we lose. In fact, our encryption system prevents even our own engineers and administrators from reading patient data, so we couldn’t sell it even if we wanted to. How’s that for a guarantee?

Reliability: ZenVault uses ZeroNines' Always Available™ technology designed to protect the world's most sensitive financial and military computer systems. There is virtually no "downtime" or data loss with ZenVault. A Cloud-based infrastructure helps keep costs down, ensures scalability, and supports universal accessibility. Use of Always Available allays any concerns over Cloud reliability. In fact, we intend to use ZenVault as an example of a highly reliable, high-usage application deployed in the Cloud. Read more about Always Available on the ZeroNines.com website ZeroNines.com website.

Convenience: Users can update or read their records anywhere they have Internet access. They can send their records to any doctor with just a few clicks using a secure message system. Have you ever wasted time at a doctor appointment filling out a clipboard full of medical history forms? Use ZenVault to send them your PHR instead! Doctors can send patients their records, lab results, and x-rays with equal ease.

Affordable: A free account is available, offering a basic PHR with full security, encryption, and privacy protection. A premium account adds advanced features for a small monthly charge.

Secure Emergency Room Access: ZenVault offers emergency rooms their own accounts with their own special encryption keys. They get controlled access to six key fields in a patient’s record such as history of heart disease, drug sensitivities, and emergency contact information. This gives them the basic information they need to save a life and contact loved ones yet protects the majority of personal information until the patient or their family elects to release it.

Take Your Personal Health Record with You

If you have Internet access, you can use ZenVault. I hope none of you ever has a medical emergency like the one that sent me to the hospital two years ago. But if you do, ZenVault could save your life by putting the needed information in the right place, at the right time. I have no doubt that one day a universal health record database will be a reality, but until then you can have all the benefits while keeping control yourself. Try it out and let me know what you think:  www.zenvault.com/medical.

Visit the ZeroNines.com website to find out more about how our disaster-proof architecture can protect businesses of any description from downtime.

Alan Gin – Founder & CEO, ZeroNines