{"id":334810,"date":"2017-03-03T02:53:52","date_gmt":"2017-03-02T18:53:52","guid":{"rendered":"http:\/\/techcrunch.com\/?p=1459781"},"modified":"2017-03-03T02:53:52","modified_gmt":"2017-03-02T18:53:52","slug":"aws-cloudsplains-what-happened-to-s3-storage-on-monday","status":"publish","type":"post","link":"https:\/\/people.utm.my\/asmawisham\/aws-cloudsplains-what-happened-to-s3-storage-on-monday\/","title":{"rendered":"AWS cloudsplains what happened to S3 storage on Monday"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/01\/getty-cloud-computing-concept-image.jpg?w=738\" class=\"\"\/><\/p>\n<p>AWS took a lot of heat <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2017\/03\/01\/the-day-amazon-s3-storage-stood-still\/\">when its S3 storage component went down for several hours<\/a> on Monday, and rightly so, but today <a target=\"_blank\" href=\"https:\/\/aws.amazon.com\/message\/41926\/\">they published a post-mortem<\/a>\u00a0explaining exactly what happened complete with technical details and how they plan to prevent a similar event from occurring again in the future.<\/p>\n<p>At the core of the problem was unsurprisingly human error. Some poor engineer, we\u2019ll call him Joe, was tasked with entering a command to shut down some storage sub-systems. On a typical day this doesn\u2019t cause any issue whatsoever. It\u2019s a routine kind of task, but on Monday something went terribly wrong.<\/p>\n<p>Joe was an authorized user, and he\u00a0entered the command according to procedure based on what Amazon calls \u201can established playbook.\u201d The problem was that Joe was supposed to issue a command to take down a small number of servers on an S3 sub-system, but he made a mistake, and instead of taking down just that small set of servers, Joe took down a much larger set.<\/p>\n<p>In layman\u2019s terms, that\u2019s when all hell broke loose.<\/p>\n<p>Amazon explains it much more technically, but suffice to say that error had a cascading impact on the S3 storage in the Northern Virginia datacenter. To make a long story short, Joe\u2019s error took down some crucial underlying sub-systems, which removed a significant amount of storage capacity, which caused the systems to restart. As this happened, S3 couldn\u2019t service requests, which caused even AWS\u2019s own dashboard to go down (which is, you know, kind of embarrassing).<\/p>\n<p>By now, the outside world started to feel the impact and your favorite websites, apps and cloud services were beginning to behave in a wonky fashion.<\/p>\n<p>As the afternoon wore on, the company was working feverishly to get the service\u00a0back online, but the size of the systems was working against them. When the system shut down, something that AWS says it hasn\u2019t had to do in many years, it became a victim of its own success. S3 capacity had grown to such an extent in the affected datacenter that when they restarted, running all of the safety checks and validating the integrity of the underlying metadata took a mite\u00a0bit longer than they expected.<\/p>\n<p>To reduce the prospect of a similar human error in the future, the company is making some changes. In their words, \u201cWe have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level.\u201d That should prevent someone like Joe from making a similar mistake in the future.<\/p>\n<p>In addition, AWS is looking at ways to break down those S3 sub-systems, which were core to the problem, into much smaller pieces or cells, as they call them, something they have tried to do in the past. Obviously, the sub-systems proved too large to recover quickly (or at least quickly enough).<\/p>\n<p>They close with an apology and a promise to do better. In the end, it was a combination of factors that caused the issue, starting with a human error and then cascading across systems that hadn\u2019t been designed to deal with an error of this magnitude.<\/p>\n<p><small>Featured Image: <a target=\"_blank\" href=\"http:\/\/www.gettyimages.com\/search\/photographer?photographer=Colin+Anderson&#038;excludenudity=true&#038;family=creative&#038;page=1&#038;phrase=cloud+computing&#038;sort=best\">Colin Anderson<\/a>\/Getty Images<\/small><\/p>\n","protected":false},"excerpt":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" width=\"680\" height=\"546\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/01\/getty-cloud-computing-concept-image.jpg?w=680\" class=\"attachment-large size-large wp-post-image\" alt=\"Person watching light column in cloud of blocks\" srcset=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/01\/getty-cloud-computing-concept-image.jpg?w=680 680w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/01\/getty-cloud-computing-concept-image.jpg?w=1360 1360w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/01\/getty-cloud-computing-concept-image.jpg?w=150 150w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/01\/getty-cloud-computing-concept-image.jpg?w=300 300w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/01\/getty-cloud-computing-concept-image.jpg?w=768 768w\" sizes=\"(max-width: 680px) 100vw, 680px\" \/>&nbsp;AWS took a lot of heat when its S3 storage component went down for several hours on Monday, and rightly so, but today they published a post-mortem\u00a0explaining exactly what happened complete with technical details and how they plan to prevent a similar event from occurring again in the future. At the core of the problem was unsurprisingly human error. Some poor engineer, we&#8217;ll call him&hellip; <a href=\"https:\/\/techcrunch.com\/2017\/03\/02\/aws-cloudsplains-what-happend-to-s3-storage-on-monday\/?ncid=rss\">Read More<\/a><\/p>\n<div class=\"feedflare\">\n<a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=IF6FmrRMiv8:-nsYEr8k4Js:2mJPEYqXBVI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=2mJPEYqXBVI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=IF6FmrRMiv8:-nsYEr8k4Js:7Q72WNTAKBA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=7Q72WNTAKBA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=IF6FmrRMiv8:-nsYEr8k4Js:yIl2AUoC8zA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=yIl2AUoC8zA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=IF6FmrRMiv8:-nsYEr8k4Js:-BTjWOF_DHI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=IF6FmrRMiv8:-nsYEr8k4Js:-BTjWOF_DHI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=IF6FmrRMiv8:-nsYEr8k4Js:D7DqB2pKExk\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=IF6FmrRMiv8:-nsYEr8k4Js:D7DqB2pKExk\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=IF6FmrRMiv8:-nsYEr8k4Js:qj6IDK7rITs\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=qj6IDK7rITs\" border=\"0\"><\/img><\/a>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~r\/Techcrunch\/~4\/IF6FmrRMiv8\" height=\"1\" width=\"1\" alt=\"\"\/> <\/p>\n","protected":false},"author":5817,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[25],"tags":[59,66,26],"class_list":["post-334810","post","type-post","status-publish","format-standard","hentry","category-technology","tag-media","tag-techcrunch","tag-technology"],"_links":{"self":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/334810","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/users\/5817"}],"replies":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/comments?post=334810"}],"version-history":[{"count":0,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/334810\/revisions"}],"wp:attachment":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/media?parent=334810"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/categories?post=334810"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/tags?post=334810"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}