{"id":334192,"date":"2017-02-24T11:15:45","date_gmt":"2017-02-24T03:15:45","guid":{"rendered":"http:\/\/techcrunch.com\/?p=1456642"},"modified":"2017-02-24T11:15:45","modified_gmt":"2017-02-24T03:15:45","slug":"major-cloudflare-bug-leaked-sensitive-data-from-customers-websites","status":"publish","type":"post","link":"https:\/\/people.utm.my\/asmawisham\/major-cloudflare-bug-leaked-sensitive-data-from-customers-websites\/","title":{"rendered":"Major Cloudflare bug leaked sensitive data from customers\u2019 websites"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2014\/11\/cloudflare.jpg?w=738\" class=\"\"\/><\/p>\n<p>Cloudflare revealed a serious bug in its software today that caused sensitive data like passwords, cookies, authentication tokens to spill in plaintext from its customers\u2019 websites. The <a target=\"_blank\" href=\"https:\/\/blog.cloudflare.com\/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug\/\">announcement<\/a> is a major blow for the content delivery network, which offers enhanced security and performance for more than 5 million websites.<\/p>\n<p>This could\u00a0have allowed anyone who noticed the error to collect a variety of very personal information that is typically encrypted or obscured.<\/p>\n<p>Remediation was complicated by an additional wrinkle. Some of that data was automatically cached by search engines, making it\u00a0particularly difficult to clean up\u00a0the aftermath\u00a0as Cloudflare had to approach Google, Bing, Yahoo and other search engines and ask them to manually scrub the data.<\/p>\n<p>The leak may have been active as early as Sept. 22, 2016, almost five months before a security researcher at Google\u2019s Project Zero discovered it and reported it to Cloudflare.<\/p>\n<p>However, the most severe leakage occurred between Feb. 13 and Feb. 18, when\u00a0around 1 in every 3,300,000 HTTP requests to Cloudflare sites would have caused data to be exposed. Attackers could have accessed the data in real-time, or later through search engine caches.<\/p>\n<p>Cloudflare notes in its announcement of the issue that even at its peak, data only leaked in about 0.00003% of requests. It doesn\u2019t sound like much, but Cloudflare\u2019s massive customer base includes categories like dating websites and password managers, which host particularly sensitive data.<\/p>\n<p>\u201cAt the peak, we were doing 120,000 leakages of a piece of information, for one request, per day,\u201d Cloudflare chief technology officer\u00a0 John Graham-Cumming told TechCrunch. He emphasized that not all of those leakages would have contained secret information. \u201cIt\u2019s random stuff in there because it\u2019s random memory,\u201d he said.<\/p>\n<p>The bug occurred in an HTML parser that Cloudflare uses to increase website performance \u2014 it preps sites for distribution in Google\u2019s publishing platform AMP and upgrades HTTP links to HTTPS. Three of Cloudflare\u2019s features (<a target=\"_blank\" href=\"https:\/\/support.cloudflare.com\/hc\/en-us\/articles\/200170016-What-is-Email-Address-Obfuscation-\">email obfuscation<\/a>, <a target=\"_blank\" href=\"https:\/\/support.cloudflare.com\/hc\/en-us\/articles\/200170036-What-does-Server-Side-Excludes-SSE-do-\">Server-side Excludes<\/a> and <a target=\"_blank\" href=\"https:\/\/support.cloudflare.com\/hc\/en-us\/articles\/227227647-How-do-I-use-Automatic-HTTPS-Rewrites-\">Automatic HTTPS Rewrites<\/a>) were not properly implemented with the parser, causing random chunks of data to become exposed.<\/p>\n<p>Ultimately, even Cloudflare itself was affected by the bug. \u201cOne obvious piece of information that had leaked was a private key used to secure connections between Cloudflare machines,\u201d Graham-Cumming wrote in Cloudflare\u2019s announcement. The encryption key allowed the company\u2019s own machines to communicate with each other securely, and was implemented in 2013 in response to concerns about government surveillance.<\/p>\n<p>Graham-Cumming\u00a0emphasized that Cloudflare discovered no evidence that hackers had discovered or exploited the bug, noting that Cloudflare would have seen unusual activity on their network\u00a0if an attacker were trying to access data from particular websites.<\/p>\n<p>\u201cIt was a bug in the thing that understands HTML,\u201d Graham-Cumming explained. \u201cWe understand the modifications to web pages on the fly and they pass through us. In order to do that, we have the web pages in memory on the computer. It was possible to keep going past the end of the web page into memory you shouldn\u2019t be looking at.\u201d<\/p>\n<p>Cloudflare\u2019s teams in San Francisco and London handed off shifts to one another, working\u00a0around the clock to fix the bug once it was reported. They had stopped the most severe issue within seven hours. It took six days for the company to completely repair the bug and to work with search engines to scrub the data.<\/p>\n<p>Tavis Ormandy, an engineer at Google, first noticed the bug, which he jokingly called \u201cCloudbleed\u201d in reference to the <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2014\/04\/08\/openssl-heartbleed-bug-leaves-much-of-the-internet-at-risk\/\">Heartbleed vulnerability<\/a>. He said in a <a target=\"_blank\" href=\"https:\/\/bugs.chromium.org\/p\/project-zero\/issues\/detail?id=1139\">blog post<\/a> that he encountered unexpected data during a project and wondered at first if there was a bug in his own code. Upon further testing, he realized the leak was coming from Cloudflare.<\/p>\n<p>\u201cWe fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major Cloudflare-hosted sites from other users,\u201d Ormandy wrote. \u201cThis situation was unusual, [personally-identifiable information] was actively being downloaded by crawlers and users during normal usage, they just didn\u2019t understand what they were seeing.\u201d Ormandy added that he later destroyed the samples because of the sensitive information they contained, but he posted redacted screenshots of some of the information leaked from Uber, Fitbit and OkCupid.<\/p>\n<p>Beyond the samples Ormandy collected, it\u2019s not clear what other information may have leaked. \u201cIt\u2019s very hard to say, because this information is transient,\u201d\u00a0Graham-Cumming said. But Ormandy says his samples revealed highly sensitive data.<\/p>\n<p>\u201cWe keep finding more sensitive data that we need to cleanup. I didn\u2019t realize how much of the internet was sitting behind a Cloudflare CDN until this incident,\u201d Ormandy wrote. \u201cI\u2019m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We\u2019re talking full HTTPS\u00a0requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.\u201d<\/p>\n<p>Although Cloudflare worked with Ormandy to address the issue, he contends\u00a0that the company\u2019s final blog post on the matter \u201cseverely downplays the risk to customers.\u201d Ormandy also expressed frustration that Cloudflare didn\u2019t move faster in the remediation process.<\/p>\n<p>But Graham-Cumming says it wouldn\u2019t have been possible for Cloudflare to work any more quickly than it did. Graham-Cumming also says that Ormandy called Cloudflare\u2019s disclosure \u201ccompletely acceptable\u201d when he reviewed a copy.<\/p>\n<p>\u201cThis is subject to a 90 day disclosure. We were disclosing after six days,\u201d Graham-Cumming said. \u201cHe\u2019s saying he\u2019s frustrated but I\u2019m a little bemused at why he\u2019s frustrated with six days rather than 90. We would have disclosed even earlier, but because some of this info had been cached, we thought we had a duty to clean that up before it became public. There was a danger that info would persist in search engines like Google.\u201d<\/p>\n<p>Graham-Cumming said that Cloudflare customers like Uber and OkCupid weren\u2019t directly notified of the data leaks because of the security risks involved in the situation. \u201cThere was no backdoor communication outside of Cloudflare \u2014 only with Google and other search engines,\u201d he said.<\/p>\n","protected":false},"excerpt":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" width=\"680\" height=\"453\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2014\/11\/cloudflare.jpg?w=680\" class=\"attachment-large size-large wp-post-image\" alt=\"SAN FRANCISCO, CALIF. - SEPT 29, 2010 Photo by Max Whittaker\" srcset=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2014\/11\/cloudflare.jpg?w=680 680w, https:\/\/tctechcrunch2011.files.wordpress.com\/2014\/11\/cloudflare.jpg?w=1360 1360w, https:\/\/tctechcrunch2011.files.wordpress.com\/2014\/11\/cloudflare.jpg?w=150 150w, https:\/\/tctechcrunch2011.files.wordpress.com\/2014\/11\/cloudflare.jpg?w=300 300w, https:\/\/tctechcrunch2011.files.wordpress.com\/2014\/11\/cloudflare.jpg?w=768 768w\" sizes=\"(max-width: 680px) 100vw, 680px\" \/>&nbsp;Cloudflare revealed a serious bug in its software today that caused sensitive data like passwords, cookies, authentication tokens to spill in plaintext from its customers&#8217; websites. The announcement is a major blow for the content delivery network, which offers enhanced security and performance for more than 5 million websites. This could\u00a0have allowed anyone who noticed the error to&hellip; <a href=\"https:\/\/techcrunch.com\/2017\/02\/23\/major-cloudflare-bug-leaked-sensitive-data-from-customers-websites\/?ncid=rss\">Read More<\/a><\/p>\n<div class=\"feedflare\">\n<a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=AxXht8hEPlE:xr8CEZI6NKk:2mJPEYqXBVI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=2mJPEYqXBVI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=AxXht8hEPlE:xr8CEZI6NKk:7Q72WNTAKBA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=7Q72WNTAKBA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=AxXht8hEPlE:xr8CEZI6NKk:yIl2AUoC8zA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=yIl2AUoC8zA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=AxXht8hEPlE:xr8CEZI6NKk:-BTjWOF_DHI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=AxXht8hEPlE:xr8CEZI6NKk:-BTjWOF_DHI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=AxXht8hEPlE:xr8CEZI6NKk:D7DqB2pKExk\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=AxXht8hEPlE:xr8CEZI6NKk:D7DqB2pKExk\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=AxXht8hEPlE:xr8CEZI6NKk:qj6IDK7rITs\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=qj6IDK7rITs\" border=\"0\"><\/img><\/a>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~r\/Techcrunch\/~4\/AxXht8hEPlE\" height=\"1\" width=\"1\" alt=\"\"\/> <\/p>\n","protected":false},"author":5817,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[25],"tags":[59,66,26],"class_list":["post-334192","post","type-post","status-publish","format-standard","hentry","category-technology","tag-media","tag-techcrunch","tag-technology"],"_links":{"self":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/334192","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/users\/5817"}],"replies":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/comments?post=334192"}],"version-history":[{"count":0,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/334192\/revisions"}],"wp:attachment":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/media?parent=334192"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/categories?post=334192"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/tags?post=334192"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}