{"id":332155,"date":"2017-02-01T17:19:32","date_gmt":"2017-02-01T09:19:32","guid":{"rendered":"http:\/\/techcrunch.com\/?p=1446323"},"modified":"2017-02-01T17:19:32","modified_gmt":"2017-02-01T09:19:32","slug":"gitlab-suffers-major-backup-failure-after-data-deletion-incident","status":"publish","type":"post","link":"https:\/\/people.utm.my\/asmawisham\/gitlab-suffers-major-backup-failure-after-data-deletion-incident\/","title":{"rendered":"GitLab suffers major backup failure after data deletion incident"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2015\/09\/gitlab.png?w=738\" class=\"\"\/><\/p>\n<p>Khosla Ventures- and Y Combinator-backed open source Git repository <a target=\"_blank\" href=\"https:\/\/gitlab.com\">GitLab<\/a> is currently offline after suffering what appears to be a major\u00a0backup restoration\u00a0failure after accidentally deleting\u00a0production data.<\/p>\n<p><a href=\"https:\/\/techcrunch.com\/2017\/02\/01\/gitlab-suffers-major-backup-failure-after-data-deletion-incident\/screen-shot-2017-02-01-at-9-19-12-am\/\" rel=\"attachment wp-att-1446327\"><img loading=\"lazy\" decoding=\"async\" class=\"size-large wp-image-1446327 aligncenter\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/screen-shot-2017-02-01-at-9-19-12-am.png?w=680&#038;h=289\" alt=\"GitLab is down\" width=\"680\" height=\"289\" srcset=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/screen-shot-2017-02-01-at-9-19-12-am.png?w=680&#038;h=289 680w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/screen-shot-2017-02-01-at-9-19-12-am.png?w=150&#038;h=64 150w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/screen-shot-2017-02-01-at-9-19-12-am.png?w=300&#038;h=127 300w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/screen-shot-2017-02-01-at-9-19-12-am.png?w=768&#038;h=326 768w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/screen-shot-2017-02-01-at-9-19-12-am.png 1004w\" sizes=\"(max-width: 680px) 100vw, 680px\"\/><\/a><\/p>\n<p>Initially\u00a0the service looks\u00a0to have been suffering load time and stability problems, which led to an outage yesterday.<\/p>\n<p>But the issue\u00a0escalated into emergency database maintenance\u00a0after data was deleted accidentally \u2014 followed by\u00a0an apparent inability to restore the data from backups, according to a series of tweets from the <a target=\"_blank\" href=\"https:\/\/twitter.com\/gitlabstatus\">@GitLabStatus<\/a> account.<\/p>\n<p>The affected database includes issues and merge requests but not repositories and wikis, according to these\u00a0updates.<\/p>\n<div class=\"embed-twitter\" readability=\"6.6240875912409\">\n<blockquote class=\"twitter-tweet c4\" data-width=\"500\" readability=\"7.0656934306569\">\n<p lang=\"en\" dir=\"ltr\">we are experiencing issues with our production database and are working to recover<\/p>\n<p>\u2014 GitLab.com Status (@gitlabstatus) <a href=\"https:\/\/twitter.com\/gitlabstatus\/status\/826582847171522562\">February 1, 2017<\/a><\/p>\n<\/blockquote><\/div>\n<div class=\"embed-twitter\" readability=\"7.2727272727273\">\n<blockquote class=\"twitter-tweet c4\" data-width=\"500\" readability=\"8.1818181818182\">\n<p lang=\"en\" dir=\"ltr\">The incident affected the database (including issues and merge requests) but not the git repo&#8217;s (repositories and wikis).<\/p>\n<p>\u2014 GitLab.com Status (@gitlabstatus) <a href=\"https:\/\/twitter.com\/gitlabstatus\/status\/826662763577618432\">February 1, 2017<\/a><\/p>\n<\/blockquote><\/div>\n<p>The company\u00a0also been updating\u00a0a <a target=\"_blank\" href=\"https:\/\/docs.google.com\/document\/d\/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik\/pub\">Google Doc with info on the ongoing incident.<\/a><\/p>\n<p>A section of\u00a0the\u00a0Google Document\u00a0document detailing\u00a0\u201cproblems encountered\u201d notes that none of GitLab\u2019s\u00a0five deployed backup\/replication techniques \u201care working reliably or set up in the first place\u201d.<\/p>\n<p>Which sounds sub-optimal, to put it politely.\u00a0<a target=\"_blank\" href=\"https:\/\/www.theregister.co.uk\/2017\/02\/01\/gitlab_data_loss\/?mt=1485932441853\">The Register<\/a>\u00a0has a blunter assessment saying\u00a0the service is in \u201cmeltdown\u201d. \u201cThe world doesn\u2019t contain enough faces and palms to even begin to offer a reaction to that sentence,\u201d it writes.<\/p>\n<p>We\u2019ve reached out to GitLab for comment and will update this story with any response.<\/p>\n<p>One positive thing that can be said is the company is at least providing a high level of detail\u00a0about the incident \u2014 however cold comfort that is if you\u2019re facing the loss\u00a0of a bunch of data.<\/p>\n<p>According to The Register\u00a0a folder containing 300GB of live production data\u00a0was erroneously deleted by a GitLab sysadmin \u2014 with just\u00a04.5GB remaining by the time the delete command was cancelled.<\/p>\n<p>GitLab\u2019s most recent tweet says\u00a0the recovery of the database can be <a target=\"_blank\" href=\"http:\/\/monitor.gitlab.net\/dashboard\/db\/postgres-stats?panelId=10&#038;fullscreen&#038;from=1485871271986&#038;to=now\">tracked here<\/a>.<\/p>\n<p>The code collaboration platform, which was founded in 2014, is headquartered in San Francisco and has raised around $25.6M to date, according to <a target=\"_blank\" href=\"https:\/\/www.crunchbase.com\/organization\/gitlab-com#\/entity\">CrunchBase<\/a>\u00a0\u2014 including a <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/09\/13\/gitlab-secures-20-million-series-b\/\">$20M Series B round last\u00a0September<\/a>, led by August Capital \u2014 when founder and CEO Sid Sijbrandij said it wanted to \u201ckeep accelerating and not have to pull up\u201d. It has some 100 employees at this point, many of whom work remotely.<\/p>\n<p>GitLab competes with various other platforms for managing code for development teams \u2014 such as GitHub and\u00a0Atlassian\u00a0\u2014 but attracted investors owing to its open source twist on the code repository and collaboration tool model, which allows users to adapt the platform\u00a0to better fit their needs by building on top of it.<\/p>\n<p>As well as free community and SaaS offerings, GitLab\u00a0lets enterprise customers run its platform on their own servers, via its paid\u00a0Enterprise Edition. Customers for the paid product\u00a0include IBM, Macy\u2019s, ING, NASA and VMWare.<\/p>\n","protected":false},"excerpt":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" width=\"680\" height=\"383\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2015\/09\/gitlab.png?w=680\" class=\"attachment-large size-large wp-post-image\" alt=\"gitlab\" srcset=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2015\/09\/gitlab.png?w=680 680w, https:\/\/tctechcrunch2011.files.wordpress.com\/2015\/09\/gitlab.png?w=1360 1360w, https:\/\/tctechcrunch2011.files.wordpress.com\/2015\/09\/gitlab.png?w=150 150w, https:\/\/tctechcrunch2011.files.wordpress.com\/2015\/09\/gitlab.png?w=300 300w, https:\/\/tctechcrunch2011.files.wordpress.com\/2015\/09\/gitlab.png?w=768 768w\" sizes=\"(max-width: 680px) 100vw, 680px\" \/>&nbsp;Khosla Ventures- and Y Combinator-backed open source Git repository GitLab is currently offline after suffering what appears to be a major backup restoration failure after accidentally deleting production data. <a href=\"https:\/\/techcrunch.com\/2017\/02\/01\/gitlab-suffers-major-backup-failure-after-data-deletion-incident\/?ncid=rss\">Read More<\/a><\/p>\n<div class=\"feedflare\">\n<a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=NoO4oSkke8Q:i4kdq-Wui5M:2mJPEYqXBVI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=2mJPEYqXBVI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=NoO4oSkke8Q:i4kdq-Wui5M:7Q72WNTAKBA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=7Q72WNTAKBA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=NoO4oSkke8Q:i4kdq-Wui5M:yIl2AUoC8zA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=yIl2AUoC8zA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=NoO4oSkke8Q:i4kdq-Wui5M:-BTjWOF_DHI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=NoO4oSkke8Q:i4kdq-Wui5M:-BTjWOF_DHI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=NoO4oSkke8Q:i4kdq-Wui5M:D7DqB2pKExk\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=NoO4oSkke8Q:i4kdq-Wui5M:D7DqB2pKExk\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=NoO4oSkke8Q:i4kdq-Wui5M:qj6IDK7rITs\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=qj6IDK7rITs\" border=\"0\"><\/img><\/a>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~r\/Techcrunch\/~4\/NoO4oSkke8Q\" height=\"1\" width=\"1\" alt=\"\"\/> <\/p>\n","protected":false},"author":5817,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[25],"tags":[59,66,26],"class_list":["post-332155","post","type-post","status-publish","format-standard","hentry","category-technology","tag-media","tag-techcrunch","tag-technology"],"_links":{"self":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/332155","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/users\/5817"}],"replies":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/comments?post=332155"}],"version-history":[{"count":0,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/332155\/revisions"}],"wp:attachment":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/media?parent=332155"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/categories?post=332155"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/tags?post=332155"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}