{"id":333202,"date":"2017-02-14T04:49:04","date_gmt":"2017-02-13T20:49:04","guid":{"rendered":"http:\/\/techcrunch.com\/?p=1451925"},"modified":"2017-02-14T04:49:04","modified_gmt":"2017-02-13T20:49:04","slug":"yahoo-supercharges-tensorflow-with-apache-spark","status":"publish","type":"post","link":"https:\/\/people.utm.my\/asmawisham\/yahoo-supercharges-tensorflow-with-apache-spark\/","title":{"rendered":"Yahoo supercharges TensorFlow with Apache Spark"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/gettyimages-474451428.jpg?w=738\" class=\"\"\/><\/p>\n<p>Yahoo, model Apache Spark citizen and developer\u00a0of\u00a0CaffeOnSpark, which made it easier for developers building deep learning models in Caffe to scale with parallel processing, is open sourcing a new project called <a target=\"_blank\" href=\"http:\/\/yahoohadoop.tumblr.com\/\">TensorFlowOnSpark<\/a>. The pairing of Spark and TensorFlow should make the deep learning framework more attractive to developers who are creating models that need to run on large computing clusters.<\/p>\n<p>For those that zoned out during\u00a0the big-data boom, Apache Spark is an open source framework designed to\u00a0increase the efficiency of parallel computing. Following in the steps of tools like Hadoop, Spark made it possible for companies <a target=\"_blank\" href=\"https:\/\/spark-summit.org\/east-2017\/events\/netflixs-recommendation-ml-pipeline-using-apache-spark\/\">like Netflix<\/a> to process huge amounts of user data to offer up recommendations at scale.<\/p>\n<p>Machine learning frameworks like Google\u2019s TensorFlow and Caffe help people create deep learning models without the rigorous skill-set of a machine learning\u00a0specialist. The open source libraries exist at a higher level of abstraction that enable developers to create models without getting lost in the weeds reinventing the wheel.<\/p>\n<p>Naturally, Spark and machine learning go hand-in-hand. Deep learning in particular leans heavily on large amounts of compute as a crutch. Yahoo created\u00a0CaffeOnSpark for its own use. But mixing Caffe and Spark only benefited\u00a0a portion of the machine learning community. TensorFlow remains the most popular framework, so Yahoo decided to bring the same pairing to it and hopefully pick up some developer respect along the way.<\/p>\n<p>Yahoo explored existing tools like SparkNet and TensorFrame to bridge the gap between TensorFlow and Spark, but ultimately decided that they could do better. The result is a tool that enables developers to quickly modify their existing TensorFlow programs to work with TensorFlowOnSpark.<\/p>\n<p>Deep learning frameworks are a strangely\u00a0tribal thing among developers. <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/09\/28\/skymind-raises-3m-to-bring-its-java-deep-learning-library-to-the-masses\/\">Skymind\u2019s\u00a0Deeplearning4j<\/a> is an open source framework written for Java with Spark integrations, \u00a0but the library ultimately falls in sixth place for popularity. MXNet on the other hand is known for its ability to scale across a multitude of GPUs. It\u2019s just that ability that drew the favor of Amazon and ultimately helped MXNet become <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2017\/01\/30\/mxnet-accepted-to-the-apache-incubator\/\">part of the Apache incubator<\/a>.<\/p>\n<p>Though\u00a0TensorFlowOnSpark has been open sourced, Yahoo has plans to continue to make improvements to the tool. If you\u2019re interested in testing it out, <a target=\"_blank\" href=\"https:\/\/github.com\/yahoo\/TensorFlowOnSpark\">you can get everything on the Yahoo GitHub.\u00a0<\/a><\/p>\n<p><small>Featured Image: luismmolina\/Getty Images<\/small><\/p>\n","protected":false},"excerpt":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" width=\"680\" height=\"326\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/gettyimages-474451428.jpg?w=680\" class=\"attachment-large size-large wp-post-image\" alt=\"Servers in Data Center\" srcset=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/gettyimages-474451428.jpg?w=680 680w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/gettyimages-474451428.jpg?w=1358 1358w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/gettyimages-474451428.jpg?w=150 150w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/gettyimages-474451428.jpg?w=300 300w, https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/02\/gettyimages-474451428.jpg?w=768 768w\" sizes=\"(max-width: 680px) 100vw, 680px\" \/>&nbsp;Yahoo, model Apache Spark citizen and developer\u00a0of\u00a0CaffeOnSpark, which made it easier for developers building deep learning models in Caffe to scale with parallel processing, is open sourcing a new project called TensorFlowOnSpark. The pairing of Spark and TensorFlow should make the deep learning framework more attractive to developers who are creating models that need to run on large&hellip; <a href=\"https:\/\/techcrunch.com\/2017\/02\/13\/yahoo-supercharges-tensorflow-with-apache-spark\/?ncid=rss\">Read More<\/a><\/p>\n<div class=\"feedflare\">\n<a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=vfIPW0ZAeFA:quiXhE-p5_c:2mJPEYqXBVI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=2mJPEYqXBVI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=vfIPW0ZAeFA:quiXhE-p5_c:7Q72WNTAKBA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=7Q72WNTAKBA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=vfIPW0ZAeFA:quiXhE-p5_c:yIl2AUoC8zA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=yIl2AUoC8zA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=vfIPW0ZAeFA:quiXhE-p5_c:-BTjWOF_DHI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=vfIPW0ZAeFA:quiXhE-p5_c:-BTjWOF_DHI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=vfIPW0ZAeFA:quiXhE-p5_c:D7DqB2pKExk\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=vfIPW0ZAeFA:quiXhE-p5_c:D7DqB2pKExk\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=vfIPW0ZAeFA:quiXhE-p5_c:qj6IDK7rITs\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=qj6IDK7rITs\" border=\"0\"><\/img><\/a>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~r\/Techcrunch\/~4\/vfIPW0ZAeFA\" height=\"1\" width=\"1\" alt=\"\"\/> <\/p>\n","protected":false},"author":5817,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[25],"tags":[59,66,26],"class_list":["post-333202","post","type-post","status-publish","format-standard","hentry","category-technology","tag-media","tag-techcrunch","tag-technology"],"_links":{"self":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/333202","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/users\/5817"}],"replies":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/comments?post=333202"}],"version-history":[{"count":0,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/333202\/revisions"}],"wp:attachment":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/media?parent=333202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/categories?post=333202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/tags?post=333202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}