{"id":343130,"date":"2017-06-09T01:35:02","date_gmt":"2017-06-08T17:35:02","guid":{"rendered":"http:\/\/techcrunch.com\/?p=1501291"},"modified":"2017-06-09T01:35:02","modified_gmt":"2017-06-08T17:35:02","slug":"facebook-is-speeding-up-training-for-visual-recognition-models","status":"publish","type":"post","link":"https:\/\/people.utm.my\/asmawisham\/facebook-is-speeding-up-training-for-visual-recognition-models\/","title":{"rendered":"Facebook is speeding up training for\u00a0visual recognition models"},"content":{"rendered":"

<\/p>\n

Every minute spent training a deep learning model is a minute not doing something else and in today\u2019s fast paced world of research, that minute is worth a lot. Facebook published a paper this morning detailing its personal approach to this problem. The company says its managed to reduce the training time of a ResNet-50 deep learning model on ImageNet from 29 hours to one.<\/p>\n

Facebook managed to reduce training time so dramatically by distributing training in larger \u201cminibatches\u201d across a greater number of GPUs. In the previous benchmark case, batches of 256 images were spread across eight GPUs. But today\u2019s work involves batch sizes of 8192 images distributed across 256 GPUs.<\/p>\n

Most people don\u2019t have 256 GPUs laying around, but big tech companies and well-funded research groups do. Being able to scale training across so many GPUs to reduce training time, without a dramatic loss in accuracy, is a big deal.<\/p>\n

The team slowed down learning rates at the beginning stages of the training process to overcome some of the difficulties that made large batch sizes previously infeasible. Without getting too lost in the details, stochastic gradient descent is used to train the ResNet-50 model.<\/p>\n

One of the key variables in\u00a0stochastic gradient descent is the learning rate \u2014 the degree by which weights change during the training process. The way this variable changes as minibatch size changes is the key to optimizing effectively.<\/p>\n

Machine learning developers spend their days dealing with compromises. Greater accuracy often requires larger data sets that demand additional training time and compute resources. In this vein, it would be possible to prioritize accuracy or speed to achieve more impressive results but training a model with poor accuracy in 20 seconds isn\u2019t super valuable.<\/p>\n

Unlike most research projects, Facebook\u2019s AI Research (FAIR) and Applied Machine Learning (AML) teams worked side by side on increasing minibatch sizes. From here the groups plan to investigate some of the additional questions generated from today\u2019s work.<\/p>\n

\u201cThis work throws out more questions than it answers,\u201d said Pieter Noordhuis, a member of Facebook\u2019s AML team. \u201cThere\u2019s a tipping point beyond 8,000 images where error rates go up again and we don\u2019t know why.\u201d<\/p>\n

Facebook used Caffe2, its open source deep learning framework, and its Big Basin GPU servers for this experiment. Additional information from Facebook is available here<\/a> if you want to dig more deeply into the details.<\/p>\n

Featured Image: Toast and Jam Films<\/small><\/p>\n","protected":false},"excerpt":{"rendered":"

\"\" Every minute spent training a deep learning model is a minute not doing something else and in today’s fast paced world of research, that minute is worth a lot. Facebook published a paper this morning detailing its personal approach to this problem. The company says its managed to reduce the training time of a ResNet-50 deep learning model on ImageNet from 29 hours to one. Facebook managed… Read More<\/a><\/p>\n

\n<\/img><\/a> <\/img><\/a> <\/img><\/a> <\/img><\/a> <\/img><\/a> <\/img><\/a>\n<\/div>\n

\"\"\/ <\/p>\n","protected":false},"author":5817,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[25],"tags":[59,66,26],"class_list":["post-343130","post","type-post","status-publish","format-standard","hentry","category-technology","tag-media","tag-techcrunch","tag-technology"],"_links":{"self":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/343130","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/users\/5817"}],"replies":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/comments?post=343130"}],"version-history":[{"count":0,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/343130\/revisions"}],"wp:attachment":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/media?parent=343130"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/categories?post=343130"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/tags?post=343130"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}