![](\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/03\/brain-ai-newsletter.jpg?w=640\")
<\/p>\n<\/p>\n
Ophir Tanz<\/a> is the CEO of GumGum<\/a>, an artificial intelligence company with particular expertise in computer vision. GumGum applies its capabilities to a variety of industries, from advertising to professional sports across the globe. Ophir holds a B.S. and a M.S. from Carnegie Mellon University and currently lives in Los Angeles.<\/p>\n Cambron Carter<\/span> Contributor<\/span><\/p>\n Cambron Carter<\/a> leads the image technology team at GumGum<\/a>, where he designs computer vision and machine learning solutions for a wide variety of applications. Cambron holds B.S. degrees in physics and electrical engineering and an M.Eng. in electrical engineering from the University of Louisville.<\/p>\n<\/div>\n If you\u2019ve dug into any articles on artificial intelligence, you\u2019ve almost certainly run into the term \u201cneural network.\u201d Modeled loosely on the human brain, artificial neural networks enable computers to learn from being fed data.<\/span><\/p>\n The efficacy of this powerful branch of machine learning, more than anything else, has been responsible for ushering in a new era of artificial intelligence, ending a long-lived \u201c<\/span>AI Winter<\/span><\/a>.\u201d Simply put, the neural network may well be one of the most fundamentally disruptive technologies in existence today.<\/span><\/p>\n This<\/span> guide to neural networks<\/span>\u00a0aims to give you a conversational level of understanding of deep learning. To this end, we\u2019ll avoid delving into the math and instead rely as much as possible on analogies and animations.<\/span><\/p>\n One of the early schools of AI taught that if you load up as much information as possible into a powerful computer and give it as many directions as possible to understand that data, it ought to be able to \u201cthink.\u201d This was the idea behind chess computers like IBM\u2019s famous Deep Blue:<\/span> By exhaustively programming every possible chess move into a computer, as well as known strategies, and then giving it sufficient power, IBM programmers created a machine that, in theory, could calculate every possible move and outcome into the future and pick the sequence of subsequent moves to outplay its opponent.<\/span> This actually works, as<\/span> chess masters learned in 1997<\/span><\/a>.*<\/span><\/p>\n With this sort of computing, the machine relies on fixed rules that have been painstakingly pre-programmed by engineers \u2014 if this happens, then that happens; if this happens, do this \u2014 and so it isn\u2019t human-style flexible learning as we know it at all. It\u2019s powerful supercomputing, for sure, but not \u201cthinking\u201d per se.<\/span><\/p>\n Over the past decade, scientists have resurrected an old concept that doesn\u2019t rely on a massive encyclopedic memory bank, but instead on a simple and systematic way of analyzing input data that\u2019s loosely modeled after human thinking. Known as deep learning, or neural networks, this technology has been around since the 1940s, but because of today\u2019s exponential proliferation of data \u2014 images, videos, voice searches, browsing habits and more \u2014 along with supercharged and affordable processors, it is at last able to begin to fulfill its true potential.<\/span><\/p>\n An artificial (as opposed to human) neural network (ANN) is an algorithmic construct that enables machines to learn everything from voice commands and playlist curation to music composition and image recognition.The typical ANN consists of thousands of interconnected artificial neurons, which are stacked sequentially in rows that are known as layers, forming millions of connections. In many cases, layers are only interconnected with the layer of neurons before and after them via inputs and outputs. (This is quite different from neurons in a human brain, which are interconnected every which way.)<\/span><\/p>\n Source: GumGum<\/p>\n<\/div>\n This layered ANN is one of the main ways to go about machine learning today,<\/span> and feeding it vast amounts of labeled data enables it to learn how to interpret that data like (and sometimes better than) a human.<\/span><\/p>\n Just as when parents teach their kids to identify apples and oranges in real life, for computers too, practice makes perfect.<\/p>\n<\/blockquote>\n<\/div>\n Take, for example, image recognition, which relies on a particular type of neural network known as the convolutional neural network (CNN) \u2014 so called because it uses a mathematical process known as convolution to be able to analyze images in non-literal ways, such as identifying a partially obscured object or one that is viewable only from certain angles. (There are other types of neural networks, including recurrent neural networks and feed-forward neural networks, but these are less useful for identifying things like images, which is the example we\u2019re going to use below.)<\/span><\/p>\n So how do neural networks learn? Let\u2019s look at a very simple, yet effective, procedure called supervised learning. Here, we feed the neural network vast amounts of training data, labeled by humans so that a neural network can essentially fact-check itself as it\u2019s learning.<\/span><\/p>\n Let\u2019s say this labeled data consists of pictures of apples and oranges, respectively. The pictures are the data; \u201capple\u201d and \u201corange\u201d are the labels, depending on the picture. As pictures are fed in, the network breaks them down into their most basic components, i.e. edges, textures and shapes. As the picture propagates through the network, these basic components are combined to form more abstract concepts, i.e. curves and different colors which, when combined further, start to look like a stem, an entire orange, or both green and red apples.<\/span><\/p>\n At the end of this process, the network attempts to make a prediction as to what\u2019s in the picture. At first, these predictions will appear as random guesses, as no real learning has taken place yet. If the input image is an apple, but \u201corange\u201d is predicted, the network\u2019s inner layers will need to be adjusted.<\/span><\/p>\n The adjustments are carried out through a process called backpropagation to increase the likelihood of predicting \u201capple\u201d for that same image the next time around. This happens over and over until the predictions are more or less accurate and don\u2019t seem to be improving.<\/span> Just as when parents teach their kids to identify apples and oranges in real life, for computers too, practice makes perfect.<\/span> If, in your head, you just thought \u201chey, that sounds like learning,\u201d then you may have a career in AI.<\/span><\/p>\n Typically, a convolutional neural network has four essential layers of neurons besides the input and output layers:<\/span><\/p>\n Convolution<\/strong><\/p>\n In the initial convolution layer or layers, thousands of neurons act as the first set of filters, scouring every part and pixel in the image, looking for patterns. As more and more images are processed, each neuron gradually learns to filter for specific features, which improves accuracy.<\/span><\/p>\n In the case of apples, one filter might be focused on finding the color red, while another might be looking for rounded edges and yet another might be identifying thin, stick-like stems. If you\u2019ve ever had to clean out a cluttered basement to prepare for a garage sale or a big move \u2014 or worked with a professional organizer \u2014 then you know what it is to go through everything and sort it into different-themed piles (books, toys, electronics, objets d\u2019art, clothes). That\u2019s sort of what a convolutional layer does with an image by breaking it down into different features.<\/span><\/p>\n One advantage of neural networks is that they are capable of learning in a nonlinear way.<\/p>\n<\/blockquote>\n<\/div>\n What\u2019s particularly powerful \u2014 and one of the neural network\u2019s main claims to fame \u2014 is that unlike earlier AI methods (Deep Blue and its ilk), these filters aren\u2019t hand designed; they learn and refine themselves purely by looking at data.<\/span><\/p>\n The convolution layer essentially creates maps \u2014 different, broken-down versions of the picture, each dedicated to a different filtered feature \u2014 that indicate where its neurons see an instance (however partial) of the color red, stems, curves and the various other elements of, in this case, an apple. But because the convolution layer is fairly liberal in its identifying of features, it needs an extra set of eyes to make sure nothing of value is missed as a picture moves through the network.<\/span><\/p>\n Activation<\/strong><\/p>\n One advantage of neural networks is that they are capable of learning in a nonlinear way, which, in mathless terms, means they are able to spot features in images that aren\u2019t quite as obvious \u2014 pictures of apples on trees, some of them under direct sunlight and others in the shade, or piled into a bowl on a kitchen counter. This is all thanks to the activation layer, which serves to more or less highlight the valuable stuff \u2014 both the straightforward and harder-to-spot varieties.<\/span><\/p>\n In the world of our garage-sale organizer or clutter consultant, imagine that from each of those separated piles of things we\u2019ve cherry-picked a few items \u2014 a handful of rare books, some classic t-shirts from our college days to wear ironically \u2014 that we might want to keep. We stick these \u201cmaybe\u201d items on top of their respective category piles for another consideration later.<\/span><\/p>\n Pooling<\/strong><\/p>\n All this \u201cconvolving\u201d across an entire image generates a lot of information, and this can quickly become a computational nightmare. Enter the pooling layer, which shrinks it all into a more general and digestible form. There are many ways to go about this, but one of the most popular is \u201cmax pooling,\u201d which edits down each feature map into a<\/span> Reader\u2019s Digest<\/span><\/em> version of itself, so that only the best examples of redness, stem-ness or curviness are featured.<\/span><\/p>\n In the garage spring cleaning example, if we were using famed Japanese clutter consultant Marie Kondo\u2019s principles, our pack rat would have to choose only the things that \u201cspark joy\u201d from the smaller assortment of favorites in each category pile, and sell or toss everything else. So now we still have all our piles categorized by type of item, but only consisting of the items we actually want to keep; everything else gets sold. (And this, by the way, ends our de-cluttering analogy to help describe the filtering and downsizing that goes on inside a neural network.)<\/span><\/p>\n At this point, a neural network designer can stack subsequent layered configurations of this sort \u2014 convolution, activation, pooling \u2014 and continue to filter down images to get higher-level information. In the case of identifying an apple in pictures, the images get filtered down over and over, with initial layers showing just barely discernable parts of an edge, a blip of red or just the tip of a stem, while subsequent, more filtered layers will show entire apples. Either way, when it\u2019s time to start getting results, the fully connected layer comes into play.<\/span><\/p>\n Source: GumGum<\/p>\n<\/div>\n Fully connected<\/strong><\/p>\n Now it\u2019s time to start getting answers. In the fully connected layer, each reduced, or \u201cpooled,\u201d feature map is \u201cfully connected\u201d to output nodes (neurons) that represent the items the neural network is learning to identify. If the network is tasked with learning how to spot cats, dogs, guinea pigs and gerbils, then it\u2019ll have four output nodes. In the case of the neural network we\u2019ve been describing, it\u2019ll just have two output nodes: one for \u201capples\u201d and one for \u201coranges.\u201d<\/span><\/p>\n If the picture that has been fed through the network is of an apple, and the network has already undergone some training and is getting better with its predictions, then it\u2019s likely that a good chunk of the feature maps contain quality instances of apple features. This is where these final output nodes start to fulfill their destiny, with a reverse election of sorts.<\/span><\/p>\n Tweaks and adjustments are made to help each neuron better identify the data at every level.<\/p>\n<\/blockquote>\n<\/div>\n The job (which they\u2019ve learned \u201con the job\u201d) of both the apple and orange nodes is essentially to \u201cvote\u201d for the feature maps that contain their respective fruits. So, the more the \u201capple\u201d node thinks a particular feature map contains \u201capple\u201d features, the more votes it sends to that feature map. Both nodes have to vote on every single feature map, regardless of what it contains. So in this case, the \u201corange\u201d node won\u2019t send many votes to any of the feature maps, because they don\u2019t really contain any \u201corange\u201d features. In the end, the node that has sent the most votes out \u2014 in this example, the \u201capple\u201d node \u2014 can be considered the network\u2019s \u201canswer,\u201d though it\u2019s not quite that simple.<\/span><\/p>\n Because the same network is looking for two different things \u2014 apples and oranges \u2014 the final output of the network is expressed as percentages. In this case, we\u2019re assuming that the network is already a bit down the road in its training, so the predictions here might be, say, 75 percent \u201capple\u201d and 25 percent \u201corange.\u201d Or, if it\u2019s earlier in the training, it might be more inaccurate and determine that it\u2019s 20 percent \u201capple\u201d and 80 percent \u201corange.\u201d Oops.<\/span><\/p>\n Source: GumGum<\/p>\n<\/div>\n So, in its early stages, the neural network spits out a bunch of wrong answers in the form of percentages. The 20 percent \u201capple\u201d and 80 percent \u201corange\u201d prediction is clearly wrong, but since this is supervised learning with labeled training data, the network is able to figure out where and how that error occurred through a system of checks and balances known as backpropagation.<\/span><\/p>\n Now, this is a mathless explanation, so suffice it to say that backpropagation sends feedback to the previous layer\u2019s nodes about just how far off the answers were. That layer then sends the feedback to the previous layer, and on and on like a game of telephone until it\u2019s back at convolution. Tweaks and adjustments are made to help each neuron better identify the data at every level when subsequent images go through the network.<\/span><\/p>\n This process is repeated over and over until the neural network is identifying apples and oranges in images with increasing accuracy, eventually ending up at 100 percent correct predictions \u2014 though many engineers consider 85 percent to be acceptable. And when that happens, the neural network is ready for prime time and can start identifying apples in pictures professionally.<\/span><\/p>\n *This is different than Google\u2019s AlphaGo which used a self-learned neural net to evaluate board positions and ultimately<\/span><\/em> beat a human at Go<\/a>,<\/span><\/em> versus Deep Blue, which used a hard-coded function written by a human.<\/span><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"More posts by this contributor:\n<\/div>\n<\/div>\n\nThinking by brute force<\/h2>\n
Teaching machines to learn<\/h2>\n
Machines \u2014 they\u2019re just like us!<\/h2>\n
![\"\"](\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/04\/neural_networks_machines_like_us_gumgum1.gif?w=1600&h=588\")
<\/p>\n\n\n
All aboard the network training<\/h2>\n
So many layers\u2026<\/h2>\n
\n
\n\n
![\"\"](\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/04\/neural_networks_convolution_layers_gumgum1.gif?w=1600&h=588\")
<\/p>\n\n\n
![\"\"](\"https:\/\/tctechcrunch2011.files.wordpress.com\/2017\/04\/neural_networks_fully_connected_layers_gumgum1.gif?w=1600&h=588\")
<\/p>\nIf at first you don\u2019t succeed, try, try, try again<\/h2>\n
If you’ve dug into any articles on artificial intelligence, you’ve almost certainly run into the term “neural network.” Modeled loosely on the human brain, artificial neural networks enable computers to learn from being fed data. This guide to neural networks aims to give you a conversational level of understanding of deep learning. Read More<\/a><\/p>\n\n<\/img><\/a>
<\/img><\/a>
<\/img><\/a>
<\/img><\/a>
<\/img><\/a>
<\/img><\/a>\n<\/div>\n
<\/p>\n","protected":false},"author":5817,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[25],"tags":[59,66,26],"class_list":["post-338328","post","type-post","status-publish","format-standard","hentry","category-technology","tag-media","tag-techcrunch","tag-technology"],"_links":{"self":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/338328","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/users\/5817"}],"replies":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/comments?post=338328"}],"version-history":[{"count":0,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/338328\/revisions"}],"wp:attachment":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/media?parent=338328"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/categories?post=338328"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/tags?post=338328"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}