neural networks
seem so complicated
but they're not statquest
hello i'm josh starmer and welcome to
statquest
today we're going to talk about neural
networks part one
inside the black box
neural networks one of the most popular
algorithms in machine learning
cover a broad range of concepts and
techniques
however people call them a black box
because it can be hard to understand
what they're doing
the goal of this series is to take a
peek into the black box by breaking down
each concept
and technique into its components and
walking through how they fit together
step by step in this first part
we will learn about what neural networks
do and how they do it
in part two we'll talk about how neural
networks are fit to data with back
propagation
then we will talk about variations on
the simple neural network presented in
this part
including deep learning note
crazy awesome news i have a new way to
think about neural networks that will
help beginners
and seasoned experts alike gain a deep
insight into what neural networks do
for example most tutorials use
cool looking but hard to understand
graphs
and fancy mathematical notation to
represent neural networks
in contrast i'm going to label every
little thing on the neural network
to make it easy to keep track of the
details
and the math will be as simple as
possible while still being true to the
algorithm
these differences will help you develop
a deep understanding
of what neural networks actually do
so with that said let's imagine we
tested a drug that was designed to treat
an illness and we gave the drug to three
different groups of people
with three different dosages low
medium and high
the low dosages were not effective so we
set them to zero
on this graph in contrast
the medium dosages were effective so we
set them to one
and the high dosages were not effective
so those are set to zero
now that we have this data we would like
to use it to predict whether or not a
future dosage will be effective
however we can't just fit a straight
line to the data to make
predictions because no matter how we
rotate the straight line
it can only accurately predict two of
the three dosages
the good news is that a neural network
can fit a squiggle to the data
the green squiggle is close to zero for
low dosages
close to one for medium dosages
and close to zero for high dosages
and even if we have a really complicated
data set like this
a neural network can fit a squiggle to
it
in this stat quest we're going to use
this super simple data set
and show how this neural network creates
this green squiggle
but first let's just talk about what a
neural network
is a neural network consists of nodes
and connections between the nodes
note the numbers along each connection
represent parameter values that were
estimated when this neural network was
fit to the data
for now just know that these parameter
estimates are analogous to the slope and
intercept values that we solve for
when we fit a straight line to data
likewise a neural network starts out
with unknown parameter values
that are estimated when we fit the
neural network to a data set
using a method called back propagation
and we will talk about how back
propagation estimates these parameters
in part 2
in this series but for now
just assume that we've already fit this
neural network to this specific
data set and that means we have already
estimated these parameters
also you may have noticed that some of
the nodes have
curved lines inside of them these
bent or curved lines are the building
blocks for fitting a squiggle
to data the goal of this stat quest is
to show you how these identical curves
can be reshaped by the parameter values
and then added together to get a green
squiggle that fits the data
note there are many common bent or
curved lines that we can choose for a
neural network
this specific curved line is called soft
plus which sounds like a brand of toilet
paper
alternatively we could use this bent
line
called relu which is short for rectified
linear unit
and sounds like a robot or we could use
a sigmoid shape
or any other bent or curved line
oh no it's the dreaded terminology alert
the curved or bent lines are called
activation functions
when you build a neural network you have
to decide which activation function
or functions you want to use
when most people teach neural networks
they use the sigmoid activation function
however in practice it is much more
common to use the relu
activation function or the soft plus
activation function
so we'll use the soft plus activation
function in this stat quest
anyway we'll talk more about how you
choose activation functions
later in this series note
this specific neural network is about as
simple as they get
it only has one input node where we plug
in the dosage
only one output node to tell us the
predicted effectiveness
and only two nodes between the input and
output nodes
however in practice neural networks are
usually much
fancier and have more than one
input node more than one output node
different layers of nodes between the
input and output nodes
and a spider web of connections between
each layer of nodes
oh no it's another terminology alert
these layers of nodes between the input
and output nodes are called hidden
layers
when you build a neural network one of
the first things you do
is decide how many hidden layers you
want and how many nodes
go into each hidden layer although there
are rules of thumb
for making decisions about the hidden
layers
you essentially make a guess and see how
well the neural network performs
adding more layers and nodes if needed
now even though this neural network
looks fancy
it is still made from the same parts
used in this simple neural network
which has only one hidden layer with two
nodes
so let's learn how this neural network
creates new shapes from the curved or
bent lines in the hidden layer
and then adds them together to get a
green squiggle
that fits the data note
to keep the math simple let's assume
dosages go from zero
for low to one for high
the first thing we are going to do is
plug the lowest dosage
zero into the neural network
now to get from the input node
to the top node in the hidden layer
this connection multiplies the dosage by
negative
34.4 and then adds
2.14 and the result
is an x-axis coordinate for the
activation function
for example the lowest dosage 0
is multiplied by negative 34.4
and then we add 2.14
to get 2.14 as the x-axis coordinate for
the activation function
to get the corresponding y-axis value
we plug 2.14 into the activation
function
which in this case is the soft plus
function
note if we had chosen the sigmoid curve
for the activation function
then we would plug 2.14 into the
equation for the sigmoid curve
and if we had chosen the rail u bent
line for the activation function
then we would plug 2.14 into the relu
equation but since we are using soft
plus for the activation function
we plug 2.14 into the soft plus equation
and the log of one plus e raised to the
2.14
power is 2.25
note in statistics machine learning and
most programming languages
the log function implies the natural log
or the log base e
anyway the y axis coordinate for the
activation function
is 2.25 so let's extend this y-axis
up a little bit and put a blue dot
at 2.25 for when dosage equals zero
now if we increase the dosage a little
bit
and plug 0.1 into the input
the x-axis coordinate for the activation
function
is negative 1.3 and the corresponding
y-axis value
is 0.24
so let's put a blue dot at 0.24
for when dosage equals 0.1
and if we continue to increase the
dosage values all the way to 1
the maximum dosage we get this blue
curve note
before we move on i want to point out
that the full range of dosage values
from 0 to 1 corresponds to this
relatively narrow range of values from
the activation function
in other words when we plug dosage
values
from 0 to 1 into the neural network
and then multiply them by negative 34.4
and add 2.14
we only get x-axis coordinates that are
within the red box
and thus only the corresponding y-axis
values in the red box are used to make
this new blue curve
bam now we scale the y-axis values for
the blue curve
by negative 1.3
for example when dosage equals zero the
current y-axis
coordinate for the blue curve is 2.25
so we multiply 2.25 by negative 1.3
and get negative 2.93
and negative 2.93 corresponds to this
position
on the y-axis likewise
we multiply all of the other y-axis
coordinates on the blue curve
by negative 1.3 and we end up with a new
blue curve
bam
now let's focus on the connection from
the input node
to the bottom node in the hidden layer
however this time we multiply the dosage
by negative 2.52
instead of negative 34.4
and we add 1.29
instead of 2.14 to get the x-axis
coordinate for the activation function
remember these values come from fitting
the neural network to the data with back
propagation and we'll talk about that in
part two
in this series now
if we plug the lowest dosage zero into
the neural network
then the x-axis coordinate for the
activation function
is 1.29 now we plug
1.29 into the activation function
to get the corresponding y-axis value
and get 1.53
and that corresponds to this yellow dot
now we just plug in dosage values from 0
to 1
to get the corresponding y-axis values
and we get this orange curve
note just like before i want to point
out that the full range of dosage values
from 0 to 1 corresponds to this narrow
range of values from the activation
function
in other words when we plug dosage
values from 0 to 1
into the neural network we only get
x-axis coordinates that are within the
red box
and thus only the corresponding y-axis
values in the red box are used to make
this new orange curve
so we see that fitting a neural network
to data
gives us different parameter estimates
on the connections
and that results in each node in the
hidden layer
using different portions of the
activation functions
to create these new and exciting shapes
now just like before we scale the y-axis
coordinates on the orange curve only
this time
we scale by a positive number 2.28
and that gives us this new orange curve
now the neural network tells us to add
the y-axis coordinates from the blue
curve
to the orange curve
and that gives us this green squiggle
then finally we subtract 0.58
from the y-axis values on the green
squiggle
and we have a green squiggle that fits
the data
bam now if someone comes along and says
that they are using dosage
equal to 0.5 we can look at the
corresponding
y-axis coordinate on the green squiggle
and see that the dosage will be
effective
or we can solve for the y-axis
coordinate by plugging
dosage equals 0.5 into the neural
network
and do the math
[Music]
and we see that the y-axis coordinate on
the green squiggle
is 1.03 and since
1.03 is closer to 1 than 0
we will conclude that a dosage equal to
0.5
is effective double bam
now if you've made it this far you may
be wondering why this is called a neural
network
instead of a big fancy squiggle fitting
machine
reason is that way back in the 1940s and
50s when neural networks were invented
they thought the nodes were vaguely like
neurons
and the connections between the nodes
were sort of like synapses
however i think they should be called
big fancy squiggle fitting machines
because that's what they do note
whether or not you call it a squiggle
fitting machine
the parameters that we multiply are
called weights
and the parameters that we add are
called biases
note this neural network starts with two
identical activation functions
but the weights and biases on the
connections slice them
flip them and stretch them into new
shapes
which are then added together to get a
squiggle that is entirely new
and then the squiggle is shifted to fit
the data
now if we can create this green squiggle
with just
two nodes in a single hidden layer just
imagine what types of green squiggles we
could fit with more hidden layers
and more nodes in each hidden layer
in theory neural networks can fit a
green squiggle to just about
any data set no matter how complicated
and i think that's pretty cool triple
bam
now it's time for some shameless
self-promotion
if you want to review statistics and
machine learning offline
check out the statquest study guides at
statquest.org
there's something for everyone hooray
we've made it to the end of another
exciting stat quest
if you like this stat quest and want to
see more please subscribe
and if you want to support statquest
consider contributing to my patreon
campaign
becoming a channel member buying one or
two of my original songs or a t-shirt or
a hoodie
or just donate the links are in the
description below
alright until next time quest on