机器学习中的神经网络Neural Networks for Machine Learning：Lecture 5 Quiz

Lecture 5 QuizHelp
Center

Warning: The hard deadline has passed. You can attempt it, but you will not get credit for it. You are welcome to try it as a learning exercise.

Question 1

For which of the following tasks can we expect that the problem of "dimension hopping" will occur (given that the data is input correctly)? Check all that apply.

Determining
whether a given image shows a bike or a car. The bike or car might appear anywhere in the image. The input is the whole set of pixels for the image.
Predicting
the next word in a sequence given the previous 3 words.
Determining
whether a wave has high frequency or low frequency. The input is a set of time values along with their corresponding vertical displacements.
Estimating
the market price of a house given the number of rooms in the house, the number of schools and the average income of the surrounding neighbourhood, and the average sale price of the surrounding houses.

Question 2

We have a convolutional neural network for images of 5 by 5 pixels. In this network, each hidden unit is connected to a different 4 x 4 region of the input image:

The first hidden unit, h1,
is connected to the upper left 4x4 portion of the input image (as shown below).
The second hidden unit, h2,
is connected to the upper right 4x4 portion of the input image (as shown below).
The third hidden unit, h3,
is connected to the lower left 4x4 portion of the input image (not shown in the diagram).
The fourth hidden unit, h4,
is connected to the lower right 4x4 portion of the input image (not shown in the diagram).

Because it's a convolutional network, the weights (connection strengths) are the same for all hidden units: the only difference between the hidden units is that each of them connects to a different part of the input image. In the second diagram, we show the
array of weights, which are the same for each of the four hidden units.





For h1,
weight w11 is
connected to the top-left pixel, i.e. x11,
while for hidden unit h2,
weight w11 connects
to the pixel that is one to the right of the top left pixel, i.e. x12.

Imagine that for some training case, we have an input image where each of the black pixels in the top diagram has value 1, and each of the white ones has value 0. Notice that the image shows a "3" in pixels.

The network has no biases. The weights of the network are given as follows:

w11w21w31w41=1=0=1=0w12w22w32w42=1=0=1=0w13w23w33w43=1=1=1=1w14w24w34w44=0=0=0=0

The hidden units are logistic.

For the training case with that "3" input image, what is the output of each of the four hidden units?

h1=0.982, h2=0.982, h3=0.982, h4=0.982
h1=0.982, h2=1.000, h3=0.881, h4=0.982
h1=0.881, h2=0.982, h3=0.982, h4=1.000
h1=0.982, h2=0.881, h3=1.000, h4=0.982

Question 3

Recall that pooling is the process of combining the outputs of several hidden units to create a single hidden unit. This introduces some invariance to local transformations in the input image.

In this example we are pooling hidden units h1, h2,
and h3.
Let's denote the output of hidden unit hi as yi.
The hidden unit that we're creating by pooling is called hpool,
with output ypool. Sum
pooling is defined as ypool=y1+y2+y3.

This form of pooling can be equivalently represented by making hpool a
regular hidden unit that takes the output of the other three hidden units, multiplies them by some weights w1, w2 and w3,
adds up the resulting numbers and then outputs some function of this sum. This is illustrated in the figure below.



For this to be equivalent to sum pooling, what type of neuron should hpool be,
and what should the weights be?

w1=1, w2=1, w3=1,
and hpool is
a logistic neuron.
w1=13, w2=13, w3=13,
and hpool is
a logistic neuron.
w1=13, w2=13, w3=13,
and hpool is
a linear neuron.
w1=1, w2=1, w3=1,
and hpool is
a linear neuron.

Question 4

Suppose that we have a vocabulary of 3 words, "a", "b", and "c", and we want to predict the next word in a sentence given the previous two words. For this network, we don't want to use feature vectors for words: we simply use the local encoding, i.e. a 3-component
vector with one entry being 1 and
the other two entries being 0.

In the language models that we have seen so far, each of the context words has its own dedicated section of the network, so we would encode this problem with two 3-dimensional inputs. That makes for a total of 6 dimensions. For example, if the two preceding
words (the "context" words) are "c" and "b", then the input would be (0,0,1,0,1,0).
Clearly, the more context words we want to include, the more input units our network must have. More inputs means more parameters, and thus increases the risk of overfitting. Here is a proposal to reduce the number of parameters in the model:

Consider a single neuron that is connected to this input, and call the weights that connect the input to this neuron w1,w2,w3,w4,w5,
and w6. w1 connects
the neuron to the first input unit, w2 connects
it to the second input unit, etc. Notice how for every neuron, we need as many weights as there are input dimensions (6 in our case), which will be the number of words times the length of the context. A way to reduce the number of parameters is to tie certain
weights together, so that they share a parameter. One possibility is to tie the weights coming from input units that correspond to the same word but at different context positions. In our example that would mean that w1=w4, w2=w5,
and w3=w6 (see
the "after" diagram below).



Are there any significant problems with this approach?

Yes:
weight tying only makes sense when we are working with images.
No:
the new model after weight tying is an example of a convolutional neural network, and these are more powerful than a non-convolutional network because they are invariant to small transformations in the data.
Yes:
the network loses the knowledge of the location at which a context word occurs, and that is valuable knowledge.
No:
this method is an appropriate solution in that it will reduce the number of parameters and therefore always improve generalization.

Question 5

Let's look at what weight tying does to gradients, computed using the backpropagation algorithm. For this question, our network has three input units, x1,x2,x3,
two logistic hidden units h1,h2,
four input to hidden weights w1,w2,w3,w4,
and two hidden to output weights u1,u2.
The output neuron y is
a linear neuron, and we are using the squared error cost function.



Consider a single training case with target output t.
The sequence of computations required to compute the error (this is called forward propagation) are as follows:

z1=z2=h1=h2=y=E=w1x1+w2x2w3x2+w4x3σ(z1)σ(z2)u1h1+u2h212(t−y)2

E is
the error. Reminder: σ(x)=11+exp(−x)

z1 and z2 are
called the total inputs to hidden units h1 and h2 respectively.

Suppose we now decide to tie the weights so that w2=w3=wtied.
What is the derivative of the error E with
respect to wtied?

Hint: forget about weight tying for a moment and simply compute the derivatives ∂E∂w2 and ∂E∂w3.
Now set ∂E∂wtied=∂E∂w2+∂E∂w3.

∂E∂wtied=−2(t−y)(u2h2(1−h2)x2)
∂E∂wtied=−(t−y)(u1h1(1−h1)x1+u2h2(1−h2)x3)
∂E∂wtied=−2(t−y)(u1h1(1−h1)x2)
∂E∂wtied=−(t−y)[u1h1(1−h1)+u2h2(1−h2)]x2

Question 6

Oh no! Brian's done it again! Claire had a dataset of 28 x 28 pixel handwritten digits nicely prepared to train a neural network, but Brian has gone and accidentally scrambled the images by re-ordering the pixels in some totally meaningless way, and now they
can't get the original dataset back! Below is an image of a handwritten '2' before and after being scrambled by Brian.

Before	After

Luckily, all of the images (in both the training set and the test set) were changed in the same way. For example, if pixels number 1 and number 3 switched places in one image, then they switched places in every other image as well. Because of that, Claire thinks
that perhaps she can train a network after all.

Whether Claire is right or not depends largely on the type of neural network that she has in mind. Which of the following neural networks will be at a disadvantage because of Brian's mistake? Check all that apply.

A
convolutional neural network where the size of each weight filter is 10 x 10.
A
feed-forward neural network with no hidden layer and linear units (and no convolution).
A
feed-forward neural network with one hidden layer of linear units (and no convolution).
A
convolutional neural network where the size of each weight filter is 8 x 8.