深度学习-note-RNN(循环神经网络)

（本文为本人学习工作总结，如有雷同，不胜荣幸。可联系本人立即修改或者删除）

RNN（循环网络）

问题：什么是RNN？RNN有什么应用？怎么用？

解答：

RNN

概念

循环卷积网络（RNN，Recurrent Neural Networks），将数据关联单个NN，RNN在x(t)时刻得出状态描述S(t)，由此输出Y(t);在x(t+1)时刻，产生S(t+1),而Y(t+1)由S(t)和S(t+1)产生。

缺陷：梯度消失（e.g:state传递下去会出现0.1^n，趋向于0），梯度爆炸(e.g:1.1^n,趋向于∞)；解决：使用LSTM Cell（Long-Short Term Memory Cell），结构：在RNN 的cell添加Input Gate,output Gate,forget Gate.这些Gate是一些参数，Input记住memory，output输出memory，forget忘记memory。

论文

参考部分

总结

应用

用于序列化数据

1. RNN描述照片

2. 写程序

3. 创作歌曲

4. 创作诗歌论文

代码实现

1.classfication

TensorFlow实现：

RNN_classfication.py

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import numpy as np
import matplotlib.pyplot as plt

#1.3.0-0
print(tf.__version__)
print(np.__version__)
#设置种子数
tf.set_random_seed(1)
np.random.seed(1)

#超参数（自定义的参数）
BATCH_SIZE = 64
TIME_STEP = 28
INPUT_SIZE =28
LR = 0.01

#数据
mnist = input_data.read_data_sets('./mnist', one_hot=True)
test_x = mnist.test.images[:2000]
test_y = mnist.test.labels[:2000]

#输出测试
print(mnist.train.images.shape)
print(mnist.train.labels.shape)
plt.imshow(mnist.train.images[0].reshape((28,28)),cmap='gray')
plt.title('%i' % np.argmax(mnist.train.labels[0]))
plt.show()

#tensorflow placeholders
##def placeholder(dtype, shape=None, name=None)：shape(batch , 28*28=784)
tf_x = tf.placeholder(tf.float32,[None, TIME_STEP *  INPUT_SIZE])
##def reshape(tensor, shape, name):shape(batch, height, width, channel)
image = tf.reshape(tf_x, [-1, TIME_STEP, INPUT_SIZE])
tf_y = tf.placeholder(tf.int32, [None, 10])

#RNN结构
rnn_cell = tf.contrib.rnn.BasicLSTMCell(num_units=64)
##def dynamic_rnn(cell, inputs, sequence_length=None, initial_state=None,
##                dtype=None, parallel_iterations=None, swap_memory=False,
##                time_major=False, scope=None):
outputs, (h_c,h_n) = tf.nn.dynamic_rnn(
rnn_cell,           #An instance of RNNCell
image,              #inputs
initial_state=None, #初始化的隐藏状态
dtype=tf.float32,   #隐藏状态为None时,dtype必须设置
time_major=False,   #False: (batch, time step, input); True: (time step, batch, input)
)
print(outputs)
output = tf.layers.dense(outputs[:, -1, :], 10)
##softmax算法,
loss = tf.losses.softmax_cross_entropy(onehot_labels=tf_y, logits=output)
##adam(学习率)
train_op = tf.train.AdamOptimizer(LR).minimize(loss)
##def accuracy(labels, predictions, weights=None, metrics_collections=None,
##             updates_collections=None, name=None):　return (acc, update_op), and create 2 local variables
###def argmax(input,axis=None,name=None,dimension=None,output_type=dtypes.int64):
accuracy = tf.metrics.accuracy(
labels=tf.argmax(tf_y,axis=1), predictions=tf.argmax(output,axis=1),)[1]

sess = tf.Session()
init_op = tf.group(tf.global_variables_initializer(),tf.local_variables_initializer())
# 在图中初始化参数
sess.run(init_op)

#训练
for step in range(1200):
##返回值,return self._images[start:end], self._labels[start:end]
b_x, b_y = mnist.train.next_batch(BATCH_SIZE)
##run(self, fetches, feed_dict=None, options=None, run_metadata=None) :Runs operations and evaluates tensors in `fetches`.
_, loss_ = sess.run([train_op,loss], {tf_x: b_x, tf_y: b_y})
if step % 50 == 0:
accuracy_ =sess.run(accuracy, {tf_x:test_x, tf_y:test_y})
print('train loss: %.4f' % loss_, '| test accuracy: %.2f' % accuracy_)

#输出10个预测值
test_output = sess.run(output, {tf_x:test_x[:10]})
pred_y = np.argmax(test_output, 1)
print(pred_y, 'prediction number')
print(np.argmax(test_y[:10], 1), 'real number')

注意：程序运行时正在下mnist手写训练集合，会卡住控制台，最好的办法就是提前下载好训练集放到对应的目录．

输出:

/usr/local/anaconda3/bin/python3.5 /usr/local/IdeaProjects/MyCSDN/Tensorflow_test/RNN_classification.py
1.3.0
1.11.1
Extracting ./mnist/train-images-idx3-ubyte.gz
Extracting ./mnist/train-labels-idx1-ubyte.gz
Extracting ./mnist/t10k-images-idx3-ubyte.gz
Extracting ./mnist/t10k-labels-idx1-ubyte.gz
(55000, 784)
(55000, 10)

(process:7948): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Did not re
4000
ceive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.

(python3.5:7948): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_register_object: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_register_object: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_get_unique_name: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_register_object: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_register_object: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_get_unique_name: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
Qt: Session management error: None of the authentication protocols specified are supported
Tensor("rnn/transpose:0", shape=(?, 28, 64), dtype=float32)
2017-11-12 21:15:47.124601: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-12 21:15:47.124622: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-12 21:15:47.124626: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-11-12 21:15:47.124630: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-12 21:15:47.124634: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
train loss: 2.3040 | test accuracy: 0.13
train loss: 0.8905 | test accuracy: 0.41
train loss: 0.6223 | test accuracy: 0.52
train loss: 0.4808 | test accuracy: 0.60
train loss: 0.4486 | test accuracy: 0.65
train loss: 0.1816 | test accuracy: 0.70
train loss: 0.4413 | test accuracy: 0.73
train loss: 0.0996 | test accuracy: 0.75
train loss: 0.1570 | test accuracy: 0.77
train loss: 0.3565 | test accuracy: 0.78
train loss: 0.0979 | test accuracy: 0.80
train loss: 0.2561 | test accuracy: 0.81
train loss: 0.1261 | test accuracy: 0.82
train loss: 0.1363 | test accuracy: 0.83
train loss: 0.1119 | test accuracy: 0.84
train loss: 0.2960 | test accuracy: 0.85
train loss: 0.0805 | test accuracy: 0.85
train loss: 0.2942 | test accuracy: 0.86
train loss: 0.2955 | test accuracy: 0.86
train loss: 0.0553 | test accuracy: 0.87
train loss: 0.0917 | test accuracy: 0.87
train loss: 0.1553 | test accuracy: 0.87
train loss: 0.2084 | test accuracy: 0.88
train loss: 0.1777 | test accuracy: 0.88
[7 2 1 0 4 1 4 9 5 9] prediction number
[7 2 1 0 4 1 4 9 5 9] real number

Process finished with exit code 0

特别感谢莫烦大神的教程

参考

Towards End-to-End Speech Recognition

with Recurrent Neural Networks

2.格雷夫斯，亚历克斯 “用循环神经网络生成序列

3.莫烦大神的教程

4.github llSourcell/LSTM_Networks