RNN, LSTM 图文详解

编程基础 • 2025-01-06 15:33 • 阅读 12

文章目录

- 往期文章链接目录
- Sequence Data
- Why not use a standard neural network for sequence tasks
- RNN
- LSTM
- - Types of gates
  - formulas and illustration of formulas
- Variants of RNNs
- - Bi-directional RNN
  - Deep RNN
- 往期文章链接目录

往期文章链接目录

Sequence Data

There are many sequence data in applications. Here are some examples

Machine translation
- from text sequence to text sequence.
Text Summarization
- from text sequence to text sequence.
Sentiment classification
- from text sequence to categories.
Music Generation
- from nothing or some simple stuff (character, integer, etc) to wave sequence.
Name entity recognition (NER)
- From text sequence to label sequence.

Why not use a standard neural network for sequence tasks

Inputs, outputs can be different lengths in different examples. This can be solved by standard neural network by paddings with the maximum lengths but it’s not a good solution since there would be too many parameters.
Doesn’t share features learned across different positions of text/sequence. Note that Convolutional Neural Network (CNN) is a good example of parameter sharing, so we should have a similar model for sequence data.

RNN

A Recurrent Neural Network (RNN) can be thought of as multiple copies of the same network, each passing a message to a successor. This chain-like nature reveals that recurrent neural networks are intimately related to sequences. Therefore, they’re the natural architecture of neural network to use for sequence data. Note that it also allows previous outputs to be used as inputs.

For each time step $t$ , the activation $a^{<t>}$ and the output $y^{<t>}$ are expressed as follows:

$a^{<t>}=g_{1}\left(W_{a a} a^{<t-1>}+W_{a x} x^{<t>}+b_{a}\right)$
$y^{<t>}=g_{2}\left(W_{y a} a^{<t>}+b_{y}\right)$

These calculation could be visualized in the following figure

Note:

dimension of $W_{a a}$ : (number of hidden neurons, number of hidden neurons)
dimension of $W_{a x}$ : (number of hidden neurons, length of $x$ )
dimension of $W_{y a}$ : (length of $y$ , number of hidden neurons)
The weight matrix $W_{a a}$ is the memory the RNN is trying to maintain from the previous layers.
dimension of $b_a$ : (number of hidden neurons, 1)
dimension of $b_y$ : (length of

今天的文章 RNN, LSTM 图文详解分享到此就结束了，感谢您的阅读。

RNN, LSTM 图文详解

文章目录

往期文章链接目录

Sequence Data

Why not use a standard neural network for sequence tasks

RNN

相关推荐