本文正在参加「Python主题月」，详情查看活动链接
上周分享关于卷积神经网的实现，不过只是实现前向传播，虽然卷积神经网络看似要复杂一些，但是实现起来可能没有想象那么难，其实难的东西都在今天内容里，我尝试去给大家解释，也尽量将一个推导公式给大家详细列出来。
训练模型
重点是训练环节，也就是在训练环节如何计算梯度和然后用梯度来更新参数，在训练过程中，通常包括 2 个阶段，前向传播和后项传播
前向传播阶段 : 输入数据经神经网网络一层一层向前传递，这个过程就是前向传播后向传播阶段 : 在整个网络反向逐层更新梯度
在训练 CNN 过程中也包含前向传播和反向传播两个阶段，以及如何具体将其实现
在前向传播过程中，会将数据(例如输入数据和中间变量)缓存起来起来，以备在反向传播过程使用。就意味着每个反向传播一定会存储与其对应的前向传播
在反向传播过程中，神经网络每一层都接受一个梯度，计算后返回一个梯度。这里
∂
L
∂
o
u
t
\frac{\partial L}{\partial out}
∂ o u t ∂ L
∂
l
∂
i
n
\frac{\partial l}{\partial in}
∂ in ∂ l
基于以上 2 个思路来实现代码，可以保证代码整洁和层次感，大多时候我们会说先思考然后再去 coding，不过想象这是对一些已经积累一些经验程序员而言，如果对于经验不多人我们还是先动手然后再去。所以我们的 CNN 代码看起来应该是类似下面代码的样子
gradient = np.zeros(10 )
gradient = softmax.backprop(gradient)
gradient = pool.backprop(gradient)
gradient = conv.backprop(gradient)
链式法则: 例如对 f [ g ( h ( x ) ) ] f[g(h(x))] f [ g ( h ( x ))] g [ h ( x ) ] g[h(x)] g [ h ( x )] g ′ [ h ( x ) ] h ′ ( x ) g^{\prime}[h(x)]h^{\prime}(x) g ′ [ h ( x )] h ′ ( x ) f [ g ( h ( x ) ) ] = f ′ [ g ( h ( x ) ) ] g ′ [ h ( x ) ] h ′ ( x ) f[g(h(x))] = f^{\prime}[g(h(x))]g^{\prime}[h(x)]h^{\prime}(x) f [ g ( h ( x ))] = f ′ [ g ( h ( x ))] g ′ [ h ( x )] h ′ ( x )
反向传播: Softmax
与前向传播相反，当前向传播完成后，就开始反向传播组成传递梯度，接下来我们就来看看在反向求导是如何进行的。首先来看的 cross-entropy loss(交叉熵损失函数)
L
=
−
ln
(
p
c
)
L = – \ln(p_c)
L = − ln ( p c )
公式里的 p c p_c p c
∂
L
∂
o
u
t
s
(
i
)
→
0
i
f
i
≠
c
−
1
p
i
i
f
i
=
c
\frac{\partial L}{\partial out_s(i)} \rightarrow \begin{aligned} 0 \, if\, i \neq c \\ -\frac{1}{p^i} \, if \, i = c \end{aligned}
∂ o u t s ( i ) ∂ L → 0 i f i = c − p i 1 i f i = c
这里 c 表示该样本图片属于类别,所以交叉熵计算损失函数只会考虑在正确类别上模型给出概率值，所以其他类别不会考虑
在 softmax 的前向传播(forward) 需要对 3 个变量进行缓存，分别是
input 是未展平前的形状input 经过展平后totals 表示传入到 softmax 激活函数前的值
在前向传播做好准备后，我们就可以开始反向传播。因为在交叉熵损失函数仅对真实标签所对应的，
，
首先，计算 o u t s ( c ) out_s(c) o u t s ( c ) t i t_i t i o u t s ( c ) out_s(c) o u t s ( c )
o
u
t
s
(
c
)
=
e
t
c
∑
i
e
t
i
=
e
t
c
S
S
=
∑
i
e
t
i
out_s(c) = \frac{e^{t_c}}{\sum_i e^{t_i}} = \frac{e^{t_c}}{S}\,\,\, S = \sum_i e^{t_i}
o u t s ( c ) = ∑ i e t i e t c = S e t c S = i ∑ e t i
首先考虑类别 k 满足条件 k ≠ c k \neq c k = c
o
u
t
s
(
c
)
=
e
t
c
S
−
1
out_s(c) = e^{t_c}S^{-1}
o u t s ( c ) = e t c S − 1
∂
o
u
t
s
(
c
)
∂
t
k
=
∂
o
u
t
s
(
c
)
∂
S
(
∂
S
∂
t
k
)
−
e
t
c
S
−
2
(
∂
S
∂
t
k
)
=
−
e
t
c
S
−
2
(
e
t
k
)
=
−
e
t
c
e
t
k
S
2
\frac{\partial out_s(c)}{\partial t_k} = \frac{\partial out_s(c)}{\partial S}(\frac{\partial S}{\partial t_k})\\ -e^{t_c}S^{-2} (\frac{\partial S}{\partial t_k})\\ = -e^{t_c}S^{-2} (e^{t_k})\\ = \frac{- e^{t_c}e^{t_k}}{S^2}
∂ t k ∂ o u t s ( c ) = ∂ S ∂ o u t s ( c ) ( ∂ t k ∂ S ) − e t c S − 2 ( ∂ t k ∂ S ) = − e t c S − 2 ( e t k ) = S 2 − e t c e t k
∂
o
u
t
s
(
c
)
∂
t
c
=
S
e
t
c
−
e
t
c
∂
S
∂
t
c
S
2
=
S
e
t
c
−
e
t
c
e
t
c
S
2
=
e
t
c
(
S
−
e
t
c
)
S
2
\frac{\partial out_s(c)}{\partial t_c} = \frac{Se^{t_c} – e^{t_c}\frac{\partial S}{\partial t_c}}{S^2}\\ = \frac{Se^{t_c} – e^{t_c}e^{t_c}}{S^2}\\ = \frac{e^{t_c}(S – e^{t_c})}{S^2}
∂ t c ∂ o u t s ( c ) = S 2 S e t c − e t c ∂ t c ∂ S = S 2 S e t c − e t c e t c = S 2 e t c ( S − e t c )
如果上面两个公式看起来不算很好理解，我通过一个具体例子给大家一步一步推导
∂
L
∂
w
2
,
1
=
∂
L
∂
a
1
∂
a
1
∂
z
1
∂
z
1
∂
w
2
,
1
+
∂
L
∂
a
2
∂
a
2
∂
z
1
∂
z
1
∂
w
2
,
1
\frac{\partial L}{\partial w_{2,1}} = \frac{\partial L}{\partial a_1} \frac{\partial a_1}{\partial z_1} \frac{\partial z_1}{\partial w_{2,1}} + \frac{\partial L}{\partial a_2} \frac{\partial a_2}{\partial z_1} \frac{\partial z_1}{\partial w_{2,1}}
∂ w 2 , 1 ∂ L = ∂ a 1 ∂ L ∂ z 1 ∂ a 1 ∂ w 2 , 1 ∂ z 1 + ∂ a 2 ∂ L ∂ z 1 ∂ a 2 ∂ w 2 , 1 ∂ z 1
这里我们以更新 w 2 , 1 w_{2,1} w 2 , 1 w 1 ， 2 w_{1，2} w 1 ， 2 ∂ L ∂ a 1 \frac{\partial L}{\partial a_1} ∂ a 1 ∂ L ∂ a 1 z 1 \frac{\partial a_1}{z_1} z 1 ∂ a 1
∂
L
∂
a
1
=
∂
∂
a
1
[
∑
j
h
−
y
i
ln
(
a
j
)
]
∂
∂
a
1
[
∑
j
h
−
y
1
ln
(
a
1
)
]
y
a
1
\frac{\partial L}{\partial a_1} = \frac{\partial}{\partial a_1} [\sum_j^h -y_i \ln(a_j)]\\ \frac{\partial}{\partial a_1} [\sum_j^h -y_1 \ln(a_1)]\\ \frac{y}{a_1}
∂ a 1 ∂ L = ∂ a 1 ∂ [ j ∑ h − y i ln ( a j )] ∂ a 1 ∂ [ j ∑ h − y 1 ln ( a 1 )] a 1 y
∂
a
1
∂
z
1
=
∂
z
1
[
e
z
1
∑
j
n
e
z
j
]
\frac{\partial a_1}{\partial z_1} = \frac{\partial}{z_1} [\frac{e^{z_1}}{\sum_j^n e^{z^j}}]
∂ z 1 ∂ a 1 = z 1 ∂ [ ∑ j n e z j e z 1 ]
这个求导看似负责，我们用 f ( x ) = e e z 1 f(x) = e^{e^{z_1}} f ( x ) = e e z 1 g ( x ) = ∑ j = 1 n e z j g(x) = \sum_{j=1}^n e^{z_j} g ( x ) = ∑ j = 1 n e z j
f
(
x
)
g
(
x
)
=
g
(
x
)
f
′
(
x
)
−
f
(
x
)
g
′
(
x
)
[
g
(
x
)
]
2
\frac{f(x)}{g(x)} = \frac{g(x)f^{\prime}(x) – f(x)g^{\prime}(x)}{[g(x)]^2}
g ( x ) f ( x ) = [ g ( x ) ] 2 g ( x ) f ′ ( x ) − f ( x ) g ′ ( x )
根据这个公式我们来计算上面偏导
∑
j
=
1
n
e
z
j
∂
∂
z
1
[
e
z
1
]
−
e
z
1
∂
∂
z
1
[
∑
j
=
1
n
e
z
j
]
[
∑
j
=
1
n
e
z
j
]
2
=
[
∑
j
=
1
n
e
z
j
]
e
z
1
−
e
z
1
e
z
1
]
[
∑
j
=
1
n
e
z
j
]
2
=
z
1
(
[
∑
j
=
1
n
e
z
j
]
−
e
z
1
)
[
∑
j
=
1
n
e
z
j
]
2
=
e
z
1
[
∑
j
=
1
n
e
z
j
]
∑
j
=
1
n
e
z
j
]
[
∑
j
=
1
n
e
z
j
]
a
1
(
1
−
a
1
)
\frac{\sum_{j=1}^n e^{z_j} \frac{\partial}{\partial z_1}[e^{z_1}] – e^{z_1} \frac{\partial}{\partial z_1} [\sum_{j=1}^n e^{z_j}] }{[\sum_{j=1}^n e^{z_j}]^2}\\ =\frac{[\sum_{j=1}^n e^{z_j}]e^{z_1} – e^{z_1} e^{z_1}]}{[\sum_{j=1}^n e^{z_j}]^2}\\ = \frac{z_1([\sum_{j=1}^n e^{z_j}] – e^{z_1} )}{[\sum_{j=1}^n e^{z_j}]^2}\\ =\frac{e^{z_1}}{[\sum_{j=1}^n e^{z_j}]} \frac{\sum_{j=1}^n e^{z_j}]}{[\sum_{j=1}^n e^{z_j}]}\\ a_1(1-a_1)
[ ∑ j = 1 n e z j ] 2 ∑ j = 1 n e z j ∂ z 1 ∂ [ e z 1 ] − e z 1 ∂ z 1 ∂ [ ∑ j = 1 n e z j ] = [ ∑ j = 1 n e z j ] 2 [ ∑ j = 1 n e z j ] e z 1 − e z 1 e z 1 ] = [ ∑ j = 1 n e z j ] 2 z 1 ([ ∑ j = 1 n e z j ] − e z 1 ) = [ ∑ j = 1 n e z j ] e z 1 [ ∑ j = 1 n e z j ] ∑ j = 1 n e z j ] a 1 ( 1 − a 1 )
因为推导过程比较详细，所以这里就不做过多解释。另一条路线大家自己去尝试一下给出答案是 − a 1 a 2 -a_1a_2 − a 1 a 2
class Softmax :
def backprop (self, d_L_d_out ):
''' Performs a backward pass of the softmax layer. Returns the loss gradient for this layer's inputs. - d_L_d_out is the loss gradient for this layer's outputs. '''
for i, gradient in enumerate (d_L_d_out):
if gradient == 0 :
continue
t_exp = np.exp(self.last_totals)
S = np.sum (t_exp)
d_out_d_t = -t_exp[i] * t_exp / (S ** 2 )
d_out_d_t[i] = t_exp[i] * (S - t_exp[i]) / (S ** 2 )
因为在cross-entropy 在损失函数只考虑模型预测中对应真实标签类别 c 那一个预测值，所以只需要考虑 d_L_d_out 梯度不为 0 就可以。一旦计算梯度 ∂ o u t s ( i ) ∂ t \frac{\partial out_s(i)}{\partial t} ∂ t ∂ o u t s ( i )
i
f
k
≠
c
∂
o
u
t
s
(
k
)
∂
t
=
−
e
t
c
e
t
k
S
2
i
f
k
=
c
∂
o
u
t
s
(
k
)
∂
t
=
e
t
c
(
S
−
e
t
c
)
S
2
if \, k \neq c \, \frac{\partial out_s(k)}{\partial t} = \frac{- e^{t_c}e^{t_k}}{S^2}\\ if \, k = c \, \frac{\partial out_s(k)}{\partial t} = \frac{e^{t_c}(S – e^{t_c})}{S^2}\\
i f k = c ∂ t ∂ o u t s ( k ) = S 2 − e t c e t k i f k = c ∂ t ∂ o u t s ( k ) = S 2 e t c ( S − e t c )
接下来就是计算权重、偏置和输入的梯度
计算权重梯度
∂
L
∂
w
\frac{\partial L}{\partial w}
∂ w ∂ L
计算偏置的梯度
∂
L
∂
b
\frac{\partial L}{\partial b}
∂ b ∂ L
在反向求导中将返回变量梯度作为前一层的梯度输入
d_t_d_w = self.last_input
d_t_d_b = 1
d_t_d_inputs = self.weights
d_L_d_t = gradient * d_out_d_t
d_L_d_w = d_t_d_w[np.newaxis].T @ d_L_d_t[np.newaxis]
d_L_d_b = d_L_d_t * d_t_d_b
d_L_d_inputs = d_t_d_inputs @ d_L_d_t
要计算权重、偏置和输入对于损失的梯度，下面这个公式可以将这些变量和上面我们计算 t 变量建立关系，t 就是这些变量的函数
t
=
w
∗
i
n
p
u
t
+
b
t = w * input +b
t = w ∗ in p u t + b
∂
t
∂
w
=
i
n
p
u
t
∂
t
∂
b
=
1
∂
t
∂
i
n
p
u
t
=
w
\frac{\partial t}{\partial w} = input\\ \frac{\partial t}{\partial b} = 1\\ \frac{\partial t}{\partial input} = w\\
∂ w ∂ t = in p u t ∂ b ∂ t = 1 ∂ in p u t ∂ t = w
接下来根据链式法则将权重、偏置和输入梯度进行整理
∂
L
∂
w
=
∂
L
∂
o
u
t
∂
o
u
t
∂
t
∂
t
∂
w
∂
t
∂
b
=
∂
L
∂
o
u
t
∂
o
u
t
∂
t
∂
t
∂
b
∂
t
∂
i
n
p
u
t
=
∂
L
∂
o
u
t
∂
o
u
t
∂
t
∂
t
∂
i
n
p
u
t
\frac{\partial L}{\partial w} = \frac{\partial L}{\partial out} \frac{\partial out}{\partial t} \frac{\partial t}{\partial w} \\ \frac{\partial t}{\partial b} = \frac{\partial L}{\partial out} \frac{\partial out}{\partial t} \frac{\partial t}{\partial b} \\ \frac{\partial t}{\partial input} = \frac{\partial L}{\partial out} \frac{\partial out}{\partial t} \frac{\partial t}{\partial input} \\
∂ w ∂ L = ∂ o u t ∂ L ∂ t ∂ o u t ∂ w ∂ t ∂ b ∂ t = ∂ o u t ∂ L ∂ t ∂ o u t ∂ b ∂ t ∂ in p u t ∂ t = ∂ o u t ∂ L ∂ t ∂ o u t ∂ in p u t ∂ t
首先，可以预计算 d_L_d_t 这是因为在计算权重、偏置和输入的梯度时候都会用到。
d_L_d_w 应该是 2 维矩阵，维度应该是 i n p u t × n o d e s input \times nodes in p u t × n o d es d_t_d_w 和d_L_d_t 都是 1 维，所以用 np.newaxis 为这两向量增加一个维度为，也就是 ( i n p u t , 1 ) (input,1) ( in p u t , 1 )
d_L_d_b
d_L_d_inputs 我们在看看其维度，由于权重的维度为 ( i n p u t , n o d e s ) (input,nodes) ( in p u t , n o d es ) ( n o d e s , 1 ) (nodes,1) ( n o d es , 1 ) i n p u t \j l e n input \j len in p u t \j l e n
class Softmax
def backprop (self, d_L_d_out, learn_rate ):
for i, gradient in enumerate (d_L_d_out):
if gradient == 0 :
continue
t_exp = np.exp(self.last_totals)
S = np.sum (t_exp)
d_out_d_t = -t_exp[i] * t_exp / (S ** 2 )
d_out_d_t[i] = t_exp[i] * (S - t_exp[i]) / (S ** 2 )
d_t_d_w = self.last_input
d_t_d_b = 1
d_t_d_inputs = self.weights
d_L_d_t = gradient * d_out_d_t
d_L_d_w = d_t_d_w[np.newaxis].T @ d_L_d_t[np.newaxis]
d_L_d_b = d_L_d_t * d_t_d_b
d_L_d_inputs = d_t_d_inputs @ d_L_d_t
self.weights -= learn_rate * d_L_d_w
self.biases -= learn_rate * d_L_d_b
return d_L_d_inputs.reshape(self.last_input_shape)
由于这部比较难，自己可能分享不够透彻，如果感觉有疑问地方还希望大家多多留言以便共同讨论。
版权声明：本文内容由互联网用户自发贡献，该文观点仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 举报，一经查实，本站将立刻删除。https://bianchenghao.cn/20300.html
