神經風格轉移

神經風格轉移(neural style transfer)中的風格(style)指的是影像中不同空間尺度的紋理、顏色和視覺圖案

神經風格轉移主要有兩個優化目標

原影像與生成影像之間內容損失差盡可能小
原影像與生成影像之間風格損失差盡可能小

內容損失

一般用兩個激活間的L2范數

一個激活是預訓練的卷積神經網路在更靠近頂部某層的影像計算得到的
另一個是同一層在生成影像上計算的激活

風格損失

在目標內容影像和生成影像之間保持相似的高層激活，從而能保留內容
在較低層和較高層的激活之間保持相互關系，從而能夠保持風格

使用Keras實作神經風格遷移

我們使用VGG19來實作這一功能，其程序如下

創建一個網路，能夠同時參考原始影像、目標影像以及VGG19影像
可以使用這三個影像計算激活層來定義之間的損失函式
設定梯度下降來使得整個程序的損失函式最小化

8-14 定義初始變數

from keras.preprocessing.image import load_img, img_to_array

# 想要變換的影像路徑
target_image_path = r'E:\code\PythonDeep\styleTransfer\ori1.jpg'
# 風格影像的路徑
style_reference_image_path = r'E:\code\PythonDeep\styleTransfer\objstyle.jpg'

# 生成影像的尺寸
width, height = load_img(target_image_path).size
img_height = 400
img_width = int(width * img_height / height) # 等比例縮放

Using TensorFlow backend.

8-15 輔助函式

import numpy as np
from keras.applications import vgg19

def preprocess_image(image_path):
    img = load_img(image_path, target_size=(img_height, img_width))
    img = img_to_array(img)
    img = np.expand_dims(img, axis=0)
    # 減去ImageNet的平均像素值，使其中心為0
    img = vgg19.preprocess_input(img)
    return img

def deprocess_image(x):
    # Remove zero-center by mean pixel
    x[:, :, 0] += 103.939
    x[:, :, 1] += 116.779
    x[:, :, 2] += 123.68
    # 將影像由BGR轉為RGB，這也是VGG19逆操作的一部分
    x = x[:, :, ::-1]
    x = np.clip(x, 0, 255).astype('uint8')
    return x

8-16 加載預訓練的VGG19網路并將其應用于三張圖

from keras import backend as K
target_image = K.constant(preprocess_image(target_image_path))
style_reference_image = K.constant(preprocess_image(style_reference_image_path))

# 這個占位符用于保存或者生成影像
combination_image = K.placeholder((1, img_height, img_width, 3))

# 將三張影像合并為1張影像
input_tensor = K.concatenate([target_image,
                              style_reference_image,
                              combination_image], axis=0) # 以行的方式疊加

# 將三張影像組成的batch作為輸入來構建VGG19網路
model = vgg19.VGG19(input_tensor = input_tensor,
                    weights = 'imagenet',
                    include_top = False)

print('Model loaded.')

Model loaded.

8-17 內容損失

# 計算內容損失
def content_loss(base, combination):
    return K.sum(K.square(combination - base))

8-18 風格損失

def gram_matrix(x):
    features = K.batch_flatten(K.permute_dimensions(x, (2, 0, 1)))
    gram = K.dot(features, K.transpose(features))
    return gram


def style_loss(style, combination):
    S = gram_matrix(style)
    C = gram_matrix(combination)
    channels = 3
    size = img_height * img_width
    return K.sum(K.square(S - C)) / (4. * (channels ** 2) * (size ** 2))

8-19 總變差損失

def total_variation_loss(x):
    a = K.square(
        x[:, :img_height - 1, :img_width - 1, :] - x[:, 1:, :img_width - 1, :])
    
    b = K.square(
        x[:, :img_height - 1, :img_width - 1, :] - x[:, :img_height - 1, 1:, :])
    
    return K.sum(K.pow(a + b, 1.25))

我們需要最小化損失的是這三項損失的加權平均，為了計算損失，我們只需一個頂部的層，即block5_conv2層

而對于風格損失，我們需要使用一系列的層，既包括頂層也包括底層，最后還要添加總變差損失

8-20 定義需要最小化的最終損失

# 將層的名稱映射為激活張量的字典
outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])

# 用于內容損失的層
content_layer = 'block5_conv2'

# 用于風格損失的層
style_layers = ['block1_conv1',
                'block2_conv1',
                'block3_conv1',
                'block4_conv1',
                'block5_conv1']

# 損失分量的加權平均所使用的權重
total_variation_weight = 1e-4
style_weight = 1.
content_weight = 0.025

# 添加內容損失
loss = K.variable(0.)
# 在定義損失時候將所有的分量都添加到這個標量變數中
layer_features = outputs_dict[content_layer]
target_image_features = layer_features[0, :, :, :]
combination_features = layer_features[2, :, :, :]
loss = loss + content_weight * content_loss(target_image_features,
                                      combination_features)
# 添加每個目標層的風格損失分量
for layer_name in style_layers:
    layer_features = outputs_dict[layer_name]
    style_reference_features = layer_features[1, :, :, :]
    combination_features = layer_features[2, :, :, :]
    sl = style_loss(style_reference_features, combination_features)
    loss = loss + (style_weight / len(style_layers)) * sl

# 添加總變差損失
loss = loss + total_variation_weight * total_variation_loss(combination_image)

8-21 設定梯度下降

# 獲得損失相對于生成影像的梯度
grads = K.gradients(loss, combination_image)[0]

# 用于獲取當前損失值和當前梯度值的函式
fetch_loss_and_grads = K.function([combination_image], [loss, grads])

# 這個類將fetch_loss_and_grads包裝起來
# 讓你可以使用兩個單獨的方法來獲取損失和梯度
class Evaluator(object):

    def __init__(self):
        self.loss_value = None
        self.grads_values = None

    def loss(self, x):
        assert self.loss_value is None
        x = x.reshape((1, img_height, img_width, 3))
        outs = fetch_loss_and_grads([x])
        loss_value = outs[0]
        grad_values = outs[1].flatten().astype('float64')
        self.loss_value = loss_value
        self.grad_values = grad_values
        return self.loss_value

    def grads(self, x):
        assert self.loss_value is not None
        grad_values = np.copy(self.grad_values)
        self.loss_value = None
        self.grad_values = None
        return grad_values

evaluator = Evaluator()

8-22 風格遷移回圈

from scipy.optimize import fmin_l_bfgs_b
import time
import imageio

result_prefix = 'style_transfer_result'
iterations = 20


# 這個是初始狀態：目標影像
# 將影像展平，因為scipy.optimize.fmin_1_bfgs_b只能處理展平的向量
x = preprocess_image(target_image_path)
x = x.flatten()
for i in range(iterations):
    print('Start of iteration', i)
    start_time = time.time()
    # 對生成的影像的像素進行L-BFGS優化
    # 將神經風格的損失最小化
    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x,
                                     fprime=evaluator.grads, maxfun=20)
    print('Current loss value:', min_val)
    
    # 保存當前生成的影像
    img = x.copy().reshape((img_height, img_width, 3))
    img = deprocess_image(img)
    fname = result_prefix + '_at_iteration_%d.png' % i
    
    # 這里同樣要改變保存的函式，與上一Part的相同
    imageio.imsave(fname, img)
    
    end_time = time.time()
    print('Image saved as', fname)
    print('Iteration %d completed in %ds' % (i, end_time - start_time))

Start of iteration 0
Current loss value: 572887550.0
Image saved as style_transfer_result_at_iteration_0.png
Iteration 0 completed in 137s
Start of iteration 1
Current loss value: 262687630.0
Image saved as style_transfer_result_at_iteration_1.png
Iteration 1 completed in 143s
Start of iteration 2
Current loss value: 188103090.0
Image saved as style_transfer_result_at_iteration_2.png
Iteration 2 completed in 146s
Start of iteration 3
Current loss value: 158912400.0
Image saved as style_transfer_result_at_iteration_3.png
Iteration 3 completed in 144s
Start of iteration 4
Current loss value: 137502610.0
Image saved as style_transfer_result_at_iteration_4.png
Iteration 4 completed in 144s
Start of iteration 5
Current loss value: 122368510.0
Image saved as style_transfer_result_at_iteration_5.png
Iteration 5 completed in 143s
Start of iteration 6
Current loss value: 111903220.0
Image saved as style_transfer_result_at_iteration_6.png
Iteration 6 completed in 144s
Start of iteration 7
Current loss value: 102213060.0
Image saved as style_transfer_result_at_iteration_7.png
Iteration 7 completed in 144s
Start of iteration 8
Current loss value: 95047980.0
Image saved as style_transfer_result_at_iteration_8.png
Iteration 8 completed in 144s
Start of iteration 9
Current loss value: 90722060.0
Image saved as style_transfer_result_at_iteration_9.png
Iteration 9 completed in 145s
Start of iteration 10
Current loss value: 85071980.0
Image saved as style_transfer_result_at_iteration_10.png
Iteration 10 completed in 144s
Start of iteration 11
Current loss value: 80578936.0
Image saved as style_transfer_result_at_iteration_11.png
Iteration 11 completed in 144s
Start of iteration 12
Current loss value: 74976456.0
Image saved as style_transfer_result_at_iteration_12.png
Iteration 12 completed in 143s
Start of iteration 13
Current loss value: 70854060.0
Image saved as style_transfer_result_at_iteration_13.png
Iteration 13 completed in 145s
Start of iteration 14
Current loss value: 68048904.0
Image saved as style_transfer_result_at_iteration_14.png
Iteration 14 completed in 144s
Start of iteration 15
Current loss value: 65703100.0
Image saved as style_transfer_result_at_iteration_15.png
Iteration 15 completed in 144s
Start of iteration 16
Current loss value: 63404410.0
Image saved as style_transfer_result_at_iteration_16.png
Iteration 16 completed in 144s
Start of iteration 17
Current loss value: 61337212.0
Image saved as style_transfer_result_at_iteration_17.png
Iteration 17 completed in 143s
Start of iteration 18
Current loss value: 59199800.0
Image saved as style_transfer_result_at_iteration_18.png
Iteration 18 completed in 143s
Start of iteration 19
Current loss value: 57377176.0
Image saved as style_transfer_result_at_iteration_19.png
Iteration 19 completed in 143s

# 畫圖
from matplotlib import pyplot as plt

# Content image
plt.imshow(load_img(target_image_path, target_size=(img_height, img_width)))
plt.figure()

# Style image
plt.imshow(load_img(style_reference_image_path, target_size=(img_height, img_width)))
plt.figure()

# Generate image
plt.imshow(img)
plt.show()