需要 ImageNet-1k 資料集的來這篇博文: https://blog.csdn.net/qq_39377134/article/details/103128970
但是要準備好 240 GB 大小的磁盤空間哈，因為資料集壓縮包是 120 GB 多一些，
本文是關于 ResNet-50 在 ImageNet 上的實驗研究，目前的話，實驗資料集分別是 ImageNet-240 和 ImageNet-1k，其中前者是后者的一個子集，
接下來直接上實驗結果吧，第一次實驗，我是 freeze all layer exclude last layer，注意此處我加載了在 ImageNet-1k 上預訓練的模型，實驗結果是: train_acc = 93.8, val_acc = 93.44, test_acc = 93.48，
第二次實驗(這一次實驗加載的預訓練模型是加載第一次保存下來的模型，原因是我們要在第一次訓練完 last layer引數的情況下做微調，而不是直接 unfreeze layer4 和 fc layer，這樣會破壞預訓練學習到的資訊，這里還要提一下，微調的話，學習率最好是第一次訓練的 1/10)，我是 freeze all layer exclude layer4 and fc layer(也就是上面說的 last layer)，實驗結果是: train_acc = 95.24，val_acc = 93.6，test_acc = 93.7，對比第一次和第二次的實驗結果，我們可以發現在 val 和 test 上獲得了些許提升，但是從 train 可以看出開始過擬合了，
第三次實驗(這一次實驗加載的預訓練模型是加載第一次保存下來的模型)，我是 freeze all layer exclude layer3 and layer4 and fc layer，實驗結果是: train_acc = 95.81, val_acc = 93.67, test_acc = 93.9，對比第三次實驗和第一次實驗，可以發現，unfreeze 更多的網路層數，能略微提升準確率，但是也不太多吧，
對于過擬合，我說說我的看法吧，加大資料量可以緩解過擬合，但也僅僅只是緩解，除非你的資料集包含了所有現實情況，不然這個無法避免，我們能做的只是縮小 train_acc 和 val_acc 之間的 gap，
最后再說一下在 ImageNet-1k 上的 acc = 87.43，對比 ImageNet-240 和 ImageNet-1k 上的結果，我們可以發現，模型在 ImageNet-1k 上做 pre-train，然后 transfer 到 ImageNet-240 上，可以明顯提升模型效果，不過兩者都是同屬于一個 domain，我們需要更多的在不同 domain 上測驗 transfer 的效果才行，
對于資料量和模型泛化性的研究，有一篇論文寫的很好，Big Transfer (BiT): General Visual Representation Learning，該論文在 ImageNet-1k(1.28M 張圖片)，ImageNet-21k(14.2M 張圖片)，JFT-300M(300M 張圖片)，上分別實驗，發現資料量越大，效果越好，可以在 papers with code 上的 benckmark 查到 ResNet-50 在 ImageNet-1k 上的 test_acc = 77.15，但是在 JFT-300M 上做預訓練之后，再做 transfer 的話，可以達到 test_acc = 87.54，這里又不得不感慨，大力出奇跡，只是這次換成了資料集，，，，，
想了下，還是要放代碼的，我這里放一下 train.py:
import torch
from utils.eval import calc_acc
from utils.utils import setup_seed, get_dataloader, show_img, predict_batch, set_gpu
from utils.model import define_model, define_optim, start_train
from torch import nn
# 設定哪塊顯卡可見
device = set_gpu('0, 1')
# 設定亂數種子，使結果可復現
setup_seed(20)
# 資料讀取
train_batch = 160
test_batch = 160
EPOCH = 200
trainloader, testloader, classes = get_dataloader(train_batch, test_batch)
# 對訓練集的一個batch圖片進行展示
show_img(trainloader, classes, batch_size=train_batch)
# 網路結構定義
net = define_model(classes)
net = nn.DataParallel(net)
net.to(device)
# 定義優化器和損失函式
criterion, optimizer = define_optim(net)
net.load_state_dict(torch.load('resnetV1-50-9519-93.pth'))
# 開始模型訓練
start_train(net, EPOCH, trainloader, device, optimizer, criterion, testloader)
# 訓練后在測驗集上進行評測
net.load_state_dict(torch.load('resnet50Cls.pth'))
print(calc_acc(net, testloader, device))
# 進行模型預測
predict_batch(net, testloader, classes, test_batch, device)
完整代碼的話，我放到 Github了: https://github.com/MaoXianXin/PycharmProjects
補充論文解讀: Deep Residual Learning for Image Recognition，主要解決網路加深，模型優化困難問題，
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously.
解讀: 訓練深度神經網路是困難的，當然淺層的網路不困難，該篇論文提出殘差學習架構來解決該問題，
We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions.
解讀: 這里可以理解為引入殘差連接，
The depth of representations is of central importance for many visual recognition tasks.
Deep networks naturally integrate low/mid/high-level features and classifiers in an end-to-end multi-layer fashion, and the “levels” of features can be enriched by the number of stacked layers(depth).
解讀: 對于很多視覺識別任務來說，網路的深度是非常重要的，可以理解為分別提取到 low/mid/high 三種層次的特征，
Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers ? An obstacle to answering this question was the notorious problem of vanishing/exploding gradients, which hamper convergence from the beginning. This problem, however, has been largely addressed by normalized initialization and intermediate normalization layers, which enable networks with tens of layers to start converging for stochastic gradient descent(SGD) with back-propagation.
解讀: 對于驗證模型深度越深，網路的性能是否越好之前，這里還存在一個阻礙，就是梯度消失和爆炸，不過這個問題很大程度上可以通過正則初始化以及中間正則化層解決，
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error.
The degradation(of training accuracy) indicates that not all systems are similarly easy to optimize.
解讀: 在解決了網路的梯度消失和爆炸之后，我們的網路可以正常的收斂了，但是隨著網路加深，出現了準確率飽和以及快速退化的問題，并且這個問題不是由過擬合造成的，我們還可以這樣理解，對于一個適當深度的網路，如果你在這個基礎之上再添加層數，這個時候你不會獲得更好的性能，反而會得到更大的誤差，
We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping.
解讀: 這個假設是這篇論文的關鍵，也是提出 residual mapping 的立足點，
To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
疑惑: 還不是特別理解，待后面解決吧，目前的理解就是，優化添加了殘差連接的網路比直接優化堆疊起來的非線性層更容易，
In our case, the shortcut connections simply perform identity mapping, and their outputs are added to the outputs of the stacked layers. Identity shortcut connections add neither extra parameter nor computational complexity. The entire network can still be trained end-to-end by SGD with backpropagation, and can be easily implemented using common libraries.
解讀: 此處提到的 identity mapping 既不增加額外的引數，也不增加計算復雜度，
We evaluate our method on the ImageNet 2012 classification dataset that consists of 1000 classes. The models are trained on the 1.28 million training images, and evaluated on the 50k validation images. We also obtain a final result on the 100k test images, reported by the test server.
解讀: ResNet 模型是在 ImageNet 2012，可以稱為 ImageNet-1k 資料集上訓練的，有 1.28M 張訓練圖片，50k 張驗證圖片，以及 100k 張測驗圖片，
we also note that the 18-layer plain/residual nets are comparably accurate, but the 18-layer ResNet converges faster.
解讀: 從上圖，確實可以看出 ResNet-18 在初始階段收斂速度快于 Plain-18，
Bottleneck Architectures: a stack of 3 layers, 1x1, 3x3, and 1x1 convolutions, where the 1x1 layers are responsible for reducing and then increasing(restoring) dimensions leaving the 3x3 layer a bottleneck with smaller input/output dimensions.
The parameter-free identity shortcuts are particularly important for the bottleneck architectures. If the identity shortcut is replaced with projection, one can show that the time complexity and model size are doubled, as the shortcut is connected to the two high-dimensional ends. So identity shortcuts lead to more efficient models for the bottleneck designs.
解讀: 此處說的是 Bottleneck 的結構，兩端是 1x1，中間是 3x3，同時兩端的 dimension 大，中間的 dimension 小，可以節省引數量和計算量
The 50/101/152-layer ResNets are more accurate than the 34-layer ones by considerable margins.
解讀: 從上圖確實可以看出來 50/101/152 層的 ResNet 準確率比 34 層的高，
We also notice that the deeper ResNet has smaller magnitudes of responses, as evidenced by the comparisons among ResNet-20, 56, and 110. When there are more layers, an individual layer of ResNets tends to modify the signal less.
解讀: 此處展示不同網路的各個層的 magnitudes of responses
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/301329.html
標籤：其他
下一篇：2021-09-18