論文名稱：Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
論文下載地址：https://arxiv.org/abs/1606.00915
論文對應開源專案：http://liangchiehchen.com/projects/DeepLab.html

視頻講解：https://www.bilibili.com/video/BV1gP4y1G7TC

這是一篇2016年發布在CVPR上的文章，接著上一篇DeepLab V1網路簡介，本文對DeepLab V2網路進行簡單介紹，個人感覺相對DeepLab V1，DeepLab V2就是換了個backbone（VGG -> ResNet，簡單換個backbone就能漲大概3個點）然后引入了一個新的模塊ASPP（Atros Spatial Pyramid Pooling），其他的沒太大區別，

DCNNs應用在語意分割任務中問題

和上篇文章一樣，在文章的引言部分作者提出了DCNNs應用在語意分割任務中遇到的問題，

• 解析度被降低（主要由于下采樣stride>1的層導致）
• 目標的多尺度問題
• DCNNs的不變性(invariance)會降低定位精度

文中對應的解決辦法

• 針對解析度被降低的問題，一般就是將最后的幾個Maxpooling層的stride給設定成1(如果是通過卷積下采樣的，比如resnet，同樣將stride設定成1即可)，然后在配合使用膨脹卷積，

In order to overcome this hurdle and efficiently produce denser feature maps, we remove the downsampling operator from the last few max pooling layers of DCNNs and instead upsample the filters in subsequent convolutional layers, resulting in feature maps computed at a higher sampling rate. Filter upsampling amounts to inserting holes (‘trous’ in French) between nonzero filter taps.

• 針對目標多尺度的問題，最容易想到的就是將影像縮放到多個尺度分別通過網路進行推理，最后將多個結果進行融合即可，這樣做雖然有用但是計算量太大了，為了解決這個問題，DeepLab V2 中提出了ASPP模塊（atrous spatial pyramid pooling），具體結構后面會講，

A standard way to deal with this is to present to the DCNN rescaled versions of the same image and then aggregate the feature or score maps. We show that this approach indeed increases the performance of our system, but comes at the cost of computing feature responses at all DCNN layers for multiple scaled versions of the input image. Instead, motivated by spatial pyramid pooling, we propose a computationally efficient scheme of resampling a given feature layer at multiple rates prior to convolution. This amounts to probing the original image with multiple filters that have complementary effective fields of view, thus capturing objects as well as useful image context at multiple scales. Rather than actually resampling features, we efficiently implement this mapping using multiple parallel atrous convolutional layers with different sampling rates; we call the proposed technique “atrous spatial pyramid pooling” (ASPP).

• 針對DCNNs不變性導致定位精度降低的問題，和DeepLab V1差不多還是通過CRFs解決，不過這里用的是fully connected pairwise CRF，相比V1里的fully connected CRF要更高效點，在DeepLab V2中CRF漲點就沒有DeepLab V1猛了，在DeepLab V1中大概能提升4個點，在DeepLab V2中通過Table4可以看到大概只能提升1個多點了，

Our work explores an alternative approach which we show to be highly effective. In particular, we boost our model’s ability to capture fine details by employing a fully-connected Conditional Random Field (CRF) [22]. CRFs have been broadly used in semantic segmentation to combine class scores computed by multi-way classifiers with the low-level information captured by the local interactions of pixels and edges [23], [24] or superpixels [25]. Even though works of increased sophistication have been proposed to model the hierarchical dependency [26], [27], [28] and/or high-order dependencies of segments [29], [30], [31], [32], [33], we use the fully connected pairwise CRF proposed by [22] for its efficient computation, and ability to capture fine edge details while also catering for long range dependencies.

DeepLab V2的優勢

和DeepLab V1中寫的一樣：

• 速度更快
• 準確率更高（當時的state-of-art）
• 模型結構簡單，還是DCNNs和CRFs聯級

From a practical standpoint, the three main advantages of our DeepLab system are: (1) Speed: by virtue of atrous convolution, our dense DCNN operates at 8 FPS on an NVidia Titan X GPU, while Mean Field Inference for the fully-connected CRF requires 0.5 secs on a CPU. (2) Accuracy: we obtain state-of-art results on several challenging datasets, including the PASCAL VOC 2012 semantic segmentation benchmark [34], PASCAL-Context [35], PASCAL-Person-Part [36], and Cityscapes [37]. (3) Simplicity: our system is composed of a cascade of two very well-established modules, DCNNs and CRFs.

ASPP(atrous spatial pyramid pooling)

個人覺得在DeepLab V2中值得講的就這個ASPP模塊了，其他的都算不上啥亮點，這個ASPP模塊給我感覺就像是DeepLab V1中LargeFOV的升級版（加入了多尺度的特性），下圖是原論文中介紹ASPP的示意圖，就是在backbone輸出的Feature Map上并聯四個分支，每個分支的第一層都是使用的膨脹卷積，但不同的分支使用的膨脹系數不同（即每個分支的感受野不同，從而具有解決目標多尺度的問題），

下圖(b)有畫出更加詳細的ASPP結構（這里是針對VGG網路為例的），將Pool5輸出的特征層（這里以VGG為例）并聯4個分支，每個分支分別通過一個3x3的膨脹卷積層，1x1的卷積層，1x1的卷積層（卷積核的個數等于num_classes），最后將四個分支的結果進行Add融合即可， 如果是以ResNet101做為Backbone的話，每個分支只有一個3x3的膨脹卷積層，卷積核的個數等于num_classes（看原始碼分析得到的），

在論文中有給出兩個ASPP的配置，ASPP-S（四個分支膨脹系數分別為2,4,8,12）和ASPP-L（四個分支膨脹系數分別為6,12,18,24），下表是對比LargeFOV、ASPP-S以及ASPP-L的效果，這里只看CRF之前的（before CRF）對比，ASPP-L優于ASPP-S優于LargeFOV，

DeepLab V2網路結構

這里以ResNet101作為backbone為例，下圖是根據官方原始碼繪制的網路結構（這里不考慮MSC即多尺度），在ResNet的Layer3中的Bottleneck1中原本是需要下采樣的（3x3的卷積層stride=2），但在DeepLab V2中將stride設定為1，即不在進行下采樣，而且3x3卷積層全部采用膨脹卷積膨脹系數為2，在Layer4中也是一樣，取消了下采樣，所有的3x3卷積層全部采用膨脹卷積膨脹系數為4，最后需要注意的是ASPP模塊，在以ResNet101做為Backbone時，每個分支只有一個3x3的膨脹卷積層，且卷積核的個數都等于num_classes，

Learning rate policy

在DeepLab V2中訓練時采用的學習率策略叫poly，相比普通的step策略（即每間隔一定步數就降低一次學習率）效果要更好，文中說最高提升了3.63個點，真是煉丹大師，poly學習率變化策略公式如下：
l r × ( 1 ? i t e r m a x _ i t e r ) p o w e r lr \times (1 - \frac{iter}{max\_iter})^{power} lr×(1?max_iteriter?)power
其中 l r lr lr為初始學習率， i t e r iter iter為當前迭代的step數， m a x _ i t e r max\_iter max_iter為訓練程序中總的迭代步數，關于煉丹對比，見原文TABLE2：

消融實驗

下表是原論文中給出的一些消融實驗對比：

其中：

• MSC表示多尺度輸入，即先將影像縮放到0.5、0.7和1.0三個尺度，然后分別送入網路預測得到score maps，最后融合這三個score maps（對每個位置取三個score maps的最大值），

Multi-scale inputs: We separately feed to the DCNN images at scale = { 0.5, 0.75, 1 } , fusing their score maps by taking the maximum response across scales for each position separately.

• COCO就代表在COCO資料集上進行預訓練

Models pretrained on MS-COCO.

• Aug代表資料增強，這里就是對輸入的圖片在0.5到1.5之間隨機縮放，

Data augmentation by randomly scaling the input images (from 0.5 to 1.5) during training.

• LargeFOV是在DeepLab V1中講到過的結構，
• ASPP前面講過了
• CRF前面也提到過了

每個方法帶來的增益看原論文表述，這里不太敘述：

we evaluate how each of these factors, along with LargeFOV and atrous spatial pyramid pooling (ASPP), affects val set performance. Adopting ResNet-101 instead of VGG-16 significantly improves DeepLab performance (e.g., our simplest ResNet-101 based model attains 68.72%, compared to 65.76% of our DeepLab-LargeFOV VGG-16 based variant, both before CRF). Multiscale fusion [17] brings extra 2.55% improvement, while pretraining the model on MS-COCO gives another 2.01% gain. Data augmentation during training is effective (about 1.6% improvement). Employing LargeFOV (adding an atrous convolutional layer on top of ResNet, with 3×3 kernel and rate = 12) is beneficial (about 0.6% improvement). Further 0.8% improvement is achieved by atrous spatial pyramid pooling(ASPP). Post-processing our best model by dense CRF yields performance of 77.69%.