MOCO： Momentum Contrast for Unsupervised Visual Representation Learning

動機：

Unsupervised representation learning is highly successful in natural language processing，but supervised pre-training is still dominant in computer vision. The reason may stem from differences in their respective signal spaces, Language tasks have discrete signal spaces, Computer vision, in contrast, as the raw signal is in a continuous, high-dimensional space and is not structured
【CC】無監督在NLP領域大獲成功，但在CV領域沒啥動靜，大佬認為可能是兩個領域的資訊空間差異比較大：NLP是離散化的/低維度的結構化資訊，CV是連續的/高維度非結構化資訊

意義：

These results show that MoCo largely closes the gap between unsupervised and supervised representation learning in many computer vision tasks
【CC】本文的方法在CV大幅抹平監督-無監督的GAP

前置知識- 對比學習 as dictionary look-up
Though driven by various motivations, these methods can be thought of as building dynamic dictionaries. The“keys” (tokens) in the dictionary are sampled from data (e.g., images or patches) and are represented by an encoder network. Unsupervised learning trains encoders to perform dictionary look-up: an encoded “query” should be similar to its matching key and dissimilar to others. Learning is formulated as minimizing a contrastive loss.
【CC】大佬認為對比學習的本質是“構造字典-查字典”. 一個key，類比NLP的token，是由一個encoder（學習出來的NN網路）從一幅圖片或者圖片的一部分編碼而成. 假設這個encoder已經訓練好了，現在來做“查字典”：給定已經編碼好的一條“query”（即待確認的一副圖片），該“query”應該跟正樣本的距離更近而跟負樣本的距離更遠（很像triplet loss）. 整個程序就是學習一個encoder使得這個contrastive loss最小

Contrastive learning [29], and its recent developments, can be thought of as training an encoder for a dictionary look-up task Consider an encoded query q and a set of encoded samples {k0, k1, k2, …} that are the keys of a dictionary. Assume that there is a single key (denoted as k+) in the dictionary that q matches. A contrastive loss [29] is a function whose value is low when q is similar to its positive key k+ and dissimilar to all other keys (considered negative keys for q). With similarity measured by dot product, a form of a contrastive loss function, called InfoNCE [46], is considered in this paper:

where τ is a temperature hyper-parameter
【CC】query q是待確認的圖片編碼， {k0, k1, k2, …} 是整個字典. 假定在字典中一定存在一個k+跟q匹配（實際這個k+就是通過圖片樣本經過資料增強得到的，所以字典中一定有一個k+，這是pretext任務完成的）. contrastive loss 目標函式使得：q跟k+的距離很近，跟其他負樣本很遠. 這里引入NCE這個目標函式形式就是一個softmax函式，只不過分母中包含的是一個正樣本+K個負樣本，就像一個K+1的分類的CE

In general, the query representation is q = fq(xq) where fq is an encoder network and xq is a query sample (likewise, k = fk(xk)).
【CC】fq就是正樣本的encoder，xq是正樣本的編碼；fk是字典的encoder， xk是字典的編碼

創新點：

From this perspective, we hypothesize that it is desirable to build dictionaries that are: (i) large and (ii) consistent as they evolve during training. Intuitively, a larger dictionary may better sample the underlying continuous, high dimensional visual space, while the keys in the dictionary should be represented by the same or similar encoder so that their comparisons to the query are consistent.
【CC】一個好的字典要滿足兩個條件： 1）字典越大越好 2）字典編碼要有一致性

We maintain the dictionary as a queue of data samples: the encoded representations of the current mini-batch are enqueued, and the oldest are dequeued. The queue decouples the dictionary size from the mini-batch size, allowing it to be large. Moreover, as the dictionary keys come from the preceding several mini-batches, a slowly progressing key encoder, implemented as a momentum-based moving average of the query encoder, is proposed to maintain consistency. Our hypothesis is that good features can be learned by a large dictionary that covers a rich set of negative samples, while the encoder for the dictionary keys is kept as consistent as possible despite its evolution.
【CC】通過queue來維護字典：當前mini-batch的編碼進佇列，最老的一批出佇列，這么做queue的size（即字典大小）跟mini-batch的size解耦了，這樣可以把字典做很大，另一個好處是可以通過動量更新的方式去更新encoder，這樣就保持了一定的編碼一致性，說白了，就是針對上面兩個必要條件做優化

Dictionary as a queue
At the core of our approach is maintaining the dictionary as a queue of data samples. The introduction of a queue decouples the dictionary size from the mini-batch size. Our dictionary size can be much larger than a typical mini-batch size, and can be flexibly and independently set as a hyper-parameter.
【CC】這里不用翻譯了，還是在講queue的好處

Momentum update
We hypothesize that such failure is caused by the rapidly changing encoder that reduces the key representations’ consistency. Formally, denoting the parameters of fk as θk and those of fq as θq, we update θk by:

Here m ∈ [0, 1) is a momentum coefficient. As a result, though the keys in the queue are encoded by different encoders (in different mini-batches), the difference among these encoders can be made small.
【CC】首先，我們的encoder fq/fk是NN網路, 其引數為θq/ θk. 這里通過式子（2）來更新θk, 即上一個時刻的θk加上θq更新當前的θk，說白了就是一個簡單的線性權重相加（非常像遺傳演算法里面的啟發函式），這里的m超參

框架&實作

Momentum Contrast (MoCo) trains a visual representation encoder by matching an encoded query q to a dictionary of encoded keys using a contrastive loss. The dictionary keys {k0, k1, k2, …} are defined on-the-fly by a set of data samples.The dictionary is built as a queue, with the current mini-batch enqueued and the oldest mini-batch dequeued, decoupling it from the mini-batch size. The keys are encoded by a slowly progressing encoder, driven by a momentum update with the query encoder.This method enables a large and consistent dictionary for learning visual representations.
【CC】contrastive loss是上面的NCE，整個流程就如上圖：xq通過encoder fq得到token q，跟字典里面的各種xk做 contrastive loss. xk的encoder是通過式子（2）的方式對fk進行更新

Code: https://github.com/facebookresearch/moco
【CC】偽代碼和實作都在上面了，福利啊

• Pretext Task

we use a simple one mainly following the instance discrimination task. Following [61], we consider a query and a key as a positive pair if they originate from the same image, and otherwise as a negative sample pair.
【CC】這里使用了非常原始的instance discrimination task：當前圖片通過資料增強（就是簡單的corp）得到對應的正樣本，資料集中其他圖片都是負樣本

• encoder

We adopt a ResNet as the encoder,whose last fully-connected layer (after global average pooling) has a fixed-dimensional output (128-D [61]). This output vector is normalized by its L2-norm. This is the representation of the query or key.
【CC】encoder網路直接使用resnet，輸出的矢量（128D）L2正則化即為編碼

• Shuffling BN

The model appears to“cheat” the pretext task and easily finds a low-loss solution. For the key encoder fk, we shuffle the sample order in the current mini-batch before distributing it among GPUs (and shuffl back after encoding); the sample order of the mini-batch for the query encoder fq is not altered.
【CC】因為BN容易產生static info的資訊泄露，而模型會通過泄露的資訊去cheat（找到梯度下降最快的路徑）. Mini-batch內打亂字典圖片的順序，查詢圖片的順序不變

比較其他對比學習

Conceptual comparison of three contrastive loss mechanisms (empirical comparisons are in Figure 3 and Table 3). Here we illustrate one pair of query and key. The three mechanisms differ in how the keys are maintained and how the key encoder is updated.
(a): The encoders for computing the query and key representations are updated end-to-end by back-propagation (the two encoders can be different).
(b): The key representations are sampled from a memory bank .
?: MoCo encodes the new keys on-the-fly by a momentum-updated encoder, and maintains a queue (not illustrated in this figure) of keys.
【CC】end-to-end的方式比較native一點，coder q和k 都是通過反向傳播自行訓練， 可以預料到每個batch間的encoder方式不一樣效果不好； memory bank的方式必然收到記憶體大小的限制； moco就不說了， 其實本文還有后續研究， 發現通過moco方式訓練出來的represetation跟監督方式訓練出來的表達差別非常大：比如在后面fine-tune時，moco需要特別大的學習率，比如30