SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation

動機：

in this paper that predicts a 3D bounding box for each detected object by combining a single keypoint estimate with regressed 3D variables. As a second contribution, we propose a multi-step disentangling approach for constructing the 3D bounding box, which signifificantly improves both training convergence and detection accuracy.
【CC】首先對于3D BB位置預測放棄了2D BB的RPN，使用keypointde+3D變數回歸的方式. 其次，對于3D BB的構造采用解耦的多階段方式提升了訓練的便利性和精度

相關作業：

Previous state-of-the-art monocular 3D object detection algorithms [25, 1, 21] heavily depend on region-based convolutional neural networks (R-CNN) or region proposal network (RPN) structures [28, 18, 7]. Based on the learned high number of 2D proposals, these approaches attach an additional network branch to either learn 3D information or to generate a pseudo point cloud and feed it into point cloud-detection network.
【CC】老的方式都是基于先proposal一堆2D BB，然后要么 1）增加額外的網路層去學習3D資訊 要么2）生成偽點云然后塞到點云檢測網路

In this paper, we propose an innovative single-stage 3D object detection method that pairs each object with a single keypoint. We transform these variables together with projected keypoint to 8 corner representation of 3D boxes and regress them with a unifified loss function. The second contribution of our work is a multi-step disentanglement approach for 3D bounding box regression.
【CC】將目標檢測變成了key point的估計；3D-BB基礎表達為8個3D點，解耦去回歸3D-BB

形式化描述：

Given a single RGB image I ∈ R W×H×3, with W being the width and H being the height of the image, find for each present object its category label C and its 3D bounding box B, where the latter is parameterized by 7 variables (h, w, l, x, y, z, θ). Here, (h, w, l) represent
the height, weight, and length of each object in meters, and (x, y, z) is the coordinates (in meters) of the object center in the camera coordinate frame. Variable θ is the yaw orientation of the corresponding cubic box.
【CC】輸入圖片I，輸出類別C和3D-BB B；B表示為7維變數 (h, w, l, x, y, z, θ)， 其中(h, w, l)表示高/寬/長，(x, y, z) 相機坐標系下（其實就是自車坐標系）的中心點，θ表示航向角

網路架構：

Figure 2. Network Structure of SMOKE. We leverage DLA-34 [41] to extract features from images. The size of the feature map is 1:4 due to downsampling by 4 of the original image. Two separate branches are attached to the feature map to perform keypoint classification(pink) and 3D box regression (green) jointly. The 3D bounding box is obtained by combining information from two branches.
【CC】Backbone是DLA-34，1/4的下采樣；兩個Header分別是keypoint 分類/3D-BB回歸

Backbone

We use a hierarchical layer fusion network DLA-34 [41] as the backbone to extract features since it can aggregate information across different layers. Following the same structure as in [42], all the hierarchical aggregation connections are replaced by a Deformable Convolution Network (DCN). Compared with the original implementation, we replace all BatchNorm (BN) operation with GroupNorm (GN)
【CC】采用DLA-34進行不同層次的特征融合（類似FPD）；網路參考了centPoint論文做了些改變：分層連接改成了DCN；BN改成了GN

Keypoint Branch

We define the keypoint estimation network similar to [42] such that each object is represented by one specific keypoint.
【CC】參考centPoint論文，一個keypoint代表一個object

Let[x y z]? represent the 3D center of each object in the camera frame. The projection of 3D points to points [xc yc]? on the image plane can be obtained with the camera intrinsic matrix K in a homogeneous form:

【CC】這是個經典的3D世界到相機平面投影的公式，K是相機內參，更具體可以參考《slam十四講》

Each 3D box on the image is represented by 8 2D points[x_b,1～8 y_b,1～8]? and the standard deviation is computed by the smallest 2D box with {x_b_min, y_b_min, x_b_max, y_b_max} that encircles the 3D box.
【CC】同樣，將3D-BB的8個角點投影到相機平面得到8個2D的點，用[x_b,1～8 y_b,1～8]表示，該2D點在平面的標準差可以用 {x_b_min, y_b_min, x_b_max, y_b_max}來約束

Figure 3. Visualization of difference between 2D center points (red) and 3D projected points (orange). Best viewed in color.
【CC】上圖表明2D–BB中心點跟3D-BB中心點投射到2D后的點存在差異

Regression Branch:

The 3D information is encodedas an 8-tuple τ = [δz, δxc, δyc, δh, δw, δl, sin α, cos α]?. Here δz denotes the depth offset, δxc, δyc
is the discretization offset due to downsampling, δh, δw, δl denotes the residual dimensions, sin(α), cos(α) is the vectorial representation of the rotational angle α.
【CC】3D資訊通過8元組表達[δz, δxc, δyc, δh, δw, δl, sin α, cos α]?， δz深度偏置（參看公式2），δxc, δyc下采樣偏置（參看公式3），δh, δw, δl 變換后的表達（參看公式4），sin a/cos a 表示轉角更進一步表達θ（參看公式5）； 實際上這里的8元組通過變換可以得到原始的“3D-BB B表示為7維變數 (h, w, l, x, y, z, θ)”

For each object, its depth z can be recovered by pre-defined scale and shift parameters σz and μz as

【CC】深度z作為一個線性表達：σz為預定義的縮放因子，預定義的μz為偏置， δz為縮放因子下的偏置值

Given the object depth z, the location for each object in the camera frame can be recovered by using its discretized projected centroid [xc, yc]? on the image plane and the downsampling offset [δxc, δyc]?:

【CC】深度z由公式（2）給出，這里給出[xc, yc]?根據公式（3）計算得到[x, y ,z]； 實際上就是公式（1）的逆變換

In order to retrieve object dimensions[h w l]?, we use a pre-calculated category-wise average dimension[hˉ wˉ lˉ]? computed over the whole dataset. Each object dimension can be recovered by using the residual dimension offset [δh δw δl]?:

【CC】在整個資料集進行category得到h/w/l的縮放因子[hˉ wˉ lˉ]，然后dot上[δh δw δl]即得到3D空間的[h w l]

we choose to regress the observation angle α instead of the yaw rotation θ for each object. We further change the observation angle with respect to the object head αx, instead of the commonly used observation angle value αz, by simply adding π2.

Figure 4. Relation of the observation angle αx and αz. αx is provided in KITTI, while αz is the value we choose to regress
【CC】αx vs αz有固定的幾何關系-π2，而θ跟αz又有公式（5）的幾何關系，因此可以用αx來表達θ； 在訓練是回歸αx即是在回歸θ

Moreover, each α is encoded as the vector[sin(α) cos(α)]?. The yaw angle θ can be obtained by utilizing αz and the object location:

【CC】這里的αz可以用向量[sin(α) cos(α)]來表示，通過公式（5）計算得到θ

Finally, we can construct the 8 corners of the 3D bounding box in the camera frame by using the yaw rotation matrix Rθ, object dimensions[h w l]? and location[x y z]?:

【CC】這里是最后我們要回歸的3D BB的真實量，由公式（6）給出；這里明顯是一個3D的量（就跟后面Lreg函式對上了）

Loss Function

• Keypoint Classification Loss

Let si,j be the predicted score at the heatmap location (i, j) and yi,j be the ground-truth value of each point assigned by Gaussian Kernel. Define y?i,j and s?i,j as:

【CC】yi，j是高斯核函式關于每個點的真值函式值； si，j是熱力圖上每個點的預測得分

For simplicity, we only consider a single object class here. Then, the classification loss function is constructed as

where γ and β are tunable hyper-parameters and N is the number of keypoints per image. The term (1 ? yi,j )corresponds to penalty reduction for points around the groundtruth location.
【CC】整個公式（7）看起來像是一個CE函式；結合上面yi，j和si，j個人認為可以這么理解，y看做資料的真實分布, s看做網路對資料的預測分布；(1 ? yi,j )懲罰項，理解為在真值yi，j附近點得分越高，會導致Lcls越高

• Regression Loss:

We regress the 8D tuple τ to construct the 3D bounding box for each object. We also add channelwise activation to the regressed parameters of dimension and orientation at each feature map location to preserve consistency. The activation functions for the dimension and the orientation are chosen to be the sigmoid function σ and the ?2 norm, respectively:

【CC】 8D tuple τ本身經過網路有激活函式處理分別是：sigmoid和?2 norm，如上式

we define the 3D bounding box regression loss as the ?1 distance between the predicted transform B? and the groundtruth B:

where λ is a scaling factor.
【CC】整個回歸的Loss 就是一個簡單的L1距離； 當然它是3Dim的參看公式（6）

In Eq. (3), we use the projected 3D groundtruth points on the image plane[xc yc]? with the network predicted discretization offset[?δxc δ?yc]?
and depth z? to retrieve the location[x? y? z?]? of each object. In Eq. (5), we use the groundtruth location[x y z]? and the predicted observation angle ?αz to construct the estimated yaw orientation θ?.
【CC】這一段化其實就是Regression Branch開頭介紹的各個量之間的轉換關系，因為后面要歸納總的Loss func

The final loss function can be represented by:

where i represents the number of groups we define in the 3D regression branch.
【CC】整個總的Loss Func參見公式（9），就是簡單的線性相加

實作&Appollo擴展：

論文：https://github.com/lzccccc/SMOKE

Appollo7.0 其針對SMOKE的改進如下：
Here we mainly focus on the modifications based on SMOKE, more detail about SMOKE model please reference paper.

Deformable conv can not convert onnx or libtorch. Therefore, we modify the deformable convolution in the backbone to normal convolution, which will lead to the decline of mAP;
【CC】DCN不好實作，直接使用普通的Conv

Because the 3D center points of some obstacles may appear outside the image, these obstacles will be filtered out during training, resulting in missed detection. Therefore, we take the center point of 2D bounding boxes to represent the obstacle, and add a head prediction offset term to recover the 3D center point;
【CC】有肯能預測的3D中心點超出了圖片導致Obj檢測失敗；這里還是采用的2D BB的中心點作為Obj的中心點，加了一個header去估計2D BB中心點關于3D BB中心點的offset

We add the head to predict the width and height of the 2D bounding box, and directly calculate the 2D bbox of the obstacle with 2D center;
【CC】加了header去估計2D BB的[w, h]

Using 2D bounding box and other 3D information, we use post-processing geometric constraints to optimize the predicted position information. Firstly, we use the 3D information predicted by the model to calculate the 3D bounding box of the obstacle, as shown in Formula 1. θ \theta θ represents the rotation of obstacle， h , w , l h,w,l h,w,l is the dimensions and x , y , z x,y,z x,y,z represent location，

Then, according to the corresponding relationship between the bounding boxes as the constraint condition, we optimized the position information of the obstacle as shown in formula 2.

【CC】具體如何處理還得看APPOLO的代碼； 大體思路先做B的估計（Formula 1），然后做二次型優化（ formula 2）

The final network structure is shown below

重要參考文獻：

[41] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In CVPR, 2018.
[42] Xingyi Zhou, Dequan Wang, and Philipp Kr¨ahenb¨uhl. Objects as points. arXiv preprint arXiv:1904.07850, 2019.
https://github.com/ApolloAuto/apollo/blob/9f6bfa281999dc5f7592dea2ae870ee13e954ac3/modules/perception/camera/README.md