Decoupling Classifier for Boosting Few-shot Object Detection and Instance Segmentation

Main Results

Task1: Few-shot object detection

Task2: Few-shot instance segmentation

Disscussion

results visualization.

Visualization results of our method and the strong baseline (Mask-DeFRCN) on MS-COCO validation images under the gFSIS setting with K=10.

These bounding boxes and segmentation masks are visualized using scores larger than 0.6. The top two rows show success cases with our method and the baseline while the middle two rows show success cases with our method but partly failure ones with the baseline. The baseline may tend to incorrectly recognize positive object regions as background due to the biased classification. The bottom row shows some failure cases from left to right, small objects (e.g., the small boats and the person), coarse boundary segmentation (e.g., the surfer), occlusion (e.g., two bears are detected to one), and misclassification of similar appearance objects (e.g., the shadow of wine glass is recognized to wine glass and the train is detected to bus).

Reduce bias classification.

The decoupling classifier is helpful to mitigate the bias classification thus boosting the FSOD and FSIS performance. Comparison on mRecall and Recall of the proposed decoupling classifier (DC) and standard classification head (CE) under FSIS and gFSIS settings. The mean and standard deviation results are computed on all 10 seeds for each shot.

Core Code with Pytorch

The proposed decoupling classifier is very simple (core implementation only uses one line of code, Eq. 8) but really effective (e.g., 5.6+ AP50 improvements for 5-shot detection and 4.5+ AP50 improvements for 5-shot instance segmentation on challenging MS-COCO.)

def dc_loss(x, y, m):
	"""
	Compute loss for the decoupling classifier.
	Return scalar Tensor for single image.

	Args:
		x: predicted class scores in [-inf, +inf], x's size: N x (1+C), where N is the 
			number of region proposals of one image.
		y: ground-truth classification labels in [0, C-1], y's size: N x 1, where [0,C-1] 
			represent foreground classes and C-1 represents the background class.
		m: image-level label vector and its element is 0 or 1, m's size: 1 x (1+C)

	Returns:
		loss
	"""

	# background class index
	N = x.shape[0]
	bg_label = x.shape[1]-1

	# positive head
	pos_ind = y!=bg_label
	pos_logit = x[pos_ind,:]
	pos_score = F.softmax(pos_logit, dim=1) # Eq. 4
	pos_loss = F.nll_loss(pos_score.log(), y[pos_ind], reduction="sum") #Eq. 5

	# negative head
	neg_ind = y==bg_label
	neg_logit = x[neg_ind,:]
	neg_score = F.softmax(m.expand_as(neg_logit)*neg_logit, dim=1) #Eq. 8
	neg_loss = F.nll_loss(neg_score.log(), y[neg_ind], reduction="sum")  #Eq. 9

	# total loss
	loss = (pos_loss + neg_loss)/N #Eq. 6

return loss