Affinity Enhanced Image-Specific Prototypes for Weakly Supervised Semantic Segmentation

I. INTRODUCTION

The goal of weakly supervised semantic segmentation (WSSS) is to learn how to generate pixel-level labels from limited supervision, usually in the form of image-level class labels^[2]. The introduction of the Class Activation Map (CAM)^[3] was a significant advancement towards achieving this goal, as it provides a means of generating pixel-level per-class scores based on image classification. However, it has been observed that meaningful CAM scores are often only assigned to a selective number of the most discriminative pixels, leading to limitations in directly using CAM as a segmentation solution.

Nonetheless, CAM proves to be a highly efficient technique for utilizing image-level annotations to make pixel-level predictions. It has frequently served as a base upon which multiple methods have been proposed to enhance and optimize the acquisition of pixel-level class probabilities.

 One approach is to erase^[5] or suppress^[6] the more discriminative regions, further mine discrimenative pixels. Another approach is to assign the limited discriminative regions as seeds and expand them into full segmentation labels using conventional region growing algorithms^[7,8], based on the similarities of local pixel values. Further methods extended this approach by incorporating pixel adaptive refinement^[4], random walks on semantic features^[9], or multitask inference of displacement and class boundary^[9,10].

Many recent methods are based on self-supervised learning. A contrastive learning framework, with positive image pairs defined by pairing an image with its linear transform and negative pairs of different images, were applied in^[11,12]. Another approach uses network features to create a per-class feature prototype-based alternative score map, providing supervision to guide the network towards generating consistent features with pixel affinities and image-level class labels^[1]. Combining these methods with others has shown benefits, as seen in recent works^[11,12,1]. The improved CAM-like score maps generated by these methods are used to enhance pixel affinities and generate pseudo-labels^[13,10], which are used to train a fully supervised semantic segmentation network^[14].

In this paper, we propose a method to incorporate pixel-adaptive mask refinement (PAMR)^[4] so that pixel affinity is maximized when generating score maps within the self-supervised image-specific prototype exploration (SIPE) method^[1]. Experimental results demonstrate that our proposed method provides substantial improvements over the baseline method SIPE. We also propose additional modifications that further improve quantitative results.

Fig. 1.

We enhance pixel-level affinity when generating image-specific CAM (IS-CAM) within the self-supervised image-specific prototype exploration (SIPE)[1]

II. RELATED WORK

Using image-level labels in WSSS tasks has the advantage of lower label generation burden compared to tasks using other labels. Consequently, research on learning image segmentation using image-level labels is actively progressing. Most existing techniques apply the CAM to generate semantic pseudo-masks. However, the conventional use of CAM in WSSS is limited to representing only distinctive parts. To address this limitation in WSSS, various methods have been proposed, such as Growing Seed Regions with Constraints, Erasing, Self-supervised Manner, and Prototyping.

Growing Seed Regions with Constraints Growing Seed Regions with Constraints involves expanding regions around seeds that represent the object's location and refining them to predict pixel-wise labels that closely match the actual class object. In SEC^[7], seeds are used to expand regions, and a Conditional Random Field (CRF) is employed to explore object boundaries based on the probability of object existence and color information per pixel. This process aims to predict masks close to the actual objects.

Erasing Conventional CAM represents only the most distinctive parts of an image. To broaden the CAM area, the Erasing method involves regenerating CAM by covering the initially created CAM area with masked images. This method, known as Adversarial Erasing (AE)^[5], expands the expected object area.

Self-Supervised Manner Most WSSS techniques utilize only one pre-processed image for learning. The Self-supervised Equivariant Attention Mechanism (SEAM)^[11] improves performance by using CAM results obtained from both the original and transformed images together.

Prototype Several techniques, such as Pixel-to-Prototype Contrast (PPC)^[12], SIPE^[1], define CAM-based prototypes for each class and utilize them for learning. PPC^[12], based on the SEAM^[11] technique, defines prototypes for each class using the top CAM-scored pixels, enhancing SEAM^[11] performance. Regional Semantic Contrast and Aggregation (RCA)^[25] defines class prototypes based on CAM and uses a memory bank containing training information from the entire dataset to improve CAM. SIPE ^[1] defines prototypes for each class per image, improving IS-CAM generation from conventional CAM. Defining prototypes for each class allows recognizing features not captured by CAM and removing noise.

Research utilizing prototypes for WSSS is actively progressing, and this paper enhances performance by generating improved prototypes based on SIPE^[1].

III. PROPOSED METHOD

1. Framework

A visual summary of the proposed method is presented in Fig. 2. The baseline method, SIPE^[1], comprises 1) the encoder module which generates pixel-level features, 2) the image classification module, which provides image-level supervision and generates the CAM, 3) the structure analysis module, which generates semantic structure seeds, and 4) the prototype module, which generates image-specific per-class prototype features and pixel-level per-class scores, denoted as image-specific CAM (IS-CAM). We note that we are using our own terminology, which we believe provides a more intuitive understanding of the framework.

Fig. 2.

Visual summary of the proposed method. We build upon the previous framework of self-supervised image-specific prototype exploration (SIPE) method[1] to enhance prototypes based on affinity using pixel-adaptive mask refinement (PAMR)[4], leading to substantial improvements in quantitative evaluations

In the proposed method, we incorporate the affinity enhancement (AE) module to the framework. In the AE module, a refined IS-CAM is generated, which is then used to generate refined region seeds, which are used to refine the prototypes and generate an improved IS-CAM. To aid the description of the AE module in 2.2, we provide a brief summary of the modules of SIPE^[1] as follows:

Encoder comprises a backbone CNN, pre-trained on image classification. The feature tensor generated from this encoder ε for the input image J is denoted as F=ε(J), and each feature vector at grid coordinate (i, j) is denoted as f_ij.

Classification comprises a layer to compute the CAM, and the global average pooling layer to connect the CAM with the image-level supervision through the classification loss.

Structure analysis relates the spatial distribution of features to the CAM to create seeds for prototypes. The pixel-wise spatial structure of grid coordinate (i, j) is first defined as S_ij = ReLU(S_C(f_ij,F)), with the cosine similarity function S_C(ㆍ) being broadcast for the elements of F. This is then compared to CAM to determine the semantic structure seed label, $$ {\mathrm{S}S}_{ij}=\underset{k}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{I}\mathrm{o}\mathrm{U}}\left(S_{ij},{\mathcal{M}}^k\right)$$denoting intersection-over-union with the M^k for the kth class.

Prototypes p^k are generated as $$ p^k=\frac{1}{\left|{\mathcal{S}}^k\right|}{\sum }_{\left(i,j\right)\in {\delta }^k}{f}_{ij}$$ denotes the set of coordinates with $$ {\mathcal{S}}^k=\left\{\left(\mathrm{i},\mathrm{j}\right)|{\mathrm{S}\mathrm{S}}_{ij}==\mathrm{k}\right\}$$ That is, p^k is the mean of the features with seed label k. The IS-CAM $$ {\tilde{\mathcal{M}}}^{\mathrm{k}}$$ is defined as $$ {\tilde{\mathcal{M}}}_{\mathrm{i}\mathrm{j}}^{\mathrm{k}}=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{S}\mathrm{c}\left({\mathrm{f}}_{\mathrm{i}\mathrm{j}},{\mathrm{p}}^{\mathrm{k}}\right)\right)$$ .

Training loss comprises the classification loss, defined as the cross-entropy between ground truth and the inferred image-level labels, and the general-specific consistency (GSC) loss, defined as the pixel-level L1 distance between the initial CAM M^k and refined IS-CAM $$ {\tilde{\mathcal{M}}}^{\mathrm{k}}$$, for all classes k.

Ⅳ. EXPERIMENTS

2. Comparative Evaluation

Quantitative evaluation results are summarized in Table 1. The incorporation of the AE module alongside the introduced modifications leads to improvements of 2.2% and 1.4% points over the baseline^[1] on the validation and test sets, respectively. Qualitative comparisons against the baseline are depicted in Figures 1 and 3. These examples illustrate instances where the proposed method yields enhanced segmentations by more effectively distinguishing both the background and the semantic classes of foreground objects.

Table 1.

Comparative evaluation of proposed AE-SIPE with SOTA on PASCAL VOC 2012 dataset. Models that rely only on image-level supervision are included for fair comparison

Fig. 3.

Qualitative results of segmentation labels for sample images of the PASCAL VOC 2012 dataset for the baseline SIPE[1] and the proposed method

3. Ablative Study

In this section, we delve into the specific effects of each proposed components: AE, structure analysis with HF, IS-CAM normalization, and background rescaling, as part of ablative analysis, presented in Table 2. Our observations highlight that the primary improvements from the AE module, with marginal enhancements arising from supplementary modifications. We also provide results from various combinations of PAMR, prototype generation, and IS-CAM generation, which constitute the submodules of the AE module, in Table 3. Notably, iterations of PAMR or prototype, and IS-CAM generation did not consistently yield improvements. The optimal results were achieved through the proposed AE module.

Table 2.

Ablation performance (mIoU %) of the baseline IS-CAM and AE-IS-CAM of the proposed method on the PASCAL VOC 2012 train set, refined by [13]

Table 3.

Comparison of various combinations of PAMR, prototype (Pr) and IS-CAM (IS) comprising affinity enhancement (AE) on the PASCAL VOC 2012 train set, refined by [13].

Ⅴ. DISCUSSION

In the case of AE-SIPE, the utilization of the conventional prototype generation technique serves to eliminate less significant areas, while simultaneously leveraging the PAMR^[4] that utilizes the RGB characteristics of prominent areas. These complementary operations effectively enhance performance.

In the comparative evaluation in Table 1, highlights that the RecurSeed method^[23] attains the highest performance. This method employs PAMR^[4] for refining pseudo-labels, coupled with a self-correlation map generation(SCG) module^[24]. Coincidentally, this SCG process, initially proposed for weakly supervised object localization, bears resemblance to the structure analysis module in SIPE^[1]. Upon Further comparison revealed that while prototypes are used to generate pseudo-semantic segmentation labels in the proposed AE-SIPE, RecurSeed employs a decoder to infer these pseudo-labels. Additionally, while iterations improve results in RecurSeed, they do not consistently do so in the proposed method.

 Also, the impact of AE is not always beneficial. When the initial Seed Region misclassifies areas outside the actual class existence region as the class, and these areas are extensive, there is a tendency for AE to further expand the erroneously predicted areas. Examining the sofa class in the third row of Fig. 3, it is evident that the initial Seed Region misclassified the gift box area as the sofa class. Through AE, the misclassified area as a sofa is expanded, resulting in a broader misclassification area compared to the original SIPE^[1]. Additional techniques should be considered to address and improve this phenomenon.

We believe that the prototype approach offers simplicity, while the decoder approach may offer greater capacity. There exists a relative scarcity of works that explicitly address the decoder structure within the self-supervised framework for WSSS, warranting further research. Additionally, we aim to identify refinement processes amenable to iteration for enhanced performance improvements.

Acknowledgments

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2021R1A2C2095452)

References

• Q. Chen, L. Yang, J. Lai, and X. Xie, "Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation," in CVPR, pp. 4278-4288, 2022. [https://doi.org/10.48550/arXiv.2203.02909]
• S. Hong, S. Kwak, and B. Han, "Weakly supervised learning with deep convolutional neural networks for semantic segmentation: Understanding semantic layout of images with minimum human supervision," IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 39-49, 2017. [https://doi.org/10.1109/MSP.2017.2742558]
• B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, "Learning deep features for discriminative localization," in CVPR, pp. 2921-2929, 2016. [https://doi.org/10.48550/arXiv.1512.04150]
• N. Araslanov and S. Roth, "Single-stage semantic segmentation from image labels," in CVPR, pp. 4252-4261, 2020 . [https://doi.org/10.48550/arXiv.2005.08104]
• Y. Wei, J. Feng, X. Liang, M.-M. Cheng, Y. Zhao, and S. Yan, "Object region mining with adversarial erasing: A simple classification to semantic segmentation approach," in CVPR, pp. 6488-6496, 2017. [https://doi.org/10.48550/arXiv.1703.08448]
• B. Kim, S. Han, and J. Kim, "Discriminative region suppression for weakly-supervised semantic segmentation," AAAI, vol. 35, no. 2, pp. 1754-1761, May 2021. [https://doi.org/10.48550/arXiv.2103.07246]
• A. Kolesnikov and C.H. Lampert, "Seed, expand and constrain: Three principles for weakly-supervised image segmentation," in ECCV, pp. 695&711, 2016. [https://doi.org/10.48550/arXiv.1603.06098]
• Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang, "Weakly-supervised semantic segmentation network with deep seeded region growing," in CVPR, pp. 7014-7023, 2018. [https://doi.org/10.1109/CVPR.2018.00733]
• J. Ahn and S. Kwak, "Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation," in CVPR, pp. 4981-4990, 2018. [https://doi.org/10.48550/arXiv.1803.10464]
• J. Ahn, S. Cho, and S. Kwak, "Weakly supervised learning of instance segmentation with inter-pixel relations," in CVPR, June 2019. [https://doi.org/10.48550/arXiv.1904.05044]
• Y. Wang, J. Zhang, M. Kan, S. Shan, and X. Chen, "Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation," in CVPR, pp. 12272-12281, 2020. [https://doi.org/10.48550/arXiv.2004.04581]
• Y. Du, Z. Fu, Q. Liu, and Y. Wang, "Weakly supervised semantic segmentation by pixel-to-prototype contrast," in CVPR, pp. 4310-4319, 2022. [https://doi.org/10.48550/arXiv.2110.07110]
• P. Kr¨ahenb¨uhl and V. Koltun, "Efficient inference in fully connected crfs with gaussian edge potentials," in NeurIPS, vol. 24, 2011. [https://doi.org/10.48550/arXiv.1210.5644]
• L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A.L. Yuille, "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs," IEEE Trans. PAMI, vol. 40, no. 4, pp. 834-848, 2018. [https://doi.org/10.48550/arXiv.1606.00915]
• C. Tomasi and R. Manduchi, "Bilateral filtering for gray and color images," in ICCV, pp. 839-846, 1998. [https://doi.org/10.1109/ICCV.1998.710815]
• K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in CVPR, pp. 770&778, 2016. [https://doi.org/10.48550/arXiv.1512.03385]
• L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, "Rethinking atrous convolution for semantic image segmentation," 2017. [https://doi.org/10.48550/arXiv.1706.05587]
• B. Hariharan, P. Arbel`aez, L. Bourdev, S. Maji, and J. Malik, "Semantic contours from inverse detectors," in ICCV, pp. 991-998, 2011. [https://doi.org/10.1109/ICCV.2011.6126343]
• J. Lee, E. Kim, and S. Yoon, "Anti-adversarially manipulated attributions for weakly and semi-supervised semantic segmentation," in CVPR, pp. 4070-4078, 2021. [https://doi.org/10.48550/arXiv.2103.08896]
• H. Kweon, S.-H. Yoon, H. Kim, D. Park, and K.-J. Yoon, "Unlocking the potential of ordinary classifier: Class-specific adversarial erasing framework for weakly supervised semantic segmentation," in ICCV, pp. 6974-6983, 2021. [https://doi.org/10.1109/ICCV48922.2021.00691]
• F. Zhang, C. Gu, C. Zhang, and Y. Dai, "Complementary patch for weakly supervised semantic segmentation," in ICCV, pp. 7222-7231, 2021. [https://doi.org/10.48550/arXiv.2108.03852]
• M. Lee, D. Kim, and H. Shim, "Threshold matters in wsss: Manipulating the activation for the robust and accurate segmentation model against thresholds," in CVPR, pp. 4320-4329, 2022. [https://doi.org/10.48550/arXiv.2203.16045]
• S. Jo, I.-J. Yu, and K. Kim, "Recurseed and edgepredictmix: Single-stage learning is sufficient for weakly supervised semantic segmentation," 2022. [https://doi.org/10.48550/arXiv.2204.06754]
• X. Pan, Y. Gao, Z. Lin, F. Tang, W. Dong, H. Yuan, F. Huang, and C. Xu, “Unveiling the potential of structure preserving for weakly supervised object localization,” in CVPR, pp. 11637-11646, 2021. [https://doi.org/10.48550/arXiv.2103.04523]
• Zhou, Tianfei, et al. "Regional semantic contrast and aggregation for weakly supervised semantic segmentation." In CVPR, 2022. [https://doi.org/10.48550/arXiv.2203.09653]