A Study on the Vulnerability of Semantic Segmentation Model to Data Transformation

1. Semantic Segmentation

In autonomous driving, semantic segmentation plays a crucial role in understanding road scenes. For example, semantic segmentation detects and classifies various semantic classes such as pedestrians, vehicles, and trees within an image, allowing for an understanding of the spatial relationships among scene components. This comprehension enables autonomous driving systems to effectively extract road information and make sophisticated decisions^[2].

To achieve a comprehensive understanding of semantic segmentation in road scenes, it is essential to have a dataset that includes a variety of road environments. Datasets capturing real-world road scenes, such as CityScapes^[4], CamVid^[5], KITTI^[6], IDD^[7] are primarily used for training semantic segmentation models. These road datasets contain scenes captured under diverse conditions, including varying weather, lighting, times of day, and traffic scenarios. Furthermore, the road datasets include critical classes: vehicles, persons, sidewalks, traffic lights, and traffic signs that must be identified while driving.

However, since these real datasets require a significant amount of time for pixel-level annotation tasks, it is challenging to build new training data for all road scenes. To address these limitations, there has been an increasing effort to utilize synthetic data created with computer graphics^[8,9,10] or to augment real-world data with generated objects^[11].

This paper applies style transfer techniques to both real and synthetic datasets to create a new dataset with generative characteristics. This new dataset is then used with a semantic segmentation model trained on real road datasets. The peculiarities of the generated data are identified by comparing the segmentation results between the existing datasets and the generated datasets.

2. Data Augmentation

Image data augmentation is a technique that generates multiple datasets from a single dataset to address overfitting issues caused by insufficient or imbalanced training data^[12]. With advancements in deep learning, methods utilizing image generation models and style transfer models^{[13,14,15,16]} have been developed to augment data.

Style transfer involves applying the style image to a content image to create a new image^[17]. Recently, vision-language models have achieved high performance, leading to active research into transforming the style of content images using only text, without requiring reference style images. Notable examples include the StyleCLIP^[15] and the CLIPstyler^[16] models, both of which utilize CLIP^[18], a text-image embedding model developed by OpenAI. The CLIPstyler exhibits superior transfer performance compared to StyleCLIP and has been utilized in this study.

Additionally, methods that use video game engines to render virtual environments similar to the real world are gaining attention for generating synthetic datasets^[8,9,10]. By combining basic blocks provided by game engines, new data can be easily generated, significantly enhancing the visual diversity of the dataset. For instance, Virtual KITTI^[9] uses the Unity game engine to create a large-scale virtual urban scene dataset for semantic segmentation.

Another method of augmentation is video compression. Video compression is a technique that reduces the redundancy existing in the data to transmit and store the information at a lower bit rate while maintaining its quality^[19]. When compressing videos, the quantization parameter, which determines the loss criteria and indicates the strength of the compression, can be used to create transformation datasets with varying qualities. During the video compression process, the internal distribution of data changes, resulting in data augmentation.

Applying data augmentation techniques in semantic segmentation allows for the generation of semantic annotations for augmented data without additional annotation efforts. This approach offers the advantage of acquiring large volumes of data at a low cost. Moreover, these methods maximize the efficiency of data augmentation and contribute to the development of models with improved performance.

1. Prompt Selection for Style Transfer Model

To augment the data, this study employs the CLIPstyler^[16] model, which facilitates style transfer using only text, to transform existing road datasets. The CLIPstyler provides various prompts such as ‘anime’, ‘stonewall’, ‘desert sand’, and ‘sketch black pencil’. The results of style transfer using these prompts are compared and analyzed to select a prompt that preserves the original image information while also imparting generative characteristics. As shown in Figure 2, the ‘sketch black pencil’ prompt produces grayscale images that lost the color information of the original images. For the ‘stonewall’ and ‘desert sand’ prompts, key classes such as vehicles and persons are rendered too blurred, making it difficult to identify the shapes in the original image. Consequently, it has been determined that semantic segmentation models may struggle to perform accurate segmentation. In contrast, the ‘anime’ prompt preserves the colors and shapes of objects while adding a distinct generative style. Therefore, this study selects the ‘anime’ prompt for the experiments.

Fig. 2.

Comparison of prompt results for CLIPstyler

2. Dataset Selection for the Experiment

To select datasets for the experiments, we compared and analyzed four real datasets—CityScapes^[4], CamVid^[5], KITTI^[6], and IDD^[7]—and three synthetic datasets—SYNTHIA-SF^[8], Virtual KITTI 2^[9], and GTAV^[10]. The results are summarized in Table 1. CityScapes, a benchmark dataset for semantic segmentation, is utilized in the ProtoSeg^[20], CTNet^[21], and SeaFormer^[22] models. KITTI, widely used for training autonomous driving systems, utilized 200 annotated images with semantic segmentation labels for this experiment. CamVid and SYNTHIA-SF are widely recognized road datasets used for training semantic segmentation models and are also included in the experiments. Specifically, CamVid is used for training with real-world data, while SYNTHIA-SF is utilized for training with synthetic data. GTAV, a dataset of urban scenes extracted from a video game, is partially adopted due to its realistic representation of day and night transitions.

Table 1.

Comparison between the road datasets

On the other hand, IDD contains unstructured road scenes compared to CityScapes, resulting in a significant decrease in segmentation accuracy when models trained on CityScapes are applied to IDD images. Furthermore, Virtual KITTI 2 lacks persons, which are a crucial class for road scenes. Consequently, both IDD and Virtual KITTI 2 are excluded from this study.

3. Models Comparison and Analysis

Semantic segmentation models can be categorized into fully convolutional network (FCN)-based models and vision transformer (ViT)-based models, depending on their backbone architecture. Traditional segmentation models utilize fully convolutional networks for feature extraction; however, with the advent of vision transformers, research has increasingly focused on these models, which have shown superior performance.

In this paper, we compare and analyze the latest FCN-based models, ProtoSeg^[20] and CTNet^[21], in conjunction with ViT-based models, SeaFormer^[22] and TopFormer^[23], to select the most suitable models for our experiments. Table 2 provides a summary of the key characteristics of each model.

Table 2.

A comparison of FCN based models and ViT based models

ProtoSeg, an FCN-based model, effectively captures diverse intra-class variations by utilizing multiple prototypes per class without requiring additional training. CTNet enhances semantic segmentation performance by exploring both spatial and semantic relationships between pixels and channels through its Channel Context Module and Spatial Context Module. On the other hand, ViT-based models like SeaFormer and TopFormer are optimized for mobile devices. SeaFormer captures rich contextual and spatial information through fusion blocks and SeaFormer layers, while TopFormer builds more robust hierarchical features by integrating spatial and semantic information through its Semantics Injection Module.

As shown in Table 2, SeaFormer is approximately three times more efficient than TopFormer in terms of floating point operations per seconds (FLOPs), making it the model used in this study. In addition, ProtoSeg and CTNet were selected to compare the performance of FCN-based semantic segmentation models on the style-transformed dataset generated by the CNN-based model ClipStyler.

CTNet does not follow the class color scheme of CityScapes but instead uses its own color system for segmentation results. Therefore, when comparing the segmentation results of CTNet with those of other models, such as ProtoSeg and SeaFormer, on the same images, the classes in CTNet’s results may be represented in colors that differ from the standard CityScapes palette.

1. Experimental Setup

The original batch sizes for the ProtoSeg and CTNet models were set at 8 and 16, respectively, while the SeaFormer model utilized 8 GPUs with a batch size of 2.

However, due to resource constraints in this study, the batch sizes for ProtoSeg and CTNet are reduced to 4, and the batch size for SeaFormer is set to 8. The parameters for iteration, loss function, and learning rate are all configured to align with those of the original models. The models were implemented using PyTorch. ProtoSeg and SeaFormer are trained on NVIDIA RTX 3080 GPUs, while CTNet is trained on Google Colab’s NVIDIA A100-SXM4 GPU.

As shown in Fig. 3, the experimental datasets include style-transferred and video compression datasets. The style transfer is performed using the prompt ‘anime.’ In addition, to perform video compression using the high efficiency video coding (HEVC) codec, each image from the experimental datasets is converted into a 5 second video. After compression, the videos exhibit characteristics of the YUV420 color space, 4:2:0 chroma subsampling, and an 8 bit depth. The value of QP is set to 10 when observing the performance of ProtoSeg, CTNet, and SeaFormer, and to 30, 40, and 50 when investigating the factors that affect the performance of ProtoSeg. The file sizes before and after compression are presented in Table 3. To apply the semantic segmentation model, the first frame of each video is extracted as an image.

Fig. 3.

Road datasets used in the experiment (the upper part shows the video compression, and the lower part shows the style transfer)

Table 3.

File sizes of datasets before and after video compression

2. Results Analysis

In this experiment, the CityScapes, KITTI, CamVid, SYNTHIA-SF, and GTAV datasets, along with their style transformed and video compressed datasets, are applied to ProtoSeg, CTNet, and SeaFormer to compare and analyze semantic segmentation performance, and the results are presented in Table 4. The bold and blue numbers in Table 4 mean the first and second highest performance results for each model, respectively. As seen in the results, the transformed datasets generally presented lower mean intersection over union (mIoU) performance compared to the original datasets. One notable point is the mIoU results of CTNet on the style transformed and video compressed CityScapes: When compared to the results of ProtoSeg and SeaFormer, CTNet shows the most significant mIoU variation between the original dataset and the transformed datasets. This suggests that CTNet is overfitted to the original CityScapes dataset and has difficulty adapting to the distribution changes caused by data transformations. Additionally, the performance of the SeaFormer on the style transfer datasets is significantly superior to that of ProtoSeg and CTNet. While ProtoSeg and CTNet show a considerable performance gap between the original and style transfer datasets, SeaFormer shows a relatively small performance difference of approximately 10% in the mIoU. This indicates that SeaFormer is more flexible in adapting to changes in data distribution. The key point to note in the performance results of video compression is the performance of CTNet on the GTAV and SeaFormer on the SYNTHIA-SF. The compressed GTAV and SYNTHIA-SF show slightly higher mIoU compared to the original data for CTNet and SeaFormer, respectively. This result suggests that the video compression process effectively preserved the essential visual features of the data, likely contributing to the performance improvements.

Table 4.

Quantitative results of the Experiment

The segmentation results for ProtoSeg, CTNet, and SeaFormer across each dataset can be observed in Figures 4 and 5. While ProtoSeg and CTNet perform relatively well on the original datasets, their performance significantly declines on the generated datasets. This decline can be attributed to overfitting to natural datasets, which results in an inability to adapt to the distributional differences present in the style transfer datasets. For instance, the segmentation results of ProtoSeg and CTNet on the CityScapes and KITTI datasets, as shown in Figure 5, reveal that different colors appear in the middle of the road, indicating a failure to accurately identify the ‘road’ class. Additionally, ProtoSeg does not classify the ‘road’ as ‘vegetation’ in the style transformed SYNTHIA-SF dataset, failing to recognize the structure of the original image. This shows the vulnerability of FCN-based models like ProtoSeg and CTNet when applied to generated datasets. Furthermore, although SeaFormer also shows reduced performance on the style transfer datasets compared to the original datasets, it demonstrates clear object segmentation capabilities in specific areas. In particular, in Figure 4, it effectively segments ‘buildings’ and ‘roads’ in SYNTHIA-SF, as well as grassy areas on the right sidewalk of GTAV.

Fig. 4.

Qualitative results of the style transfer datasets

Fig. 5.

Qualitative results of the original datasets

We conduct additional research to analyze the factors that cause performance differences and variations in sensitivity in ProtoSeg. We measure the mIoU of the ProtoSeg by setting various quantization parameters for video compression data. As seen in Table 4, the results for video compression with QP set to 10 demonstrate that a low compression rate preserves high data quality, allowing the model to accurately segment object boundaries. However, as shown in Table 5, the mIoU decreases with increasing QP values, recording 55.1 at QP 30, 34.4 at QP 40, and 7.3 at QP 50. This performance degradation occurs because detailed information in the data is gradually lost as the QP increases during the video compression process. These results show that ProtoSeg is sensitive to data quality degradation and struggles to adapt to distribution shifts between the training and testing data.

Table 5.

The mIoU of ProtoSeg

As mentioned earlier, it is confirmed that ProtoSeg is sensitive and unable to adapt to changes in the internal distribution of data. Therefore, we test whether ProtoSeg would show high adaptability when trained with style-transformed and video-compressed datasets along with CityScapes. The model is trained using CityScapes, an ‘anime’ style-transformed dataset, and a dataset compressed with a QP value of 40. The performance of the trained model is evaluated on datasets transformed with the ‘Desert sand’ and ‘Sketch black pencil’ styles and compressed with QP values set to 30, 40, and 50.

As shown in Table 5, for the style-transformed datasets, the model trained with the transformed datasets achieves mIoU values of 12.1 and 24.8, respectively, improving performance compared to the model trained solely on CityScapes. This shows that the ProtoSeg has been enhanced to withstand changes in color and texture caused by style transformations. Additionally, the model shows stronger performance on video compression dataset transformed using various quantization parameters. Particularly, for the dataset with a QP value of 50, the model trained on the transformed datasets achieves an mIoU of 46.3, which is more than six times higher than the 7.3 achieved by the baseline model. These results confirm its robustness in noise induced environments caused by compression. Therefore, these demonstrate that training with datasets transformed by various styles and quantization parameters enables the model to exhibit stronger performance in handling diverse transformation environments.