Past Issue

Due to the recent advancements and great success of deep learning, deep neural networks have become the dominant approach in many computer vision and image processing fields. CNNs have widely been used in image classification^{[14],[24],[41]}, object detection^[35],[50], semantic segmentation^[11],[20], natural scene understanding^[71],[73], human pose estimation^[57],[70] and so on. However, only a few IQA/VQA methods have adopted CNNs to assess the quality of images or videos. The first attempt was made by Kang et al.^[21], where CNN was used to solve an NR-IQA problem instead of using any hand-designed features. The author used a patch-based training process by following the state-of-the- art training mechanism where each patch quality was needed. However, the IQA benchmark datasets do not provide patch-wise ground truth quality scores but a single visual quality value for the whole image. As a result, they assigned all patch quality with the ground truth quality score.

To remedy the patch quality problem, Kim et al.^[22] proposed an FR-IQA method where the network takes the distorted image and an error map as input and estimates the sensitivity map which is then used in the weighted pooling stage. Bosse et al.^[9] introduced an impressive method that can effectively be used for both the FR-IQA and NR-IQA problems. The network is first trained to predict the quality score and the weight for each patch. At the final stage, the overall quality score of the whole image is estimated by applying a weighted average pooling method. Prashnani et al.^[46] reveal that pairwise learning helps to improve IQA performance and based on that they proposed PieAPP which offers much higher prediction accuracy.

To overcome the limited dataset problem, Liu et al.^[36] exploited synthetically distorted images with various levels of distortion and used a Siamese network to rank the images based on their quality. And finally, this knowledge is transferred to a traditional CNN to estimate the quality of a single image. Another interesting approach for NR-IQA was proposed in^[32]. To compensate for the absence of a reference image, Lin et al.^[32] generated a hallucinated reference image and then the pair of the hallucinated reference and corresponding distorted image is passed to the regressor to estimate the quality. Since human is the ultimate receiver of any image or video, it is necessary for the IQA/VQA methods to consider the HVS properties during the quality prediction. As a result, Seo et al.^[48] proposed an HVS inspired IQA network that incorporated two important HVS properties named visual saliency (VS) and just noticeable difference (JND).

On the other hand, the Video Quality Metric (VQM)^[43] is an early VQA method, that employs losses in the spatial gradients of luminance, along with features based on the product of luminance contrast and motion. The MOVIE index^[49] uses the idea of motion tuning by tracking perceptually relevant artifacts along motion trajectories to measure video quality. ST-MAD^[59] index uses spatiotemporal video slices and applies the idea of “most apparent distortion”^[25] to assess quality. SpEED-VQA^[6] computes spatial and temporal entropic differences in a band-pass domain to measure quality deviation.

Recently, data-driven approaches have become increasingly popular because of their high prediction performance. For example, the Video Multi-method Fusion (VMAF)^[31] model developed by Netflix is a widely used quality predictor built on existing IQA/VQA models on features that are combined using a Support Vector Regressor (SVR).

However, the existing methods are developed based on uncompressed images/videos that are not focused on any particular scene or object. As a result, they poorly performed on compressed saliency videos, which suggests the need for a dedicated VQA method.

The human visual system (HVS) has the capability of automatically focusing on the distinctive regions (called salient regions) in any visual scene. This powerful property of HVS has widely been studied by cognitive scientists. However, due to its excellence, it has recently been adopted in computer science to solve various computer vision problems. In computer vision, the visual saliency detection models aim to simulate this HVS attention mechanism. According to [7], image saliency detection methods can be categorized into bottom-up models which are stimulus-driven, and top-down models which are task-driven. Bottom-up models extract low-level vision features and use visual priors to describe the properties of salient objects.

Visual priors include contrast prior^[12], background prior ^{[10],[63],[86]}, compactness prior^[84], etc. Further- more, frequency domain analysis^[2], sparse representation^[30], cellular automata^[74], random walks^[10], low-rank recovery^[42], and Bayesian theory^[28] are also used to achieve image saliency detection. On the other hand, top-down models take advantage of supervised learning strategy, especially, deep learning techniques including hierarchical deep networks^[15],[34], multi-scale or multi-context deep network^[29],[83], encoder-decoder networks^[47],[82], etc.

Among several well-established models, some of the fast and well-known saliency detection algorithms are [2], [18], [39], [47] Hou et al.^[18] proposed a spectral residual method that focuses on the background properties instead of the target object's property. This method analyzes the log spectrum of an image, extracts the spectral residual in the spectral domain, and then reconstructs the saliency map in the spatial domain. Achanta et al.^[2] proposed a frequency-tuned approach that preserves the boundary information by retaining a sufficient amount of high-frequency contents and produces a full-resolution saliency map. Montabone & Soto introduced a method that was originally designed for fast human detection in a scene by proposing novel features derived from the visual saliency mechanism^[39].

Later on, this feature extraction mechanism was generalized for other forms of saliency detection. Qin et al.^[47] exploited the power of deep CNN and proposed a boundary-aware saliency detection network (BASNet) that consists of a densely supervised Encoder-Decoder network and a residual refinement module to produce saliency prediction and to perform saliency map refinement, respectively.

One of the most important and challenging tasks in IQA/VQA research is to collect the dataset. IQA/VQA dataset needs to provide the ground truth quality values of the collected images or videos. However, human beings are the ultimate receivers of those contents and as a result, it is necessary to evaluate those contents by humans, which is highly time-consuming and cumbersome. LIVE^[85], TID2008^[45], CSIQ^[25], and TID2013^[44] are the four benchmark datasets that are widely used to evaluate the IQA/VQA metrics. LIVE^[85] consists of 779 distorted images which are generated from 29 reference images using 5 distortion types. 161 subjects evaluated the dataset. CSIQ^[25] consists of 866 distorted images which are generated from 30 reference images using 6 distortion types and the dataset is evaluated by 35 subjects.

Another well-known dataset is proposed by Ponomarenko et al.^[45] where there are 1700 distorted images which are generated from 25 reference images using 17 distortion types and evaluated by 838 subjects. After that, the author of TID2008 extended the dataset by introducing 7 more distortion types, resulting in an increased number of distorted images. TID2013^[44] consists of 3000 distorted images that are the largest number of images among existing IQA datasets. Those images are generated from 25 reference images using 24 distortion types and evaluated by 917 subjects. Consequently, TID2013^[44] has become the most reliable dataset to evaluate IQA/VQA metrics.

However, there is no dataset available for highly concentrated videos that are compressed to use in an ultra-narrow band transmission system. Since, this kind of video usually focuses on a specific target object or region and requires a high compression rate, existing IQA/VQA metrics are not suitable to predict their visual quality. This phenomenon motivated us to collect a new VQA dataset that will help to develop a better quality assessment metric that can be used to automatically estimate the visual quality of ultra-narrow band transmission system contents.

We collect candidate video sequences for performance evaluation from largely two sources: self-collected video sequences and standard video sequences. In the collection of self-collected videos, we use several types of cameras: IR camera (built by i3 Systems, resolution 320 × 480) and EO cameras: GoPro (built by GoPro Inc., resolution 1920 × 1080). Using these cameras, we capture tremendous battlefield-like surveillance raw video sets (since we cannot acquire “real” battlefield surveillance videos, we organize a pseudo-battlefield environment). Furthermore, to add more videos to our test sequence pool, we additionally collect video sequences from YouTube. In order to maintain the ‘generality of videos’, we collect standard videos provided by high-efficiency video coding (HEVC) and MPEG. Consequently, we have three groups of test sequences based on the source of videos: 1) Standard test sequences; 2) Collected-EO test sequences; and 3) Collected-IR test sequences.

The collected sequence also can be categorized based on pixel variation statistics. We categorize all the collected sequences into two groups: 1) stationary camera sequences; and 2) moving camera sequences. The reason behind categorizing the videos based on camera motion is that we are interested in two types of cameras: 1) a camera installed on a tripod - which will be similar to the stationary camera sequences; and 2) a camera mounted on a helmet - which will be similar to the moving camera sequences. Then the stationary camera sequences are resized to CIF (352 × 288) resolution, clipped to make 10 seconds long of 5 frames per second (fps), and re-encoded to YUV 4:2:0 format of 8-bit color depth. Similarly, the moving camera sequences are resized to QCIF (176 × 144) resolution, clipped to make 10 seconds long of 10 fps, and re-encoded to YUV 4:2:0 format of 8-bit color depth. Each resolution and fps specification for moving and stationary target sequences are driven by the system-level requirement of video transmission.

According to ITU recommendation, there should be a high variation in the test sequences when evaluating their perceptual quality, since the quality perception usually depends on their spatial-temporal characteristics^[55]. Therefore, we analyze candidate test sequences using the following metrics: spatial variation (SV), temporal variation (TV), and sharpness. In spatial and temporal variation evaluation, the MPEG Edge Histogram Descriptor^[69] for SV and the average magnitude of motion vectors^[55] for TV are used, respectively. For the sharpness measure, we use cumulative probability of blur detection (CPBD) metric^[40]. We first categorize the candidate sequences based on SV and TV, and verify the adequacy of selected sequences based on CPBD. As mentioned in the previous section, we categorize overall collected test sequences into 2 (stationary, moving) by 3 (standard, collected-EO, collected-IR) groups, which results in a total of 6 groups.

Fig. 1. 
Reference images of the proposed dataset. In each category there are six videos, and for nine categories there are total 54 videos.

Fig. 2. 
Statistics of the collected MOS values i.e., the score wise histogram

For each group and each test sequence, we measure the SV and TV characteristics and based on the measured SV and TV, the video sequences are classified into the following nine categories: LL, ML, LL, ML, HL, LM, MM, HM, LH, MH, HH, where the first letters indicate Low (L), Medium (M) and High (H) spatial complexity and the second letters indicate the temporal complexity in the same manner. After that, from each category, one sequence is selected as a representative of that class, i.e., nine representative videos for each group. For verification of representative video selection, we use several aforementioned sharpness metrics and check the distribution of the selected videos. Figure 3 shows the scatter plots of the selected videos from the perspective of spatial complexity and temporal variation. As shown in Figure 3, the videos are well distributed all over the range of spatial complexity and temporal variation.

Fig. 3. 
Scatter plot of the selected videos in perspective of spatial complexity and temporal variation. (a) Distribution of the moving camera videos. (b) Distribution of the stationary camera videos.

Therefore, we finally get 6 groups with 9 representative test sequences in each, resulting in 54 selected test sequences. The first frames of each selected test sequence are shown in Figure 1. The selected ‘moving’ sequences show ‘moving camera’ characteristics as we intended, and the ‘stationary’ sequences show ‘stationary camera’ characteristics. It can be seen from Figure 1 that most of the collected EO videos contain natural scenes (complex patterns) that do not present in the standard sequences. Usually, military operations take place near ‘borders’ i.e., where man-made things can hardly be found such as buildings, traffic lights, traffic signs, etc. Moreover, usually, military equipment and/or personnel are with camouflage, and therefore their appearances are very similar. As a result, the test sequences need to contain natural scenes. Figure 1 also shows that the IR video sequences have different characteristics from the EO sequences.

However, IR sequences are widely used for military and surveillance tasks, and consequently, we include IR video sequences in this study. Similar to the EO sequences, IR video sequences also contain a large portion of natural scenes.

Using the selected 54 test sequences, we conduct a subjective video quality assessment study. Subjective quality assessment in a contextual manner is one of the most fundamental and important for video quality assessment. In this paper we perform the subjective quality assessment experiments following the ITU-R. BT500-11^[19]. A total of 16 persons participated which satisfied the recommendation^[19]. The age range of the participants is between 22 to 35 years. The environmental conditions are satisfied according to the recommendation and we adopt double-stimulus impairment scale (DSIS) method.

We generate impaired sequences for three target bit-rates i.e., 20, 45, and 65 kilobits per second (kbps) for each reference video using the x265 encoding tool^[56] where the GOP size is 10 and the number of consecutive B frames is four encoded with the two reference frames. The participants give a score for every impaired sequence within a range of 1 to 5 mainly considering the quality of the target object/region.

Based on the collected scores, we evaluate the Mean Opinion Score (MOS) value. The outlier scores (which are unnatural compared to others) are distinguished based on z- score method^[13], and discarded from MOS evaluation. The z-score measures the extent of deviation from the mean score with the following formula

where x- and σ are the mean and standard deviation of the data. The z-score ranges from -3σ and 3σ and if any score belongs outside of this range, it is considered as an outlier and removed from the data. After that, the MOS value is calculated as follows:

where N is the number of subjects and S_i is the ith subject’s score. Statistics about the collected MOS values are shown in Figure 2.

Since the collected dataset contains videos of YUV color format i.e., the color space is defined by one luminance component (Y) and two chrominance components, called blue projection (U) and red projection (V), respectively, and also the videos are stored as a one-dimensional vector. To feed into a CNN, the videos need to be converted to RGB color space.

Furthermore, we have a quality score neither for each frame nor for any image patch, but rather for the whole video. As a result, we extract all the RGB frames from a YUV video and stack them as a single input, and use its subjective score as the target label for training our network. Since the video frames are in color format i.e., they are of three channels, the dimension becomes W × H × 3N, where W, H, and N represent the width, height, and number of frames in a video. It is worth noting that patch-based quality assessment methods assign each patch quality score similar to the quality of the whole image or video, which is undesired as described in Section I. Using the whole image frame instead of the image patch offers two benefits: it does not require any patch quality score and pooling stage at the end of the network (see Section IV-C for more details).

We apply a well-recognized and widely accepted saliency detection algorithm proposed by Montabone & Soto^[39] to estimate the saliency map. It is not restricted to^[39], rather any saliency detection algorithm can be used here (see Section V-C for more details). Since, Montabone & Soto^[39] is a state-of-the-art method and easily available as a OpenCV library, we incorporate this saliency detection in our method. First, we extract the saliency maps of the reference video frames and then stack them together, similar to the video input. Since the saliency maps are gray scale image, the dimension becomes W × H × N, where W, H and N represent the width, height and the number of frames in a video.

Our proposed network consists of three main parts i.e., a video feature extraction part; a saliency feature extraction part and the final quality estimation part. We use a Siamese network architecture that has two branches of networks but with shared parameters to extract the features of the reference and distorted video. This kind of network is popular where the goal is to find out the similarities/differences between two different inputs. On the other hand, the saliency feature extraction module takes the corresponding saliency map and extracts the important features. The joint effect of video and saliency feature extraction offers a more powerful feature ex- traction that enables the network to extract more meaningful, significant, and effective feature maps. Figure 4 shows the architecture of the proposed network. It is noted that one video clip consists of three channels and a saliency map which has a single channel.

Fig. 4. 
Graphical representation of the proposed network architecture.

Therefore, we apply a 3 × 3 convolution on the input video where the input channel is 3 and the output channel is 32. Similarly, for the input saliency map, we apply 3 × 3 convolution with input channel 1 and output channel 32. The features (href and hdist) are extracted in a series of conv3-32, conv3-32, maxpool, conv3-64, conv3-64, maxpool, conv3-128, conv3-128, maxpool layers where conv3-64 means the convolution layer with total 64 filters sized of 3 × 3 and the maxpool layers perform with the stride value of 2. The extracted features are then passed to the feature fusion stage. The feature fusion first calculates the differences between reference and distorted features (href - hdist) and then concatenates this difference with reference and distorted features as cat((href - hdist), href, hdist).

This fused feature map now needs to be passed to the regression module. It is worth noting that the input videos are of different resolutions and the whole video (instead of patch) is considered as input. These various resolutions of input will produce feature maps with different sizes, which are not allowed to feed into the fully connected layer-based regression module. To address this problem, we apply global average pooling (GAP) on the fused features before feeding them to the fully connected layer for quality estimation. The regression module is made up of a sequence of two fully connected layers i.e., one FC-3*128 and one FC-128 layer. It is worth noting that the proposed model does not require any pooling stage. Because, unlike other IQA/VQA methods, we utilize the full image/video as input instead of a patch. Therefore, we directly estimate the overall quality of the given input rather than estimating each patch’s quality.

This results in about 5.2 million trainable network parameters of the proposed CNN. All convolutional layers apply 3×3 pixel size convolution kernels and are activated through a rectified linear unit (ReLU) activation function g = max (0, x)^[17]. To obtain an output of the same size as the input, convolutions are applied with zero padding. All max-pool layers have 2 × 2 pixel-sized kernels. Dropout regularization with a ratio of 0.5 is applied to the fully connected layers in order to prevent overfitting^[52].

The proposed networks are trained iteratively by the standard error back-propagation^[26],[27] over a number of epochs, where one epoch is defined as the period during which all the samples from the training set have been used. It is noted that since the videos are of different sizes, batch-wise training is not possible. The learning rate for the optimization is controlled per parameter adaptively using the ADAM method^[23] based on the variance of the gradient. Parameters of ADAM are chosen as recommended in [40] as β1 = 0.9, β2 = 0.999, E = 10−8 and α = 10−4. The mean loss over all images during validation is computed in evaluation mode (i.e. dropout is replaced with scaling) after each epoch. The final model that is used in the evaluation is the one with the best validation loss.

The overall samples in the dataset are randomly split by 70% training and 30% test set. This split is done by reference video to make sure that no test video has been seen by the network during training. Results are reported based on 5 random splits. Models are trained for 500 epochs. In order to evaluate the proposed method, four well-known and widely accepted evaluation metrics are used i.e., Spearman Rank-Order Correlation coefficient (SROC), Kendall Rank-Order Correlation coefficients (KROC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE). The SROC, KROC, and PLCC are used to measure the monotonicity between MOS and their estimated values by the IQA/VQA methods, while RMSE is used to measure the prediction error. Usually, SROC and KROC are the representative performance measures in IQA/VQA methods. Due to the non-linear mapping function between the estimated IQA/VQA scores and their measured subjective visual quality scores, we adopt a logistic regression when measuring the PLCC and RMSE. For SROC, KROC, and PLCC, the higher the score the better the quality. While for RMSE, the lower the score the better the quality. It is worth noting that, in order to compare the existing IQA methods for VQA we apply the IQA method for each frame and then take the average to estimate the final quality score.

Table 1 compares the performances of existing FR-IQA methods with the proposed FR-VQA method. Here the per-formance is compared in terms of prediction accuracy. The best results are highlighted using black boldface text. It can be seen that all other methods except the proposed method achieved low prediction accuracy when applied to the collected saliency videos. This is because the characteristics of those videos are significantly different from the characteristics of existing benchmark datasets. Since the proposed method employs deep CNN to extract the meaningful features and also utilizes the visual saliency of highly important regions, it offers a higher prediction map of the distorted videos to help the model focus on accuracy in terms of SROCC, KROCC, PLCC, and RMSE. The prediction accuracy of the proposed method is significantly higher than all other methods in comparison. Compared to the nearest competitor GMSD^[72], the proposed method achieves 0.12%, 0.13%, 0.09%, and 0.15% higher accuracy in terms of SROCC, KROCC, PLCC, and RMSE, respectively.

Table 2 compares the performances of existing FR-VQA methods with the proposed FR-VQA method. For comparison, we collected the most representative and publicly available FR-VQA methods, i.e., DVQA^[54], GREED^[37], and VMAF^[31]. Here the performance is also compared in terms of prediction accuracy. All the methods that are compared here are developed for video quality estimation. According to the results, the existing VQA methods poorly estimate the video quality for the proposed dataset. Among them, DVQA^[37] and GREED^[37] offer poor prediction performance, and VMAF^[31] offers rela-tively competitive prediction performance. The proposed method outperforms all other methods in comparison. Compared to the nearest competitor VMAF^[31], the proposed method achieves 0.10%, 0.11%, 0.08%, and 0.40%.

Our network takes the VS map of the input video and those VS maps are generated by using a third-party VS detection method. Since the quality of the VS map varies from method to method, here we investigate the effect of incorporating different saliency detection methods in our proposed VQA. We use four well-recognized saliency detection algorithms i.e., Montabone & soto^[39], Spectral Residual^[18], Frequency Tuned^[2], and BSANet^[47] to extract the VS map of the training videos and use those VS maps during experiments. Figure 5 shows the performance comparison when the proposed VQA method is applied with VS maps extracted by various saliency detection algorithms. The results suggest that the effect of the quality of generated VS maps varies by a small point. And since the network has a sufficient number of learnable parameters, it can compensate the VS map quality variation. As a result, the network is capable of offering almost similar prediction accuracy despite of the Saliency detection algorithm used in the model. However, we have used^[39] to extract the saliency map because it is well accepted and easily available as OpenCV library.

Fig. 5. 
Effects of various Saliency detection algorithms on the proposed VQA method

As shown in Figure 4, the proposed network consists of two branches, one is to extract features of the input video and another one is to extract features of VS map of the corresponding input video. Those two branches of the network share identical configurations and the output of each branch is concatenated before passing to the regression network. However, this concatenation may take place in the early stage, middle stage, or later stage. To find out the optimum place to concatenate the features, we performed experiments with early stage fusion, middle stage fusion, and later stage fusion. The experimental results are shown in Table 3. It can be seen that the prediction accuracy is poor for early stage fusion. This happened since the characteristics of input video and its corresponding VS map are significantly different and when we first concatenate them and feed them into the network, the model fails to get meaningful representations. Middle and later stage fusion works as a feature attention mechanism at feature level. Because VS map highlights the important pixels of an image or video and mixing this information with input video features followed by convolution helps the model to focus on significant pixels, resulting in higher prediction accuracy. However, the later stage fusion has a better ability to extract important features since it contains multiple layers of convolutions and thereby offers the highest prediction accuracy.