Past Issue

In [8], using cropped face images for a deepfake detection network provide much higher accuracy compared to using the entire images. On the other hand, utilizing all frames of a given video not only takes a high computational complexity for the decision of a video but also requires huge training and computation time. Therefore, we propose to use cropped face images of keyframes in a video, which can incorporate useful information compactly by removing less informative spatial backgrounds and temporal duplicated frames at the same time. Let V=Vii=1M denote the set of videos. During the preprocessing step, people present in a video V_i are traced by a trained tracker to maintain the identity of a person throughout sequential frames. Next, keyframe extraction to select the most representative frames is conducted. A simplified method from the complex keyframe extraction method^[12] including computation of color histograms edge direction histograms, wavelet statistics, etc. extracts keyframes by computing the difference between successive frames and choosing N frames with the high variation score. Lastly, the set of face regions with the same identity in keyframes are obtained by a trained face detector to be used as inputs of a network that receives the sequential face images of the same human. The process is shown in Fig. 1 (a) and can be simply written as,

where V_i is i - th video in the dataset, Xidi=xi,jidj=1N is a face image set of i - th video with an identity id. F_D, K_E, and Track indicate a face detector, a keyframe extractor, and a tracker, respectively. In the case of only a single person present in the video, xi,jid can be simplified as x_i,j. In the following sections, we assume one person per video and omit id for a better understanding. Then the set of extracted face images of i - th video can be represented as: Xi=xi,jj=1N.

Deepfake generation methods produce highly realistic fake images indistinguishable by a single image, on the other hand, their movement in a video is often unnatural and detected as a synthetic result. Therefore, we propose a deepfake detection framework that receives multiple face images of a single human as input to consider the movement information in a video. Also, under the input conditions, configuring multiple face images of a single human extracted from keyframes becomes one of the crucial points to avoid almost identical face image pairs in a video.

As shown in Fig. 1 (b), the feature extractor ϕ_f extracts feature vectors from the sequential face image of the identical person, and the classifier ϕ_c estimates the probability of fakeness. Let X = {x_i,j | x_i,j ∈ X_i, 1 ≤ i ≤ M, 1≤ j ≤ N}denote the set of face images from keyframes through the preprocessing steps, then, obtaining the fakeness probability can be written as:

where p_i,j is the output fakeness probability of the given input set, (x_i,j-1, x_i,j), and f_i,j is an extracted feature by ϕ_f . The fakeness decision of the paired keyframes is conducted by comparing the probability with the given threshold as c_i,j = Ψ_t(p_i,j; τ_img), where c_i,j is a predicted class, τ_img is a threshold for an paired keyframe, and the decision function, Ψ_t, is defined as

where 0 and 1 indicate the real and fake, respectively. Deciding whether a video is manipulated or not is computed by measuring the average probability over all paired keyframes as

where c_i is a predicted class of a video and τ_vid is a threshold for a video.

Since the results of the deepfake discrimination are highly related to the fake region, the input contribution of the results is highly relevant to the synthesized face part. Therefore, to estimate the fake face area, input attribution in the deepfake decision model is measured by computing the relationship between the input and the output through a gradient-based class activation mapping (CAM)^[9]. The input attribution of the proposed network is obtained by using the CAM as,

where Ψ_m denotes the CAM method and M_i,j is an input attribution represented by its heatmap.

1) Dataset: The deepfake data set, FaceForensics++^[8], consisted of 6, 000 videos including 1, 000 real videos from Youtube and 5, 000 generated videos by Deepfakes^[3], Face2Face^[6], Faceshifter^[5], FaceSwap^[4], and Neural-Textures^[7]. For our experiments, the H.264-compressed videos with a compression rate factor of 23 are used in the dataset. For the training, validation, and test sets, we split the videos into training (80%), validation (10%), and test (10%) sets. Also, a maximum keyframe from a video is set 15 for the preprocessing of videos in the dataset. To evaluate the performance of the proposed structure, the detection accuracy of image-wise and video-wise was respectively measured to recognize the generated image and video by deepfake methods. Thresholds τ_img and τ_vid to decide the fakeness are set to 0.5. In addition, we employ a logloss method to evaluate the quality of predicted probabilities.

2) Training setup: We implemented the proposed network using EfficientNet, EfficientNetV2, and Swin Transformer as a backbone network. To fairly compare with previous research experiments of^[8], we also report the results when XceptionNet is used as a backbone network under identical settings with the proposed method. The SGD (Stochastic Gradient Descent) optimizer with a learning rate of 0.01 was used for the network based on EfficientNets and AdamW optimizer with a learning rate of 0.0001.

Under various backbone networks, the proposed method was compared with a single input condition like the conventional method^[13]. As shown in Table 1, the proposed method using two sequential keyframes, multiple (2), achieves higher accuracy compared with the detection method using an image under backbone networks. In addition, the proposed scheme with three sequential keyframes, multiple (3), shows better performance than the proposed method with two sequential keyframes. Table 2 shows video-wise deepfake detection accuracy according to fake generation methods that include Deepfakes (DF), Face2Face (F2F), FaceShifter (FSH), FaceSwap (FSW), and NeuralTextures (NT). The accuracy of real videos and generated videos by NeuralTextures is lower than the results of the other methods. Because the NeuralTexures method only focuses on changing the mouth region, real videos are confused with them.

To estimate the fake face region by an explanation of the proposed network, we employ GradCAM++^[9] to obtain input attribution maps that reflect the influence of inputs on results. As shown in Fig. 2, the overlaid images of input and their maps show the input attribution to results in the first row. The red and blue colors indicate the high and low attribution scores, respectively. Also, to analyze the influence characteristic of the input according to the deepfake generation methods, the average attribution map according to the generation methods is calculated using all test videos, as shown in the second row in Fig. 2. The proposed detection network well focuses on the face that has a high contribution score to the decision. Furthermore, statistical analysis using the input attribution mean shows a shape related to the characteristics of the deepfake technique, and the fake region is concentrated according to the deepfake methods. In more detail, the attribution maps of face-swapping methods (Deepfakes, FaceShifter, and FaceSwap) show high attention scores on crucial face components; eyes, nose, and mouth region. Since these methods change the identity of a face by replacing the entire face region with that of another face. In the reenactment schemes (Face2Face and NeuralTextures), attribution maps on the face components have lower scores than the swapping methods, on the other hand, the broad area of the face contributes to the attribution map of input. The attention region of Face2Face is wider than that of NeuralTexture since Face2Face modifies all face components for the reenactment, while NeuralTextures only modifies the mouth region.

Fig. 2. 
The images in the top three rows, from left to right, show a face image overlaid with an attribution map of a real image (a), a synthesized face image by Deepfakes (b), Face2Face (c), FaceShifter (d), FaceSwap (e), and NeuralTextures (f), respectively. The images in the last row are the accumulated input attribution maps of test videos, which provide an explanation of the proposed network based on the generation schemes.