Gaussian-based Realistic Avatar Reconstruction With Monocular Geometric Priors

1. Monocular Geometric Cue Estimation

MonoSDF^[1] demonstrates that the use of a general-purpose monocular estimator significantly improves both reconstruction quality and geometric accuracy for large scenes. Omnidata^[3], used in [1], estimates monocular depth cues for a wide range of scenes. DN-Splatter^[20] utilizes Omnidata and ZoeDepth^[4] to enhance the geometry alignment in Gaussian Splatting using monocular depth and normal cues. ZoeDepth extends the relative depth prediction of MiDaS. Metric3Dv2^[9] is a monocular estimation method based on Vision Transformer and ConvGRU. It addresses depth ambiguity by transforming images into the canonical camera space. To overcome the scarcity of surface normal datasets, Metric3Dv2 uses joint learning for depth and normal. DepthAnythingv2^[8] employs a teacher-student framework, where the teacher is trained on synthetic datasets to circumvent the noise and incompleteness in real-world datasets. Recently, diffusion models like Marigold^[6] and GeoWizard^[7] have been proposed. Marigold, a depth estimation model, proposes training solely on synthetic datasets. GeoWizard introduces a cross-attention mechanism to ensure consistency between depth and normal predictions. However, diffusion-based models are known to be time-consuming.

On the other hand, ECON^[13] introduces a normal integration method for reconstructing clothed humans from a single image. It estimates the front and back normal maps using a GAN-based model, as seen in [12], and this normal estimator can serve as a monocular cue. Additionally, the recent model Sapiens^[10], based on a Vision Transformer, offers broad applicability to various human-related tasks. To enhance generalizability, it uses the Human-300M dataset for pretraining, achieving state-of-the-art performance.

2. Gaussian splatting in Avatar Reconstruction

Gaussian splatting (GS) has been applied to various fields of 3D reconstruction due to its reconstruction quality along with fast training and rendering times. However, the reconstructed Gaussians are not aligned with the actual geometric surfaces. To address this, SuGaR^[23] introduces regularization for Gaussian shapes and locations, while 2DGS^[24] and GaussianSurfel^[21] propose 2D Gaussians to resolve multi-view inconsistencies. Nevertheless, they do not address dynamic scenes or have not explored recent monocular estimators.

Some GS-based avatar reconstruction methods have been proposed. [15] uses pose features to decode Gaussian parameters for dynamic texture representation. [16] reconstruct a template by multi-view stereo then learns a network to estimate Gaussian maps from a position map at each time step. GART^[14], on the other hand, employs a learnable skeleton to represent loose clothing. Though these models show high rendering quality, they still suffer from inaccurate 3D surfaces reconstructed by Gaussians. We improve 3D avatar reconstruction using monocular geometric cues. We compare different monocular geometric estimators in terms of depth and normal accuracy on human datasets.

1. Preliminary: Gaussian Splatting

3D Gaussian Splatting (3DGS)^[2] introduces volumetric 3D Gaussians to represent 3D scenes. Each 3D Gaussian represents a small cloud, having positional and radiometric properties such as mean position μ, rotation R, scale S, opacity o, and view dependent color c. Then, the Gaussian distribution at position x is parametrized by μ and 3D covariance matrix Σ=RSS^TR^T as below:

The 3D Gaussian is splatted onto the image plane by the elliptical weighted average (EWA) approximation since there is no analytic form to represent Gaussian projection. To accelerate the Gaussian rendering process and make it differentiable, the Gaussian renderer is implemented as the GPU-parallelized differentiable rasterizer. The rasterizer sorts Gaussians by depth from the view-point in fast tile-based approach. Then, Gaussians are blended with other Gaussians using the alpha-blending technique as shown in Equation 2:

where c(x) is color of a pixel x, $$ G_j^{2D}\left(x\right)$$ is a distribution of the ith Gaussian along ray x, o_j and g_j are an opacity and a distribution of the jth intersected Gaussian before the ith Gaussian.

The goal of optimizing 3D Gaussians is to encourage rendered images I_rend aligned with ground-truth images I_gt so that Gaussians represent the appearance of the 3D scene. To this end, the difference between two images is computed as an objective function of the optimization process. This photometric loss L_rgb is computed as the combination of0 L₁ loss and D－SSIM loss.

During optimization, overly-large Gaussians or sparsely-distributed Gaussians are struggling to reconstruct the surface and its texture. To alleviate this problem, 3DGS clone Gaussians in under-reconstructed regions and split large Gaussians in over-reconstructed regions.

2. Overview

We set GART^[14], which is the state-of-the-art avatar reconstruction model by combining 3DGS with avatar reconstruction, as the baseline for our clothed human reconstruction. We first initialize Gaussians on the 3D body template mesh defined in the canonical space in Section 3.3. Then, Gaussians are deformed into the frame space by LBS in Section 3.4. and rendered into images. Our Gaussians learn the avatar’s shape and appearance while minimizing the loss introduced in Section 3.5. The implementation details are described in Section 3.6.

3. Initialization

We define the canonical space of the avatar as the space with a “Da”-posed 3D body template as shown in Figure 1. “Da”-pose is a stance where the arms and legs are spread out wide to the both sides and it allows to reconstruct concave regions since a stretching-out stance reduces occlusions and overlaps between body parts. We locate Gaussians on the canonical mesh vertices. The orientation of each Gaussian aligns with normal of the mesh, scale is proportional to the face area, the opacity and the color are set to 0.9 and 0.5, respectively.

Fig. 1.

Overview of Gaussian Splatting avatar reconstruction using monocular geometric cues

4. Deformation

Gaussians in the canonical space are animated according to the SMPL parameters θ_t at time t based on LBS. While the pose changes, the effects of key points on adjacent surfaces vary. We update the blending weight W_k of the kth key-point motion at each ith Gaussian by Δw:

In clothed human reconstruction, it is challenging to describe cloth deformation by the traditional skinning method which represents deformation of a human skin. To account for dynamic cloth surfaces, we employ learnable latent bones $$ \tilde{B}\left({\theta }_t\right)=\left[{\tilde{B}}_{t,1},\dots ,{\tilde{B}}_{t,n_2}\right]$$ to increase the expressivity of the surface deformation model:

where i indexes over Gaussians, μ_t,i^' is a deformed Gaussian mean position to time t frame space, W is the blending weight of latent bones and B(θ_t)=[B_t,1,…,B_t,nb] are SMPL bones. n_b and n_l are number of SMPL bones and latent bones respectively. Then, Gaussians in the posed space are rendered into a RGB image, a normal map, and a depth map to compute loss and optimize Gaussians by following Section 3.5.

5. Optimization

We train Gaussians defined in the canonical space, their temporal deformation field for a given pose sequence during optimization. To make Gaussians of an avatar learn the appearance and 3D surface, we penalize the difference between rendered images and captured images and leverage the monocular geometric prior to enhance the surface details.

The photometric consistency loss L_rgb is the radiometric difference between the reconstructed video frames I_t,rendI_t,rend and observations I_t,rend:

where t is a time index. We penalize the photometric consistency loss to enhance the reconstruction fidelity.

The photometric consistency loss may not be sufficient to reconstruct smooth and natural surfaces. To resolve this issue, we leverage monocular geometric priors [1], which represent general surface shapes for given observation, in order to facilitate natural surface reconstruction. We encourage the reconstruction to have general surface orientations by fitting normal with monocular observation:

where N_t,rend is rendered normal and N_t,mono is monocularly estimated normal.

We also utilize a monocular depth prior, representing the relative distance among pixels. We align the monocular depth map D_t,mono to the reconstructed depth map D_t,rend by scaling w and shifting q since there is the scale and shift ambiguity in monocular depth estimation:

We encourage the smooth appearance by minimizing the standard deviation of Gaussian properties among K-nearest points:

where property_i is a property of the ith Gaussian. In addition, We regularize non-rigid motion and size of Gaussians:

where i means the i^th Gaussian.

The total loss is sum of all loss terms for N Gaussians:

6. Implementation Details

We jointly train Gaussian properties and motion parameters by applying the Adam optimizer with learning rates for each parameter the same as in [14], and other Adam hyperparameters are left at their default values of β_l=0.9, β₂=0.999, and ϵ=10^－8. We set λ for each loss term as λ_rgb=1.0, λ_norm=λ_dep=0.05 and $$ {\lambda }_{\stackrel{⏜}{W}},{\lambda }_{\tilde{W}},{\lambda }_s,{\lambda }_{property }$$ the same as in [14]. The Gaussian parameters are optimized by stochastic gradient descent method for each time step t where t is randomly sampled from the training input video frame sequence. The entire framework is trained and tested on a single NVIDIA RTX A6000 GPU, with training times within 5 minutes and rendering times about 150 FPS.

1. Comparison between Monocular Estimators

Our Gaussian avatar learns surface of the object from monocular geometric cues. This makes the reconstruction quality of our method depend on monocular cue estimator performance. Thus, we analyze the state-of-the-art monocular normal and depth estimation models by qualitative and quantitative comparisons.

We evaluate the performance of normal and depth estimation of monocular estimators. We use the Thuman3.0^[17] dataset, a real-world capture dataset of clothed humans which includes a variety of human and clothing scenes. It consists of 20 different clothed humans, each including between 15 to 35 pose sequences. For our evaluation, we select 5 poses from each clothed human. Since captured images are not provided, we synthesize multi-view images of scenes using the provided meshes and textures. For rendering, 8 point lights are placed at vertices of the bounding box surrounding the human mesh. We render four images from the front, back, and sides. The RGB images are used as input for the monocular estimation models, while rendered normal and depth maps are served as ground-truth data for evaluation.

1.1. Comparison on Surface Normal Estimation

We compare monocular estimators in terms of normal prediction. The evaluation metric “angular mean” represents the average angular distance, measured in degree, between rendered normals and the ground-truth. Lower values indicate better performance. The “ratio within x” metric represents the ratio of pixels whose angular distance with the ground-truth is less than x degrees. Higher ratio indicates higher performance. For Omnidata^[3], we use two approaches proposed in DN-Splatter^[20]: Omnidata-low and Omnidata-hd. The Omnidata-low approach resizes an input image, estimates normals, then resizes it back to the original resolution. The Omnidata-hd approach divides an input image into patches with overlaps and aggregates estimated normals of patches by aligning them with others.

In the case of Marigold^[6], whose original version only estimate depth, we use the normal pretrained model available on Hugging Face. For Sapiens^[10], we use the Sapiens-2B model trained with 2 billion parameters.

Table 1 shows that Omnidata-hd performs better than Omnidata-low. However, both Omnidata-based methods exhibit lower performance than other methods, indicating inadequacy of the model for human scenes. Vision Transformer-based models like Metric3Dv2^[9] and Sapiens-2B outperform diffusion-based models like Marigold and GeoWizard^[7], and ECON^[13] demonstrates competitive results. GeoWizard, in particular, shows suboptimal performance in normal prediction. In contrast, Sapiens, which leverages pretraining on a large human-centric dataset, achieves the highest performance among the evaluated models.

Table 1.

Quantitative comparison of monocular normal estimation models (Red: 1st place, Blue: 2nd place)

Figure 2 illustrates the qualitative performance of monocular estimators. Omnidata-low lacks detail due to its resolution limitations, and while Omnidata-hd addresses alignment in the overlap regions, it still struggles with overall consistency. ECON captures large wrinkles well but fails to estimate precise normal direction. Marigold captures details effectively but lacks a sense of depth. Metric3Dv2 is highly accurate but suffers from inaccurate facial normals due to limited exposure to human scenes during training. Sapiens produces results close to the ground truth.

Fig 2.

Qualitative comparison of monocular surface normal estimation models

1.2 Comparison on Monocular Depth Estimation

We measure two depth metrics for the comparison of monocular estimators: AbsRel and δ¹. AbsRel represents the relative difference $$ \left|d-\widehat{d}\right|/d$$ between the ground-truth depth map and the estimated depth map $$ \widehat{d}$$ with scale and shift adjustment. δ¹ represents the percentage of pixels whose ratio between the ground-truth and predicted values is below the threshold (1.25). We use the implementation of ZoeDepth^[4] in DN-Splatter.

Table 2 demonstrates that Metric3Dv2 and Sapiens outperform other methods significantly. GeoWizard shows comparable performance to Marigold and ZoeDepth. Conversely, DepthAnythingv2^[8] produces results that deviates significantly from the ground truth. Overall, Sapiens demonstrates the most realistic results.

Table 2.

Quantitative comparison of monocular depth estimation models (Red: 1st place, Blue: 2nd place)

Figure 3 illustrates qualitative performance of monocular estimators. DepthAnythingv2 fails to align the scale and shift correctly due to the large difference between the minimum and maximum depth scales. ZoeDepth, Marigold, and GeoWizard appear to lack high-frequency details. In contrast, Metric3Dv2 and Sapiens produce results that closely approximates the ground truth.

Fig. 3.

Qualitative comparison of monocular depth estimation models

2. Avatar Reconstruction

We evaluate the geometric quality of avatar reconstruction. We use the RANA^[18] dataset, a photorealistic synthetic dataset of clothed humans that provides normal maps, camera parameters, and SMPL^[19] parameters. We select 5 subjects for evaluation. Each subject contains 150 frames, with the first 100 frames used for Gaussian optimization and the remaining 50 frames reserved for testing.

We conduct qualitative and quantitative comparison. We use Sapiens^[10] as a model for monocular geometric consistency due to its superior performance demonstrated in Sections 4.1.2 and 4.1.3. Table 3 quantitatively verifies that the monocular geometric consistency constraint improves both rendering quality and geometric quality. Specifically, after the GS process, the accuracy of the normal map is significantly improved.

Table 3.

Quantitative comparison of Gaussian Splatting avatar reconstruction using monocular cues

Figure 4 shows the qualitative comparison between the baseline and reconstruction with monocular consistency constraint. The normal map reconstructed with the monocular consistency prior is smoother, aligned to the ground-truth, and reducing noise in the texture of the rendered RGB images. Notably, it is challenging for the Gaussian to reconstruct wrinkles in clothing, but Sapiens’ precise normal and depth estimation plays a key role in improving the geometric quality of the clothing, especially in capturing the folds.

Fig. 4.

Qualitative comparison of Gaussian Splatting avatar reconstruction using monocular cues