TongueReenact: First AI Framework to Animate Tongues in Face Swaps

A July paper from an Australia–China academic collaboration claims the first framework for transferring tongue motion between identities in AI facial reenactment, and it more than doubles the closest competitor's accuracy on key tongue metrics. The project, called TongueReenact, reportedly achieved a Tongue IoU (Intersection over Union, an overlap measure between generated and reference regions) of 0.3582 and a Tongue Area Similarity of 0.4338 across 9,000 test pairs — figures the authors describe as "more than double the nearest competitor." One caveat is unavoidable: the promised code has not been released, and no supplementary video exists to let outsiders verify the results.

The stakes are more than anatomical pedantry. Amateur diffusion-based face reconstruction has reached a level of realism where the tongue is one of the remaining tells — fully concealed at rest, intermittently visible during speech, and dominant during laughter or disgust. Nearly all reenactment systems, the authors note, treat the mouth interior as a textureless void.

What does TongueReenact actually solve?

The paper's central claim is architectural, not cosmetic. The authors state: "The central insight of our work is that tongue synthesis cannot be treated as a generic inpainting problem. The tongue must be spatially grounded to the source face geometry, which itself changes with every driving frame."

In other words, painting a plausible tongue into an open mouth fails because the mouth geometry shifts frame by frame. The tongue has to be anchored to a face that is itself moving. The paper describes itself as "the first framework for cross-identity tongue dynamics transfer in face reenactment," transferring both facial motion and tongue articulation from a driving video onto a source identity.

The authors frame the gap bluntly: "During expressive actions such as laughter, disgust, or surprise, [the tongue] can dominate the visible oral region entirely. Yet nearly all existing reenactment and avatar systems either model the oral interior as a textureless void or leave it entirely unconstrained." They attribute the neglect to the tongue's transience — "fully concealed at rest, intermittently exposed during speech, and suddenly dominant in extreme expressions, making it fundamentally resistant to the static parametric modeling applied to the rest of the face."

This is the first non-medical AI work to address the tongue at all, and the first to tackle the annotated-data shortage that made the problem intractable. Firsts in underexplored territory cut both ways: benchmarks are thin, rivals barely attempt the task, and outsized margins are easier to post.

How was the pipeline built?

The framework stacks several established methods. It uses 3D Gaussian Splatting (a scene-representation technique that renders from clouds of Gaussian primitives), the FLAME parametric face model, Neural Radiance Fields, and diffusion-based video generation. GANs did not make the cut.

The pipeline runs in two stages:

• First, the driving person's facial movements are transferred onto the source face using 3D Gaussian Splatting and FLAME. This produces the correct pose and expression — with an empty mouth.
• Then a Stable Diffusion model generates the missing tongue, guided by the tongue's appearance in the driving video and the geometry of the reconstructed face.

Solving the data problem required building a tongue-segmentation stack from existing components. The researchers drew on the NERSemble dataset of multi-view human head recordings and synthetic frames from VFHQ, a dataset of more than 16,000 high-quality facial videos. The processing chain ran as follows:

• MaskGAN, a face-parsing model, located the mouth in each image.
• TongueSAM, a medical tongue detector, identified the likely tongue region.
• SegmentAnything produced an initial outline — a pseudo-label — of the visible tongue.
• Masks were sorted into three groups by tongue visibility; incorrect examples were discarded and promising masks were manually corrected to separate tongue from lips, teeth, and surrounding tissue.
• The refined annotations trained BiSeNet, an image segmentation network, which was then applied to more NERSemble and VFHQ frames, with human correction and retraining repeated until performance on unseen subjects stabilized.

For synthesis, the oral region was slightly enlarged before being passed to the diffusion model, so the generated tongue could extend beyond the initial segmentation as the mouth changed shape. Generation ran in the latent space of Stable Diffusion Image Variations, with separate guidance from reconstructed facial geometry and the driving frame's tongue. Only the identified mouth and tongue region was regenerated.

What do the numbers say?

The training configuration is fully specified, which is more than many releases manage. BiSeNet trained on two classes, background and tongue, over 50,000 iterations at a learning rate of 10⁻³, using 512×512-pixel images at batch size 4 on a single NVIDIA A40 GPU with 48GB of VRAM.

For the reenactment stage, the researchers used the pretrained GAGAvatar Gaussian Splatting backbone and the Stable Diffusion Image Variations checkpoint, with the VAE and CLIP encoder frozen. Guidance encoders and the denoising UNET trained on NERSemble data for 85,000 iterations at batch size 8, also with 512×512 inputs. Inference used 20 DDIM sampling steps — a high count for current diffusion models, favoring detail at some cost in speed.

The comparison set consisted of four rival avatar frameworks: GPAvatar, LivePortrait, X-NeMo, and X-Portrait, evaluated across 9,000 cross-identity VFHQ source/driving pairs. Four metrics were used:

• LPIPS (Learned Perceptual Image Patch Similarity), a perceptual quality measure
• Tongue Presence — the proportion of frames where a tongue is detected
• Tongue IoU — overlap between generated and driving tongue regions
• Tongue Area Similarity — how closely the generated tongue's size matches the driving tongue

The authors report: "Our method achieves the best performance across all four tongue transfer metrics, with a Tongue IoU of 0.3582 and Tongue Area Similarity of 0.4338, more than double the nearest competitor, confirming substantially more accurate tongue dynamics transfer. Tongue Presence of 0.7891 further demonstrates that our method reliably synthesizes visible tongue articulation where existing methods frequently produce none."

A Tongue Presence of 0.7891 means the system still fails to render a tongue in roughly one in five frames where one should appear. It leads the field; it does not close the problem.

Temporal consistency was tested with T-LPIPS. The authors report: "Our full model achieves substantially lower flow jitter (0.218543 vs. 0.622901) and flow magnitude (0.513165 vs. 1.254073), confirming that geometry-anchored denoising produces smooth and stable tongue motion. Lower T-LPIPS further confirms that tongue synthesis does not compromise temporal coherence." No independent verification of these motion results is possible, since the supplementary video has not been released.

Why the results need cautious reading

Two caveats frame any assessment. First, testing a first-of-its-kind system against frameworks that barely attempt the task makes the reported margins structurally flattering. The source itself concedes that tests of this kind are hard to contrive "when prior frameworks address the new challenge minimally, or not at all." Second, the absence of code and video months after publication leaves the paper's claims checkable only on the authors' word.

There is a broader evaluation problem forming. As the original commentary observes, the uncanny valley is being gradually overcome by the latest diffusion-based video models, but a new psychological barrier is emerging: viewers are starting to rate AI-generated material negatively for reasons other than its quality. That bias threatens to contaminate human evaluations of AI realism, and metrics built on "blind" tests — where viewers want to downgrade AI output but fear downgrading real footage — may be needed to establish a reliable shadow scale.

For now, tongue motion joins hair, emotion simulation, and gaze as a solved-in-principle secondary identity trait whose real-world impact depends on delivery. Until TongueReenact ships its code and video, the most accurate talking tongues in AI remain a claim, not a demo.