How AI lipsync actually works
The model transcribes your audio into a timed sequence of phonemes, then maps each one to a viseme — the mouth shape a human makes for that sound. 'M' and 'B' close the lips, 'F' tucks the lip under the teeth, open vowels drop the jaw. Those shapes are blended frame-by-frame onto the face in your footage while everything else (eyes, hair, background) is preserved.
Because the sync is driven by the audio timeline rather than the original footage, you can replace dialogue entirely: swap languages, fix a flubbed line, or make a still portrait deliver a monologue. Nidhogg re-renders only the lower face region, which keeps identity and lighting stable across the whole clip.
Getting a clean sync: what actually matters
Audio quality matters more than video quality. A dry, close-miked voice with no music bed gives the phoneme detector clean timings; background music or reverb smears them and the mouth starts to lag. If you must use a mixed track, sync to the isolated voice first and add music back in your editor.
On the video side, favor a face that stays roughly frontal — up to about a three-quarter turn is fine, but a full profile hides half the mouth and the model has less to work with. Avoid clips where hands, microphones, or hair cross the lips mid-sentence, and keep sentences under a natural breath length so pauses land where a speaker would actually pause.
What creators use it for
The heaviest users are localization teams (one presenter video, eight languages), UGC-style ad producers who generate a spokesperson with Nidhogg's image models and then give them a script, and educators who maintain a consistent on-screen host without booking studio time every week.
It also rescues real footage: re-record a single botched sentence and sync it back in, rather than re-shooting a whole take. Pair it with the AI Talking Avatar workflow when you don't have any source footage at all.