myned-ai/audio2face-emotion-arkit-teacher · Datasets at Hugging Face

audio2face-emotion-arkit-teacher

Nyx avatar (Gaussian-splat head, ARKit-52 blendshape rig) driven by a surprise clip's blendshape labels derived from this dataset.

14,082 emotional-speech clips, each annotated with two parallel 52-channel ARKit blendshape sequences (NVIDIA Audio2Face-3D-v2.3.1-James and LAM_Audio2Expression) plus a 26-dimensional NVIDIA Audio2Emotion conditioning vector.

Reference-only dataset — the original audio is not shipped. Each row contains a clip_id and audio_path_hint that let you join with the source audio you download yourself from CREMA-D, RAVDESS, TESS, or JL Corpus.

Released by myned-ai to support teacher–student distillation research: condensing heavy GPU-bound audio2face teacher models — NVIDIA Audio2Face-3D (James actor) and LAM_Audio2Expression (identity 2) — into small student networks that run in real time on CPU. Both teacher chains here are speaker-locked, so the labels reflect those specific actors' styles rather than a generic talking face; the dataset is therefore aimed at studying distillation of these particular teachers, not at training character-agnostic audio2face models.

What's in it

The 52 ARKit channels follow the standard ARKit order: browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, …, tongueOut.

Splits

Stratified by (source × emotion_label) so each split keeps the same per-class proportions.

Quick start

from datasets import load_dataset
import numpy as np

ds = load_dataset("myned-ai/audio2face-emotion-arkit-teacher")

row = ds["train"][0]
print(row["clip_id"], row["source"], row["emotion_label"])
nim_bs = np.array(row["nim_bs"], dtype=np.float32)  # (T, 52)
lam_bs = np.array(row["lam_bs"], dtype=np.float32)  # (T, 52)
print(f"T = {nim_bs.shape[0]} frames @ 30 fps")

Joining with audio

The dataset does NOT ship audio. To use it for training, download the source corpora separately and join via audio_path_hint:

import soundfile as sf
import os

CORPUS_ROOTS = {
    "cremad":    "/path/to/CREMA-D",       # contains AudioWAV/
    "ravdess":   "/path/to/RAVDESS",       # contains Actor_01/, Actor_02/, ...
    "tess":      "/path/to/TESS",          # contains OAF_angry/, YAF_happy/, ...
    "jl_corpus": "/path/to/JL_corpus",     # flat .wav files
}

audio_path = os.path.join(CORPUS_ROOTS[row["source"]], row["audio_path_hint"])
audio, sr = sf.read(audio_path)
# audio: original sample rate (16k for CREMA-D, 48k for RAVDESS, etc.); resample to 16k as needed.

See examples/join_with_audio.py in the dataset repo for a complete data loader that streams audio + teacher labels together.

How nim_bs and lam_bs were generated

For each source audio file at its original sample rate:

1. NIM (nim_bs) — NVIDIA/Audio2Face-3D-v2.3.1-James with emotion_strength = 0.6, live_blend_coef = 0.7, conditioned on emotion_26d (which itself comes from running NVIDIA/Audio2Emotion-v2.2 on the same audio). Then apply_locked_tune per-channel cosmetic post-processing was not applied at teacher-generation time — these are raw model outputs.
2. LAM (lam_bs) — aigc3d/LAM_Audio2Expression streaming model (lam_audio2exp_streaming.tar), identity index 2 of LAM's 12-identity embedding (id_idx = 2, one-hot of length 12). LAM is identity-conditioned, so different ids produce different outputs on the same audio. If you want a different speaker style, re-generate lam_bs with the LAM model under a different id and replace that column.
3. Both are emitted at 30 fps at the audio's original duration; T = ceil(audio_duration_s × 30).

Both models read 16 kHz audio internally; if your source audio is at a different rate, resample it before any comparison.

Source corpora

You need to download each corpus separately from its official source. All four allow research use; CREMA-D and TESS allow broad reuse.

Citations:

@article{cao2014cremad,
  title={CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset},
  author={Cao, Houwei and Cooper, David G and Keutmann, Michael K and Gur, Ruben C and Nenkova, Ani and Verma, Ragini},
  journal={IEEE Transactions on Affective Computing},
  year={2014}, volume={5}, number={4}, pages={377--390},
}

@misc{pichora2010tess,
  title={Toronto emotional speech set (TESS)},
  author={Pichora-Fuller, M. Kathleen and Dupuis, Kate},
  year={2020}, publisher={Borealis},
  doi={10.5683/SP2/E8H2MF},
}

@article{james2018jl,
  title={An Open Source Emotional Speech Corpus for Human Robot Interaction Applications},
  author={James, Jesin and Tian, Li and Watson, Catherine},
  journal={Interspeech},
  year={2018},
}

@article{livingstone2018ravdess,
  title={The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)},
  author={Livingstone, Steven R. and Russo, Frank A.},
  journal={PLoS ONE},
  year={2018}, volume={13}, number={5}, pages={e0196391},
}

Teacher models

The blendshape teacher labels were generated by:

1. NVIDIA Audio2Face-3D-v2.3.1-James — https://huggingface.co/nvidia/Audio2Face-3D-v2.3.1-James (NVIDIA Open Model License)
2. NVIDIA Audio2Emotion-v2.2 — https://huggingface.co/nvidia/Audio2Emotion-v2.2 (NVIDIA Open Model License)
3. LAM_Audio2Expression (streaming) — https://github.com/aigc3d/LAM_Audio2Expression (Apache-2.0). Identity index 2 (one of 12 LAM identity slots) was used for every lam_bs row in this dataset.

The NVIDIA Open Model License (§2.4) explicitly states "NVIDIA claims no ownership rights in outputs" and (§1.1a) "An output is not a Derivative Model". We redistribute these generated blendshape arrays under that grant. LAM outputs are redistributed under Apache-2.0.

Intended uses

• Distillation — train a smaller, CPU-friendly audio2face student model on nim_bs and/or lam_bs labels.
• Multi-teacher experimentation — compare the two teachers' annotations frame-by-frame on the same audio, or combine them per-channel (e.g. NIM for upper face, LAM for lower face).
• Emotion classification — emotion labels are clean and balanced; can serve as a SER benchmark even without using the blendshape labels.
• Audio-to-facial-expression research — generally any task that needs paired (emotional speech, facial expression parameters).

What this is NOT

• Not a complete dataset out of the box — you must download the source audio yourself for any task that needs audio. The four corpora collectively are ~2 GB.
• Not ground-truth motion capture — the blendshape labels come from regression models, not from human face tracking. Treat them as teacher labels, not as gold annotations.
• Not specific to any one downstream architecture — the labels are at 30 fps in standard ARKit-52 ordering; you can use them with any model that accepts that contract.

Repository layout

audio2face-emotion-arkit-teacher/
├── README.md                 ← this card
├── data/
│   ├── train.parquet         386 MB
│   ├── validation.parquet     22 MB
│   └── test.parquet           21 MB
├── assets/
│   └── nyx_surprise.gif      ← preview animation
├── parse_metadata.py         ← filename → metadata parsers (source of truth)
├── build_dataset.py          ← NPZ → Parquet converter (for reproducibility)
└── examples/
    └── join_with_audio.py    ← reference data loader that joins parquet + audio

License

The dataset (this repository's content) is released under Apache-2.0. Source audio is not included; if you download it, it is governed by each corpus's own license linked above.

Cite

If you use this dataset, please cite the source corpora (above) and the two teacher models (NVIDIA A2F-3D, LAM_Audio2Expression), plus this dataset:

@misc{myned2026audio2face,
  title={Audio2Face Emotion ARKit Teacher Labels},
  author={myned-ai},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/myned-ai/audio2face-emotion-arkit-teacher},
}