2016 2026

Physics-Based
Humanoid Animation

From one backflip in simulation
to a foundation model on a real robot
Jingbo Wang  ·  NVIDIA GEAR Lab
The hook

Same recipe, seven years apart

2018 · DeepMimic · a simulated character (blue) imitates one mocap clip (green)
2025 · SONIC · a real Unitree G1
one policy · one clip
one policy · 700 hours of human motion
Both are trained the same way: reinforcement learning to imitate human motion capture inside a physics simulator. This lecture is about everything that had to change in between.
BasicsImitatePriorsModelsScaleInteractReality
02
Motivation

Why simulate at all?

A kinematic model outputs poses. A physics-based controller outputs torques, and the simulator decides the motion.
2023 · PhysDiff · zoom on feet and hands, line = floor
01

Plausible by construction

Feet stay on the floor and hands do not sink into it. Contact comes from the simulator, not from a loss term.

2019 · DReCon · boxes launched at it at 5 m/s
02

Reactive for free

Each box knocks it off balance and it catches itself. The response is computed, not recorded: no dataset holds every push.

2025 · SONIC · trained in simulation, runs on a real G1
03

It is a controller

The output is a policy that commands motors, so the same recipe can leave the screen and drive a robot.

The price: you now have to control a body with dozens of joints, no direct handle on its root, and contacts that switch on and off. Every paper in this lecture is a way of paying that price.
Clips: PhysDiff project page (Yuan et al., ICCV 2023), its two zoom-in views shown side by side · DReCon (Bergamin et al., SIGGRAPH Asia 2019) · SONIC (Luo et al., 2025)
BasicsImitatePriorsModelsScaleInteractReality
03
Chapter 0  ·  before the papers

Foundations

Just enough physics and RL for the rest of the lecture.
AWhat a physics-based character is bodies · joints · contacts
BHow it is actuated PD control
CHow a controller is learned policy gradients · PPO
DWhere the supervision comes from rewards · the lens
04
Foundations · the body

A character is a tree of rigid bodies

Links connected by joints, motors at the joints, and nothing holding the body up.
Simulated squat. The bars are the numbers in q and v at this instant.
Simulated with the torso clamped: the motors move their joints, one group at a time.
Not simulated: the root is moved by hand. No motor could do this.
Simulated hop. Blue arrows: contact forces from the ground, zero in the air.
Kinematicthe pose is written directly
Simulatedsame angles, sent to the motors
Both are told to lift their feet. Only the kinematic one can hover.
Actuated DoF of the humanoids in this lecture
This model21
ASE · sword-and-shield character31
SMPL body · PHC69
SMPL-X body with fingers · Omnigrasp153
Actuated DoF from the papers: ASE, 31-D action space (its “37 DoF” count the 6 of the root as well) · PHC, 23 joints × 3 · Omnigrasp, 51 joints × 3
BasicsImitatePriorsModelsScaleInteractReality
05
Foundations · the simulator

What the simulator does at every step

$M(q)\,\dot v$
$c(q, v)$
$\textcolor{#76B900}{\tau}$
$\textcolor{#4fc3f7}{J^{\top} f}$
+
=
+
mass matrix × acceleration
2 N·m at each shoulder. Folded arm: 3.6× less inertia, turns 3.6× as far. No gravity, joint springs or dampers. 3× slow motion.
gravity, Coriolis, centrifugal
Motors off: nothing holds the joints against gravity any more. The character collapses.
joint torques · all we control
PD torques (next slide) hold the legs and move the arms and the waist. Green bars: the 21 motor torques.
contact forces from the world
Dropped from 30 cm, then shifting its weight. Blue arrows: the ground pushes back; 400 N, its weight, once it stands.
BasicsImitatePriorsModelsScaleInteractReality
06
Foundations · the simulator

Three properties of simulation to keep in mind

Why it is hard

Contact is non-smooth

Why it is easy

A black box is enough

Why it scaled

Speed is a feature

A contact happens or it does not. A 1 % change in the action, a different outcome.

Bullet
DeepMimic
PhysX
Isaac Gym: ASE, PHC
MuJoCo
every clip on these slides

The learner only calls s′ = f(s, a). It never opens the box, so any engine will do.

4096characters at once on one GPU
10 yearsof experience in about 10 days

RL needs billions of steps: ASE took 10 billion. Above: 60 simulations with random actions.

Hence reinforcement learning: trial and error on a black box, with as many trials as the hardware allows.
BasicsImitatePriorsModelsScaleInteractReality
07
Foundations · actuation

The policy sets targets; a PD controller makes the torques

$\textcolor{#f5b301}{\tau} \;=\; \textcolor{#76B900}{k_p\,(q_{\text{target}} - q)} \;\textcolor{#4fc3f7}{-\;k_d\,v}$
A simulated joint: $I = 0.03$ kg·m², $k_p = 120$ N·m/rad, $k_d = 2.7$ N·m·s/rad, no gravity · shown 30× slower
Simulation · DeepMimic
Real robot · SONIC on a Unitree G1
Action spaces compared: Peng & van de Panne, “Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?” (SCA 2017) · clips: DeepMimic (2018), SONIC (2025)
BasicsImitatePriorsModelsScaleInteractReality
08
Foundations · actuation

PD gains: how stiff, how damped

One simulated body, one jump of the target pose (green ghost), three pairs of gains for all its joints.
Soft$k_p = 30,\ \ k_d = 6$
Slow, and gravity holds the leg short of its target
Balanced$k_p = 300,\ \ k_d = 20$
Ten times stiffer: fast, and it stops on the target
Under-damped$k_p = 300,\ \ k_d = 2$
A tenth of the damping: overshoots by 40°, then rings
Damping ratio $\zeta = k_d \,/\, \big(2\sqrt{k_p\,I}\,\big)$in the charts: the knee, $I \approx 0.32$ kg·m²
$\zeta$ about 0.7 to 1: no ringingthen $k_p$ decides how fast and how firm
Heavier limb, same gains: rings morelarger $I$ lowers $\zeta$, so gains are set per joint
MuJoCo · DeepMind humanoid, torso pinned in the air · the same $k_p$ (N·m/rad) and $k_d$ (N·m·s/rad) on all 21 joints · clips at half speed · charts: right knee angle
BasicsImitatePriorsModelsScaleInteractReality
09
Foundations · actuation

PD gains in practice

Every joint has its own pair of gains. A simulated character and a real robot choose very different numbers.
Real settings: $k_p$ / $k_d$  (N·m/rad / N·m·s/rad)
Joint
DeepMimic humanoidsimulation · policy at 30 Hz
Unitree G1, legsreal robot · policy at 50 Hz
Chest / waist
1000 / 100
—
Hip
500 / 50
100 / 2
Knee
500 / 50
150 / 4
Ankle
400 / 40
40 / 2
Shoulder
400 / 40
—
Elbow
300 / 30
—
Neck
100 / 10
—
Gains and torque limits from xbpeng/DeepMimic (humanoid3d_ctrl.txt, humanoid3d.txt) and unitreerobotics/unitree_rl_gym (g1_config.py, g1_12dof.urdf: hip pitch)
BasicsImitatePriorsModelsScaleInteractReality
10
Foundations · the control problem

Why control is hard

All 21 motors at work · dashed: free-fall parabola
01

Under-actuated

In the air the motors move every limb, yet the centre of mass keeps its free-fall path. Only contact forces can redirect it.

One fixed pose, PD-held · green: the run without a push
02

Unstable

60 N · 0.2 sfalls within 1.5 s

An inverted pendulum: once the centre of mass leaves the feet, the error grows unless the controller reacts.

Six untrained policies: random targets, 30 per second
03

High-DoF, long-horizon

21 motors × 30 per secondall 6 fall

Chance cannot find the right commands. And each fall shows only after 40 or more actions: which one was wrong?

So a controller has to use contact to move, correct itself all the time, and choose actions whose result shows only later. How was that done before learning?
All three clips are simulated for this lecture in MuJoCo with its standard humanoid model (DeepMind): 40.8 kg, 27 degrees of freedom, 21 motors under PD control.
BasicsImitatePriorsModelsScaleInteractReality
11
Foundations · the control problem

What came before learning

Pushed (red line), it keeps walking

Hand-designed feedback

A state machine of target poses and one balance rule for the swing leg, tuned by hand for each gait.

Orange: simulation · blue: mocap

Sampling-based tracking

Sample many PD targets, simulate each, keep the best. A cluster run per clip, and the result is open-loop.

Only the goal is given: get up and stand

Trajectory optimization

Optimize the whole motion and its contacts offline. One trajectory per goal, no feedback controller.

Heess et al. 2017 · reward: go forward

Deep RL

A neural-network policy improves by trial and error on a reward. No per-motion design.

Each earlier approach worked, but needed expert design or a new optimization for every motion. Deep RL offers one recipe for all motions, provided you can say what “good” means.
Clips: Yin et al., SIGGRAPH 2007 · Liu et al., SIGGRAPH 2010 · Mordatch, Todorov, Popović, SIGGRAPH 2012 · Heess et al., arXiv 2017
BasicsImitatePriorsModelsScaleInteractReality
12
Foundations · reinforcement learning

Reinforcement learning in one slide

Learn by trial and error: no gradients through the simulator, only samples.
Objective: maximize
$J(\theta) = \mathbb{E}_{\pi}\Big[\sum_t \gamma^{t}\, r_t\Big]$
The policy

Stochastic on purpose

The action noise is the exploration. At test time we take the mean.

The reward

One number per step

Everything we want is squeezed into it: the real design problem of the field.

The cost

Billions of samples

Standing took 6 × 105 samples. ASE used 1010: ten years of practice, in ten days.

BasicsImitatePriorsModelsScaleInteractReality
13
Foundations · reinforcement learning

The policy and the return, in pictures

What the network outputs for one joint, and how far ahead the return looks.
The policy: a Gaussian around the network’s output
  • Every policy step, one target is sampled; the PD joint chases it.
  • The network moves the mean $\mu(s)$; the samples follow it.
  • $\sigma$ sets how far it explores. At test time: no noise, just the mean.
The return: the future, discounted
  • A reward $t$ steps ahead counts only $\gamma^{t}$ of its value.
  • All weights add up to $1/(1-\gamma)$ steps: the effective horizon.
  • $\gamma$ = 0.99 looks five times further than $\gamma$ = 0.95.
DeepMimic uses γ = 0.95 at 30 steps per second: its return looks about two-thirds of a second ahead, enough when every step is rewarded. For tasks that need a longer horizon, such as dribbling, AMP uses 0.99.
BasicsImitatePriorsModelsScaleInteractReality
14
Foundations · the optimizer

Policy gradients, and why everyone uses PPO

$\nabla_\theta J = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a_t \mid s_t)\,\hat A_t\big]$
Policy gradient. Make the actions that turned out better than expected more likely.
$\hat A_t = r_t + \gamma\,V(s_{t+1}) - V(s_t)$  (+ GAE)
Advantage. A second network, the critic V, predicts the reward still to come. The advantage is the surprise.
$L = \mathbb{E}\big[\min\big(\rho_t \hat A_t,\ \mathrm{clip}(\rho_t,\,1-\epsilon,\,1+\epsilon)\,\hat A_t\big)\big]$
PPO (Schulman et al., 2017). ρ = new probability / old probability of the action. Clipping it keeps every update small.
Why everyone uses it: simple, stable and parallel. Thousands of simulated characters collect, one update, repeat.
BasicsImitatePriorsModelsScaleInteractReality
15
Foundations · the optimizer

PPO in pictures: the clipped objective

One sampled action, seen by the old and by the new policy
What PPO maximizes for that one sample
The ratio

ρ = new / old probability of one sampled action. ρ = 1: nothing has changed.

A good action

Making it more likely pays, but only up to ρ = 1 + ε. Beyond that the objective is flat.

A bad action

Making it less likely pays only down to 1 − ε. Making it more likely costs without limit.

DeepMimic uses ε = 0.2: a sample stops pushing after a 20 % change, so the new policy stays close to the old one.
BasicsImitatePriorsModelsScaleInteractReality
16
Foundations · the optimizer

PPO in pictures: the training loop

1 · Collectrun the current policy, store every step
2 · Scorecritic V + GAE(λ): an advantage for every step
3 · Updateclipped gradient steps on random minibatches
Eight characters, one untrained policy, different noise
Numbers: DeepMimic (2018)
Samples per update4,096
Policy rate30 Hz
Physics rate1.2 kHz
Discount γ0.95
GAE / TD λ0.95
Critic step size10−2
Minibatch256
Clip ε0.2
Policy step size5 × 10−5
Scale changes the numbers, not the loop: ASE (2022) runs 4,096 characters on one GPU, for over 10 billion samples.
DeepMimic (Peng et al., 2018): sections 5.1, 10 and appendix D · ASE (Peng et al., 2022): section 10.1 · clip: MuJoCo humanoid, simulated for this slide
BasicsImitatePriorsModelsScaleInteractReality
17
Foundations · supervision

The tracking reward, term by term

Each term is 1 on a perfect match and falls toward 0 as its error grows:  $r = \sum_i w_i\,\exp(-k_i\,\mathrm{error}_i^{\,2})$
0.65
0.1
0.15
0.1
Posejoint rotations
$0.65\,\exp\!\Big[-2\sum_j \lVert \hat q_j \ominus q_j \rVert^2\Big]$
Velocityjoint angular velocities
$0.1\,\exp\!\Big[-0.1\sum_j \lVert \hat{\dot q}_j - \dot q_j \rVert^2\Big]$
End-effectorshand and foot positions
$0.15\,\exp\!\Big[-40\sum_e \lVert \hat p_e - p_e \rVert^2\Big]$
Centre of masswhole-body position
$0.1\,\exp\!\Big[-10\,\lVert \hat p_c - p_c \rVert^2\Big]$

The reference clip says where the body should be at every instant. The simulated character chases it with a hand-written PD controller, nothing learned: it lags behind, and it cannot jump. How much reward does this step deserve?

Pose: is every joint bent like the reference? The wedge at a joint is its rotation error. The angles are squared and summed over all joints: this one term decides 65 % of the reward.

Velocity: is every joint turning as fast as the reference? The arrow at a joint is its angular-velocity error. This kernel is forgiving: with the joints 2.6 rad/s off in total, half of the term is still there.

End-effectors: are the hands and feet where they should be in the world? Small joint errors add up along an arm or a leg, and this kernel is strict: 13 cm off in total and half of the term is gone.

Centre of mass: is the body as a whole in the right place? Watch the hop: the reference leaves the floor, the character cannot, and this term drops to 0.66.

Every policy step gets one number between 0 and 1. A reward this dense tells the policy at every step how close it is.

BasicsImitatePriorsModelsScaleInteractReality
18
Foundations · supervision

The style reward: a learned judge

AMP has no reference pose to compare with. A discriminator scores how much a motion looks like the data.
$r_{\text{style}} = \max\big[\,0,\ 1 - 0.25\,(D - 1)^2\,\big]$
A task reward alone says nothing about how to move: the policy does push the ball toward the flag, but no human moves like this.AMP adds a second reward, given by a learned judge: does this motion look like the data?
The judge is a discriminator $D(s, s^\prime)$. It sees one transition: two consecutive frames.It is trained by least squares to answer +1 for the dataset and −1 for the policy.
The policy is paid by the score of its own transitions.Nothing while it still looks like itself, full reward when it passes for data. Dense and bounded, like the tracking reward.
To earn more, the policy has to move like the data.The judge keeps training too, on the policy’s newest motion: the two get harder to tell apart, and what fooled it before stops working.
$r = 0.5\,r_{\text{task}} + 0.5\,r_{\text{style}}$The task says what to do, the judge says how. Same dribbling task, now in the style of the dataset: no reference clip to follow.
Task reward only
the same task,
with a style reward?
Task + style reward (AMP)
Clips: AMP supplementary video (Peng et al., 2021), “Humanoid: Dribble”, panels “No Data” and “AMP (Ours)”
BasicsImitatePriorsModelsScaleInteractReality
19
Foundations · supervision

Four ways to tell the policy what “good” means

Task reward

Reward the outcome

The simulator pays for progress toward the goal. Easy to write, but silent about how to move.

Emergence of Locomotion
Imitation

Reward looking like data

Tracking: stay close to a reference clip, frame by frame. Adversarial: convince a discriminator.

Tracking: DeepMimic · PHC · SONIC
Adversarial: AMP · ASE
World model

Differentiate a model

Learn a neural copy of the simulator. Back-propagate the loss through it: the policy gets a gradient, not just a score.

SuperTrack · ControlVAE
MoConVQ · SuperMimic
Distillation

Copy a teacher

A student imitates a trained teacher’s actions, with less information or a new interface.

PULSE · MaskedMimic
InterMimic · OmniH2O
Every paper in this lecture combines these four columns in its own way.
The lens needs two more questions: what goal is the policy given, and how much human motion has it seen?
BasicsImitatePriorsModelsScaleInteractReality
20
Foundations · the lens

One loop, three design choices

Every paper keeps this loop and changes one of three things.
1Supervisionwhat is the learning signal?
task rewardEmergencetrack mocapDeepMimicadversarial priorAMPlearned world modelSuperTrackdistill a trackerPULSE
2Interfacewhat is the goal g?
nothingEmergenceclip phaseDeepMimicreference posesDReConlatent zASEtokensMoConVQmasked goals · textMaskedMimic
3Scalehow much human motion?
noneEmergence1 clipDeepMimic30 minutesASE40 hours (AMASS)PHC700 hoursSONIC
Read every paper as one choice per row:Emergence2017DeepMimic2018ASE2022MaskedMimic2024SONIC2025
BasicsImitatePriorsModelsScaleInteractReality
21
Roadmap

Six questions, one thread

Each chapter answers one question; each paper inherits a problem from the one before it.

1 · Imitate

How do we make it look human?
DeepMimic · a backflip
Emergence2017
DeepMimic2018
DReCon2019

2 · Priors

Do we need a reference at all?
CALM · steering
AMP2021
ASE2022
CALM2023
C·ASE2023
NCP2023
Composite2023
AdaptNet2023
SMP2026

3 · Models

Can we learn the simulator and differentiate it?
SuperTrack · tracking
SuperTrack2021
ControlVAE2022
MoConVQ2024

4 · Scale

One policy for all human motion, that we can direct?
MaskedMimic · go there
PHC2023
PULSE2024
MaskedMimic2024
CLoSD2025
GPC2026

5 · Interact

Scenes, objects, sports, hands, and data beyond mocap?
InterPhys · sit, lie down
InterPhys2023
UniHSI2024
TokenHSI2025
Tennis · video2023
SMPLOlympics2024
Omnigrasp2024
InterMimic2025

6 · Reality

Can the character leave the screen?
SONIC · a real Unitree G1
OmniH2O2024
SONIC2025
ScaleBFM2026
SuperMimic2026
Supervision along the way: task rewardtrackingadversarial prior / world modeldistillation
BasicsImitatePriorsModelsScaleInteractReality
22
Chapter 1  ·  2017 – 2019

Imitate

How do we make a simulated human look human?
Emergence of Locomotion Behaviours DeepMind · 2017
DeepMimic SIGGRAPH 2018
DReCon SIGGRAPH Asia 2019
23
DeepMind · arXiv 2017
Reward-only RL

Emergence of Locomotion Behaviours

Reward ≈ forward progress. Let varied terrain, not reward shaping, teach running, jumping and crouching.
Planar walker · hurdles: it jumps
a platform overhead?
Planar walker · platforms: it crouches
gaps in the track?
Quadruped · gaps: it jumps across
What was it rewarded for?
The whole reward · humanoid
$r \;=\; \textcolor{#76B900}{\min(v_x,\ v_{\max})}$
speed along the track,
capped (usually at 4 m/s)
$-\;0.005\,(v_x^2 + v_y^2)$
small penalty
on speed
$-\;\textcolor{#f5b301}{0.05\,y^2}$
stay near the centre
of the track
$-\;0.02\,\lVert u \rVert^2$
small penalty
on the torques
$+\;0.02$
every step,
for not falling
Nothing in it rewards jumping, crouching or turning.Walker and quadruped: no such term either.
Heess, TB, Sriram, Lemmon, Merel, Wayne, Tassa, Erez, Wang, Eslami, Riedmiller, Silver. Emergence of Locomotion Behaviours in Rich Environments. arXiv:1707.02286 · clips: the paper’s video
BasicsImitatePriorsModelsScaleInteractReality
24
How it works

The terrain is the teacher

Curriculumobstacles get harder along the track
Distributed PPOgradients averaged across workers
Reward $\approx v_x$speed along the track
Procedural terraina new random course every episode
Policy $\pi(a \mid s)$no goal input, only what it senses
MuJoCoplanar walker · quadruped · humanoid
torques
body state
$v_x$
reward
update
course
heights ahead
Humanoid · slalom walls
Humanoid · hurdles
Humanoid · gaps
Humanoid · steps
The reward: the faster the body moves along the track, the more it earns. The policy outputs joint torques directly, and nothing tells it how to move. On flat ground, running is all this reward asks for.
Procedural terrain: every episode gets a new random course, built from hurdles, gaps, platforms, slalom walls or uneven ground. An obstacle that stops the body stops the reward: getting past it is the only way to earn more.
Perception: next to its own joint angles and velocities, the policy receives the terrain heights at sample points that move with the body, a little behind it and further ahead. It sees the hurdle coming.
Curriculum: obstacles get harder along the track, so the agent meets tall hurdles only after it clears the low ones. The planar walker learns faster this way than on a course that mixes low and tall hurdles at random.
Distributed PPO: many workers run the same policy, each on its own courses. Every worker computes a gradient; the gradients are averaged into one update, and all workers continue with the updated policy.
The humanoid: the same recipe on a 3D humanoid with 21 actuated joints. It walks around walls, gets over hurdles and jumps gaps. Which gait it finds varies strongly from one random seed to the next.

Unlocked

Deep RL scales to a 3D humanoid: robust locomotion emerges from the environment.

Still open

It runs, but nobody moves like this. Reward alone does not define “natural”.

The course under the pipeline is a schematic, not a simulation · after Heess et al., 2017 (arXiv:1707.02286)
Clips: the paper’s video (Heess et al., 2017)
BasicsImitatePriorsModelsScaleInteractReality
25
SIGGRAPH 2018
Motion imitation

DeepMimic

Give RL a motion-capture clip to imitate — and the motion finally looks human.
Before · deep RL, even with mocap (GAIL, Merel et al. 2017)
No reference clip · DeepMimic’s humanoid develops a peculiar gait
the same body,
with one mocap clip to imitate?
DeepMimic · one mocap clip to imitate: a human-looking run
a stunt,
and a target to hit?
DeepMimic · clip + task reward: the spin-kick hits the target
Deep RL can make a simulated humanoid get somewhere. It does not make it move like a person.
Paper teaser · the humanoid’s cartwheel, a simulated Atlas robot’s spin-kick
Peng, Abbeel, Levine, van de Panne. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. ACM TOG (SIGGRAPH) 2018 · clips: BAIR blog, authors’ video
BasicsImitatePriorsModelsScaleInteractReality
26
How it works

The recipe that never left

Track one reference clip with RL: a clock goes in, PD targets come out, the clip pays the reward.
Goal $g$e.g. where the target is
PPOupdates the policy
Task reward $r^{G}$e.g. is the target hit?
Phase $\phi$a clock, 0 → 1
Policy $\pi(a \mid s, \phi)$Policy $\pi(a \mid s, \phi, g)$sees no reference pose
PD + simulatorpolicy acts at 30 Hz
Tracking reward $r^{I}$four terms, as before
Reference clipone mocap clip, 0.5–5 s
PD targets
state $s$
pose
pose at $\phi$
$r^{I}$
update
$r^{G}$
The policy sees its own state and a clock, the phase. It answers with PD targets, 30 times a second.
The clip’s poses never enter the policy. They pay the reward: how close is the body to the clip at this phase?
A task reward can be mixed in: $r = 0.7\,r^{I} + 0.3\,r^{G}$. The policy then also gets a goal $g$: here, where the target is.
Phase · where the clip is right nowthe policy gets this number, not the pose
what tells the policy
that this is a run?
Blue simulated · green the clip it tracks$r^{I}$: posevelocityend-effectorscentre of mass
and if the kick
has to hit something?
Spin-kick clip + task · the kick finds each targethit rate 99 % with both · clip only 19 % · task only 55 %
Clips: authors’ video (youtu.be/vppFvq2quQ0), “Humanoid: Run” and “Humanoid: Spinkick – Strike” · success rates: Table 4 · the phase figure is a schematic
BasicsImitatePriorsModelsScaleInteractReality
27
How it trains

Two small details make hard skills learnable

Same policy, same reward. What changes is where an episode starts and where it stops.
Default episodes start at the first frame and run a fixed time: the flip is hardly ever reached, most samples lie on the ground.
Reference state initialization: start at a random frame of the clip. The policy meets the landing before it can take off.
Early termination: torso or head on the ground ends the episode, with no reward after it. No samples wasted on the floor.
With both, the same reward learns the backflip. The return alone would not tell: watch the motion, not only the number.
trained
without RSI
Without RSI · a small hop, never a flipnormalized return 0.730
trained
without early termination
Without ET · falls, then mimes it lying downnormalized return 0.379
trained
with both
RSI + ET · the full backflipnormalized return 0.791
Clips: authors’ video (youtu.be/vppFvq2quQ0), backflip ablation; blue = simulated, green = reference · returns: Table 5 · the episode figure is a schematic
BasicsImitatePriorsModelsScaleInteractReality
28
Under the hood

DeepMimic: Phase-conditioned policy, tracking reward, RSI + early termination

Value $V(s, g)$similar MLP, one linear output
PPOclip $\epsilon = 0.2$ · $\gamma = 0.95$
$r = 0.7\,r^{I} + 0.3\,r^{G}$imitation + task
Phase $\phi$ + goal $g$$\phi \in [0, 1]$, part of $s$ goal: heading, target, clip id
Policy $\pi(a \mid s, g)$MLP 1024-512 → $\mu(s)$, fixed $\Sigma$
PD + simulatorpolicy 30 Hz · Bullet 1.2 kHz
Imitation reward $r^{I}$pose, velocity, end-effectors, centre of mass
Reference clip $\{\hat q_t\}$0.5–5 s of mocap or keyframes, retargeted
Early terminationtorso or head touches the ground → zero reward
Reference state initsampled uniformly from the reference motion
PD targets
state $s$
pose $q_t$
target $\hat q_t$
RSI: start state
ET: end episode
task reward $r^{G}$
update
GAE($\lambda$)
TD($\lambda$)
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: phase and goal in, PD targets out, state backEvery step is scored against the clip: four kernelsRSI sets where an episode starts, ET where it stopsPPO maximizes imitation + task reward; a critic helpsWhat it costs, and what the two tricks buy
Objective
  • $r^{p}_t = \exp\big[-2 \sum_j \|\hat q^{j}_t \ominus q^{j}_t\|^2\big], \quad r^{v}_t = \exp\big[-0.1 \sum_j \|\hat{\dot q}^{j}_t - \dot q^{j}_t\|^2\big]$
  • $r^{e}_t = \exp\big[-40 \sum_e \|\hat p^{e}_t - p^{e}_t\|^2\big], \quad r^{c}_t = \exp\big[-10\,\|\hat p^{c}_t - p^{c}_t\|^2\big]$
  • $r_t = 0.7\,r^{I}_t + 0.3\,r^{G}_t, \quad r^{I}_t = 0.65\,r^{p}_t + 0.1\,r^{v}_t + 0.15\,r^{e}_t + 0.1\,r^{c}_t$
  • PPO: $L^{CLIP} = \mathbb{E}\big[\min\big(w_t A_t,\ \mathrm{clip}(w_t, 1-\epsilon, 1+\epsilon)\,A_t\big)\big]$, $\ w_t = \pi_\theta / \pi_{\theta_{old}}$, $\epsilon = 0.2$
Training recipe
  1. Retarget one 0.5–5 s clip to the character; start π and V from random weights
  2. Start episodes at random clip frames (RSI); stop when torso or head hits the ground (ET)
  3. Every 4,096 samples: minibatches of 256; update V on TD(λ) targets, π on the clipped loss
  4. Tasks: add $r^{G}$ and $g$. Terrain: pre-train on flat ground, then add heightmap conv layers
≈ 60 Msamples per skill
≈ 2 dayson an 8-core CPU
0.5–5 sof mocap per skill
197 → 36state → action dimensions
The detail that makes it work

RSI shows the policy high-reward mid-air states before it can reach them; ET stops fallen-state samples from dominating the data.

Versus its predecessor

vs Emergence (task-only reward) and GAIL (2017, reward learned by a discriminator): an explicit per-frame tracking reward toward one reference clip.

BasicsImitatePriorsModelsScaleInteractReality
29
Results

One clip in, one robust skill out

Cartwheel · sim vs mocap
Side flip
Kip-up
Speed vault
… with a task on top?
Imitate + task: hit the target
… on a much heavier body?
Retargeted to Atlas · boxes thrown
… while it is being hit?
Atlas spin-kick · perturbed
… with no mocap at all?
Keyframed T-Rex
Blue = simulated
Green = motion-capture reference
4
charactershuman · Atlas · T-Rex · dragon
1 : 1
one clip, one policyper skill

Unlocked

Mocap quality, physically robust: survives pushes, adds task rewards, retargets to other bodies.

Still open

One policy per clip, days of training each. Steering and skill switching only within a handful of clips.

Clips: BAIR blog “Towards a Virtual Stuntman” (Peng, 2018)
BasicsImitatePriorsModelsScaleInteractReality
30
SIGGRAPH Asia 2019

DReCon: a physics character
you can drive with a gamepad

Bergamin, Clavet, Holden, Forbes · Ubisoft La Forge / McGill
31
How it works

Split the problem: plan kinematically, track physically

Gamepaddirection, heading, speed, style
Motion matchinga search, nothing is learned
Kinematic characterposes only, no physics
Mocap database≈ 10 min of locomotion
Policy $\pi(a \mid s)$MLP 128-128, 30 Hz
PD + simulatorsimulated body, 60 Hz
Tracking rewardhow far from the kinematic character
open-loop PD targets
kinematic state
simulated state
offsets
target
PPO
A gamepad steers an animated character. It is pure animation: nothing can push it, and nothing has to balance.Motion matching: every 10 frames, find the mocap frame that best fits the current pose and the wanted path. Play on from there.Its joint angles become the PD targets of a simulated body. Open loop comes close, but it cannot keep its balance.A small policy watches both characters and adds corrective offsets to the PD targets: 110 numbers in, 25 out.PPO trains it to stay on the kinematic character while a random user keeps steering. A fall ends the episode.
$r \;=\; e_{fall}\,\big(\,r_p + r_v + r_{local} + r_{vCM}\,\big)$
fall factor  $e_{fall} = \mathrm{clamp}\big(1.3 - 1.4\,d_{head},\ 0,\ 1\big)$
Kinematic character: follows the gamepad, not simulated
Early in training: blue falls and is put back on white
Trained: blue = simulated, white = the kinematic target
Bergamin, Clavet, Holden, Forbes. DReCon: Data-Driven Responsive Control of Physics-Based Characters. ACM TOG (SIGGRAPH Asia) 2019 · clips: authors’ video · figure: schematic
BasicsImitatePriorsModelsScaleInteractReality
32
Under the hood

DReCon: A tiny corrective policy on top of motion matching

Value $V(s)$MLP 128-128
PPO$\gamma = 0.99$, 40,000 samples per epoch
Tracking reward$e_{fall}\,(r_p + r_v + r_{local} + r_{vCM})$
User controlgamepad: direction, heading, speed, style
Policy $\pi(a \mid s)$MLP 128-128 tanh, 30 Hz, 110-d state → 25-d
PD + Bullet, 60 Hztarget = kinematic pose ⊕ smoothed offsets
Mocap database≈ 10 min unstructured locomotion, 60 fps
Motion matchingnot learned · 27-d match every 10 frames
Kinematic characteranimation stream that follows the user
desired $v$
kinematic state
offsets
open-loop targets
simulated state, previous offsets
simulated state
tracking target
update
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Not learned: motion matching turns the gamepad into posesThe loop: kinematic pose as PD target, plus smoothed offsetsFour terms compare the body with the kinematic characterSummed, then scaled by a fall factor on the head distancePPO trains policy and value against a random userWhat it costs, and what makes it work
Objective
  • $r_p = \exp\!\big(-\tfrac{10}{N}\sum_{i,j}\|\hat p_{ij} - p_{ij}\|\big)$, $\ r_v = \exp\!\big(-\tfrac{1}{N}\sum_{i,j}\|\hat v_{ij} - v_{ij}\|\big)$ · 6 points $i$ × $N$ bodies $j$
  • $r_{local} = \exp\!\big(-\tfrac{10}{N}\sum_{j}\|\hat a_j \ominus a_j\|_q\big)$, $\ r_{vCM} = \exp\!\big(-\|\hat v_{CM} - v_{CM}\|\big)$
  • $r_t = e_{fall}\,\big(r_p + r_v + r_{local} + r_{vCM}\big)$, $\ e_{fall} = \mathrm{clamp}\big(1.3 - 1.4\,\|\hat p_{head} - p_{head}\|,\ 0,\ 1\big)$
  • hats $\hat p,\ \hat v,\ \hat a$ = kinematic character · $e^{-\sum_j d_j} = \prod_j e^{-d_j}$: the worst body caps each term
Training recipe
  1. Drive motion matching with a random user: each frame, P = 0.001 of a new command
  2. Initialize the last policy layer in ±0.01, so the open-loop targets dominate at first
  3. Start each episode on the kinematic pose; smooth the offsets with an EMA, β = 0.2
  4. Stop the episode on a fall (heads > 1 m apart) or after 20 s; train ≈ $3 \times 10^8$ steps (≈ 30 h)
128 × 2hidden units: a very small policy
340 µswhole pipeline, one CPU thread
60 Hzphysics (DeepMimic: 1.2 kHz)
30 Hzpolicy rate
The detail that makes it work

Inside each reward term the per-body errors multiply (exp of a sum), so the worst-tracked body caps the reward and cannot be traded against the others.

Versus its predecessor

vs DeepMimic: no phase and no fixed clip. The reference is generated online from user input, and the network is 128-128 instead of 1024-512.

BasicsImitatePriorsModelsScaleInteractReality
33
Results

What the split buys

ResponsiveIt goes where the stick points.
A gamepad user traces the black path (red = walked)
1.2 s

rise time of a 90° turnDeepMimic 2.0 s, Liu & Hodgins 5.1 s (read off their videos)

RobustA hit lands on the body, not on the plan.
Impact test: 0.2 m cubes, launched at 5 m/s

The cubes never touch the plan. The policy only has to bring the body back onto it.Robust even when trained without projectiles

CheapSmall enough for a game.
Runtime of each stage on one CPU thread (paper, Table 4)
340 µs

for the whole pipelineMLP 128-128, physics at 60 Hz (DeepMimic: 1024-512, 1.2 kHz)

Unlocked

Planner decides what, tracker decides how. Responsive, robust, and cheap enough for a game.

Still open

The tracker only knows its locomotion data, and RL is slow. Three ways out.

Clips: authors’ video · numbers: Tables 2, 3, 4 and Sec. 7.1 of the paper
BasicsImitatePriorsModelsScaleInteractReality
34
The fork

Three roads out of 2019

DReCon leaves two problems open. The following years answer them in three ways.
Still openA tracker is only as good as its reference
DReCon · blue tracks white
Still openModel-free RL is slow

Drop the reference

Chapter 2 · Priors

Do we need a reference at all?

CALM · walks to the flag

Learn the simulator

Chapter 3 · Models

Can we learn the simulator and differentiate it?

SuperTrack · tracking

Scale the tracker

Chapter 4 · Scale

One policy for all human motion, that we can direct?

PHC · kicks and a cartwheel
2019DReCon splits control into a kinematic planner and a physics tracker. Where does the field go from here?
Road ALearn what human motion looks like as a prior; let the task reward decide what to do.
Road BFit a differentiable world model, and train controllers by backpropagating through it.
Road CMake one model-free policy track all human motion, then worry about who supplies the reference.
SpoilerThe roads keep crossing: priors get distilled from trackers (2024), and the world model returns on real robots (2026).
BasicsImitatePriorsModelsScaleInteractReality
35
Chapter 2  ·  2021 – 2026

Priors

Do we need a reference motion at all?
AMP adversarial motion prior · 2021
ASE reusable skill latent · 2022
CALM directable · 2023
C·ASE skill labels · 2023
Neural Categorical Priors discrete codes · 2023
Composite Motion per body part · 2023
AdaptNet fast adaptation · 2023
SMP diffusion prior as reward · 2026
36
SIGGRAPH 2021
Adversarial imitation

AMP: Adversarial Motion Priors

Don’t track the data. Ask a discriminator whether the motion looks like the data.
Obstacle course: run, leap, roll
Rewardmove forward at a target speed
Clips10 clips, 22 s: running, leaping, rolling
Walk to the target and punch it
Rewardreach the target and hit it
Clips15 clips, 248 s: walking only or punching only
Tracking reward · DeepMimic, DReCon
“How far are you from the reference at frame t?”
DeepMimic: a phase to stay in syncDReCon: a planner to pick the clip
Style reward · AMP
“Does this motion look like the data?”
no phaseno clip selectionno planner
Peng, Ma, Abbeel, Levine, Kanazawa. AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. ACM TOG (SIGGRAPH) 2021 · clips: authors’ video
BasicsImitatePriorsModelsScaleInteractReality
37
How it works

Two rewards: a task you write, a style you show

PPOmaximises the reward
Task goal $g$where the target is
Policy $\pi(a \mid s, g)$
PD + simulator
Discriminator $D$data, or policy?
Motion clipsno labels, no order
action $a$
state $s$
task reward
update
$(s, s')$
$(s, s')$
style reward

A task you write. The reward pays for coming close to the target, then for hitting it, and PPO trains the policy on it. On its own it gives this: the target is hit, by an unnatural, awkward motion.

A style you show. 15 clips of walking and of punching: no labels, no actions. A discriminator learns to tell their transitions $(s, s^\prime)$ from the policy’s: the learned judge from Foundations.

Two rewards meet in the policy. The judge’s score becomes the style reward, and PPO maximises $r = 0.5\,r_{\text{task}} + 0.5\,r_{\text{style}}$. Same task: now it walks up and punches.

Nobody picks the clip. The discriminator accepts whatever looks like the data: a walk, or a punch. The task reward decides which one pays: walking while the target is far, a punch once it is within reach.

Task reward only
how should it move?
The data: punching clips, walking clips
with both rewards?
Task + style reward (AMP)
who decides when to walk
and when to punch?
Schematic: the glow marks the clips the motion resembles
Clips: AMP supplementary video (Peng et al., 2021), “Humanoid: Strike”, panels “No Data” and “AMP (Ours)”, and “Dataset: Walk + Punch”
BasicsImitatePriorsModelsScaleInteractReality
38
Under the hood

AMP: A discriminator on state transitions replaces the tracking reward

Value $V(s, g)$similar MLP, separate network
PPO$\gamma$ = 0.95 (imitation), 0.99 (tasks)
$r = 0.5\,r^{G} + 0.5\,r^{S}$task + style
Style reward$\max[0,\ 1 - 0.25\,(D-1)^2]$
Task goal $g$heading, location or strike target
Policy $\pi(a \mid s, g)$MLP 1024-512 · no phase
PD + simulatorpolicy 30 Hz · Bullet 1.2 kHz
Discriminator $D(s, s')$similar MLP · least squares + gradient penalty
Motion clipsunstructured data, no labels, no segmentation
Replay buffer$10^5$ past policy transitions
PD targets
state $s$
$(s, s')$
store
fake
real
task reward
update
GAE($\lambda$)
TD($\lambda$)
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: goal in, PD targets out, state backA discriminator compares policy transitions with the clipsIts score becomes the style reward, mixed with the task rewardPPO updates the policy; a critic supplies the advantageWhat it costs, and what makes it stable
Objective
  • $\min_D\ \mathbb{E}_{\mathcal{M}}\big[(D-1)^2\big] + \mathbb{E}_{\pi}\big[(D+1)^2\big] + \tfrac{w^{gp}}{2}\,\mathbb{E}_{\mathcal{M}}\,\|\nabla_{\phi} D\|^2$
  • gradient penalty on dataset samples, $w^{gp} = 10$ · $\phi = (\Phi(s), \Phi(s'))$, the features $D$ sees
  • $r^{S} = \max\big[\,0,\ 1 - 0.25\,(D(s,s') - 1)^2\,\big]$
  • $r = 0.5\,r^{G} + 0.5\,r^{S}$
Training recipe
  1. Initialize D, π and V from scratch: there is no pre-training stage
  2. Roll out with random start frames (RSI); stop the episode on a fall
  3. Score every transition with D; mix style and task reward
  4. Update D on 256 dataset + 256 replay-buffer transitions, then π and V with PPO
100–300 Msamples per policy
30–140 hon 16 CPU cores
≤ 434 slargest dataset: 56 clips
10⁵replay buffer for D
The detail that makes it work

The gradient penalty on real samples is “the most vital component”: without it training oscillates and visible artifacts appear.

Versus its predecessor

vs DeepMimic: no phase, no clip selection, no hand-built tracking reward. The cost: slower learning and mode collapse on diverse data.

BasicsImitatePriorsModelsScaleInteractReality
39
Results

One recipe: swap the clips, swap the task

Go to the flag: locomotion style, then zombie style

Swap the clips, swap the style

Same task, same reward. Trained once with locomotion clips, once with a single zombie clip of 18 seconds.

Dribbling: style from locomotion clips only

A task no clip shows

The data has walking and running, no ball. The dribbling comes from the task reward, the gait from the clips.

Knocked down by boxes: gets up, keeps going

Add clips, add a skill

With get-up clips added to the locomotion data, a character that falls gets up and carries on with its task.

≤ 434 s
of motion clips
in the largest dataset
100–300 M
samples
for every policy
30–140 h
on 16 CPU cores
for every policy

Unlocked

No tracking, no planner: style from unstructured clips, any task reward, skills compose on their own.

Still open

Trained from scratch for every task. The prior is not reused — and GANs drop modes.

Clips: authors’ supplementary video (youtu.be/wySUxZN_KbM)
BasicsImitatePriorsModelsScaleInteractReality
40
SIGGRAPH 2022
Reusable skills

ASE: Adversarial Skill Embeddings

Pre-train one low-level controller over a latent skill space. Reuse it for every new task.
Pre-training: dozens of the 4,096 parallel characters
A new latent (the marker changes colour), a new skill
30 min
of mocap187 clips · the whole dataset
10 years
of simulated experience> 10 billion samples
4,096
parallel charactersIsaac Gym · one GPU
Peng, Guo, Halper, Levine, Fidler. ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters. ACM TOG (SIGGRAPH) 2022 · clips: authors’ video
BasicsImitatePriorsModelsScaleInteractReality
41
How it works

Pre-train once, then steer with a latent

Latent $z$on the unit sphere
Low-level policy$\pi(a \mid s, z)$ · frozen$\pi(a \mid s, z)$
PD + simulatorIsaac Gym
Discriminatorfrozen · style reward$D(s, s')$: like the clips?
Motion clips30 min, no labels
High-level policy$\omega(z \mid s, g)$, one per task
Encoder$q(z \mid s, s')$: which $z$?
$a$
$(s, s')$
real
pre-training reward $-\log(1 - D)$
pre-training reward $+\,\beta \log q$
picks $z$
reward $0.9\,r_{\text{task}} + 0.1\,r_{\text{style}}$
One policy for every skill. A latent $z$, a random point on a sphere, tells it which one.
Natural: the reward $-\log\big(1 - D(s, s')\big)$ is high when a transition looks like the clips, whatever $z$ is.
Distinct: the reward $\beta \log q(z \mid s, s')$ is high when the motion gives away which $z$ produced it.
Reuse: freeze the low-level policy and $D$. For each task, train only a high-level policy: it picks $z$ 6 times a second.
The goal: a new latent (marker colour), a new skill
Six of the 187 motion clips
Strike task: run up, swing, stand idle
Clips: authors’ video (youtu.be/hmV4v_EnB0E) · the figure on the left is a schematic
BasicsImitatePriorsModelsScaleInteractReality
42
Under the hood

ASE: Skill-conditioned policy: adversarial + mutual-information pre-training

Stage 1: pre-training the low-level policy
Stage 2: task training, once per task
Diversity lossaction KL ≈ latent distance, $w_{div} = 0.01$
PPO + value $V(s, z)$$\gamma = 0.99$ · GAE $\lambda = 0.95$
$r = -\log\big(1 - D(s, s')\big) + \beta\,\log q(z \mid s, s')$looks like the data + $z$ can be read back from the motion · $\beta = 0.5$
Latent $z \sim p(z)$64-d unit sphere, uniform, held 1–150 steps
Low-level $\pi(a \mid s, z)$MLP 1024-1024-512, 31-d Gaussian action
PD + Isaac Gympolicy 30 Hz · physics 120 Hz
$D(s, s')$ and $q(z \mid s, s')$discriminator + skill encoder, one network
Motion clips187 clips ≈ 30 min, unsegmented, unlabeled
Task goal $g$reach, speed, steering, location, strike
High-level $\omega(\bar z \mid s, g)$MLP 1024-512 · 6 Hz, latent held 5 steps
$r = 0.9\,r^{G} + 0.1\,r^{S}$task + style · $\pi$ and $D$ frozen
PD targets
120-d state $s$
$(s, s', z)$
real
$z = \bar z / \|\bar z\|$
task $r^{G}$
style $-\log(1 - D)$
PPO
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: a latent in, PD targets out, state backOne network scores each transition: real? which z?PPO updates π; actions must differ as much as latents doStage 2: π and D frozen; one high-level policy per taskWhat it costs, and the detail that makes it work
Objective
  • Stage 1: $r_t = -\log\big(1 - D(s_t, s_{t+1})\big) + \beta\,\log q(z_t \mid s_t, s_{t+1})$, $\ \beta = 0.5$
  • $q \propto \exp\big(\kappa\,\mu_q(s, s')^{\top} z\big)$: $\ \max_q\ \mathbb{E}_{\pi}\big[\kappa\,\mu_q(s, s')^{\top} z\big]$ · $D$: GAN loss + gradient penalty, $w_{gp} = 5$
  • Diversity: $-\,w_{div}\,\mathbb{E}\big[(D_{KL}(\pi_{z_1}, \pi_{z_2})\,/\,D_z(z_1, z_2) - 1)^2\big]$, $\ D_z = 0.5\,(1 - z_1^{\top} z_2)$, $\ w_{div} = 0.01$
  • Stage 2: $r_t = 0.9\,r^{G} - 0.1\,\log\big(1 - D(s_t, s_{t+1})\big)$, with $\pi$ and $D$ frozen
Training recipe
  1. Sample latents on the sphere, hold each for 1–150 steps; start 10 % of episodes fallen
  2. Roll out π; reward = looks real (D) + z can be recovered from the motion (q)
  3. Update q on policy samples, D on dataset vs policy transitions, then π and V with PPO
  4. Freeze π and D; train one high-level policy per task at 6 Hz, with no motion data
4,096environments on one V100
> 10 Bsamples ≈ 10 simulated years in 10 days
187 clips≈ 30 minutes of mocap
131 ksamples per update
The detail that makes it work

A bounded hypersphere avoids out-of-distribution latents; the random fallen starts are why get-up recovery emerges with no get-up data.

Versus its predecessor

vs AMP: the prior becomes a reusable, pre-trained low-level policy instead of a reward that is retrained with every task.

BasicsImitatePriorsModelsScaleInteractReality
43
Speed: match the target speed (green arrow)
Steering: move one way, face another
Reach: sword tip to the red target
Strike · trained from scratch
Strike · ASE
Hit by a box: falls, gets back up
ASE · results

Five tasks, one low-level policy

Unlocked

The foundation-model shape: large-scale pre-training, then reusable motor skills.

Still open

The latent is tied to no motion: you cannot ask for a particular one, and coverage of the data is uneven.

From scratch: higher return on 4 of the 5 tasks, by moving unnaturally.
Pushed 500 times, it got up every time. No get-up clip in the data.
BasicsImitatePriorsModelsScaleInteractReality
44
SIGGRAPH 2023

CALM: ask for a motion, get that motion

Tessler, Kasten, Guo, Mannor, Chechik, Peng.
Each latent is the encoding of a real motion.
45
How it works

A discriminator that also sees the latent

Motion window $M$2 s of a real clip
Encoder $E$$z = E(M)$, on a 64-d unit sphere
Policy $\pi(a \mid s, z)$low level, 30 Hz
PD + simulatorIsaac Gym
Discriminator $D(s, s' \mid z)$is it the motion of $z$?
$z$
$a$
$(s, s')$
the same $z$
real: frames of $M$
reward $r$
gradient

ASE: the latent z is drawn at random on a sphere. Every z gives some skill, but no z is tied to a clip: there is no reliable way to ask for the kick.

Encoder. The clips are cut into overlapping 2-second windows. Each window M is encoded to a point z = E(M) on the sphere: every motion has an address.

Conditional discriminator. It is given the same z, and it learns to reject real motion that is paired with another latent. A natural sprint satisfies the old judge, but it does not pass for the kick.

Reward $r = -\log\big(1 - D(s, s^\prime \mid z)\big)$: the policy is paid when its motion passes for M. The policy’s gradient flows on through z and also trains the encoder.

On demand. Encode a motion, hand z to the policy, get that motion. Raters judged 78 % of CALM’s motions similar to the requested one; with ASE, 35 %.

A map of motions. Slide z from E(sprint) to E(crouch-idle): the character slows down and sinks, walking all the way.

Four requests in a row: name top left, reference top right
Interpolating in latent space: sprint → crouch-idle
Tessler, Kasten, Guo, Mannor, Chechik, Peng. CALM: Conditional Adversarial Latent Models for Directable Virtual Characters. SIGGRAPH 2023 · clips: authors’ video · stick figures: a schematic
BasicsImitatePriorsModelsScaleInteractReality
46
Under the hood

CALM: Policy and discriminator both conditioned on the encoding of one motion

Stage 1: low-level training
Stage 2: precision training
Alignment + uniformityoverlapping windows close, random pairs spread out
PPOupdates the policy and, through $z$, the encoder
Reward$-\log\big(1 - D(s, s' | z)\big)$
Encoder $z = E(M)$60 frames → MLP → 64-d unit hypersphere
Low-level $\pi(a \mid s, z)$input head $H(z)$, then MLP → 31-d action
PD + Isaac Gym37-DoF sword-and-shield humanoid · policy 30 Hz
Discriminator $D(s, s' | z)$sees 10-state windows · negatives: data with random $z'$
Motion clipsunlabeled · split into overlapping sub-motions
Rewardheading $d^*$ + latent similarity ${\exp(-4\,\|z_t - \hat z\|^2)}$
High-level policyMLP · 6 Hz · sees $s$, $d^*$, requested motion
Low-level $\pi(a \mid s, z_t)$frozen together with the encoder $E$
random 2-s window $M$
$z$
gradient
PD targets
120-d state $s$
fake
real, from the same $M$
same $z$ · stop-gradient
latent $z_t$
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The loop: a window M becomes z, the policy acts on (s, z)D sees the same z: real, fake, and data with a random z′Its score is the reward: PPO trains π and, through z, ETwo more losses shape where the windows land on the sphereStage 2: freeze π and E, train a policy that picks zWhat it costs, and why a motion can now be requested
Objective
  • $\mathcal{L}_D = -\mathbb{E}_{\pi}\log(1 - D(s, s' | z)) - \mathbb{E}_{M}\big[\log D(\hat s, \hat s' | z) + \log(1 - D(\hat s, \hat s' | z'))\big] + w_{\text{gp}}\mathbb{E}_{M}\|\nabla D\|^2$
  • $r = -\log\big(1 - D(s, s' | z)\big)$, $z = E(M)$: one reward trains $\pi$ and, through $z$, the encoder $E$
  • $\mathcal{L}_{\text{align}} = \mathbb{E}\,\|E(M) - E(M')\|_2^2$ (overlapping), $\mathcal{L}_{\text{uniform}} = \log \mathbb{E}\,e^{-2\|E(M) - E(M')\|_2^2}$ (i.i.d.)
  • High level: $r = \exp\!\big(-0.25\,\|d^* - \dot x^{\text{root}} / \|\dot x^{\text{root}}\|\,\|^2\big) + \exp\!\big(-4\,\|z_t - \hat z\|^2\big)$, $\hat z = E(M)$
Training recipe
  1. Encode a random 2-s window M; roll out π; swap M at random steps to learn transitions
  2. Train D on real, fake and negative pairs (data with a random z′); no gradient from D into E
  3. Train π and E with PPO; E also gets the alignment and uniformity losses
  4. Freeze E and π; train the high-level policy; an FSM commands it or sends ẑ to π
4,096environments on one A100
5 Blow-level training steps
160 clips> 30 minutes of mocap
78 % vs 35 %controllability, CALM vs ASE
The detail that makes it work

The discriminator sees the same latent as the policy, so imitating just any motion no longer fools it: this is what counters mode collapse.

Versus its predecessor

vs ASE: z comes from an encoder of real motions, so a chosen motion can be requested; ASE’s latents had no such correspondence.

BasicsImitatePriorsModelsScaleInteractReality
47
How it is directed

A steering policy, then a state machine

State machinehand-written rules
High-level policypicks $z_t$, 6 Hz
Low-level $\pi(a \mid s, z)$frozen, with the encoder
PD + simulator
Rewardalong $d^*$ + near $\hat z$
$\hat z,\ d^*$
$z_t$
$a$
trains
isolated motions: $\hat z$ directly

Frozen low-level policy. Ask for a sprint with ẑ = E(sprint) and it sprints. But nothing tells it where to.

Precision training. A high-level policy picks zt. It is rewarded for moving along d* and for keeping zt near the requested ẑ.

State machine. Hand-written rules choose ẑ and d*: move to the target, slash, celebrate. Nothing is trained.

No task training. The location and strike tasks score 0.96 – 1.0, in three locomotion styles with three endings each.

Low-level policy alone: asked to sprint, then to walk
a sprint, a crouch-walk …
in a direction you choose?
Keeps the crouch-walk (ẑ), turns with the arrow (d*)
reach a target, strike it, celebrate:
a new reward for every task?
State machine: move to the target, slash, celebrate

Unlocked

Directable skills: a latent per motion, one steering policy, and a state machine replace per-task rewards.

Still open

With many motions, some are still imitated far more than others. Coverage.

Tessler, Kasten, Guo, Mannor, Chechik, Peng. CALM: Conditional Adversarial Latent Models for Directable Virtual Characters. SIGGRAPH 2023 · clips: authors’ video and project page
BasicsImitatePriorsModelsScaleInteractReality
48
SIGGRAPH Asia 2023
Conditional skill embeddings

C·ASE: give every skill a label

Label every clip with its skill. The label becomes the control handle: ask for a skill by name.
The user picks the skill (label at the top) and the direction (blue arrow)
the same handle,
on a path that the user gives?
Path following: the user gives the path (blue line) and picks the skill
How do you ask for a skill?
ASE · sample a latent za random point on a sphere, tied to no clip
CALM · show a motionz is the encoding of an example clip
C·ASE · name it: a skill label cthe latent z only picks the variation
Dou, Chen, Fan, Komura, Wang. C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters. SIGGRAPH Asia 2023 · clips: authors’ video
BasicsImitatePriorsModelsScaleInteractReality
49
How it works

Divide the data by skill, then conquer

Skill $c$ + latent $z$which skill, which variation
Low-level ${\pi(a \mid s, z, c)}$one policy, all skills
PD + simulatorIsaac Gym
Discriminator ${D(s, s', c)}$like skill $c$?
Labelled clipsone subset per skill label
Encoder ${q(z \mid s, s', c)}$which $z$ was it?
Focal skill samplinglow score, drawn more often
$a$
$(s, s')$
real
score $b_c$
Reward of the policy
$r_t = -\log\big(1 - D(s_t, s_{t+1}, c)\big)$
$+\ \beta\,\log q(z_t \mid s_t, s_{t+1}, c)$
Every clip gets a skill label c. The policy takes the label as one more input.c says which skill. The latent z, kept from ASE, now only says which variation.
The discriminator gets the label too. It no longer asks “like the data?” but “like skill c?”Standing idle is real motion, but it is not a kick.
The encoder gets the label too. Inside skill c it must tell from the motion which z was used.So different z have to look different: variations of one skill.
Focal skill sampling. A skill whose motion the discriminator still rejects (low average score bc) is drawn more often as reference:$w_c \leftarrow (1-\alpha)\,w_c + \alpha\,\big(1 - b_c \,/ \textstyle\sum_{c^\prime} b_{c^\prime}\big)$
Authors’ clip: one fixed skill label, random latents
Filtered motion coverage · 87 clips
C·ASE≈ 91 %
CALM71 %
ASE66 %
A clip counts only if enough of the random rollouts match it.
On 265 skills: 82 %  ·  CALM 58 %  ·  ASE 51 %
ASE, epoch 30,000: boxed characters stay inactive
C·ASE, same epoch: actively practising various skills

Unlocked

Explicit skill control and balanced coverage across 265 skill types: one hard matching problem becomes many easy ones.

Still open

Still adversarial, and it needs labels. Is the GAN necessary at all?

Clips: authors’ video (youtu.be/Cgq6JbQ1VW4) · stick figures and bars are schematic, not data
BasicsImitatePriorsModelsScaleInteractReality
50
Under the hood

C·ASE: ASE with a skill label: π, D and encoder all conditioned on c

Stage 2: high-level control
PPO + value $V$$V$: MLP similar to $\pi$ · + diversity term on $\pi$
Reward $r_t$$-\log(1 - D) + \beta \log q$
Focal skill samplingskills with a low $D$ score are sampled more
Skill $c$ + latent $z$64-d embedding of $c$ · 16‑d unit sphere for $z$
Low-level $\pi(a \mid s, z, c)$MLP 1024-1024-512 → PD + residual forces
PD + Isaac Gympolicy 30 Hz · simulation 120 Hz
Encoder $q(z \mid s, s', c)$MLP 1024-1024-512 · von Mises-Fisher
Discriminator $D$MLP 1024-512 + dropout, one output per skill $c$
Labelled clipsup to 778 clips · 265 skill labels
User or task goalskill $c$ + direction or target location
High-level policy $\omega$MLP 1024-512 · 6 Hz · task reward
state $s$
$(s, s')$
real
score $b_c$
sampling prob.
latent $z$ (+ $c$)
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: skill c and latent z in, PD targets outTwo judges read the motion and the label: D and encoder qTheir two scores are the reward; PPO updates the policyFocal sampling: low D score, more reference samplesStage 2: a 6 Hz policy steers; the user names the skillWhat it costs, and the detail that makes it work
Objective
  • $\min_D\ -\mathbb{E}_{\mathcal{M}}\big[\log D(s,s',c)\big] - \mathbb{E}_{\pi}\big[\log(1 - D(s,s',c))\big] + w_{gp}\,\mathbb{E}_{\mathcal{M}}\,\|\nabla D\|^2$ · 20% dropout
  • $r_t = -\log\big(1 - D(s_t, s_{t+1}, c)\big) + \beta\,\log q(z_t \mid s_t, s_{t+1}, c)$ · encoder: $\max_q\ \mathbb{E}\big[\kappa\,\mu_q(s,s',c)^{\top} z\big]$
  • $\max_\pi\ \mathbb{E}\big[\textstyle\sum_t \gamma^t r_t\big] - \lambda_D\,\mathbb{E}\big[(D_{KL}(\pi(\cdot|s,z_1,c),\,\pi(\cdot|s,z_2,c))\,/\,[0.5\,(1 - z_1^{\top} z_2)] - 1)^2\big]$
  • Focal sampling: $w_c \leftarrow (1-\alpha)\,w_c + \alpha\,\big(1 - b_c / \textstyle\sum_{c'} b_{c'}\big)$, $\alpha = 0.2$, $b_c$ = mean $D$ on $\pi$'s skill $c$
Training recipe
  1. Label each clip with a skill c; start episodes at a random state with a random label
  2. Roll out π with z held 1–150 steps; train D (real vs policy) and q; train π and V with PPO
  3. Start focal sampling at ≈ 2,500 steps; allow body-ground contact after 15,000 epochs
  4. Stage 2: reuse π; train a 6 Hz high-level ω that outputs z; the user (or ω) picks skill c
1.5 Bsamples · 1.5 days · one A100
265skills in 778 clips
82%filtered coverage, 265 skills (CALM 58, ASE 51)
44.3%transition coverage, 87 clips (ASE 25.4)
The detail that makes it work

Focal skill sampling: the discriminator's per-skill score is a free difficulty signal, so hard skills are sampled more and are not dropped.

Versus its predecessor

ASE: one unconditional D over all clips, so coverage is uneven. C·ASE conditions π, D and encoder on a skill label, one homogeneous subset each.

BasicsImitatePriorsModelsScaleInteractReality
51
SIGGRAPH Asia 2023
Discrete priors

Neural Categorical Priors

No discriminator: track with RL through a VQ bottleneck, then learn a categorical prior over the codes.
Codes sampled at random from the prior
Coverage by random sampling
NCP94.6 %
ASE77.1 %
CALM69.6 %
clip frames reproduced · sword & shield data
Strike task: knock the pillar over
Reuse

A task policy only picks codes for the frozen decoder. The strike task trains within 4 hours.

task set up as in ASE
Two-player boxing
Self-play

The policy trains against past versions of itself. Defense and dodging emerge.

prioritized fictitious self-play
Zhu, Zhang, Lan, Han. Neural Categorical Priors for Physics-Based Character Control. SIGGRAPH Asia 2023 (Tencent Robotics X) · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
52
How it works

One codebook of skills, three ways to pick a code

Motion clips≈ 26 or 30 min of mocap
Encoder $p^e(z^e \mid s, f)$sees 2 future clip frames
Codebook $\mathbf{e}$$K$ = 512 or 256 codes
Decoder $\pi^d(a \mid s, e_k)$state + code → PD targets
PD + simulatorIsaac Gym · policy 30 Hz
Prior $\pi^p(k \mid s)$sees only the state
Upper-level policy$\pi^h(k \mid s, g)$ · sees the goal
$f$
$z^e$
$e_k$
tracking reward · PPO trains both halves
distill
code $k$
count reward
KL
code $k$
task reward
?
Stage 1: one policy tracks every clip, DeepMimic-style. It is cut in two, and only the encoder sees the clip.Between the halves, the encoder’s output snaps to the nearest of K codes: one code per control step.Stage 2: a prior learns the encoder’s pick from the state alone. Sample from it: motion, with no clip to track.The prior copies the data: a combination of punches stays rare. A count reward pays for rarely visited clip frames until visits even out.Stage 3: a task policy picks the code, the decoder stays frozen, and a KL term keeps its choice near the prior.
Tracking the boxing clips
Schematic: the nearest code wins
Codes sampled from the prior
count reward $r = \sqrt{N_k / \hat N(s)}$
Schematic: visits even out
Two-player boxing

Unlocked

A discrete skill space without a GAN: sharp, distinct skills, good enough for competitive boxing.

Still open

Every new behavior still means a new policy. Can skills be combined, or adapted?

Clips: project page (tencent-roboticsx.github.io/NCP) · the two figures are schematics
BasicsImitatePriorsModelsScaleInteractReality
53
Under the hood

NCP: A VQ codebook of skills learned by tracking, plus a categorical prior

Stage 1 · imitation
Stage 2 · prior: distill, then shift
Stage 3 · downstream task
Motion clips≈ 26 or 30 min in total sampled $\propto (1 - R_m)^3$
Encoder $p^e(z^e \mid s, f)$MLP 1024-1024-512 sees the next 2 clip frames
Codebook $\mathbf{e}$$K$ = 512 or 256 codes, 64-d keeps the nearest code
Decoder $\pi^d(a \mid s, z^q)$MLP 1024-1024-512 → residual PD targets
PD + Isaac Gympolicy 30 Hz · sim 120 Hz → state $s$: 362-d or 323-d
Tracking rewardjoints, key bodies, root: pose and velocity terms
Prior $\pi^p(k \mid s)$MLP 1024-1024-1024 categorical over $K$ codes
Decoder $\pi^d$frozen
PD + Isaac Gym
Count reward$\sqrt{N_k / \hat{N}(s)}$ 0 if no clip frame matches
Task goal $g$strike target (15-d) or opponent state (289-d)
Upper-level policy$\pi^h(k \mid s, g)$ · picks a code MLP 1024-512-512
Decoder $\pi^d$frozen
PD + Isaac Gymboxing: opponent is a past policy snapshot (PFSP)
Task rewardstrike or boxing damage + facing + velocity terms
$f$
$z^e$
$z^q$
PPO + commitment loss → encoder, codebook, decoder
distill
sampled code $e_k$
PPO: prior shifting
KL
code $e_k$
PPO + entropy bonus
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Stage 1: encoder → nearest code → decoder → PD targetsA tracking reward and the commitment loss train all threeStage 2: a prior that sees only s copies the encoder’s picksA count reward shifts the prior to rarely visited framesStage 3: a task policy picks the codes, held near the priorWhat it costs, and why the prior has to be shifted
Objective
  • Imitation: $\max_{\theta,\phi,e}\ \mathbb{E}\sum_t \big(\gamma^t r_t - \|\mathrm{sg}[z^e_t] - e\|_2^2 - \beta\,\|z^e_t - \mathrm{sg}[e]\|_2^2\big)$, $\beta = 0.25$
  • Distill: $\min_{\varphi}\ \mathbb{E}\sum_t \mathrm{KL}\big(p^e(\cdot \mid s_t, f_t)\ \|\ \pi^p(\cdot \mid s_t)\big)$, $s_t$ from the prior's rollouts, $f_t$ from clips
  • Shift: $r(s) = (N_k / \hat{N}(s))^{1/2}$, $\hat{N}$ = visit count of the best-matching clip frame
  • Task: $\max_{\vartheta}\ \mathbb{E}\sum_t \gamma^t r_t - 0.05\,\mathrm{KL}\big(\pi^h \,\|\, \pi^p\big) + 0.01\,\mathrm{H}(\pi^h)$
Training recipe
  1. Track clips with PPO through encoder → nearest code → decoder; favor hard clips
  2. Roll out the prior through the frozen decoder; fit it to the code the encoder picks
  3. Shift the prior with PPO on the count reward so all clip frames are visited evenly
  4. Train the upper-level policy over codes: PPO + KL to the prior; boxing adds self-play
≈ 2 daysimitation stage
12 hprior training
≤ 4 hsword-and-shield strike task
4–4.5 daystwo-player boxing
The detail that makes it work

A rare combination punch is rarely sampled from the distilled prior, so a count-based reward shifts the prior until every clip frame is visited about evenly.

Versus its predecessor

vs ASE / CALM: no discriminator. Skills are discrete codes learned by tracking; compared against β-VAE and GAN-based priors.

BasicsImitatePriorsModelsScaleInteractReality
54
SIGGRAPH 2023
Spatial composition

Composite Motion Learning with Task Control

Different body parts imitate different motions, while the character also solves a task.
Policy $\pi(a \mid s, g)$one network, whole body
PD + simulatorthe state of every body link
Discriminator $D^{\text{upper}}$upper body only: punch clip, or policy?
Discriminator $D^{\text{lower}}$lower body only: run clip, or policy?
Imitation rewardsone for the upper body, one for the lower
One critic per rewardstandardized advantages, summed: $\sum_k \omega_k \bar{A}^k$
Task rewardse.g. follow a target heading
Frozen meta policypre-trained locomotion
PPO update
action mean: $\mu + w\,a^{\text{meta}}$

One policy, the whole body. The goal: punch like one clip and run like another, at the same time. No clip shows both.

Split the body. Each discriminator sees only its own body links and its own clips, and pays its own reward.

One critic per reward. Each advantage is standardized, then PPO adds them with fixed weights: no reward drowns out another.

Reuse what works. A frozen locomotion policy adds its action, scaled per degree of freedom by weights the new policy outputs.

Insets: the two references · centre: the trained character
no clip shows both:
who judges the motion?
Schematic: one motion, two partial views
Schematic: one dot = one sample’s advantage
a new upper-body skill:
learn to walk again?
Red: frozen meta policy · blue: new policy
Xu, Shang, Zordan, Karamouzas. Composite Motion Learning with Task Control. ACM TOG (SIGGRAPH) 2023 · clips: authors’ video
BasicsImitatePriorsModelsScaleInteractReality
55
Under the hood

Composite: One discriminator per body-part group, one critic per objective

PPOadvantage ${\sum_k \omega_k \bar{A}^k_t}$, ${\gamma = 0.95}$
$K$-head criticone value head per objective, PopArt normalized
Imitation reward $r^{D^i}$$\mathrm{mean}_n\ \mathrm{clip}\big(D^i_n(o^i_t),\,-1,\,1\big)$
Goal $g_t$heading, target, aim direction, ball states
Policy $\pi(a_t \mid s_t, g_t)$GRU 256 → 2 FC → $\mu_t, \sigma_t$ (and $w_t$)
PD servo + Isaac Gym28-DoF humanoid, 512 envs, policy 30 Hz, sim 120 Hz
Discriminators $D^{\text{upper}}$upper-body links × 5 frames, GRU 256, $N$ heads
Reference clips $\mathcal{M}^i$one clip set per group: punch (upper), run (lower)
Discriminators $D^{\text{lower}}$lower-body links × 5 frames, GRU 256, $N$ heads
Meta policy $\pi^{\text{meta}}$optional · pre-trained, fixed, e.g. locomotion
state $s_t$, 4 frames
$g^{\text{meta}}_t$
$s^{\text{meta}}_t$
$a^{\text{meta}}_t$
$o^{\text{lower}}_t$: lower-body link poses
one reward per group
one reward per task
update
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: goal in, PD targets out, state backPer body part: a discriminator ensemble, its clips, its rewardOne critic head per reward; PPO on the weighted advantagesOptional: a frozen meta policy’s action is added, scaled by wWhat it costs, and what keeps the objectives balanced
Objective
  • $\mathcal{L}_{D^i} = \tfrac{1}{N}\sum_n \big(\mathbb{E}[\max(0,\,1 + D^i_n(o^i))] + \mathbb{E}[\max(0,\,1 - D^i_n(\tilde{o}^i))] + 10\,\mathbb{E}[(\|\nabla D^i_n(\hat{o}^i)\|_2 - 1)^2]\big)$
  • $\smash[t]{r^{D^i}_t} = \tfrac{1}{N}\sum_n \mathrm{clip}\big(D^i_n(o^i_t),\,-1,\,1\big)$ · group $i$: $o^i$ policy, $\tilde{o}^i$ reference, $\hat{o}^i = \alpha o^i + (1-\alpha)\,\tilde{o}^i$
  • $\mathcal{L}_\pi = \mathbb{E}_t\big[\big(\sum_k \omega_k \bar{A}^k_t\big)\log\pi(a_t \mid s_t, g_t)\big]$, $\ \bar{A}^k_t$ standardized per $k$; goal tasks share $\omega = 0.5$
  • Incremental: $\pi(a_t \mid s_t, g_t, a^{\text{meta}}_t) = \mathcal{N}\big(\mu_t + w_t\,\mathrm{Stop}(a^{\text{meta}}_t),\ \sigma_t^2\big)$
Training recipe
  1. Split the body into groups (upper / lower, or arms / rest), each with its own clips
  2. Roll out; update each discriminator ensemble: half policy, half reference samples
  3. Per objective: reward, critic head, standardized advantage; PPO on the weighted sum
  4. Incremental: add a frozen meta policy’s action, scaled by learned per-DoF weights w
1.5 hpure imitation policy · 20 M samples · V100
+ 300 Msamples (+ 20 h) without incremental learning · Aiming+Run
120 / 30 Hzsimulation / policy
15–30 hgoal-directed task from scratch
The detail that makes it work

Per-part discriminators let upper and lower body imitate different clips; one critic and one standardized advantage per objective keep them balanced.

Versus its predecessor

vs AMP: spatial composition (two motions at once), which AMP left open. Builds on the authors’ GAN-like approach (ICCGAN, 2021).

BasicsImitatePriorsModelsScaleInteractReality
56
Results

Two motions and task goals, in one policy

Each task reward is one more objective, with its own critic.
Lower body: locomotion · upper body: aiming
Arms: juggling · rest of the body: walk to a target
Lower body: footwork · upper body: racket swing
Twist + lunge · left: ours · right: rewards added
Rewards simply added: one motion is lost
Aiming + Run, to similar performance
What the frozen policy saves
Gamepad: steer, aim, switch crouch / walk / run
Steered live from a gamepad

Unlocked

Two motions at once: the composition AMP left open, with a single multi-objective policy.

Still open

A trained policy is still rigid. A new style or terrain means starting over?

Clips: authors’ supplementary video (youtu.be/mcRAxwoTh3E)
BasicsImitatePriorsModelsScaleInteractReality
57
SIGGRAPH Asia 2023
Policy adaptation

AdaptNet: edit a trained policy

A locomotion policy takes ≈ 26 hours to train. Adapt it to a new style, body or terrain instead of training again.
Few-shot style transfer
New body shapes · split view: with / without AdaptNet
Rough terrain
Obstacles · grey ghost: pre-trained policy
Xu, Xie, Andrews, Kry, Neff, McGuire, Karamouzas, Zordan. AdaptNet: Policy Adaptation for Physics-Based Character Control. ACM TOG (SIGGRAPH Asia) 2023 · clips: authors’ video
BasicsImitatePriorsModelsScaleInteractReality
58
How it works

Freeze the policy. Add branches that start at zero.

Latent injectorcopy of the encoder → zero-init FC
Internal adaptorsa zero-init FC beside each internal FC layer
PPO, branches onlynew discriminators and critic
State $s_t$character + goal
Encoder $\mathcal{E}_\xi$pre-trained · frozen
Latent $z^0_t$the encoded state
FC layers $\pi_\theta$pre-trained · frozen
PD + simulatorIsaac Gym
$s_t$ and, if needed, a new input $c_t$: e.g. a height map
offset
$+\,\Delta z^i$
task + imitation reward
The trained policy stays as it is: encoder → latent z⁰ → FC layers → PD targetsLatent injector: a copy of the encoder that adds an offset to the latent z⁰Internal adaptors: a new FC layer in parallel with each internal FC layer, for larger changesBoth branches start at zero: training begins at the old behaviour and changes only the branchesAt run time, a scale α on both branches blends the original policy (α = 0) into the adapted one (α = 1)
The pre-trained policy: a plain, steerable walk
Scale α from 0 to 1: the walk becomes a jaunty skip

Unlocked

Policies become editable: new styles, bodies and terrains from a pre-trained controller, in minutes to hours.

Still open

Every prior so far is tied to the policy it was trained with. A prior you train once and hand around?

Clips: authors’ video (youtu.be/WxmJSCNFb28) · imitation errors: Table 2 of the paper · the two diagrams in the cards are schematic
BasicsImitatePriorsModelsScaleInteractReality
59
Under the hood

AdaptNet: Adapt a pre-trained policy by editing its latent space

Stage 2 · AdaptNet, trained
Stage 1 · pre-trained policy, frozen
Latent injector $\mathcal{I}_\phi$copy of $\mathcal{E}_\xi$ (+ CNN for $c_t$) → zero-init FC
Internal adaptors $\mathcal{F}^i_\eta$full-rank, zero-init FC beside each FC layer
Multi-critic PPOpolicy update: only $\phi$, $\eta$ · new 2-head critic
Imitation rewardmean of $\mathrm{clip}(D_n, -1, 1)$
State $s_t = \{o_t, g_t\}$784-d: body links over 4 frames + 4-d goal
Encoder $\mathcal{E}_\xi$GRU, 256-d hidden, on $o_t$ · concat goal $g_t$
Latent $z^0_t$ · 260-dthe bottleneck of the policy network
Policy layers $\pi_\theta$FC, 1024 / 512-d latents → Gaussian action
PD + Isaac Gympolicy 30 Hz · sim 120 Hz · 512 envs
32 new discriminatorsfrom scratch · hinge loss + gradient penalty
Reference motion0.8–2.2 s style clip, or the walk / run set
$s_t$, optional new input $c_t$
offset
$+\,\Delta z^i$
character state $o_t$
real
task reward
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Stage 1: the pre-trained policy, frozen, in the control loopTwo zero-initialized branches: the only policy weights trainedNew discriminators, trained from scratch, score the motionPPO updates only φ and η; two penalties keep the change smallWhat it costs, and why starting at zero matters
Objective
  • $z^0_t = \mathcal{E}_\xi(s_t) + \alpha\,\mathcal{I}_\phi(s_t, c_t)$, $z^i_t = \mathcal{F}^i_\theta(z^{i-1}_t) + \alpha\,\mathcal{F}^i_\eta(z^{i-1}_t)$; run-time scale $\alpha \in [0, 1]$, $1$ = as trained
  • $r^{imit} = \tfrac{1}{N}\sum_n \mathrm{clip}\big(D_n(o_t), -1, 1\big)$, $N = 32$; $\ \omega$ = 0.35 imitation / 0.65 task for style transfer
  • $\min_{D_n} \mathbb{E}[\max(0, 1 + D_n(o_t))] + \mathbb{E}[\max(0, 1 - D_n(\tilde{o}_t))] + 10\,\mathbb{E}[(\|\nabla D_n(\hat{o}_t)\|_2 - 1)^2]$
  • $\max_{\phi,\eta}\ \mathbb{E}_t\big[\big(\textstyle\sum_k \omega_k \bar{A}^k_t\big)\log \pi_\theta(a_t \mid z^0_t;\, \eta) - \beta\,\|\mathcal{I}_\phi\|_2 - \kappa\,\|\eta\|_2\big]$, $\ \beta = \kappa = 0.01$
Training recipe
  1. Pre-train a walk or run policy with multi-critic PPO + discriminators (≈ 26 h)
  2. Freeze θ and ξ; copy the encoder into the injector; zero-init the last FC of each branch
  3. Discard the old discriminators and critic; train new ones from scratch on the new task
  4. Update only φ and η with PPO: 10–30 min for a new style, up to 4 h for rough terrain
10–30 minsimple adaptation tasks
≤ 4 hrough-terrain adaptation
≈ 26 hpre-training the base policy
512parallel environments
The detail that makes it work

Zero-initialized adapters start from the pre-trained behaviour, so exploration begins near a working gait instead of from random actions.

Versus its predecessor

vs fine-tuning or training from scratch: reuses a policy in minutes. Its adaptors are full-rank (LoRA did worse here); zero-init low-rank adapters return in GPC (2026).

BasicsImitatePriorsModelsScaleInteractReality
60
SIGGRAPH 2026
Reusable priors

SMP: score-matching motion priors

Pre-train a motion diffusion model once. Freeze it. Use it as the style reward for every policy.
AMP with a reused discriminator: steering
AMP · 2021

One discriminator per policy, trained together with it, on the motion data. Reused frozen, it gets exploited.

Reused for steering: task return 0.24
the same task,
judged by a frozen diffusion model?
SMP, the same task: steering with a frozen diffusion prior
SMP · 2026

One diffusion model, trained first and alone, then frozen. Policies only query it: for a reward and for start states.

Frozen prior on steering: task return 0.91
Mu, Zhang, Shi, et al., Peng. SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control. ACM TOG (SIGGRAPH) 2026 · clips: authors’ video
BasicsImitatePriorsModelsScaleInteractReality
61
How it works

The judge is a frozen diffusion model

Stage 1 · once
Motion datasetneeded only here
Diffusion model $f$learns to predict noise
Policy $\pi(a \mid s, g)$any task goal $g$
PD + simulator
Noised clip10 frames + noise $\epsilon$
Frozen $f$predicts the noise: $\hat\epsilon$
Reward $r^{\mathrm{smp}}$$\exp(-w_s\,\lVert\hat\epsilon - \epsilon\rVert^{2})$
freeze
$\tilde{x}$
$\hat\epsilon$
the noise that was added, $\epsilon$
+ task reward, PPO
start states
Stage 1, done once: a diffusion model learns to predict the noise that was added to real 10-frame clips.Policy training: the policy’s last 10 frames get noise added, exactly like the data did.The frozen model predicts that noise. It was trained on real motion only, so it answers as if this clip were real.A wrong prediction means the clip is not like the data. The error becomes a reward between 0 and 1.Add any task reward, train with PPO. The model is never updated, so the next policy reuses it.Episodes start from states sampled from the model too: the dataset is never opened again.
noise that was added, $\epsilon$noise predicted by $f$, $\hat\epsilon$error
$r^{\mathrm{smp}} = \exp\!\big(-w_s\,\lVert \hat\epsilon - \epsilon \rVert^{2}\big)$
error $\lVert \hat\epsilon - \epsilon \rVert$
BasicsImitatePriorsModelsScaleInteractReality
62
Under the hood

SMP: A frozen motion diffusion model, used as the style reward

Stage 1: pre-train the prior
PPO$\pi$ and $V$ · $\gamma = 0.99$
$r = w^{\mathrm{prior}} r^{\mathrm{smp}} + w^{g} r^{g}$prior + task
SMP reward $r^{\mathrm{smp}}$$\exp\big(-w_s\,\mathrm{mean}_{i}\,\|\hat\epsilon_i - \epsilon_i\|^2\big)$
Task goal $g$target location, speed, heading or ball state
Policy $\pi(a \mid s, g)$MLP 1024-512
PD + Isaac Gym1,024 parallel environments
Noised clip $x^i$10 frames + noise $\epsilon_i$, levels $i \in \{22, 15, 8\}$ of $N = 50$
Frozen prior $f$reused across tasks · optional style label $c$
Generative state initthe frozen prior samples the start states
Motion datasetneeded only here · 3 s to 20 h (100STYLE)
Diffusion model $f$2-layer transformer encoder, 3 M parameters
PD targets
state $s$
clip $\tilde{x}$
noise $\epsilon_i$
predicted $\hat\epsilon_i$
task reward
update
clips $x^0$
freeze
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: goal in, PD targets out, state backStage 1: pre-train a small diffusion model, then freeze itEvery step: noise the last 10 frames, score the predictionPrior + task reward; PPO updates π and V, never the priorStart states are sampled from the prior: no dataset leftWhat it costs, and the detail that makes it work
Objective
  • Stage 1: $\mathcal{L}_{\mathrm{simple}} = \mathbb{E}_{i,\,x^0,\,\epsilon}\,\|\epsilon - f(x^i)\|^2$, $N = 50$ steps, EMA weights
  • $\hat\epsilon_i = f\big(\sqrt{\bar\alpha_i}\,\tilde{x} + \sqrt{1-\bar\alpha_i}\,\epsilon_i,\ c\big)$; errors normalized by a running mean $\mu_i$
  • $r^{\mathrm{smp}} = \exp\big(-\tfrac{w_s}{|\mathbb{K}|}\sum_{i\in\mathbb{K}}\|\hat\epsilon_i - \epsilon_i\|^2\big)$, $\mathbb{K} = \{22, 15, 8\}$, $w_s$ = 4–8 by task
  • $r_t = w^{\mathrm{prior}}\,r^{\mathrm{smp}}_t + w^{g}\,r^{g}_t$ (target location: $0.9\,r^{\mathrm{smp}} + 0.7\,r^{g}$)
Training recipe
  1. Pre-train the diffusion model on motion data: 400k–800k iterations, within 5 h on one RTX 4090
  2. Freeze it. Every step: noise the last 10 frames at 3 fixed levels; the model predicts each noise
  3. Update π and V with PPO on prior + task reward; reuse the same prior for the next task
  4. Drop the dataset: start episodes from states sampled from the model (GSI)
≈ 3 Mparameters in the prior
≤ 5 hto pre-train the prior (one RTX 4090)
100+ stylesin one prior (100STYLE, > 20 h)
≤ 12 hfor most task policies
The detail that makes it work

A randomly sampled noise level gives a high-variance reward. SMP averages three fixed levels, {22, 15, 8}: high noise corrects large errors, low noise fine ones.

Versus its predecessor

vs AMP: the prior is pre-trained once and frozen instead of retrained with every policy. It costs more per policy: 11.5 h vs 6.2 h on one task.

BasicsImitatePriorsModelsScaleInteractReality
63
Why it matters

One prior, many policies, no dataset

The prior’s data: running (LaFAN1)
Dodgeball, trained with that prior
One prior, many tasks

The same frozen prior trains steering, target location and dodgeball. Dodging is not in its data.

Target location in many styles
AeroPlane + HighKnees, Elated + FlickLegs
One model, 100 styles

A style label picks the prior. Blending the predictions for two labels gives a style that is not in the data.

Carrying, from a human–object prior
Getting up
Other data, same recipe

Train the model on other motion: carrying succeeds 99.7 % of the time, getting up 99.8 %.

Unitree G1, steered with a gamepad
Unitree G1: spin kick
On a real robot

Policies trained in simulation with this reward run on a Unitree G1.

Unlocked

The prior is finally a pre-trained model, not a by-product of one policy: train once, reuse across tasks.

Still open

Still one policy per task, and style datasets of about 20 hours. Other roads went for scale.

Clips: authors’ video (youtu.be/jBA2tWk6vzU)
BasicsImitatePriorsModelsScaleInteractReality
64
Chapter 3  ·  2021 – 2024

Models

What if we learn the simulator,
and differentiate through it?
SuperTrack SIGGRAPH Asia 2021
ControlVAE SIGGRAPH Asia 2022
MoConVQ SIGGRAPH 2024
65
SIGGRAPH Asia 2021

SuperTrack: motion tracking
by supervised learning

Fussell, Bergamin, Holden · Ubisoft La Forge
No reinforcement learning.
66
How it works

Learn the simulator, then differentiate through it

Kinematic target $K$the next mocap frame
Policy $\Pi$outputs PD targets $T$
PD + simulatora black box: no gradient
Data bufferframes of $(S, K, T)$
Tracking losspredicted $P$ vs target $K$
World model $W$state, $T$ → next state
$T$
state $S$
every frame
regression on 8-frame windows
$T$
$P$
gradient
The tracking loop. To reinforcement learning the physics engine is a black box: it returns the next state, never a gradient.
Collect. 256 characters run in the simulator. Every frame is stored: the state S, the target K, the PD targets T.
Fit a world model. W predicts the next state from a state and its PD targets. Rolled out over 8 stored frames; the label is what the simulator did.
Train the policy inside W. Roll Π and W out for 32 frames. The loss is the distance to the target; its gradient flows back through W and updates only Π.
Alternate one gradient step each; the new Π goes back to the simulator. Two supervised problems: no reward, no critic, no sampled policy gradient.
Yellow = simulated character · green = its kinematic target
Trail of frames: red = world-model prediction, yellow = simulation
Yellow = simulation · green = kinematic target
Clips: authors’ video (Fussell, Bergamin, Holden, SIGGRAPH Asia 2021) · the frame strip is a schematic, not data
BasicsImitatePriorsModelsScaleInteractReality
67
Under the hood

SuperTrack: Replace RL with supervised learning through a learned world model

Tracking lossL1 in local space: $P$ vs $K$ + offset penalty
World model $W(P_i, T_i)$MLP 5 × 1024 ELU → accelerations, integrated
World-model lossL1 on the unrolled window: simulated $S$ vs $P$
Kinematic target $K_{i+1}$next mocap frame · LaFAN ≈ 6.5 h, one policy
Policy $\Pi$MLP 5 × 1024, ELU · sees local $S$ and $K_{i+1}$
PD targets $T$offset ⊗ kinematic pose · noise $\sigma = 0.1$
PD + simulator256 characters · PhysX at 240 Hz
Cyclic data buffer≈ 150,000 samples of $(S, K, T)$ · off-policy
target $K$
offsets $o$
predicted $P$
next state $P_{i+1}$
gradient through $W$
state $S$
episodes
8-frame windows: $S$, $T$
update
32-frame windows: $S_0$, $K$
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The gym: target and state in, PD targets outEvery simulated frame goes into one cyclic bufferFit W: unroll it over 8 stored frames, L1 to the simulatorFit Π: unroll it inside W; the loss flows back through WAlternate one step each: what it costs, what makes it work
Objective
  • PD targets: $t = \exp\!\big(\tfrac{\alpha}{2}\,\hat{o}\big) \otimes k^{t}$, $\ \hat{o} = o + \sigma\,\mathcal{N}(0, 1)$, $\ \alpha = 120$, $\sigma = 0.1$
  • $\mathcal{L}_W = \sum_i \big[\,w_{pos}\|s^{p}_i - p^{p}_i\|_1 + w_{vel}\|\dot{s}^{p}_i - \dot{p}^{p}_i\|_1 + w_{rot}\|s^{r}_i \ominus p^{r}_i\|_1 + w_{ang}\|\dot{s}^{r}_i - \dot{p}^{r}_i\|_1\big]$
  • $\mathcal{L}_\Pi = \sum_i \big[\,\sum_x w_x \|p^{x}_i - k^{x}_i\|_1 + w_{lreg}\|o_i\|_2^2 + w_{sreg}\|o_i\|_1\big]$, $\ p_i$: $\Pi$ unrolled in $W$
  • $x$: local pos, vel, rot, ang. vel, height, up · $w$ balanced at the start, regularizers ≈ 100× less
Training recipe
  1. Run the noisy Π on 256 simulated characters (RSI resets); episodes fill the buffer
  2. Fit W: sample 2,048 windows of 8 frames, unroll W, L1 loss to the simulated states
  3. Fit Π: sample 1,024 windows of 32 frames, unroll Π in W, back-propagate into Π only
  4. Alternate one RAdam step each; copy Π to the gym; ≈ 100–200k iterations, 20–40 h
≈ 5,000simulator samples per second
48–512frames per collection episode
0reward functions or value networks
The detail that makes it work

The world model predicts accelerations in the character’s local frame and is integrated forward, so one model serves every position and heading.

Versus its predecessor

vs DReCon (same lab): a similar tracking set-up, but PPO’s sampled gradient estimates are replaced by analytical gradients through a learned world model.

BasicsImitatePriorsModelsScaleInteractReality
68
Results

Against PPO, in the same gym

One policy for ≈ 6.5 hours of LaFAN mocap. PPO does well on small databases, not on this one.
SuperTrack, in PhysX
PPO, in PhysX
70 %

of the collected episodes thrown away: PPO trains twice as slowly, SuperTrack is largely unaffected.

× 10

Walk over-sampled (motion-matching data): after 10 hours PPO is stable only on walking; SuperTrack runs, jogs and crouches after ≈ 1.5 hours.

Tracking on rough terrain
Gamepad control: human and dog

The catch: a short horizon. On long roll-outs W (violet) drifts from the simulator (white), so Π is trained only 32 frames ahead.

Unlocked

No RL at all: with a learned, differentiable model, tracking is supervised learning, and more sample-efficient than PPO.

Still open

Still a tracker: it needs a reference. Can the same trick learn a generative skill space?

Fussell, Bergamin, Holden. SuperTrack: Motion Tracking for Physically Simulated Characters using Supervised Learning. ACM TOG (SIGGRAPH Asia) 2021 · clips: authors’ video · figure: schematic
BasicsImitatePriorsModelsScaleInteractReality
69
SIGGRAPH Asia 2022
Generative control, model-based

ControlVAE: a latent space of motor skills

A VAE whose decoder is the control policy, trained by direct supervision through a learned world model.
Motion clips≈ 30 min with mirroring
Encoder $q(z \mid s, \tilde{s}')$sees the next frame
Policy $\pi(a \mid s, z)$the decoder
World model $\omega(s, a)$a learned simulator
Reconstruction $\mathcal{L}_{\text{rec}}$distance to that frame
Prior $p(z \mid s)$sees only the state
PD + simulatorblack box: no gradient
$\tilde{s}'$
$z$
$a$
$s'$
target: the same mocap frame $\tilde{s}'$
$a$
$s$
its rollouts fit $\omega$
$\mu_p$
A VAE for motion: the encoder writes the next mocap frame $\tilde{s}'$ as a code $z$; the decoder is the control policy, acting through PD and physics.The simulator passes no gradient. A world model fitted to its rollouts does: the error flows back into policy and encoder.A prior guesses $z$ from the state alone. The encoder may only correct it by $\mu_q$, and the KL term keeps $\mu_q$ short.Why must the prior see the state? With a fixed $\mathcal{N}(0, I)$ instead, random samples turn unstable (authors’ ablation).
White: simulation · red: its reference
what trains them?
World-model rollout (red) vs simulation
and with no mocap frame
to look at?
Latent space, schematic (2 of 64 dims)
why not a plain
$\mathcal{N}(0, I)$ prior?
Ablation: random $z$ from a fixed $\mathcal{N}(0, I)$
$\mathcal{L}_{\text{rec}} = \sum_t \gamma^t\,\lVert W(\tilde{s}_{t+1} - s_{t+1})\rVert_1$24-step rollouts in $\omega$ · $\gamma = 0.95$
+
$\beta\,\mathcal{L}_{\text{kl}} = \beta \sum_t \gamma^t\,\lVert\mu_q\rVert_2^2 \,/\, 2\sigma_p^2$$\beta$ raised from 0.01 to 0.1
+
$\mathcal{L}_{\text{act}}$keeps actions small
Yao, Song, Chen, Liu. ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters. ACM TOG (SIGGRAPH Asia) 2022 · clips: project page, supplementary video
BasicsImitatePriorsModelsScaleInteractReality
70
Under the hood

ControlVAE: A VAE whose decoder is the policy, trained through a world model

$\mathcal{L}_{\text{rec}} + \beta\,\mathcal{L}_{\text{kl}} + \mathcal{L}_{\text{act}}$24-step rollouts in $\omega$ · $\gamma = 0.95$
World model $\omega(s_t, a_t)$MLP 4 × 512 · velocity changes + Euler step
Replay buffer $\mathcal{B}$$5 \times 10^4$ states · 2,048 new per epoch
Motion clips4 LaFAN sequences · ≈ 30 min with mirroring
Posterior $q(z_t \mid s_t, \tilde{s}_{t+1})$MLP 2 × 512 · mean $\mu_p + \mu_q$
Policy $\pi(a_t \mid s_t, z_t)$6 experts, 3 × 512 each · gating 2 × 64
PD targets $a_t$joint rotations as axis angles · ${\sigma_\pi = 0.05}$
PD + ODE simulatorpolicy 20 Hz · physics 120 Hz
Prior $p(z_t \mid s_t)$MLP 2 × 512 · $\sigma_p = 0.3$
Task policy $\pi(z_t \mid s_t, g)$MLP 3 × 512 (heading: 256) · mean $\mu_p + \mu_g$
Downstream goal $g$height, heading, steering or skill control
$\tilde{s}_{t+1}$
$\mu_p$
latent $z_t$
state $s_t$
$(s, a)$
$\mathcal{L}_w$
next state
predicted $s_{t+1}$
reference
gradient through $\omega$
$\mu_p$
$z_t$
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Collection: the posterior tracks a clip in the simulatorSimulated states fill a buffer that fits the world model24-step rollouts in ω; the loss flows back into π and qThe prior: q only adds a residual; KL grows with its squareDownstream: a task policy picks z, trained through ωWhat it costs, and what it changes
Objective
  • $\mathcal{L}_{\text{rec}} = \sum_t \gamma^t\,\|W(\tilde{s}_{t+1} - s_{t+1})\|_1$, $\ \gamma = 0.95$; $\ \mathcal{L}_{\text{act}}$ penalizes $\|a_t\|_1$ and $\|a_t\|_2^2$
  • $\mathcal{L}_{\text{kl}} = \sum_t \gamma^t\,\|\mu_q(s_t, \tilde{s}_{t+1})\|_2^2 \,/\, 2\sigma_p^2$, weighted by $\beta$, raised from 0.01 to 0.1
  • World model: $\mathcal{L}_w = \sum_t \|\bar{s}_{t+1} - \mu_w(s_t, \bar{a}_t)\|^2_{W'}$ along 8 recorded steps
  • Downstream: $\sum_t \big[\mathcal{L}_g + \mathcal{L}_{\text{fall}}\big] + w_z \sum_t \|\mu_g\|_2^2$ with $w_z = 20$, on 16-step rollouts
Training recipe
  1. Collect 2,048 simulator states per epoch: track clips with z from q, or the prior (prob. 0.4)
  2. Fit the world model on 512 buffer clips of 8 steps, 8 updates per epoch
  3. Roll out q, π and ω for 24 steps from 512 buffer states; backpropagate L, 8 updates per epoch
  4. Downstream: freeze ControlVAE; train a task policy z ~ N(μp + μg, σ²) through the world model
≈ 50 h20,000 iterations
1 GPUGTX 2080 Ti + 4 CPU threads
64-dGaussian latent
5 × 10⁴states in the replay buffer
The detail that makes it work

A state-conditional prior with posterior = prior + residual. A standard N(0, I) prior gives jerky skill switches and falls.

Versus its predecessor

vs SuperTrack: the same model-based trick, now for a generative latent space. vs ASE: the skill space is learned with no discriminator and no policy gradient.

BasicsImitatePriorsModelsScaleInteractReality
71
Using the skill space

ControlVAE: sample the prior, or steer it

Goal $g$direction, speed, skill
Task policy $\pi(z \mid s, g)$mean = $\mu_p + \mu_g$
Policy $\pi(a \mid s, z)$frozen
PD + simulatorthe real physics
Task lossgoal error + falling
Prior $p(z \mid s)$frozen
World model $\omega(s, a)$frozen
sample $z$
$a$
$z$
$\mu_p$
$a$
$s'$
gradient, back through $\omega$ and $\pi$
Training is over: drop the encoder, freeze the rest. Draw $z$ from the prior at every control step.A task policy takes over the choice of $z$: it reads the goal and shifts the prior’s mean by $\mu_g$.It is trained like the VAE: a task loss, back-propagated through the frozen world model. Half an hour for heading control.It was trained inside the world model, and it holds in the real simulator: pushed over, it gets up.All of this from about 30 minutes of motion (mirrored clips included) and a 64-d latent, trained on one GPU.
Random samples from the latent prior
and to reach a goal?
Latent space, schematic (2 of 64 dims)
what trains it,
and what comes out?
Skill + direction from the user
and if it is pushed?
Pushed over, recovers

Unlocked

ASE’s promise without the GAN: a latent skill space learned by reconstruction, sampled or steered by a high-level policy.

Still open

A modest dataset and a continuous latent. Does it scale to tens of hours, and can it talk to language models?

Clips: project page and supplementary video (Yao et al., 2022)
BasicsImitatePriorsModelsScaleInteractReality
72
SIGGRAPH 2024

MoConVQ: motor skills
as discrete tokens

VQ-VAE codebooks of physical skills,
learned from tens of hours of motion
73
How it works

Motion in, tokens in the middle, physics out

Loss $\mathcal{L}$$\|M - \tilde M_{\mathcal{W}}\|$ + VQ and action terms
World model $\mathcal{W}$learned, differentiable
Motion clips $M$23.2 h of mocap
Encoder $\mathcal{E}$1.2 s → 6 vectors
Residual VQnearest of 512 codes
Policy $\pi(s^t, u^t)$the decoder: outputs PD targets
PD + simulatorpolicy 20 Hz · physics 120 Hz
$v^k$
tokens
rollouts
$\hat a$
$\hat s$
rollout in $\mathcal{W}$
gradient through $\mathcal{W}$
target $M$
The encoder turns 1.2 s of motion (24 frames) into 6 latent vectors: one for every 0.2 s.Each vector snaps to the nearest entry of a codebook of 512: the clip is now 6 integers, its tokens.What the snap lost is quantized by the next codebook, and again: N = 8 residual layers, from coarse to fine.The decoder is a control policy: from the tokens and the simulated state it computes PD targets, and physics plays the motion back.The simulator is a black box: no gradient. A world model is fitted to its rollouts, and in training the policy acts inside it.One loss compares the input clip with its rollout in the world model; the gradient reaches policy, codebooks and encoder.ControlVAE learned from about 30 minutes of motion. This one model holds 23.2 hours.
and how is it trained?
what is left
state $s^t$
and back from tokens to motion,
with physics in the loop?
Blue: unseen, corrupted mocap · white: simulated
23.2 h
of motion in one modelLaFAN + AMASS
6.3 cm
mean body position erroron 2 h of unseen HDM05 motion
≈ 6 days
on a single RTX 309040k epochs
BasicsImitatePriorsModelsScaleInteractReality
74
Under the hood

MoConVQ: Residual-VQ tokens for motor skills, decoded by a model-based policy

After training: token generators
MoConVQ loss $\mathcal{L}$$\|M - \tilde M_{\mathcal{W}}\|$ + commitment + VQ + $\mathcal{L}_{\text{reg}}$
World model $\mathcal{W}(\hat s^t, \hat a^t)$differentiable 4-layer MLP, 512 units
Replay bufferFIFO of simulated $\tilde M_{\text{sim}}$ · size 50,000
Motion clips $M$23.2 h · LaFAN + AMASS
Encoder $\mathcal{E}$1D conv · 24 frames (1.2 s) to 6 latents $v^k$
Residual VQ$N = 8$ RVQ layers · 512 codes each, 768-d
Deconv1D conv · upsamples to one code $u^t$ per frame
Policy $\pi(s^t, u^t)$6 experts, 4 × 256 each → PD targets
PD + simulatorpolicy 20 Hz · ODE 120 Hz
MoConGPTtemporal + depth transformers · text via T5
LLM (Claude-2)1600 in-context examples, no fine-tuning
$M$
target $M$
$v^k$
$z^k$
$u^t$
state $s^t$
rollouts
$\mathcal{L}_{\mathcal{W}}$
$\hat a$
$\hat s$
reconstruction $\tilde M_{\mathcal{W}}$
gradient through $\mathcal{W}$
indices $I^k$
VQ index $I^k_0$
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The autoencoder: clip in, tokens in the middle, physics outA world model is fitted to simulated rollouts from a bufferThe loss flows back through W: policy, codebooks, encoderAfter training, a GPT or a prompted LLM writes the tokensWhat it costs, and what makes it work
Objective
  • $\mathcal{L} = \|M - \tilde M_{\mathcal{W}}\| + \beta_1\|\mathcal{E}(M) - \mathrm{sg}(Z)\| + \beta_2\|\mathrm{sg}(\mathcal{E}(M)) - Z\| + \beta_3\,\mathcal{L}_{\text{reg}}$
  • $\mathcal{L}_{\text{reg}} = \sum_t w_1\|\hat a^t - \bar a^t\| + w_2\|\hat a^t\|$, $\ \bar a^t = (1-\beta)\,\bar a^{t-1} + \beta\,\hat a^t$, $\beta = 0.8$
  • World model: $\mathcal{L}_{\mathcal{W}} = \|\tilde M_{\text{sim}} - \tilde M_{\mathcal{W}}\|$ on sequences from the replay buffer
  • Codebooks: EMA update $\mathcal{B} \leftarrow (1-\gamma)\,\mathcal{B} + \gamma\,\hat{\mathcal{B}}$, $\gamma = 0.99$; inactive codes are reset
Training recipe
  1. Track random clips in the real simulator with the current model; add 1024 frames to the buffer
  2. Replay buffer sequences in the world model $\mathcal{W}$ and minimize $\mathcal{L}_{\mathcal{W}}$
  3. Track a 24-frame clip inside $\mathcal{W}$; update policy, codebooks and encoder (RAdam, lr 1e-5)
  4. Apply quantizer dropout: decode from a random number of leading RVQ layers each iteration
≈ 6 days40k epochs on one RTX 3090
23.2 hof motion · LaFAN + AMASS
8 × 512residual codebooks
0.2 sof motion per token
The detail that makes it work

Residual VQ yields a coarse-to-fine code: the first layer alone is enough for an LLM to plan with, later layers add detail.

Versus its predecessor

vs ControlVAE: discrete tokens and far more data (23.2 h). vs NCP: model-based RL through a learned world model instead of PPO.

BasicsImitatePriorsModelsScaleInteractReality
75
Tokens in use

A vocabulary of skills that language models can speak

Texta motion description, or a task
MoConGPTa GPT over tokens · text via T5
LLM (Claude-2)1600 examples in its prompt
Tokensintegers · 0.2 s of motion each
The same decoderpolicy + simulator
every layer
first layer only
who writes the tokens?
Whatever writes tokens now drives the character. The simplest writer is a rule: replay one clip’s tokens, with jumps.MoConGPT is a GPT over motion tokens: given a text, it predicts the next token, one residual layer after another.An LLM, not fine-tuned, picks the vocabulary up from 1600 examples in its prompt, and answers a new request in integers.An LLM also reasons: told of a key on the ground and a door, it plans eight moves and writes each one in tokens.
One clip’s tokens, replayed with jumps
MoConGPT: “a man is kicking with right leg”
LLM: walk for a long time, kick, then dance
LLM: key-and-door plan, in an empty scene

Unlocked

Skills become a vocabulary: one tokenizer serves tracking, text-to-motion and LLM agents, at tens of hours of data.

Still open

A learned model is an approximation. The model-free camp bet on brute-force scale instead.

Yao, Song, Zhou, Ao, Chen, Liu. MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations. ACM TOG (SIGGRAPH) 2024 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
76
Chapter 4  ·  2023 – 2026

Scale

One policy for all human motion, and a way to direct it.
PHC ICCV 2023
PULSE ICLR 2024
MaskedMimic SIGGRAPH Asia 2024
CLoSD ICLR 2025
GPC SIGGRAPH 2026
77
ICCV 2023

PHC: Perpetual
Humanoid Control

One tracker for (almost) all of AMASS —
that gets back up by itself
78
How it works

Grow the policy one hard subset at a time

Primitive $P^{(1)}$all of AMASS: 11,313 sequences
Primitive $P^{(2)}$only what $P^{(1)}$ fails
Primitive $P^{(3)}$only what $P^{(2)}$ fails
copy weights
its failures
copy weights
its failures
One network, all of AMASS. It keeps learning new sequences and forgetting old ones, so its success rate stops improving.
Freeze it and collect the sequences it fails. A new primitive starts from a copy of its weights and trains only on those.
Repeat. Each round the set of failures is smaller and harder, and the new primitive has only that to learn. What is frozen cannot be forgotten.
Does progressive training matter? Test on noisy poses estimated from video (H36M): one network tracks 59.4 % of the sequences. A composer over primitives:
What is left is highly dynamic: high jumps, backflips. The authors’ guess: a next-frame target does not say “run up first”.
66.2 %
success on H36M videowithout progressive training
86.9 %
success on H36M videowith progressive training (PMCP)
The task: follow a mocap clip from AMASS (top left)
Hard mocap: kicks, spins, a cartwheel
and what still fails?
Failure case: the reference flips, the humanoid falls
Luo, Cao, Winkler, Kitani, Xu. Perpetual Humanoid Control for Real-time Simulated Avatars. ICCV 2023 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
79
How it works

One more primitive to get back up, a composer over all

Imitation primitives$P^{(1)}, P^{(2)}, P^{(3)}$ · frozen
Recovery primitive$P^{(F)}$ · starts fallen, or 2–⁠5 m away
Composer $C(s_t)$one weight $w_i \ge 0$ per primitive
new task
freeze all
A tracker that falls is lost: more than 0.5 m from its reference counts as failure, and the usual fix is a reset. What we want instead is in the clip.
A primitive for this task alone. It is told only where the reference root is. Reward: $\textcolor{#f5b301}{0.5\,(d_{t-1} - d_t)} + 0.5\,r^{\text{amp}} + 0.1\,r^{\text{energy}}$: get closer, move like walking and running mocap, spend little energy.
Freeze all four, train a composer. Same input, one weight per primitive. The action is the weighted product of their Gaussians: a blend, not a switch.
One policy, no resets, no external forces. What it still fails on the training set are highly dynamic motions.
98.9 %
of AMASS train imitatedUHC, no residual forces: 84.5 %
98.8 %
recover from a fallwithin 10 seconds
Fallen and far: get up, walk back
one policy for all of it?
Live two-person demo: avatar falls, gets back up
Clips: PHC project page (zhengyiluo.com/PHC-Site)
BasicsImitatePriorsModelsScaleInteractReality
80
Under the hood

PHC: Progressive primitives and a composer; recovery is one of the primitives

Stage 2: composer over frozen primitives
Stage 1: one primitive at a time, same PPO + AMP loop
PPO$\gamma = 0.99$ · value net MLP 1024-512
Reward $r_t$$0.5\,r^{g} + 0.5\,r^{amp} + r^{energy}$
Reference pose $\hat{q}_{t+1}$rotations + positions, or 3D keypoints only
Composer $C(s_t)$same input as the primitives → weights $w \ge 0$
Frozen primitivesweighted product of the Gaussians $P^{(1)}, P^{(2)}, P^{(3)}, P^{(F)}$
PD + Isaac Gympolicy 30 Hz · sim 60 Hz · no external forces
Discriminator $D(s^{p}_{t-10:t})$MLP 1024-512 · AMP loss · fine-tuned in all stages
Primitive $P^{(1)}$MLP 1024-512 · all of AMASS until success plateaus
Primitives $P^{(2)}, P^{(3)}$copy the previous weights · train on its failed clips
Recovery primitive $P^{(F)}$starts fallen or 2–5 m away · locomotion clips
AMASS mocap11,313 sequences · references and real samples
goal $s^{g}_t$
weights $w$
PD targets
same $s_t$
proprioception $s^{p}_t$
policy
real
tracking
$r^{amp}$
update
failed clips
then
freeze all
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The controller: a composer weights four frozen primitivesEvery part is trained by the same loop: reward, then PPOStage 1: one primitive at a time, on what still failsThen a recovery primitive, with a reward of its ownStage 2: freeze all four, train only the composerWhat it costs, and what makes it work
Objective
  • $r = 0.5\,r^{g} + 0.5\,r^{amp} + r^{energy}$, $\ r^{energy} = -0.0005 \sum_j |\mu_j\,\omega_j|^2$ (torque × joint velocity)
  • $r^{g} = 0.5\,e^{-100\,\|\hat{p}-p\|} + 0.3\,e^{-10\,\|\hat{q}\ominus q\|} + 0.1\,e^{-0.1\,\|\hat{v}-v\|} + 0.1\,e^{-0.1\,\|\hat{\omega}-\omega\|}$ (imitation)
  • Recovery: $r = 0.5\,(d_{t-1} - d_t) + 0.5\,r^{amp} + 0.1\,r^{energy}$, $\ d_t$ = root distance to the reference
  • $\pi_{PHC}(a \mid s) \propto \prod_i P^{(i)}(a \mid s)^{C_i(s)}$, $\ C_i(s) \ge 0$ (weighted product of the primitives' Gaussians)
Training recipe
  1. Train $P^{(1)}$ on all of AMASS until its success rate stops improving, then freeze it
  2. Copy its weights into a new primitive; train it only on the clips that still fail; repeat
  3. Train $P^{(F)}$ from fallen or far-away starts; beyond 0.5 m the goal is root-only
  4. Freeze all primitives; train the composer on all of AMASS, interleaving recovery episodes
1,536humanoids simulated in parallel
≈ 10 Bsamples · ≈ 7 days · one A100
11,313AMASS training sequences
28.8 MBfinal model size
The detail that makes it work

Capacity is added only for what still fails, and old primitives are frozen, so hard motions and recovery do not erase the easy ones.

Versus its predecessor

vs UHC (2021): no residual forces, yet 98.9 % of AMASS train (UHC: 97.0 % with residual forces, 84.5 % without), and no resets.

BasicsImitatePriorsModelsScaleInteractReality
81
Why it matters

Any reference in, physical motion out

Webcamlive, 30 fps
MeTRAbsnoisy 3D keypoints
PHCkeypoints in, PD targets out
Live webcam → noisy 3D keypoints → avatar
Text promptsa stream of them
MDMdisjoint motion clips
PHCfills the gaps itself
and a reference generated from text?
Text → motion diffusion → disjoint clips, tracked back to back

Unlocked

Tracking becomes a primitive. Video, text-to-motion, mocap: any kinematic source is now physical — no resets.

Still open

A tracker needs a full-body reference every frame. On its own it cannot solve a task.

Clips: PHC project page (zhengyiluo.com/PHC-Site)
BasicsImitatePriorsModelsScaleInteractReality
82
ICLR 2024 spotlight

PULSE: a universal
motion latent space

Every character here is driven by random noise
83
How it works

Distill the tracker into a latent space

and with no reference?
and for a new task?
Encoder $\mathcal{E}$sees body + reference
Prior $\mathcal{R}$sees the body only
Task policysees body + task goal
Latent $z$32 numbers per frame
Decoder $\mathcal{D}$body + $z$ → PD targets
PD + simulator
Teacher PHC+the frozen tracker
body state
same state
target action
KL
offset
task reward → PPO
Rebuild the tracker with a bottleneck. An encoder sees the body state and the next reference frame and writes 32 numbers, $z$. A decoder sees the body state and $z$, never the reference, and outputs the PD targets.
The student acts, the teacher labels. In every state the student reaches, the frozen tracker PHC+ says what it would do there. The loss is the squared difference of the two actions: no reward, no RL.
A prior that never sees the reference. It predicts a Gaussian over $z$ from the body state alone. A KL term pulls encoder and prior together, so the prior learns which $z$ are likely from this pose.
Drop the encoder: draw $z$ from the prior. A new random $z$ every frame and no reference at all. The decoder turns the noise into human motion, and gets up after a fall as the teacher did.
Freeze decoder and prior: $z$ is the new action space. A task policy outputs an offset from the prior mean, $z = \mu^p + z^{\text{task}}$, and is trained with PPO on the task reward alone.
99.8 %
of AMASS train trackedteacher PHC+: 100 %
72.0 %
with RL insteadsame student, over 10 B samples
The tracker PHC+ (blue) follows a mocap clip (red)
Encoder writes $z$: the student (blue) follows mocap (red)
Prior writes $z$: random motion, falls, gets up
Task policy writes $z$: training to reach the red targets
Schematic: two of the 32 dimensions of $z$
Luo, Cao, Merel, Winkler, Huang, Kitani, Xu. Universal Humanoid Motion Representations for Physics-Based Control. ICLR 2024 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
84
Under the hood

PULSE: Distill the tracker into a 32-d latent with a learned prior

Online distillation: the student explores, the teacher labels
Downstream: RL in the frozen latent space · speed, reach, strike, terrain, VR tracking
Prior $\mathcal{R}(z \mid s^p)$3-layer MLP · learned jointly
KL + smoothness$\beta\,D_{\text{KL}}(\mathcal{E}\,\|\,\mathcal{R}) + \alpha\,\mathcal{L}_{\text{regu}}$
Action loss$\|a^{\text{PHC+}}_t - a_t\|_2^2$ · supervised
Reference motionAMASS · next frame $\hat q_{t+1}$
Encoder $\mathcal{E}(z \mid s^p, s^g)$3-layer MLP → 32-d Gaussian
Decoder $\mathcal{D}(a \mid s^p, z)$3-layer MLP → 69-d action
PD + Isaac Gympolicy 30 Hz · simulation 60 Hz
Teacher PHC+RL imitator, frozen SiLU MLPs, 6 layers
PPOtask reward only, no adversarial reward
Task policy $\pi_{\text{task}}$sees $s^p$ + task goal $s^g$ · MLP 2048-1024-512 (VR: 6 layers)
Frozen $\mathcal{R}$ and $\mathcal{D}$$a = \mathcal{D}(s^p,\ z^{\text{task}} + \mu^p)$
goal $s^g$
latent $z$
PD targets
proprioception $s^p$
$(s^p, s^g)$
teacher action $a^{\text{PHC+}}$
KL
update
$z^{\text{task}}$
PD targets
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The student: reference in, 32-d latent, PD targets outThe frozen teacher labels every state the student visitsA learned prior, tied to the encoder by a KL termDownstream: a task policy adds a residual to the prior meanWhat it covers, and why sampling from it works
Objective
  • $\mathcal{L}_{\text{action}} = \|a^{\text{PHC+}}_t - a_t\|_2^2$: the teacher’s action is the label, no RL term
  • $\mathcal{L}_{\text{KL}} = D_{\text{KL}}(\mathcal{E}\ \|\ \mathcal{R})$, $\ \mathcal{E}(z_t \mid s^p_t, s^g_t) = \mathcal{N}(\mu^e_t, \sigma^e_t)$, $\ \mathcal{R}(z_t \mid s^p_t) = \mathcal{N}(\mu^p_t, \sigma^p_t)$ (diagonal)
  • $\mathcal{L}_{\text{regu}} = \|\mu^e_t - \mu^e_{t-1}\|_2^2$, $\quad\mathcal{L} = \mathcal{L}_{\text{action}} + \alpha\,\mathcal{L}_{\text{regu}} + \beta\,\mathcal{L}_{\text{KL}}$, $\ \alpha = 0.005$, $\ \beta$ annealed $0.01 \to 0.001$
  • Downstream: $a_t = \mathcal{D}\big(s^p_t,\ z^{\text{task}}_t + \mu^p_t\big)$, $\ z^{\text{task}}_t \sim \pi_{\text{task}}(\cdot \mid s^p_t, s^g_t)$; PPO, task reward only
Training recipe
  1. Train PHC+ with RL on cleaned AMASS until it tracks all 11,313 sequences; freeze it
  2. Roll out the student (E → z → D) on the imitation task; label its states with PHC+ actions
  3. Update E, D and R with the supervised loss only (no RL term); keep mining hard sequences
  4. Freeze D and R; train a task policy (and value net) with PPO; its z is added to the prior mean
32-dlatent, one per frame
99.8 %of AMASS train still tracked after distillation
≈ 40 hof mocap covered
≈ 1 Bsamples per latent-space model
The detail that makes it work

A learned, state-conditioned prior is the default action: z sampled from it already moves like a person, and the task policy only adds a residual.

Versus its predecessor

vs ASE / CALM: distilled from a tracker instead of trained adversarially. Retrained on the same 40 h of AMASS, ASE and CALM reach 38 % / 11 % on VR tracking; PULSE 93 %.

BasicsImitatePriorsModelsScaleInteractReality
85
Results

One latent space: new tasks and free-form tracking

Strike: walk up, knock the block over

A new task is one new reward

The reward only measures how far the block has tipped. PPO trains a policy that picks $z$; decoder and prior stay frozen.

Terrain: follow the path (blue dots)

Human-like without a style reward

The reward is stay on the path, plus an energy penalty. No adversarial reward, which PACER uses for this task.

Real headset and controllers in, full body out

Free-form tracking from three points

Recorded with a Quest 2: headset and two controllers. 14 of 14 recordings tracked; with an ASE latent, 7.

93.4 %
PULSE latentVR tracking success, AMASS test
37.6 %
ASE latent, 30 Hzsame test · retrained on AMASS
10.9 %
CALM latent, 6 Hzsame test · retrained on AMASS

Unlocked

The two roads merge: the coverage of a tracker, the reusability of a latent prior.

Still open

Every new task still needs a reward and RL training. Can we just tell it what we want?

Clips: PULSE project page (zhengyiluo.github.io/PULSE)
BasicsImitatePriorsModelsScaleInteractReality
86
SIGGRAPH Asia 2024

MaskedMimic: control as motion inpainting

One model. No task rewards. Any partial goal is a prompt.
87
How it works

Train on everything, then hide most of it

Full goal $g^{\mathrm{full}}$every joint, next frames
Tracker $\pi^{\mathrm{FC}}$RL-trained, then frozen
Partial goal $g^{\mathrm{partial}}$what the mask leaves
Prior $\rho(z \mid s, g^{\mathrm{partial}})$transformer encoder
Decoder $\mathcal{D}(a \mid s, z)$MLP, deterministic action
PD + simulatorterrain + objects
random mask $\mathcal{M}$
latent $z$
PD targets
state $s$
the student’s states
imitate $a^{\mathrm{FC}}$
The full-body tracker, here on stairs
Only a head target is given (green): it runs
One sofa, three characters, three ways to sit
Stage 1: a full-body tracker. Trained with RL on a tracking reward, on flat ground, on irregular terrain and around objects. Its goal says where every joint should be in the next frames.
Hide most of that goal. A random mask keeps targets for some of six body parts. A pattern persists for a while, then changes; gaps hide every joint. Text and object are given in some episodes only.
The student sees only what is left, here just the head. A transformer turns the visible targets into a latent $z$; a decoder turns the state and $z$ into PD targets for the whole body.
No reward: copy the tracker. The student drives the simulation. In every state it visits, the frozen tracker, which still sees the full goal, supplies the action to imitate.
One partial goal fits many motions. So $z$ is sampled: each sample is one way to complete the motion. The authors’ example: one sofa, three ways to sit.
Tessler, Guo, Nabati, Chechik, Peng. MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting. ACM TOG (SIGGRAPH Asia) 2024 · clips: authors’ video, project page
BasicsImitatePriorsModelsScaleInteractReality
88
Under the hood

MaskedMimic: A tracker distilled into a masked conditional VAE with a transformer prior

Stage 1: RL tracker · ≈ 30 B steps
Stage 2: masked distillation · ≈ 10 B steps
Tracking rewardjoint pose and velocity, root height, energy
A2C${\gamma = 0.99}$ · GAE ${0.95}$
Tracker $\pi^{\mathrm{FC}}$transformer + MLP critic, sees the full goal
Teacher $\pi^{\mathrm{FC}}$frozen · sees the full goal
Partial goal $g^{\mathrm{partial}}$joint targets, text, object from the user at test time
Prior $\rho(z \mid s, g^{\mathrm{partial}})$transformer encoder · 4 layers, 4 heads, 512-d
Latent $z$64-d · ${z = \mu + \sigma\,\epsilon}$ noise fixed per episode
Decoder $\mathcal{D}(a \mid s, z)$MLP 1024-1024-1024 → deterministic 69-d action
PD + Isaac Gymterrain + objects · policy 30 Hz, sim 120 Hz
Full goal $g^{\mathrm{full}}$AMASS + HumanML3D text + SAMP objects
Encoder $\mathcal{E}(z \mid s, g^{\mathrm{full}})$MLP 1024-1024-1024 used in training only
update
freeze
$\mu^{\rho}, \sigma^{\rho}$
PD targets
state $s$, heightmap
student's states
imitate $a^{\mathrm{FC}}$
random mask $\mathcal{M}$
residual $\mu^{\mathcal{E}}, \sigma^{\mathcal{E}}$
$\alpha\,D_{\mathrm{KL}}$
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
At test time: partial goal → prior → latent z → decoderIn training, z comes from an encoder that sees the full goalThe frozen tracker labels every state the student visitsWhere the teacher comes from: an RL tracker (stage 1)What it costs, and the detail that makes it work
Objective
  • Stage 1: $r = 0.5\,r^{\mathrm{pos}} + 0.3\,r^{\mathrm{rot}} + 0.2\,r^{\mathrm{height}} + 0.1\,r^{\mathrm{vel}} + 0.1\,r^{\mathrm{ang\,vel}} + 0.0005\,r^{\mathrm{energy}}$
  • Stage 2: $\max\ \mathbb{E}\big[\log \mathcal{D}(a^{\mathrm{FC}} \mid s, z) - \alpha\,D_{\mathrm{KL}}\big(\mathcal{E}(\cdot \mid s, g^{\mathrm{full}})\,\|\,\rho(\cdot \mid s, g^{\mathrm{partial}})\big)\big]$ · $\alpha$: $10^{-4} \to 10^{-2}$
  • $\mathcal{E} = \mathcal{N}\big(\mu^{\rho}(s, g^{\mathrm{partial}}) + \mu^{\mathcal{E}}(s, g^{\mathrm{full}}),\ \sigma^{\mathcal{E}}(s, g^{\mathrm{full}})\big)$: the encoder is a residual to the prior
  • $g^{\mathrm{partial}} = \mathcal{M}(g^{\mathrm{full}})$ · $z \sim \mathcal{E}$ in training, $z \sim \rho$ at test time
Training recipe
  1. Stage 1: A2C; stop at 0.25 m joint error (0.5 m on irregular terrain); oversample failing clips
  2. Stage 2 (DAgger): roll out the student; the frozen tracker labels every visited state
  3. Mask joints, frames, text and object; a mask persists over time instead of being resampled
  4. Ramp the KL weight from 1e-4 to 1e-2; at test time drop the encoder, sample z from the prior
16,384environments on 4 × A100
≈ 2 weeks30 B + 10 B steps
64-dlatent
99.2 %full-body tracking, AMASS test
The detail that makes it work

Masks are structured in time: a goal stays hidden or visible for a while. Without this, sitting on unseen objects falls from 96.9 % to 0 %.

Versus its predecessor

vs PULSE: the latent prior is conditioned on partial goals, so new tasks need no new RL. One model replaces one policy per task.

BasicsImitatePriorsModelsScaleInteractReality
89
The prompt: green = a target the user fills in
VR: head + hands only
Joystick: steering on rough terrain
Text: “performs a handstand”
Object: walks up to an armchair and sits down
Text + path: “walks with its hands raised”
MaskedMimic

One controller, many prompts

98.1 %VR tracking on unseen motions, zero extra trainingPULSE 93.4 % · ASE 37.6 %
96.9 %sits on held-out furniture0 % without structured masking

Unlocked

Zero task-specific training: VR, joystick, text, keyframes and objects are the same policy with different masks.

Still open

Someone still has to author the goals. Who plans long, semantic behavior?

BasicsImitatePriorsModelsScaleInteractReality
90
ICLR 2025 spotlight
Generative planner in the loop

CLoSD: diffusion plans, physics executes

A text-driven diffusion planner and a tracking controller, re-planning from the simulated state.
One planner and one tracker run a whole sequence of tasks
Sit down → get up → walk to the target → strike it with the left hand
The words choose the style as well
Deep squat → walk like a zombie to the target → sit down on the sofa
A task is a text prompt plus a target location for one joint. When the task is done, a state machine switches to the next pair.
“Sitting on the [sofa]”pelvis → centre of the seat
“Getting up from the [sofa]”pelvis → 0.3 m in front, 0.9 m high
“Walking to the [target]”pelvis → over the ring, any height
“Punching the [target] with the left”left hand → centre of the bag
Tevet, Raab, Cohan, Reda, Luo, Peng, Bermano, van de Panne. CLoSD: Closing the Loop between Simulation and Diffusion for Multi-Task Character Control. ICLR 2025
BasicsImitatePriorsModelsScaleInteractReality
91
How it works

Plan two seconds, track them, plan again

Text + targetwhat to do · where one joint must end
PPO, PHC rewardfine-tune in the loop, DiP fixed
Prefixthe last 20 frames
DiP plannerdiffusion, 10 denoising steps
Planthe next 40 frames = 2 s
Tracker $\pi_{PHC}$PHC, sets PD targets
PD + simulatorthe body, a bag, a sofa
every frame
every 40 frames
update
DiP, a diffusion model, continues a motion: 20 frames in, the next 40 frames out of noise, in 10 denoising steps.Two conditions steer the plan: the text says what to do, the target says where one joint must be at its last frame.A PHC tracker executes the plan one frame at a time: planned pose and simulated pose in, PD targets out.Close the loop: after 40 frames, the last 20 simulated frames become the prefix. DiP plans from what happened.The plans are not mocap, and they touch objects: fine-tune the tracker inside the loop, with PHC’s own reward.
“A person is walking”: many seeds, many walks
Prompt “High kick” · target: the bag
Blue markers: the plan · body: what the tracker does
Sit down, then get up: without and with the loop
Open loop
Closed loop
sit 0.19 · get up 0.23
sit 0.66 · get up 0.53
Sit down, then get up: before and after fine-tuning
Not fine-tuned
Fine-tuned: CLoSD
sit 0.66 · get up 0.53
sit 0.88 · get up 0.98
Tevet et al. CLoSD. ICLR 2025 · clips: project page (guytevet.github.io/CLoSD-page) · success rates: Table 1 of the paper · figure: schematic
BasicsImitatePriorsModelsScaleInteractReality
92
Under the hood

CLoSD: An autoregressive diffusion planner, replanned from the simulated state

HumanML3D(motion, text) pairs · crops of 20 + 40 frames
DiP loss$\mathcal{L}_{simple} + \lambda_{target}\,\mathcal{L}_{target}$
AMASSmocap clips · 62K PPO epochs
PPOfine-tune 4K epochs in the loop, DiP fixed
Original PHC rewardtrack $x^{ref}$ + AMP style + energy penalty
Text prompt + targetfrozen DistilBERT · joint $c_j,$ heading $c_\theta$
Diffusion Planner (DiP)8-layer transformer decoder, 512-d, 4 heads
Reference $x^{ref}$40-frame plan (2 s) · R2G to global joints
Tracker $\pi_{PHC}$single primitive · keypoints only
PD + Isaac Gym3072 parallel envs · kickboxing bag or sofa
Prefix $x^{prefix}$last 20 simulated frames, G2R
cross-attn
plan $x^{pred}$
$x^{ref}[n]$
PD targets
$x^{sim}[n]$, every frame
every 40 frames
one token per frame
train, then freeze
pre-training
update
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
One pass: text + target, DiP, a 40-frame plan, tracker, simulatorDiP: trained on HumanML3D crops of 20 + 40 frames, then frozenThe tracker: PHC reward and PPO, pre-trained on AMASSClose the loop, then fine-tune the tracker inside itWhat it costs, and the detail that makes it work
Objective
  • DiP: $\mathcal{L} = \mathcal{L}_{simple} + \lambda_{target}\,\mathcal{L}_{target}$, $\ \mathcal{L}_{simple} = \mathbb{E}_{x_0,\,t}\,\|x_0 - \hat{x}_0\|_2^2$
  • $\mathcal{L}_{target} = \sum_{j \in J} v_j\,\|R2G(\hat{x}_0[N_g])_j - c_j\|_2^2 + v_\theta\,\|R2G(\hat{x}_0[N_g])_\theta \ominus c_\theta\|_2^2$
  • Tracker: $s[n] = (x^{ref}[n] - x^{sim}[n],\ x^{ref}[n])$; PHC reward: tracking + AMP + energy penalty
  • Loop: $x^{ref} = R2G(x^{pred})$, $\ x^{prefix} = G2R(x^{sim}[-N_p{:}])$, $\ N_p = 20$, $N_g = 40$
Training recipe
  1. Train DiP on HumanML3D (MDM framework, 600K steps); use the 200K-step checkpoint
  2. Pre-train a single-primitive, keypoint-only PHC tracker on AMASS for 62K PPO epochs
  3. Close the loop: track each 40-frame plan, feed the last 20 simulated frames back as prefix
  4. Freeze DiP; fine-tune the tracker 4K epochs on random tasks: reach, strike, sit + get up
10diffusion steps per plan
3,072environments on one A100
600 KDiP training steps, one RTX 3090
4 Kclosed-loop fine-tuning epochs
The detail that makes it work

The planner is conditioned on what the simulated body actually did, so it reacts to contact; fine-tuning the tracker on its plans closes the gap.

Versus its predecessor

vs tracking text-to-motion output open loop (as in PHC): planner and tracker now exchange information at every plan.

BasicsImitatePriorsModelsScaleInteractReality
93
Results

Same task, different words

Reach the goal. The target stays where it is; the words choose how to get there.
“Jogging”
“Jump”
“Big diagonal steps”
“Dancing waltz”
Strike the bag. The words name the move; the target goes to the hand or foot that makes it.
“Push with both hands”
“High kick”
“Punch with the right hand”
1.0reach the goal
0.9strike
0.88sit down
0.98get up
Success rates: one model, four tasks

Unlocked

DReCon's split, upgraded: the kinematic planner is now a text-to-motion diffusion model that sees the simulated motion.

Still open

Planner and tracker are separate models that can disagree. Can one generative controller do both, at scale?

Clips: CLoSD project page (guytevet.github.io/CLoSD-page) · success rates: Table 1 of the paper, 1K episodes per task
BasicsImitatePriorsModelsScaleInteractReality
94
SIGGRAPH 2026
Generative pre-training

GPC: generative pre-trained controllers

Learn a discrete vocabulary of motor skills with RL, then train a GPT over it.
No reference: tokens sampled from the GPT
Generate

No reference and no goal. A GPT writes skill tokens; a decoder and physics turn them into motion.

Unconditional sampling: every character starts from the same pose.
Emergent: pushed, falls, gets up
Emergent

Pushed over, it gets back up. No reward and no special training asked for that.

not on the ground at the end, after a 9.8 m/s push
GPC52.5 %
C-VAE prior3.1 %
The FSQ tracker: sim (blue) vs reference (green)
Scale

One tracker with a discrete bottleneck reproduces 99.98 % of its clips.

hours of training motion
GPC680 h
MoConVQ23.2 h
Shi, Jiang, Tessler, Peng. GPC: Large-Scale Generative Pretraining for Transferable Motor Control. SIGGRAPH 2026 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
95
How it works

A vocabulary learned with RL, a GPT that speaks it

Reference clips680 h of mocap
Encoder $\mathcal{E}$sees 7 future frames
FSQ9 levels · nothing to learn
Decoder $\mathcal{D}(a \mid s, \hat z)$state + code → PD targets
PD + simulatorIsaac Gym
Task input $c$target, path, heightmap
CoLA adapters< 1 % extra parameters
GPT $p_\theta(\tilde z \mid s)$sees only the state
$z_t$
$\hat z_t$
$a_t$
$s_t$
tracking reward · PPO trains both halves
tokens
state $s_t$
$c$
task reward · PPO trains only the adapters
Stage 1: one policy tracks 680 hours of clips with PPO. It is cut in two, and only the encoder sees the reference.Between the halves, FSQ squashes each of the encoder’s 40 numbers and rounds it to one of 9 levels. No codebook to learn.The levels of five neighbouring numbers pack into one token with 9⁵ = 59,049 possible values: every control step is now 8 tokens.Stage 2: a GPT learns to write these 8 tokens from the state alone. Sample from it: motion with no reference to track.Stage 3: a new task trains only small adapters, with PPO. Sampling stays inside the frozen GPT’s most likely tokens.
?
later: who drives the decoder when there is no reference?
8 tokens
cross-entropy targets
$\hat z = \mathrm{round}\,(4 \tanh z)$
Stage-1 tracker: simulated (blue) vs reference (green)
Tokens sampled from the GPT: one start pose, many motions
Adapter: steering
Clips: project page (yi-shi94.github.io/gpc-page) · the figure under the pipeline is a schematic, not data
BasicsImitatePriorsModelsScaleInteractReality
96
Under the hood

GPC: FSQ skill tokens learned with RL, then a GPT over the tokens

Stage 1: skill quantization · RL tracker with an FSQ bottleneck
Stage 2: generative controller · next-token prediction
Stage 3: task adaptation · GPT and decoder frozen
Reference $\hat s_{t:t+h}$next 1, 2, 5, 7, 12, 18, 25 frames
Encoder $\mathcal{E}$MLP 3×1024-512-256 → 40 latent dims
FSQrounds each dim to 9 levels · no learned codebook
Decoder $\mathcal{D}(a \mid s, \hat z)$MLP 3×1024-512-256 → PD targets
PD + Isaac Gym23 bodies · 66 DoFs
PPOtracking reward · critic MLP 4×1024
Frozen $\mathcal{E}$ + FSQreference → 8 grouped tokens, vocab $9^5$
Cross-entropylabel smoothing · weight EMA 0.9
GPT $p_\theta(\tilde z_t \mid s_t)$causal transformer · 6 layers, 4 heads, 1024-d
Frozen decoder $\mathcal{D}$decodes tokens sampled with top-$p$ 0.9
PD + Isaac Gym
Task obs $c$target, heading, waypoints or heightmap
CoLA adaptersFiLM on DoRA · rank 64 · < 1 % extra params
PPOtask reward · actions are the 8 tokens
$z_t$
$\hat z_t$
$a_t$
$s_t$
$r_t$
update $\mathcal{E}$ and $\mathcal{D}$
$\tilde z_t$
$\tilde z_t$
$a_t$
$s_t$
state $s_t$ as context token
$c$
adapts
task reward
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Stage 1: reference in, a rounded code, PD targets outPPO trains both halves end to end, through the roundingStage 2: a GPT learns the encoder’s tokens, then replaces itStage 3: PPO trains only the adapters, on a task rewardWhat it costs, and what makes it stable
Objective
  • FSQ: $\hat z_t = \big\lfloor \lfloor L/2 \rfloor \tanh(z_t) \big\rceil$, $\ L = 9$, $d = 40$: implicit codebook $9^{40}$, straight-through gradient
  • Stage 1: $r = 0.5\,r_{\mathrm{gp}} + 0.3\,r_{\mathrm{gr}} + 0.2\,r_{\mathrm{rh}} + 0.1\,r_{\mathrm{jv}} + 0.1\,r_{\mathrm{jav}}$ · joint pos, rot, root height, vel, ang vel
  • Stage 2: $\mathcal{L}_{\mathrm{CE}} = -\sum_{j=0}^{d'-1} \log p_\theta\big(\tilde z_t^{\,j} \mid s_t, \tilde z_t^{<j}\big)$, $\ d' = 8$ tokens of vocabulary $9^5$
  • Stage 3: $W x = W_0 x + m\,B u\,/\,\|B u\|_F$, $\ u = \mathrm{diag}(\gamma(c))\,A x + \beta(c)$; PPO on the task reward
Training recipe
  1. Stage 1: train encoder and decoder end to end with PPO; Muon actor, AdamW critic
  2. Prioritize failed clips: success = max joint error under 0.5 m over the episode
  3. Stage 2: pack every 5 FSQ tokens into one (40 → 8); teacher-force the GPT on them
  4. Stage 3: train only the adapters: optional SFT, then PPO in the frozen GPT’s top-p set
680 hBones dataset · ≈ 343 k clips
99.98 %tracking success rate
24 × A100for the Bones models
40 hAMASS, used for ablations
The detail that makes it work

FSQ rounds each latent channel to a few fixed levels: no learned codebook, no commitment loss, no dead-code resets, so it trains stably inside RL.

Versus its predecessor

vs MoConVQ: FSQ and plain RL instead of VQ-VAE and a world model, on 680 h instead of 23.2 h. vs MaskedMimic: tokens + GPT, not a C-VAE.

BasicsImitatePriorsModelsScaleInteractReality
97
Tracker: flip, sim vs reference
Adapter: path following under pushes
Adapter: gaps in the terrain
Same flip: FSQ lands it, VQ-VAE and FSQ-K fall
Adapter after SFT on a crouch walk: target reaching
Adapter: parkour course
GPC

The tracker: 34.9 mm error with FSQ, 25.6 mm with no bottleneckLocomotion: PPO trains only the adapters, the GPT stays frozenScenes: a heightmap as task input, and one reward: reach the targetPre-train once, adapt with a few parameters

Unlocked

The language-model recipe, complete: tokenizer, generative pre-training, parameter-efficient fine-tuning.

Still open

Mostly locomotion so far. No objects, no furniture: real behavior is contact.

BasicsImitatePriorsModelsScaleInteractReality
98
Chapter 5  ·  2023 – 2025

Interact

Scenes, objects, sports, hands — and data beyond mocap.
Physical Character-Scene Interactions SIGGRAPH 2023
UniHSI ICLR 2024
TokenHSI CVPR 2025
Tennis Skills from Broadcast Videos SIGGRAPH 2023
SMPLOlympics arXiv 2024
Omnigrasp NeurIPS 2024
InterMimic CVPR 2025
99
SIGGRAPH 2023
Scene interaction

Physical Character-Scene Interactions

Show the discriminator the scene as well as the character: AMP learns to sit, lie down and carry.
Randomizationshape, size, place
RL updateas in AMP
Objectbounding box = goal $g$
Policy $\pi(a \mid s, g)$
PD + simulatorcharacter + object
Discriminator $D$sees body + object
Scene mocapmotion + object pose
$g$
placed in the scene
action $a$
state $s$
task reward
update
$(s, s')$
$(s, s')$
style reward

A task you write. A chair stands in the simulated scene, and the policy is told its bounding box. The reward pays for walking towards it at 1.5 m/s, then for bringing the hip onto the seat.

A style you show, with the furniture in it. Mocap of people sitting, lying down and carrying a box, no labels. Every clip also records where the object is: its position and orientation.

The judge sees the object too. The state $s$ holds the object’s position and orientation in the character’s frame, and the discriminator reads that state. “Looks like the data” now means: like the data, next to this object.

Randomize the object. The sitting and lying-down mocap has seven objects. In training, the parallel environments hold different objects, ≈ 350 from ShapeNet, rescaled every episode and placed 1–10 m away, facing any way.

Sit: walks to the chair, sits down
what shows it
how to sit?
The data: every clip has its object
what does the judge look at?
Schematic: the same two motions, shown to both discriminators
and a chair it
has never seen?
The objects: chairs, sofas, tables…
Hassan, Guo, Wang, Black, Fidler, Peng. Synthesizing Physical Character-Scene Interactions. SIGGRAPH 2023 · clips: authors’ video
BasicsImitatePriorsModelsScaleInteractReality
100
Under the hood

InterPhys: AMP, with the object inside the discriminator’s view

RL update, as in AMPvalue function MLP 1024-512
$r = w^{G} r^{G} + w^{S} r^{S}$task + style
Style reward$-\log\big(1 - D(s, s')\big)$
Goal $g$object bounding box (carry: + target position)
Policy $\pi(a \mid s, g)$MLP 1024-512 → 28-d action
PD + Isaac Gymcharacter and object
Discriminator $D(s, s')$MLP 1024-512 · object pose is part of the state
Scene mocapSAMP sit / lie down + 15 carry clips, with object poses
Object randomization≈ 350 ShapeNet objects, rescaled, 1–10 m away
PD targets
114-d state $s$: character + object pose
fake
real
task reward
update
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Control loop: bounding box in, PD targets out, state backTask reward: walk to the object, then hip onto the seatThe discriminator reads the same state: object includedIts score is the style reward, added to the task rewardDifferent objects across environments, moved each episodeThe numbers, and the detail that makes it work
Objective
  • Sit, lie down: $r^{G} = 0.7\,r^{near} + 0.3\,r^{far}$ beyond 0.5 m from the object, $0.7\,r^{near} + 0.3$ within
  • $r^{far}$: be close to the object, walk towards it at 1.5 m/s · sit: $r^{near} = \exp\big(-10\,\|x^{root*} - x^{root}\|^2\big)$
  • $\min_D\ -\mathbb{E}_{\mathcal{M}}\big[\log D(s, s')\big] - \mathbb{E}_{\pi}\big[\log\big(1 - D(s, s')\big)\big] + w^{gp}\,\mathbb{E}_{\mathcal{M}}\big[\|\nabla D\|^2\big]$
  • $r = w^{G}\,r^{G} + w^{S}\,r^{S}$, $\ r^{S} = -\log\big(1 - D(s, s')\big)$
Training recipe
  1. Different objects in the parallel environments: ≈ 350 ShapeNet instances for sit / lie down
  2. Each episode: start from a random dataset state (RSI), rescale the object, place it 1–10 m away
  3. Roll out 10 s (carry: 15 s); stop if a joint except feet / hands is within 20 cm of the ground
  4. Train π, V and D jointly, following the training strategy of AMP; one policy per task
≈ 350training objects
4,096evaluation trials per task
100 minof SAMP mocap
90–94 %success on the three tasks
The detail that makes it work

Because the discriminator sees the object, “sitting” only looks real when a chair is under the character: the prior itself encodes the interaction.

Versus its predecessor

vs AMP: the same algorithm, made scene-conditioned. vs kinematic methods (NSM, SAMP): 93.7 % against 75 % sit success in the baseline comparison.

BasicsImitatePriorsModelsScaleInteractReality
101
Results

Three tasks, and objects it has never seen

Sit: approach the chair and sit down in one motion

No planner, no phase labels

Walking, turning and sitting come from one policy: it learns when to switch.

Lie down, on sofas never seen in training

Furniture it has never seen

It sits, swings its legs up and lies down, adapting to the sofa it finds.

Pick up a box and carry it to the target

A box that is not glued on

A simulated rigid body: only forces from the character move it. The mocap has one box.

≈ 350
training objects21 unseen at test time
90 – 94 %
successsit · lie down · carry
100 min
of scene mocap+ 15 carry clips

Unlocked

Scene-conditioned priors: approach, contact and interaction in one natural motion, on unseen furniture.

Still open

One policy per task, and someone has to pick the task. Can we just say what we want?

Clips: authors’ video (youtu.be/q3hyQdaElQQ)
BasicsImitatePriorsModelsScaleInteractReality
102
ICLR 2024 spotlight
Language-driven scene interaction

UniHSI: say it, and the character does it

An LLM turns a sentence into a Chain of Contacts; one controller executes it step by step.
“I want to have a rest”: planned by the LLM in a real scanned room
a longer plan, several objects?
One plan, three objects: the chair, the laptop, then the bed
Before · a policy per task
“Run the sit policy on this chair.”
its own observationits own rewardits own policy
UniHSI · one policy, the task is data
“Which joint touches which part, in what order?”
sit = pelvis on seatlie down = pelvis on mattress, then head on pillow
Xiao, Wang, Wang, Cao, Zhang, Dai, Lin, Pang. Unified Human-Scene Interaction via Prompted Chain-of-Contacts. ICLR 2024 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
103
How it works

Words become contacts, contacts become the reward

Command + sceneparts, joint names
LLM plannerGPT, not trained
Chain of Contactssteps of contact pairs
TaskParserstep → goal + reward
Policyone for every step
SimulatorIsaac Gym
joint and part positions
task reward $\sum_k w_k R_k$
done → next step
+ style reward (AMP discriminator)

Say it. The command goes to an LLM, together with what it cannot see: the objects in the room with their named parts, and the joint names of the body. A few rules and an example plan fix the format of the answer.

The LLM writes a Chain of Contacts. Every step is a set of pairs {object, part, joint, contact type, direction}: “sit” is the pelvis on the seat, “use the laptop” adds both hands on the keyboard. GPT is prompted, nothing is trained here.

One controller for every step. For each pair of the current step, the TaskParser measures in the simulator the vector $d_k$ from the joint to the nearest point of the part. That vector, with the contact type and direction, is the policy’s goal, whatever the step asks for.

Contacts are the task reward. Each pair has a score $R_k$: high when the joint is at its part, from the right direction. Adaptive weights $w_k \propto 1 - R_k$ put the weight on the pairs not yet satisfied. Without them, success on simple plans falls from 85.5 % to 21.2 %.

Done? Next step. Once every pair holds ($\|d_k\| < 0.1$, direction matched), the TaskParser fetches the next step. Walking and standing up are steps too: a pseudo-contact pulls the pelvis to a target point.

Schematic of the paper’s example (prompt and plan: Table 7). Not a simulation.
and in the simulator?
Authors’ demo in such a room: chair, laptop, then the bed
Prompt and plan: Xiao et al., Table 7 (LLM planner input and output) · clip: UniHSI project page, “Multi-objects Interaction”
BasicsImitatePriorsModelsScaleInteractReality
104
Under the hood

UniHSI: Chain of Contacts: one task format for every interaction

LLM Planner
Unified Controller · built on AMP
Task reward $R^{G}$$\sum_k w_k R_k$ · adaptive weights ${w_k \propto 1 - R_k}$
PPO$R = w^{G} R^{G} + w^{S} R^{S}$
Style reward $R^{S}$$-\log\big(1 - D(s^{A}_{t}, s^{A}_{t+1})\big)$
GPT-3.5 / GPT-4prompted with rules + examples, not trained
Chain of Contactssteps of pairs {object, part, joint, contact, direction}
TaskParserstep → obs $s^{U}$ + reward per contact pair
Policyactor and critic: CNN layer + MLP 1024-1024-512
Isaac Gym8,192 envs · 1–4 PartNet objects per scene
Discriminator $D$MLP 1024-1024-512 on 10 adjacent frames
Language + scenecommand, object parts + layout, joint names
Motion clipsSAMP (100 min) + 20 CIRCLE reaching clips
done → next step
state + heightmap
joints, part points
real
update
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The planner: a prompt in, a Chain of Contacts outThe TaskParser turns the current step into the policy’s goalOne reward term per contact pair, mixed by adaptive weightsStyle from an AMP discriminator; PPO trains the policyWhat it costs, and the detail that makes it work
Objective
  • $R^{G} = \sum_{k=1}^{n} w_k R_k$ over the $n$ contact pairs, adaptive $w_k = (1 - R_k)\,/\,(n - \sum_k R_k + e)$
  • contact: $R_k = w_{dis}\exp(-w_{dk}\|d_k\|) + w_{dir}\max(\bar{d}_k \cdot \hat{d}_k,\, 0)$, $\ d_k$: joint → nearest part point
  • not contact: $R_k = 1 - \exp(-w_{dk}\|d_k\|)$ · a contact is done at $\|d_k\| < 0.1$, $\bar{d}_k \cdot \hat{d}_k > 0.8$
  • $R = w^{G} R^{G} + w^{S} R^{S}$, $\ R^{S} = -\log\big(1 - D(s^{A}_{t}, s^{A}_{t+1})\big)$; $D$: GAN loss + gradient penalty
Training recipe
  1. Pre-generate plans with the LLM (ScenePlan): 40 PartNet objects, 5–20 steps each
  2. Per scenario, place 1–4 random objects and use their steps; no LLM in the loop
  3. Build observation and reward from the current step; advance once all pairs hold
  4. Train the policy with PPO (Adam, lr 2e-5) and the discriminator jointly, as in AMP
8,192environments on one A100
40training objects from PartNet
85.5 → 21.2 %simple-plan success without adaptive weights
The detail that makes it work

Walking is written as a pseudo-contact between the pelvis and a target point, so locomotion and interaction share one formulation.

Versus its predecessor

vs InterPhys: one controller for all interactions instead of one per task, and the task arrives as language.

BasicsImitatePriorsModelsScaleInteractReality
105
Results

One controller, contact chains of any length

Fine-grained control: many interactions with the same chair
longer plans, real rooms?
A long multi-object plan in a scanned room: two characters, planned by LLMs
85.5 %
success, simple plans
1 object, up to 3 steps
67.9 %
medium plans
1 object, more steps
40.5 %
hard plans
several objects
73.2 %
simple plans
in real scanned rooms

Unlocked

Language as the interface to scenes: contacts are a universal task description, and an LLM can write them.

Still open

Static furniture, one contact chain at a time. Carrying, climbing, combining skills?

Clips: UniHSI project page (xizaoqu.github.io/unihsi) · numbers: Table 2, success rate on ScenePlan
BasicsImitatePriorsModelsScaleInteractReality
106
CVPR 2025 oral

TokenHSI: one transformer for scene interaction

Follow a path, carry a box, climb, sit down: one policy, one long task
107
How it works

Proprioception is one token; every task adds its own

Task label $l_t$one-hot: which task
Transformer encoder $\phi$self-attention over the tokens · a learnable output token $e$ collects the result
Action head $\mathbb{H}$MLP → PD targets
PD + simulatorIsaac Gym
Tokenizer $\mathbb{T}^{prop}$shared by all tasks
Task tokenizersone MLP per task
Tokenizer $\mathbb{T}^{new}$for the new task
Adapters $\xi^{\mathbb{A}}$zero-initialized
body state $s_t$
body token
task tokens
mask
new token
+ bypass

The body state gets a token of its own. Earlier controllers read body state and goal as one vector. Here a tokenizer turns the body state $s_t$ (proprioception) into one token; a transformer reads it and an action head sets the PD targets.

Every task gets a tokenizer. Follow a path, sit, climb, carry: each task has its own observation, and its own small MLP turns that observation into one more token of the same size.

A label picks the task. The one-hot task label $l_t$ is the attention mask: the tokens of the other tasks are masked out. The body token is never masked.

One policy, trained on all four. Every environment draws a task at reset. PPO maximizes that task’s reward plus an AMP style reward; the discriminator is told the task, too.

It matches four specialists. On every task its success rate is at least that of a policy trained for that task alone. Carrying gains most: 83.1 % → 92.2 %.

A new task is a new token. Sit down while carrying: the trained policy is frozen and its sit and carry tokenizers are reused. It gains two trainable parts: a new tokenizer, and zero-initialized adapters on the action head.

Follow a path
Sit on a chair
Climb onto an object
Carry a box to a target
New: sit down while carrying
99.7 %success rate
specialist: 98.7 %
99.6 %success rate
specialist: 98.2 %
99.8 %success rate
specialist: 99.7 %
92.2 %success rate
specialist: 83.1 %
96.3 %success rate
from scratch: 82.9 %
Pan, Yang, Dou, Wang, Huang, Dai, Komura, Wang. TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization. CVPR 2025 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
108
Under the hood

TokenHSI: A shared proprioception token and masked task tokens in one transformer

Stage 2: only these policy modules train
Stage 1: one policy, four tasks
New tokenizer $\mathbb{T}^{new}$height map or composite state, or fine-tuned $\mathbb{T}^{f}$, $\mathbb{T}^{c}$
Adapters $\xi^{\mathbb{A}}$zero-init bypass layers on the action head
PPO$r_t$: style + task · critic MLP 2048-1024-512
4 task tokenizersMLP 256-128-64 each → one 64-d token per task
Transformer $\phi$4 layers · 2 heads · FF 512 · label $l_t$ masks other tasks
Action head $\mathbb{H}$MLP 1024-512-32 on the learnable output token $e$
PD + Isaac Gym4,096 envs · task redrawn at every reset
Discriminator $D$MLP 1024-512 · conditioned on task label $l_t$
4 task observationsfollow 2×10 · sit 38 · climb 27 · carry 42-d
Tokenizer $\mathbb{T}^{prop}$MLP 256-128-64 → 64-d · attended in every task
Reference motionAMASS, SAMP, OMOMO · five skill subsets
shared token
PD targets
proprioception $s_t$ · 222-d
real
task reward
style reward
stage 1 update
new token
+ bypass
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The loop: body state → shared token → transformer → PD targetsFour task tokenizers; the label masks all but the active taskStage 1: PPO on a task reward and a task-conditioned style rewardStage 2: the policy is frozen; a new tokenizer and adapters trainWhat it costs, and what makes skills composable
Objective
  • $\pi(a_t \mid e, s_t, g^f_t, g^s_t, g^m_t, g^c_t)\ \to\ \pi(a_t \mid e, s_t, g^{l_t}_t)$: label $l_t$ masks the other task tokens
  • $r_t$ combines an AMP style reward $r^{style}_t$ and the active task's $r^{task}_t$; PPO maximizes $\sum_t \gamma^t r_t$
  • Sit: $r^s_t = 0.7\,\exp(-10\|x^{tar}_t - x^{root}_t\|^2) + 0.3\,r^{far}_t$, $r^{far}_t = \exp(-2\|1.5 - d^*_t \cdot \dot{x}^{root}_t\|^2)$ ($1$ within 0.5 m)
  • Sit + carry: $r^{s+c}_t = 0.7\,r^s_t + 0.3\,r^{pick}_t$ (hands on the box), $\ 0$ if $\|x^{obj}_{2d} - x^{root}_{2d}\| > 0.7$
Training recipe
  1. Stage 1: every env draws a task at reset: 10 % follow, 30 % each sit / climb / carry
  2. Train all tokenizers, φ and H with PPO + task-conditioned AMP: 4,096 envs, 50k iterations
  3. Stage 2: freeze the shared tokenizer, e, φ, H; add adapters and a new or fine-tuned tokenizer
  4. Train only those with PPO and a new D from scratch: 3k–10k iterations (50k: carry on stairs)
4,096environments
50kPPO iterations in stage 1
3k–10kiterations per adaptation (carry on stairs: 50k)
99.2 %climb + carry; 81.9 % without the shared token
The detail that makes it work

Keeping proprioception as its own token lets every task reuse one body representation: that is what makes skills composable.

Versus its predecessor

vs InterPhys and UniHSI: one policy covers several skill types that can be combined (sit while carrying) and extended.

BasicsImitatePriorsModelsScaleInteractReality
109
Results

New tasks plug into the frozen policy

Compose two skills. Two trained task tokenizers are reused, frozen; a new one joins them. No reference motion sits down holding a box.
Sit down while carrying a box
Climb while carrying a box
Follow a path while carrying a box
Climb + carry
TokenHSI99.2 %
w/o body token81.9 %
CML68.3 %
CML (dual)51.3 %
From scratch26.8 %

success rate after 5k iterations

Change the object or the ground. The carry tokenizer is fine-tuned in a copy; on stairs, a height map is one more token.
Carry a chair
Carry a table
Carry a box over stairs
Carry on uneven terrain
TokenHSI74.0 %
w/o adapters10.8 %
AdaptNet63.4 %
From scratch0 %

success rate after 50k iterations

Unlocked

Multi-skill scene interaction in one policy: composable (sit while carrying) and extensible by adding tokens.

Still open

Every basic skill still starts from mocap of that skill. What if the motion only exists on video?

Clips: project page (liangpan99.github.io/TokenHSI) · success rates: Tables 2 and 4 of the paper
BasicsImitatePriorsModelsScaleInteractReality
110
SIGGRAPH 2023

Tennis skills from
broadcast video

Both players are simulated. Their skills came from TV footage, not a mocap stage.
111
How it works

Video in, physics as the filter

Broadcast video13 US Open matches
Pose estimationoff-the-shelf models
Imitation policytracks a pose in the simulator
Motion VAEpose, $z$ → next pose
High-level policysees the ball and a target
noisy
cleaner
latent $z$
next pose
wrist + elbow angles
grey: how the data is made, stage by stagegreen: what runs at every frame
Input: broadcast video
Estimated kinematic motion
can a body with mass
move like this?
Estimate yellow, simulated pink
how to reuse
the corrected motion?
Sampled from the embedding
who picks
the next pose?
Many balls in, one red target

The data is television.

13 US Open broadcasts, about 4.6 hours of motion from three players. Off-the-shelf pose estimation turns the near-court player into 3-D motion: no tennis mocap, no stroke labels.

Physics as the filter.

The estimate jitters; its feet slide and sink into the ground. A tracking policy (pre-trained on AMASS mocap, fine-tuned on this motion) follows it in the simulator, and what the simulated body does becomes the new data.

A motion embedding.

A VAE is fitted on the corrected motion. Its decoder turns the current pose and a latent $z$ into the next pose: each $z$ is one way to continue.

A planner picks the latents.

A second policy, the planner, sees the incoming ball and a target and outputs $z$ frame by frame. The same imitation policy tracks the decoded pose in the simulator. The planner’s reward: racket to the ball, then ball to the target.

Overwrite what the camera cannot see.

Blur and occlusion hide the wrist. The planner also outputs three arm angles (two at the wrist, one elbow twist) that overwrite the decoder’s. Without them, bounce-in rate 0.85 → 0.46.

Clips: authors’ supplementary video (youtu.be/ZZVKrNs7_mk) and project page; broadcast footage courtesy of Tennis Channel · bars: Table 3 · bounce-in rate: Table 2, medians
BasicsImitatePriorsModelsScaleInteractReality
112
Under the hood

Tennis: A kinematic motion VAE on top, physics imitation below

Stage 4: high-level motion planning
Stages 1–3 (right to left): pose estimation, imitation, motion embedding
Random incoming ballsevery 2–2.5 s · 25–35 m/s · spin 0–50 RPS
PPO3-stage curriculum · 5 B samples
Reward: $r^r$ before contact, $r^b$ afterracket to ball at phase $\theta = \pi$ · estimated bounce vs target, with the right spin
Ball forecast + targetball's next 10 frames · target bounce + spin
High-level policyoutputs latent $z$ and 3 swing arm joint angles
MVAE decoderencoder discarded · next pose + phase $\theta$
Imitation policytracks the target pose · residual root force
PD + Isaac Gymracket · ball with air drag + Magnus force
Motion VAE (MVAE)court-frame poses · KL weight $\beta = 0.5$
Imitation policyPPO tracking · AMASS, then fine-tuned on $\mathcal{M}_{\text{kin}}$
Broadcast video13 US Open matches · ViTPose + HybrIK → SMPL
latent $z$
pose
wrist + elbow override
PD targets
state
simulated character state
racket, ball
$\mathcal{M}_{\text{kin}}$
$\mathcal{M}_{\text{corr}}$
reuse
decoder
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Run time: planner → decoder → tracker → simulatorThree arm angles go around the decoderOnly the planner is trained here: PPO, two rewardsThe tracker: video motion in, corrected motion outThe decoder: an MVAE fitted on the corrected motion
Objective
  • Before contact: $r^r_t = \exp\!\big(-\alpha_r \|x^r_t - x^b_t\|^2\big)\cdot\exp\!\big(-\alpha_\theta \|\theta_t - \pi\|^2\big)$
  • After contact: $r^b_t = \exp\!\big(-\alpha_b \|\tilde{x}^b - \hat{x}^b\|^2\big)$ if the spin direction matches, else $0$
  • Imitation: $r_t = \omega_o r^o_t + \omega_v r^v_t + \omega_p r^p_t + \omega_k r^k_t + \omega_e r^e_t$ (rotation, velocity, position, 2D keypoints, power)
  • MVAE: next-pose reconstruction $+\ \beta\,D_{\text{KL}}$, $\ \beta = 0.5$, with scheduled sampling
Training recipe
  1. Estimate SMPL motion and a court-frame root trajectory from 13 broadcast matches: Mkin
  2. Pre-train the imitation policy on AMASS, fine-tune on Mkin (adds $r^e$), replay to export Mcorr
  3. Fit the MVAE on Mcorr (pose + phase, 20 % of shots phase-labelled); keep the decoder
  4. Train the high-level policy: $r^r$ only, then 3 fixed targets (0.9 on $r^b$), then any target
≈ 1 Blow-level samples · ≈ 12 h, 1 GPU
5 Bhigh-level samples · ≈ 2 days
≈ 4.6 hof player motion from video
0.92median hit rate
The detail that makes it work

Video cannot resolve the wrist during a swing, so the high-level policy learns to override it with corrections that make the shot work.

Versus its predecessor

vs mocap-based priors: the data is monocular broadcast video, and physics imitation cleans it before any skill is learned.

BasicsImitatePriorsModelsScaleInteractReality
113
Results

Diverse strokes, player styles, extended rallies

What the simulator sees: rigid bodies and a racket

Many strokes, no stroke labels

One controller serves and hits forehands and backhands. The target also names a spin, topspin or backspin, and the matching stroke follows.

Left-handed, two-handed backhand

Other footage, another player

One embedding and one planner per player. From Federer’s footage: right-handed, one-handed backhand. From Nadal’s: left-handed, two hands.

Rally: left-hander (near) against right-hander (far)

Trained alone, they rally

Each controller trained alone, on launched balls. Put two on one court and they rally: the paper reports 38 shots in one rally.

0.92
median hit ratethe racket meets the ball
0.85
median bounce-in ratethe ball lands in the court
1.74 m
median bounce errorto the target, balls that land in

Unlocked

Mocap is not the only data. Physics turns noisy video estimates into usable skills; median hit rate 92 %.

Still open

One sport took a dedicated system. Can sports be a benchmark instead of a project?

Zhang, Yuan, Makoviychuk, Guo, Fidler, Peng, Fatahalian. Learning Physically Simulated Tennis Skills from Broadcast Videos. SIGGRAPH 2023 · clips: project page · Table 1, Federer’s controller
BasicsImitatePriorsModelsScaleInteractReality
114
arXiv 2024
Sports as a benchmark

SMPLOlympics: an arena for simulated humanoids

Golf, javelin, high jump, hurdles, tennis, table tennis, fencing, boxing, soccer, basketball: one humanoid, one pipeline.
High jump · PPO: the task reward alone
the same reward,
with a motion prior?
High jump · the same reward, acting through the PULSE motion prior
The whole reward: one term pays for getting closer to a point 2 m behind the bar; within 0.5 m of the bar a second term pays for height. No term says how to jump.
The finding: “combining strong motion priors with simple rewards can result in human-like behavior in various sports.” The humanoid is SMPL-compatible, so video and mocap demonstrations need no retargeting.
Luo, Wang, Liu, Zhang, Tessler, Wang, Yuan, Cao, Lin, Wang, Hodgins, Kitani. SMPLOlympics: Sports Environments for Physically Simulated Humanoids. arXiv:2407.00187, 2024 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
115
How it works

One simple reward per sport. Where does the prior go?

Goal state $s^g$hurdles, finish line
Task policy $\pi_{\text{task}}$trained with PPO
Isaac Gymone SMPL humanoid
Discriminatorlike the demos?
Mocap clipsAMASS locomotion
PULSE decoderfrozen, from AMASS
PHC trackerimitates it in physics
Internet videoTRAM: video → pose
action $a$
proprioception $s^p$
task reward
fake
real
style reward
latent $z$
action
real

A sport is a goal state and a simple reward. Hurdling: the policy sees the ten hurdles and the finish line 110 m ahead, and is paid for every step that brings it closer. PPO on this alone gets over the hurdles in a way no human would, and completes 57.6 % of its races.

The prior as a second reward. An AMP discriminator pays for moving like mocap locomotion: $r = 0.5\,r^{\text{task}} + 0.5\,r^{\text{disc}}$. The task is hard, and simple motion such as standing already pleases the judge: the policy stops before the first hurdle. 0 % here and in both jumps.

The prior as the action space. The policy outputs a 32-d latent; the frozen PULSE decoder, pre-trained on AMASS, turns it into joint actions. The prior is inside every action, so the task reward alone is enough: 100 %, in 17.76 s.

A sport’s own style, from video (golf, tennis, table tennis, penalty kicks; not track and field). TRAM estimates the pose, PHC tracks it in simulation, and the tracked motion feeds the discriminator, on top of PULSE. Tracking doubles table-tennis hits: 0.91 → 1.83.

PPO: the task reward alone
add a motion prior
as a second reward?
AMP: task + style reward
or build the prior
into the actions?
PULSE: task reward, latent actions
and the style of the sport itself:
demonstrations from video?
Tennis video: the estimated pose
After PHC tracks it in physics
Clips: SMPLOlympics project page (Luo et al., 2024), hurdling with PPO, AMP and PULSE, and “Data from Videos” (tennis) · numbers: Tables 1 and 5 of the paper
BasicsImitatePriorsModelsScaleInteractReality
116
Under the hood

SMPLOlympics: One SMPL-family humanoid and one hyper-parameter set for many sports

PPO$\gamma=0.99$ · clip $0.2$ same set for every sport
$w_{\text{task}}\,r^{\text{task}} + w_{\text{disc}}\,r^{\text{disc}}$task only: 1 / 0 with AMP: 0.5 / 0.5
Goal state $s^g$bar, hurdles, ball, target, opponent …
Task policy $\pi_{\text{task}}$MLP 2048-1024-512
Isaac GymSMPL / SMPL-X humanoid policy 30 Hz · sim 60 Hz
AMP discriminatorreal vs policy motion AMP variants only
AMASS locomotionsubset · AMP reference for track and field
PULSE prior + decoder32-d latent · SMPL-X: 48-d AMASS pre-trained
PHC motion trackerimitates the estimate → physically plausible
TRAMinternet video → SMPL pose + trajectory
PPO, AMP
latent $z$
action
proprioception $s^p$
fake
real
real
style reward
task reward
update
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The loop: a goal state per sport in, actions out, state backPPO trains every baseline; each sport has its own task rewardAMP adds a style reward; AMASS locomotion for track and fieldSports with video: TRAM estimates the motion, PHC tracks itPULSE variants act through a frozen latent decoderWhat it costs; curricula and self-play for the hard sports
Objective
  • High jump: $r^{\text{task}}_t = r^p_t + r^h_t$ within 0.5 m of the bar, else $r^p_t$; $\ r^h_t = p_{t,z}$ (height)
  • $r^p_t = \|p_{t-1} - p^g\|_2 - \|p_t - p^g\|_2$, clamped to $[0, 1]$ (also long jump, hurdling)
  • Tennis: $\exp(-\|p^{\text{racket}} - p^{\text{ball}}\|^2)$ until contact, then $1 + \exp(-\|p^{\text{land}} - p^{\text{tar}}\|^2)$
  • $r_t = w_{\text{task}}\,r^{\text{task}}_t + w_{\text{disc}}\,r^{\text{disc}}_t$; $\ (w_{\text{task}}, w_{\text{disc}}) = (1, 0)$ for PPO and PULSE, $(0.5, 0.5)$ with AMP
Training recipe
  1. Build each sport on one SMPL / SMPL-X humanoid: goal state, reward, termination
  2. Get demonstrations: video → TRAM → PHC; AMASS locomotion for track and field
  3. Train with PPO, AMP, PULSE (latent actions); add PULSE + AMP except in track and field
  4. Curricula on bar and hurdle height; boxing and fencing use alternating self-play
≈ 10⁹samples per baseline
1–3 dayson one RTX 3090
1.83 vs 0.91table-tennis hits, PHC-tracked vs raw
10+sports environments
The detail that makes it work

Video pose estimates are first tracked by PHC before they serve as AMP data: the physically refined motion doubles the average hits in table tennis (0.91 → 1.83).

Versus its predecessor

vs the tennis paper: a benchmark across sports instead of one system. It measures what PULSE’s prior buys over PPO and AMP.

BasicsImitatePriorsModelsScaleInteractReality
117
Long jump · PULSE
Javelin · PULSE
with a ball,
against an opponent?
Table tennis · 1v1, PULSE + AMP
Soccer · 2v2
against an opponent
that keeps learning?
Boxing · self-play
Fencing · self-play
SMPLOlympics

Track and field. PULSE and the task reward: human-like, far from records (long jump 5.1 m, javelin 9.6 m).

Ball games. 1v1 table tennis starts from the single-player policy; 2v2 soccer has only a rudimentary reward.

Boxing and fencing. Two policies learn by alternating self-play: one is frozen while the other trains.

Priors + simple rewards, from javelin to self-play boxing

Unlocked

A shared benchmark: the PULSE prior (+ AMP) turns simple sport rewards into human-like play, self-play included.

Still open

Rackets, clubs and swords replace the right hand. Picking up an arbitrary object is still unsolved.

BasicsImitatePriorsModelsScaleInteractReality
118
NeurIPS 2024

Omnigrasp: grasp diverse
objects, carry them anywhere

Full body, dexterous hands: writing its name with a held object
119
How it works

The motion prior is the action space

Object goalpath ahead + shape
Policy $\pi$
PULSE-X decoderfrozen
PD + simulator153 DoF + object
Stepwise rewardthree stages
Random 3D pathsno object MoCap
AMASS + handsdexterous AMASS
PULSE-Xa tracker, distilled
Pre-grasphand pose, one frame
state
latent $z$
PD targets
freeze
target
PPO update
153 PD targets

153 PD targets, every step

That is the action of a body with fingers. Plain RL explores by adding noise to every target. A little noise in the torso moves the hand a lot, and the hand knocks the object away.

Act in a motion latent instead

The policy outputs 48 numbers, a latent z. A frozen decoder, PULSE-X, turns z into all 153 targets. Noise on z still explores, but it comes out as coordinated, human-like motion.

Distilled from a tracker

91 % of AMASS sequences have no finger motion, so bodies are paired with recorded hand motion. PHC-X tracks it and is distilled into PULSE-X: random latents decode to human-like motion. Trained once, then frozen.

A goal about the object only

Its path over the next 20 frames, its shape, its offsets to the hands. No reference for the body, no grasp. So training paths can be random curves: no MoCap of people carrying objects.

One reward, three stages

A pre-grasp is the hand pose right before grasping: one frame, from MoCap or a grasp generator. It guides the hand through the reward only: the policy never sees it. PPO trains the policy on this reward.

Schematic: the same small noise on every joint
Trained without PULSE-X: unnatural motion
Schematic: the same small noise on every joint
Schematic: noise on the latent, decoded into a pose
Dexterous AMASS (inset), and PULSE-X imitating it
Random latents from the PULSE-X prior, decoded
Schematic: what the goal contains
Green dots: randomly generated paths (OakInk objects)
$r^{\text{approach}}$
before 1.5 s · hand over 0.2 m awayget closer to the pre-grasp
$r^{\text{pre-grasp}}$
before 1.5 s · hand within 0.2 mmatch its position and rotation
$r^{\text{obj}}$
from 1.5 s on · only while in contactkeep the object on its path
Schematic of one episode · violet hand: the pre-grasp · green dots: the object’s reference path
Luo, Cao, Christen, Winkler, Kitani, Xu. Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. NeurIPS 2024 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
120
Under the hood

Omnigrasp: RL in the latent space of a dexterous motion prior (PULSE-X)

Stage 2: grasp policy in the frozen latent space
Stage 1: PULSE-X motion prior, trained once
Trajectory generatorrandom speed and direction · no object MoCap needed
PPO$\gamma = 0.99$ · clip $0.2$
Stepwise reward $r_t$approach → pre-⁠grasp → object trajectory after 1.5 s
Pre-grasp hand poseMoCap or grasp generator · one frame, reward only
Object goal $s^g_t$20 future frames at 15 Hz + 512-⁠d BPS code, hand offsets
Policy $\pi_{\text{Omnigrasp}}$GRU 512 + 3-⁠layer MLP → 48-⁠d latent ${z_t}$
Frozen prior + decoder${a_t = \mathcal{D}(s^p_t,\ z_t + \mu^p_t)}$ ${\mu^p_t = \text{mean of } \mathcal{P}(z \mid s^p_t)}$
PD + Isaac Gym153-⁠DoF humanoid + object policy 30 Hz · sim 60 Hz
Object meshessampled $\propto$ failed lifts · random yaw + position
Dexterous AMASS14,889 AMASS sequences + GRAB / Re:InterHand hands
PHC-X imitatorRL tracker with fingers · 6-⁠layer MLP, 2048 → 512 wide
PULSE-X3-⁠layer MLP encoder, decoder + learned prior · 48-⁠d latent
reference path
residual $z_t$
PD targets
proprioception $s^p_t$
$s^p_t,\ s^g_t$
target
update
distill
freeze
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: the policy acts in the frozen latent spaceStage 1, once: track dexterous AMASS, distill into PULSE-XWhat trains it: a stepwise reward, maximised with PPOBefore 1.5 s a pre-grasp hand pose guides the handEach episode: a random object pose and a random 3D pathWhat it costs, and why the latent matters
Objective
  • $t < \lambda = 1.5$ s: $r^{\text{approach}}$ if the hand is $> 0.2$ m from the pre-grasp, else $r^{\text{pre-grasp}}$; after: $r^{\text{obj}}$
  • $r^{\text{obj}} = \mathbf{1}\{\text{C}\}\,(0.5\,e^{-100\,\|\hat{p}-p\|} + 0.3\,e^{\smash[t]{-100\,\|\hat{\theta}-\theta\|}} + 0.05\,e^{-5\,\|\hat{v}-v\|} + 0.05\,e^{-5\,\|\hat{\omega}-\omega\|} + 0.1)$
  • $r^{\text{pre-grasp}} = w_{hp}\exp(-100\,\|\hat{p}^{\text{pre-grasp}} - p^{\text{hand}}\|) + w_{hr}\exp(-100\,\|\hat{\theta}^{\text{pre-grasp}} - \theta^{\text{hand}}\|)$
  • $a_t = \mathcal{D}\big(s^p_t,\ z_t + \mu^p_t\big)$, $\ z_t \in \mathbb{R}^{48}$ · $\mathbf{1}\{\text{C}\}$: hand–object contact · $r^{\text{pre-grasp}}$: grasping hands only
Training recipe
  1. Pair AMASS bodies with recorded hand motion; train PHC-X with RL (≈ 10¹⁰ samples)
  2. Distill PHC-X online into PULSE-X (≈ 10⁹ samples); freeze its prior and decoder
  3. Each episode: random object yaw + position, random 3D path; stop if > 12 cm off it
  4. Train the GRU policy with PPO in the latent space; sample objects by failed lifts
≈ 10⁹samples · 3 days · one A100
48-dlatent action space
1,200+objects picked up
≈ 1 monthplain PPO, ≈ 10¹⁰ samples: only 55.9 %
The detail that makes it work

Raw torso noise travels down the arm and knocks the object away; acting in a human-motion latent keeps exploration coordinated.

Versus its predecessor

vs PULSE: adds fingers (PULSE-X) and shows the prior pays off on a contact task, with no paired body–object mocap.

BasicsImitatePriorsModelsScaleInteractReality
121
Different object, same policy
Unseen object: mug
Unseen object: binoculars
a thousand objects,
and no MoCap of grasps?
OakInk test objects
Large objects: a floor lamp
and what does
it cost?
1,200+objects picked upno paired body–object mocap
94 %trajectory successunseen GRAB objects
3 dayson one GPUplain PPO: one month, only 56 %
Omnigrasp

GRAB. Trained on random paths only, it follows MoCap paths, also with objects it has never seen.

OakInk. Pre-grasps from a grasp generator: 94 % grasp success, also on unseen objects. OMOMO. Large objects too.

The price. Three days for the policy, on top of a prior that is trained once. Without the prior: a month.

New objects, new paths, no paired MoCap

Unlocked

A motion latent (PULSE-X) makes dexterous RL tractable: search over human-like motion, not raw joint targets.

Still open

One hand-held object at a time. Whole-body contact with big objects needs interaction data — and that data is messy.

BasicsImitatePriorsModelsScaleInteractReality
122
CVPR 2025 highlight

InterMimic: whole-body
human–object interaction

One policy from hours of imperfect mocap · clip: a teacher
123
How it works

Perfect first, then scale up

Interaction mocapOMOMO: ≈ 10 h, 15 objects
Teachers $\pi^{(T)}$one per subject (17) · PPO
Frozen teachersrollouts = new references
Student $\pi^{(S)}$one Transformer for all
reference
contacts
freeze
reference
actions
Mocap reference: the hand floats, later sinks in
Schematic, front view: lifting a box off a table
the same frames,
in simulation?
InterMimic, same frames: the hand rests on the object
Student policy: one network, many objects
The data is flawed. In interaction mocap a hand often hovers beside the object, or sinks into it. A replay does not mind. A simulator does: a hand that does not touch holds nothing.
A teacher per subject tracks the body and the object, in physics, with PPO. There the object follows its reference only if a body part really pushes it: to earn the object term, the hands have to close the gap.
A contact term says which parts should touch. Mocap distances are unreliable, so the object’s acceleration tells when somebody is acting on it. Then: touch with the closest parts, keep off with those beyond 10 cm, neutral in between.
Freeze the teachers and roll them out. Their simulated motion replaces the mocap as the reference of the next stage: contacts repaired, every subject retargeted to one body.
One Transformer student for all skills. Its update is $w\,L_{\text{PPO}} + (1-w)\,\lVert a^{(S)} - a^{(T)} \rVert$ with $w$ rising from 0 to 1: copy the teachers first, then RL on the same reward terms. Copying alone averages teachers that disagree.
Success on
the held-out
subject
9.6 %PPO on raw mocap
91.6 %PPO on teacher rollouts
54.3 %copying only
95.5 %copy, then RL
98.1 %+ Transformer
one policy
for all skills?
Xu, Ling, Wang, Gui. InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions. CVPR 2025 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
124
Under the hood

InterMimic: Per-subject teachers repair the data; one transformer student scales

Stage 1: one teacher per subject (17 for OMOMO) · PPO in Isaac Gym
Stage 2: one student · distillation, then RL fine-tuning
Reward $R$$\exp(-\sum_i \lambda_i E_i)$ · body, object, contact, energy
PPOcritic MLP 1024-1024-512 · $\gamma = 0.99$
DAgger → PPO$w\,L_{\text{PPO}} + (1-w)\,\|a^{(S)} - a^{(T)}\|$
Reward $R$same terms, now against the teacher rollouts
Imperfect MoCapsplit by subject · OMOMO ≈ 10 h, 15 objects
Teacher $\pi^{(T)}$MLP 1024-1024-512 · sees reference frames $t{+}\{1, 16\}$
Frozen teachersone base body · rollouts = refined references
Student $\pi^{(S)}$3-layer Transformer, 4 heads, 256-d · frames $t{+}\{1, 2, 4, 16\}$
PD + Isaac Gymhuman + object · policy 30 Hz, sim 60 Hz
PSI bufferMoCap states + high-return simulated states
Kinematic generatorsHOI-Diff (text) · InterDiff (prediction)
goal $s^g$
reference
update
start states
freeze
reference
teacher action $a^{(T)}$
PD targets
state $s^s$
update
training data
reference at test time
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Stage 1: each teacher tracks one subject’s mocapReward on body, object and contact; PPO updates the teacherRollouts start from mocap or from good simulated statesFreeze: teacher rollouts are the student’s referenceCopy the teacher’s action first, then shift to PPOAt test time a kinematic generator can supply the reference
Objective
  • $R = \exp\big(-30\,E^h_p - 2.5\,E^h_\theta - 5\,E_d - 0.1\,E^o_p - 5\,E^o_\theta - 5\,E^c_b - 5\,E^c_p - 3\,E^c_h - \text{energy}\big)$
  • $E^h_p = \langle\Delta^h_p, w_d\rangle$, $E^h_\theta = \langle\Delta^h_\theta, 1 - w_d\rangle$, $E_d = \langle\Delta_d, w_d\rangle$; $w_d$: normalized $1/\|d\|^2$ to the object
  • Contact: $E^c_b = \sum\|\hat c_b - c\|\odot\hat c_b$ (promote), $E^c_p = \sum\|c\|\odot\hat c_p$ (penalize), else neutral
  • Student: $\nabla_\psi\big(w\,L_{\text{PPO}}(\psi) + (1-w)\,J(\psi)\big)$, $J = \|a^{(S)} - a^{(T)}\|$, $w = \min(t/\beta,\,1)$
Training recipe
  1. Infer reference contacts from object acceleration; train one PPO teacher per subject
  2. Start from the PSI buffer; stop on 0.5 m object drift or 10 frames of lost contact
  3. Freeze the teachers: their rollouts become the references, their actions the labels
  4. Distill with DAgger, then shift to PPO as w grows; train the critic on R throughout
8,192environments
9.6 → 91.6 %held-out success, refined references
98.1 %full method, OMOMO test
17teacher policies
The detail that makes it work

The student imitates the teachers’ physically valid rollouts, not the raw mocap: that change alone lifts held-out success from 9.6 % to 91.6 %.

Versus its predecessor

vs PHC-style tracking: adds objects and contact. vs SkillMimic: handles imperfect mocap at scale through teacher–student distillation.

BasicsImitatePriorsModelsScaleInteractReality
125
Results

Beyond the mocap it was trained on

“Pull the tripod.” HOI-Diff alone, then with InterMimic

A generator writes the reference

A kinematic generator supplies the motion. Success: 76.7 % on InterDiff’s predictions, 12.5 % on HOI-Diff’s text samples.

a sequence
nobody recorded?
Kick, lift, relocate: mocap sequences chained

A sequence nobody recorded

Mocap sequences concatenated into one reference: kicking, lifting and relocating boxes. The chain is not in the training data.

a robot body?
Simulated Unitree G1 with Inspire hands

A robot body, no retargeting step

The same framework trains a simulated Unitree G1 on human references. No external retargeting: RL absorbs the change of body.

Unlocked

Simulation as a data cleaner: teachers repair the mocap, the student generalizes; a simulated G1 learns the skills too.

Still open

Everything so far lives on a screen. Does the recipe survive real hardware?

Clips: InterMimic project page (sirui-xu.github.io/InterMimic)
BasicsImitatePriorsModelsScaleInteractReality
126
Chapter 6  ·  2024 – 2026

Reality

The character leaves the screen.
OmniH2O CoRL 2024
SONIC Science Robotics 2026
ScaleBFM arXiv 2026
SuperMimic CoRL 2026
127
H2O (IROS 2024): teleoperation from one RGB camera
OmniH2O: VR teleoperation
Squat, pick, place (2×)
Whole-body swing
Dexterous hands (2×)
Outdoor locomotion
CoRL 2024 · He, Luo, He, Xiao, Zhang, Zhang, Kitani, Liu, Shi

OmniH2O: the tracker crosses to a real humanoid

A person moves, a full-size humanoid follows: whole-body teleoperation, first from a camera, then from a VR headset.The same policy swings a racket, stands still while dexterous hands pick up toys, and walks outdoors.
BasicsImitatePriorsModelsScaleInteractReality
128
How it works

Same recipe, plus what hardware demands

Retargeted AMASS+ standing and squatting variants
Teacher $\pi_{\text{privileged}}$RL in simulation, sees the full state
Student $\pi_{\text{OmniH2O}}$sees what the robot can measure
Unitree H1the policy runs at 50 Hz
reference
labels
PD targets

Mocap for a robot body. AMASS is retargeted to the H1 and motions the robot cannot do are dropped. Every clip also gets a variant with root and legs fixed, standing or squatting: a robot that manipulates must be able to stand still.

A teacher that sees everything. PPO in simulation, rewarded for tracking the clips. It is given what only a simulator knows: position, rotation and velocity of every body, and the full-body reference.

A student with the robot’s senses. Joint angles and velocities, root angular velocity, gravity direction, its last actions, and three targets: head and hands. No global linear velocity: 25 steps of history stand in for it.

It learns by copying, not by RL (DAgger). The student drives the simulation; in every state it reaches, the teacher is asked for its action. The loss is the gap to the student’s own: $\lVert a^{\text{privileged}}_t - a_t \rVert_2^2$.

Then onto the robot. Training randomizes friction (0.2–1.1), link masses (±30 %), PD gains (±25 %) and control delay (20–60 ms), and pushes the robot every 5 s. The same network then drives the H1 at 50 Hz, and stays up when it is struck.

Does copying cost anything? In simulation the student almost matches its teacher, while RL on the same 25-step input tracks under half of the sequences. On the real robot the history pays off: without it the position error grows by three quarters.

Schematic · green: the reference motion · white: the simulated robot
and outside
the simulator?
Kicked and pushed
94.1 %
of 14k sequences trackedin simulation · the teacher: 94.8 %
47.1 %
same input, trained by RLno teacher, no DAgger
47.9
mm position error, real H1without history: 83.3 · H2O: 87.3
He, Luo, He, et al. OmniH2O. CoRL 2024  ·  He, Luo, Xiao, et al. H2O. IROS 2024  ·  clips: project pages  ·  numbers: Tables 1 and 2
BasicsImitatePriorsModelsScaleInteractReality
129
Under the hood

OmniH2O: A privileged teacher, then a sensor-only student with history

Stage 2: DAgger distillation of the student
Stage 1: privileged teacher, RL
Retargeted AMASS+ standing / squatting variants · 14k sequences
DAgger action loss$\|a^{\text{privileged}}_t - a_t\|_2^2$
Frozen teacherqueried on the student's rollouts
Teacher $\pi_{\text{privileged}}$913-d in: all rigid bodies + full-body reference
3-point goalhead + two hands · 27-d ${(\tilde p - p,\ \tilde v - v,\ \tilde p)}$
Student $\pi_{\text{OmniH2O}}$MLP 512-256-128 · 1665-d in no global linear velocity
PD + simulatorflat or rough terrain, low obstacles
PPOtracking + regularization + penalty rewards
Any pose sourceVR, RGB, MDM text, GPT-4o, diffusion policy
Real Unitree H1sim-to-real · policy at 50 Hz dexterous hands driven by IK
Domain randomizationfriction, mass, PD gains, control delay · pushes
PD targets
25-step proprioception history
privileged state
$a^{\text{privileged}}_t$
update
stage 1 rewards
update
freeze
training
test time
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The student’s control loop: three target points in, PD targets outStage 1: PPO trains a privileged teacher in the randomized simulatorStage 2: the frozen teacher labels the student’s own rolloutsTest time: any pose source sets the goal, the student drives the H1What it costs, and what makes it work
Objective
  • Stage 1 (PPO, $\gamma = 0.99$): $\max\ \mathbb{E}\big[\sum_{t=1}^{T} \gamma^{t-1} \sum_i s_{t,i}\,r_{t,i}\big]$, $\ s_{t,i} = s_{\text{current}}$ if $r_{t,i} < 0$, else $1$
  • $s_{\text{current}}$: starts at $0.5$, $\times\,0.9999$ if mean episode length $< 40$, $\times\,1.0001$ if $> 120$, capped at $1$
  • Rewards: $50\,\exp(-0.5\,\|p^{\text{real}} - \hat p^{\text{real}}\|_2^2)$ (3 VR points), $\ 1000\,\max\{h^{\text{feet}}_{\max} - 0.25,\ 0\}$ (per step), …
  • Stage 2 (DAgger): $\mathcal{L} = \|a^{\text{privileged}}_t - a_t\|_2^2$, teacher queried on the student's own states
Training recipe
  1. Retarget AMASS to the H1, drop infeasible motions, add standing / squatting variants
  2. Stage 1: train the privileged teacher with PPO under domain randomization and pushes
  3. Stage 2 (DAgger): roll out the student; the frozen teacher labels every visited state
  4. Deploy on the H1 at 50 Hz; teleoperated demos train a diffusion policy: image → 3-point goals
14ksequences: retargeted AMASS + standing / squatting
19joints driven by the policy
20–60 msrandomized control delay
1,665-dstudent input including history
The detail that makes it work

Biasing the data toward standing and squatting stops the constant small adjustment steps of H2O, so the robot can stand still and manipulate.

Versus its predecessor

vs H2O: teacher–student DAgger instead of direct RL, a 25-step history instead of global linear velocity (no MoCap at test time), 3 VR points instead of 8 keypoints, plus dexterous hands.

BasicsImitatePriorsModelsScaleInteractReality
130
Why it matters

A kinematic pose is the universal interface

Any source of posesa person, a sentence, an agent
Motion goal3 points: head and two hands
Student $\pi_{\text{OmniH2O}}$the same network every time
Unitree H1whole-body motion
PD targets
VR teleoperation: table-tennis strokes

A person

A VR headset or an RGB camera follows the operator: head and hand positions, live.

Spoken instruction → text-to-motion → robot

A sentence

MDM, a text-to-motion model, turns the words into a motion; the policy tracks it.

Autonomy: GPT-4o chooses the motion

An agent

GPT-4o looks through the head camera and picks a motion primitive. A diffusion policy trained on teleoperated demos works too.

Unlocked

The animation stack is a robot stack: mocap tracking plus teacher–student distillation runs on a full-size humanoid.

Still open

Motions stay cautious and the networks small. What if tracking is scaled like a foundation model?

BasicsImitatePriorsModelsScaleInteractReality
131
Bridge · 2025

Real humanoids catch up with the simulated ones

1Track
BeyondMimic · a cartwheel, then sprints
1One tracking pipeline, the same MDP and hyper-parameters for every motion.
2Distill, then steer
BeyondMimic · the diffusion policy, steered by guidance
2The trackers are distilled into one diffusion policy. At test time a simple cost function steers it.
Also in
2024 – 2025
ExBody2024
HumanPlus2024
HOVER2025
ASAP2025
TWIST2025
KungfuBot2025
GMT2025
UniTracker2025
On screen · Chapter 4PHC2023a tracker
PULSE2024MaskedMimic2024distilled trackers
The pattern from Chapter 4 repeats on hardware: first track, then distill the tracker into something steerable.
Clips: Liao, Truong, Huang, Gao, Tevet, Sreenath, Liu. BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion. 2025 (beyondmimic.github.io)  ·  others: project pages
BasicsImitatePriorsModelsScaleInteractReality
132
Bridge · 2026

The interaction chapter crosses over too

On screen · 2023
Chapter 5 · a simulated rally
On a robot · 2026
LATENT · rally with a human
Chapter 5: tennis skills from broadcast video, in simulation.
LATENT: a real G1 sustains multi-shot rallies with human players, learned from imperfect motion fragments.
On a robot · 2026
HumanX · basketball
HumanX · catch and pass
HumanX · pick up cargo
HumanX: interaction skills compiled from human videos, without task-specific rewards.
Chapter 5 repeats on hardware: video and imperfect motion as data, physics as the filter, interaction as the goal.
Clips: Zhang, Yuan, et al. Tennis Skills from Broadcast Videos, SIGGRAPH 2023  ·  Zhang et al. LATENT, 2026  ·  Wang et al. HumanX, 2026  ·  project pages
BasicsImitatePriorsModelsScaleInteractReality
133
Science Robotics 2026 · arXiv Nov 2025

SONIC: supersizing
motion tracking

One policy · a real Unitree G1
Tracking as the foundation task
134
The bet

Scale three axes, keep the task simple

Tracking, live: the robot follows the person’s motion
Supervisiontrack mocapas DeepMimic did
Interfacetokensnext slide
Scale700 hoursthis slide
Datamotion capture
100M+frames · 611 h at 50 Hz
Modelnetwork size
42Mparameters · up from 1.2M
ComputeGPUs in parallel
128 GPUs7 days ≈ 21k GPU hours
Unseen motionsabsent from training
99.6 %success · 98.0 % at 1.2M

Keep the task simple. It is DeepMimic’s: follow a reference motion, frame by frame. Every mocap frame is a target, so nobody writes a reward per skill. SONIC’s bet: change nothing about the task, and scale everything around it.

Data. About 700 hours of motion capture are retargeted to the G1. Motions the robot cannot do, such as stairs and sitting, are dropped: 611 hours remain, 100M+ frames in 317k clips. Subsets of 4M, 10M and 22M frames are trained for comparison.

Model. Three network sizes are compared: 1.2M, 16M and 42M parameters. The largest one is what runs on the real robot.

Compute. 16, then 32, then 128 GPUs, every run to 50k iterations, with 4,096 simulated robots on each GPU. The largest run takes 7 days: about 21k GPU hours.

Scale pays on every axis, and most on 182 kinds of motion that are absent from training: from the smallest to the largest model, success there rises from 98.0 % to 99.6 % and the error falls. On the real G1, 123 of 124 test motions succeed.

Luo, Yuan, Wang, Li, Castañeda, … Fan, Zhu. SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control. Science Robotics 11(117), 2026 · clips: authors’ project page
BasicsImitatePriorsModelsScaleInteractReality
135
How it works

Every motion command becomes the same token

Human motion $g_h$3D joints of a person
Robot motion $g_r$joint angles, velocities
Hybrid motion $g_m$head + hands, G1 legs
Encoder $\mathcal{E}_h$
Encoder $\mathcal{E}_r$
Encoder $\mathcal{E}_m$
Universal token $z$FSQ: 2 × 32 numbers
Control decoder $\mathcal{D}_c$→ 29 PD targets
PD + Unitree G1control at 50 Hz
Motion decoder $\mathcal{D}_r$→ robot motion $\hat g_r$
Auxiliary lossesrecon + token + cycle
PPOone stage, end to end
Tracking rewarddense, every frame
proprioception $s^p$
token only
$\hat g_r$
update $\mathcal{D}_c, \mathcal{E}$

The command becomes a token

What should the robot do next? The mocap clip, retargeted to the G1, says it: the next 10 frames.

An encoder squeezes them into 2 × 32 numbers, and every number is rounded to one of 32 levels. That is all the controller is told about the motion.

One decoder turns it into action

The control decoder reads the token and the robot’s own sensing: joint angles and velocities, gyro, gravity direction, the last 10 steps of each.

It sets 29 PD targets, 50 times a second.

Real G1: crawling on elbows and knees
Real G1: hit by a swinging load, it stays on its feet

Two more encoders

The same motion can arrive in other formats: the 3D joints of a human body, or only head and hands on top of the robot’s leg motion.

Each format gets its own encoder. All three write into the same token.

Three losses make them agree

Every clip exists in all three formats. Its three tokens are pulled together, and each must decode back to the robot motion.

So human encoder + motion decoder learn to retarget. Re-encoded, their output must give the robot’s token.

Trained in one stage

PPO on the tracking reward and the three auxiliary losses are optimized together. Gradients pass through the rounding.

No teacher, no distillation. Only the critic sees privileged simulator state.

$\mathcal{L} =$
$\mathcal{L}_{\text{ppo}}$the tracking reward
+
$\mathcal{L}_{\text{recon}}$token → robot motion
+
$\mathcal{L}_{\text{token}}$same motion, same token
+
$\mathcal{L}_{\text{cycle}}$human → robot → same token
BasicsImitatePriorsModelsScaleInteractReality
136
Under the hood

SONIC: Three encoders, one FSQ token space, one control decoder

Deployment: same policy, no retraining
Auxiliary losses$\mathcal{L}_{\text{recon}} + \mathcal{L}_{\text{token}} + \mathcal{L}_{\text{cycle}}$
Motion decoder $\mathcal{D}_r$MLP 2048-⁠1024-⁠512-⁠512 → robot motion $\hat g_r$
PPOprivileged critic, same widths as $\mathcal{D}_c$ · ${\gamma = 0.99}$
Tracking reward $r_t$$\mathcal{R}(s^p, s^g) + \mathcal{P}(s^p, a)$ 7 tracking terms, 5 penalties
Mocap datasetretargeted to G1 · per clip: robot, human, hybrid motion
Encoders $\mathcal{E}_r, \mathcal{E}_h, \mathcal{E}_m$MLP 2048-⁠1024-⁠512-⁠512 each input: 10 future frames
Universal token $z$FSQ, straight-⁠through: 2 tokens × 32-⁠d, 32 levels
Control decoder $\mathcal{D}_c$MLP 4096-⁠4096-⁠2048-⁠2048-⁠ 1024-⁠1024-⁠512-⁠512 → 29-⁠d
PD + Isaac LabUnitree G1 · 29 joints control at 50 Hz
Motion generatorsplanner → $\mathcal{E}_r$ · GEM, VR → $\mathcal{E}_h$ 3-⁠point VR + planner → $\mathcal{E}_m$
GR00T N1.5 VLAfine-⁠tuned on teleop data → 64-⁠d token + 14-⁠d hands
$g_r, g_h, g_m$
quantize
PD targets
10-step proprioception $s^p$
update $\mathcal{D}_c, \mathcal{E}$
token only
$\hat g_r$
target $g_r$
update $\mathcal{E}, \mathcal{D}_r$
motion command
skips the encoders
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The tracking loop: clip → token → PD targets, state backPPO on a dense tracking reward trains the whole pathThree auxiliary losses align the three encodersDeployment: feed an encoder, or write the token directlyWhat it costs, and why FSQ
Objective
  • $r_t = \mathcal{R}(s^p_t, s^g_t) + \mathcal{P}(s^p_t, a_t)$, e.g. $2.0\,\exp\!\big(-\tfrac{1}{5}\sum_{k \in \mathcal{K}} \|p^p_{t,k} - p^g_{t,k}\|^2 / 0.1^2\big)$ on head, wrists, ankles
  • $\mathcal{L}_{\text{ppo}}$ updates $\mathcal{E}$, FSQ, $\mathcal{D}_c$ and the critic; only the critic sees privileged simulator state
  • $\mathcal{L}_{\text{recon}} = \sum_{k \in \{r,h,m\}} \|\mathcal{D}_r(z_k) - g_r\|^2$, $\quad\mathcal{L}_{\text{token}} = \|z_r - z_h\|^2 + \|z_r - z_m\|^2 + \|z_m - z_h\|^2$
  • $\mathcal{L}_{\text{cycle}} = \|\mathcal{E}_r(\mathcal{D}_r(z_h)) - z_r\|^2$; these three update $\mathcal{E}$, $\mathcal{D}_r$ · $\mathcal{L} = \mathcal{L}_{\text{ppo}} + \mathcal{L}_{\text{recon}} + \mathcal{L}_{\text{token}} + \mathcal{L}_{\text{cycle}}$
Training recipe
  1. Retarget ≈ 700 h of mocap to the G1, keep 611 h; prepare each clip in all three command formats
  2. Sample 1 s bins by capped failure rate; randomize physics, pushes and the target commands
  3. Encode, quantize and decode all three; optimize the four losses jointly in a single stage
  4. Deploy the 42 M model on the G1: generators feed an encoder, or a VLA predicts the token
611 h100 M+ frames · 317k clips
128 GPUs7 days ≈ 21k GPU hours
4,096environments per GPU
42 Mparameters
The detail that makes it work

No codebook collapse, commitment loss or codebook EMA: FSQ’s straight-through gradients coexist with PPO; token dimension matters more than levels.

Versus its predecessor

vs OmniH2O: single-stage training, far more data, and one token interface for human, robot and teleoperation inputs.

BasicsImitatePriorsModelsScaleInteractReality
137
SONIC

One policy, every interface

Universal token zthe same 2 × 32 numbers, whoever writes them
control decoderUnitree G1Whatever can write the token can drive the robot. The policy is never retrained.
Gamepad + planner$\mathcal{E}_r$

A gamepad

A planner turns speed, heading and style into the next 0.8 – 2.4 s of robot motion, up to 10 times a second.

Gamepad planner: boxing
Video · text · VR$\mathcal{E}_h$

A camera, a prompt, trackers

GEM turns video, text or music into human motion; whole-body VR streams it. No retargeting at run time.

human motion?
VR whole-body teleoperation
3-point VR + planner$\mathcal{E}_m$

A headset, two controllers

Head and wrists come from the operator, the legs from the planner.

only three points?
Three-point VR: mowing the lawn
GR00T N1.5 VLAno encoder

A vision-language-action model

It predicts the token itself: 75 % average success on five tasks. Predicting body poses instead, on three of them: 27 % against 68 %.

no encoder at all?
VLA: drill into the box, box to the shelf (2×)

Unlocked

Tracking scales like a foundation model, and one token space serves teleoperation, planners and VLAs.

Still open

Data, model, compute: scaling each one paid. Which have to be scaled together, and how?

Clips: SONIC project page (nvlabs.github.io/GEAR-SONIC) · success rates: Table 1 of the paper
BasicsImitatePriorsModelsScaleInteractReality
138
arXiv 2026
Scaling a behavior foundation model

ScaleBFM: what has to be scaled together?

One pre-trained controller on a real Unitree G1. The paper studies how such a controller scales.
Dexterous manipulation: a rose goes into the vase
Agile locomotion: a cartwheel, right after the person’s
Loco-manipulation: laundry from the chair to the hamper
SONIC grew the motion data, the model and the compute, and each of them paid. Which parts have to be scaled together, and how?

1Learning paradigm

Track the reference as one whole-body trajectory in the global frame; command it through masked link targets.

2Data

In PPO it is two things: the quantity of on-policy rollouts and the diversity of the reference motions.

3Architecture

The Humanoid Transformer: the command becomes a latent on a sphere, shaped by tracking alone.

Zeng, Yin, Niu, Lu, Zhong, … Lin, Pang, Wang. Scaling Behavior Foundation Model for Humanoid Robots. arXiv:2607.15163, 2026 · clips: project page, real robot
BasicsImitatePriorsModelsScaleInteractReality
139
How it works

Three parts of one tracking loop, scaled together

any source of
link poses
Reference motionseight open datasets
Command $s^g_t$target poses of links
Goal latentnormalized to a sphere
Policy $\pi_\theta$Humanoid Transformer
PD + Unitree G1simulated in IsaacLab
Rolloutswhat PPO learns from
Mask $m$8 control modes
PPOactor and critic
Tracking rewardin the global frame
proprioception
update
on-policy data
1 · Paradigm: the reward. Still tracking, still PPO. The reward compares the whole body with the reference in the global frame: root and pose are one trajectory, so stepping on the spot cannot pass for walking forward.
1 · Paradigm: the command. It says where body links should be: target positions and rotations in Cartesian space, seen from the robot’s root; no joint angles. Whatever yields link poses can write it: motion capture, a planner, a teleoperator.
1 · Paradigm: the mask. A link-wise mask picks the commanded links: eight control modes, from the pelvis alone to 14 links. The policy fills in the rest of the body: MaskedMimic’s partial goals, now on a real robot.
2 · Data: quantity. PPO learns from on-policy rollouts, not from motion-capture frames. Their amount is GPUs × 8,192 robots × rollout horizon. More GPUs and a longer horizon together help; either one alone, not consistently.
2 · Data: diversity. Reference motions decide what the rollouts contain. Three times more of one source leaves coverage flat; other sources widen it, and help on motions from sources the model never trained on.
3 · Architecture. A Transformer reads the robot’s recent states and actions; the command enters by cross-attention, normalized onto a sphere. No extra loss shapes that latent, yet example motions trace smooth, separate paths on it.
Target, seen from the root $p^{\mathrm{ref}}_{t+1} - p^{\mathrm{root}}_{t}$
Offset from the link’s own pose $p^{\mathrm{ref}}_{t+1} - p_{t}$
Rotations likewise $\theta^{\mathrm{ref}}_{t+1} \ominus \theta^{\mathrm{root}}_{t},\ \ \theta^{\mathrm{ref}}_{t+1} \ominus \theta_{t}$

Up to 14 body links: pelvis, torso, shoulders, elbows, hands, hips, knees, feet.

Paper: “masked whole-body target poses in the root-relative Cartesian space”

Circle walk under Root Mode
Natural walking under Bimanual Mode
Slow run under End-Effector Mode
Still the whole body. A link nobody provides is simply masked. In training the reference stays whole: any link 0.5 m off ends the episode, and the critic sees the unmasked target.
Clips: project page (scalebfm.github.io), real Unitree G1, titled there by control mode · stick figures and the sphere are schematics · all numbers: the paper
BasicsImitatePriorsModelsScaleInteractReality
140
Under the hood

ScaleBFM: Masked link targets, a Transformer, a global tracking reward

Critic $V_\phi$same architecture · privileged state · unmasked target
PPO${\gamma = 0.99}$ · GAE ${\lambda = 0.95}$ clip 0.2 · desired KL 0.01
Tracking reward $r_t$global frame: 5 tracking terms, 2 penalties, survival
Reference motions102M frames at 50 FPS 8 open datasets → G1
Goal state $s^g_t$${(x_t \odot m_i,\ m_i)}$ · 6 future frames: 0–4, one of 5–32
Humanoid Transformeractor $\pi_\theta$ · 3.00M parameters 4 layers · width 256 · 4 heads
PD + IsaacLabUnitree G1 · 29 DoF 8192 environments per GPU
On-policy rollouts32 / 48 / 64 GPUs × 32 / 48 / 64 steps
Mask $m_i$8 control modes · 1 to 14 links · drawn at each reset
Real Unitree G1TensorRT 50 Hz · PD 200 Hz global or local mode
Domain randomizationfriction, joint offsets, base CoM, hand mass · pushes
targets $x_t$
goal tokens
PD targets
history: states and actions
transitions
update
GAE
value loss
full target
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The tracking loop: link targets in, joint targets outThe command: link poses seen from the root, then maskedPPO on a global reward; the critic sees the full targetThe training data: rollouts, and which motions fill themOn the robot: 50 Hz, global or local mode; what it costs
Objective
  • Target: $x_t = \big(p^{\text{ref}}_{t+1} - p^{\text{root}}_t,\ \ p^{\text{ref}}_{t+1} - p_t,\ \ \theta^{\text{ref}}_{t+1} \ominus \theta^{\text{root}}_t,\ \ \theta^{\text{ref}}_{t+1} \ominus \theta_t\big)$: link poses in the root frame
  • Goal state: $s^g_t = (x_t \odot m_i,\ m_i)$, $\ m_i \sim \mathcal{M} = \{m_0, \dots, m_7\}$: the masked target, and the mask itself
  • PPO: $\max_\theta\ \mathbb{E}_{\tau \sim p_\theta(\tau)}\big[\sum_{t=0}^{T} \gamma^t r_t\big]$, $\ r_t$ = 5 global tracking terms + 2 penalties + survival
  • e.g. body position: $\exp\!\big(-\tfrac{1}{|\mathcal{B}|}\sum_{b \in \mathcal{B}} \|p_{t,b} - p^{\text{ref}}_{t,b}\|^2 / 0.3^2\big)$, weight 1.0; likewise rotation and velocities
Training recipe
  1. Each environment reset draws one of the 8 control modes: one reference, many forms of command
  2. End an episode when any link is 0.5 m off in the global frame; restart on the reference there
  3. Collect 8192 environments × 32–64 GPUs × 32–64 steps per update; PPO for actor and critic
  4. Every 200 epochs, test every motion: failed ones are sampled more, weights kept in [0.03, 1]
102Mframes at 50 FPS · eight open datasets
64 GPUsRTX 4090 · 8192 envs each · 64-step rollouts
22,200training epochs · training time not stated
3.00Mparameters in the benchmarked model
The detail that makes it work

Under every mask the whole reference stays the target: the critic sees it unmasked, and any link 0.5 m off in the global frame ends the episode.

Versus its predecessor

vs SONIC: one masked Cartesian command instead of three encoders and a token; root and pose tracked as one global trajectory. vs BFM4Humanoid: PPO tracking, not DAgger.

BasicsImitatePriorsModelsScaleInteractReality
141
Results

Stronger in the global frame; bigger is not always better

Simulation: MuJoCo, the paper’s ‘Ours’ test set. A motion fails when a commanded link is 0.5 m off in the global frame.
Real G1 (project page): robot and person walk side by side
0.5937 → 0.9776

success rate, off-the-shelf SONIC → ScaleBFM: whole-body commands, global mode. Position error: 0.5035 → 0.0915 m. Local mode, local error: 0.0430 → 0.0388 m.

Fruit collection under Root-and-End-Effector Mode
5 of 14 links

commanded: pelvis, hands, feet. Success 0.9794; all 14 links: 0.9776; pelvis and hands only: 0.8878. Each mode is scored on its commanded links.

Paper, Fig. 9 and 27: ‘Ours’ test set, MuJoCo
3.0M parameters

in the benchmarked model; an MLP of that size reaches 0.9563. Beyond 4.4M, whole-body success dips, and the robot must infer at 50 Hz.

Unlocked

A recipe for scale: track in the global frame, grow rollouts with motion diversity. One model, full or sparse commands.

Still open

Best run: 64 GPUs, the largest setting tried; distributed training is fragile. Does scale have to cost this much?

Zeng, Yin, Niu, Lu, Zhong, … Lin, Pang, Wang. Scaling Behavior Foundation Model for Humanoid Robots. arXiv:2607.15163, 2026 · numbers: Table 3, Fig. 7, 9, 19, 21, 27 · clips: project page
BasicsImitatePriorsModelsScaleInteractReality
142
CoRL 2026 oral

SuperMimic: one
differentiable world model

Trained in imagination, zero-shot on a Unitree G1.
“Compute is no longer the bottleneck.”
143
How it works

SuperTrack’s idea, SONIC-level tracking

Motion library129,514 clips · 264 h
Reference $g_t$the next 10 frames
Policy $\pi_\theta$joint setpoints $a_t$
PD + MuJoCo Warpno gradient needed
Replay buffernewest replace oldest
Critic $V_\psi$return after step 32
Imagined return $\hat J$32 predicted rewards
World model $f_\omega$one MLP, every motion
$a_t$
observation $o_t$
fit on 8-step windows
$a_t$
$\hat o_t$
$\hat r_t$
$\nabla_\theta \hat J$
Collect. 4096 simulated robots per GPU track motions from the library. Every step is stored in a buffer; the newest replace the oldest.
Fit one world model. fω predicts the next state from the state and the action. All motions share one body, so one network serves them all.
Imagine. From a stored state the policy acts on the model’s predictions for 32 steps. The model predicts every step’s tracking reward as well.
Differentiate. Back-propagate the return through all 32 steps into the policy. PPO in this model needs about 64× more imagined steps for its best return.
Add a critic. Vψ estimates the return after step 32: the model has to be right for 32 steps only, and rewards beyond the window still count.
The task: a flip, then a roll on the floor (authors’ render)
Policy switched off: the model (blue) predicts the fall
Policy on: it acts on the model’s predictions (blue)
Authors’ film: only the model passes a gradient back
With the critic
Without it
A window
of 16 steps
as good as 32 steps without it
training fails
Pushed at
0.5 to 4 m/s
settles within 0.5 to 2.4 s
never settles; falls at 4 m/s
Paper, section 4.3: in simulation; pushes while holding a T-pose
Clips: project page and film (Liao et al., CoRL 2026); white robot = simulator, blue = world model · the frame strip is a schematic, not data
BasicsImitatePriorsModelsScaleInteractReality
144
Under the hood

SuperMimic: One learned dynamics model for every motion; policy by BPTT

Downstream interface
Critic $V_\psi(\hat s_t, g_t, \xi)$MLP 256-128 · one critic per goal interface
Imagined return $\hat J(\theta)$$H = 32$ steps in $f_\omega$ · $\gamma = 0.99$
World model $f_\omega(\hat s_t, a_t, \xi)$MLP 3 × 2048; reward, obs, termination heads
Replay buffer $\mathcal{B}$FIFO · 1000 steps per environment
Reference motionBONES-SEED 264.6 h · $K = 10$ future frames
Motion encoder $\mathrm{enc}_\theta$MLP 2048-1024-512-256 → 32-d tanh $z$
Shared decoder $\mathrm{dec}_\theta$MLP 2048-1024-512-256 · last 10 observations
PD + MuJoCo Warp4096 envs per GPU · 50 Hz control
Velocity command$(v_x, v_y, \omega_z)$ + a feature of the last 10 obs
Velocity encoderMLP 256-256 · velocity reward through $f_\omega$
Sim context $\xi$random friction, joint offsets, torso CoM
goal $g_t$
latent $z$
$a_t + \epsilon_t$
observation $o_t$
per episode
transitions
$\mathcal{L}_{WM}$ on 8-step windows
start states $(s_0, \xi)$
$a_t$
$\hat{\bar o}_t + \eta_t$
$\hat r, \hat d$
bootstrap
GAE $\hat R_t$
$\nabla_\theta \hat J$: BPTT through $f_\omega$
latent $z$
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: reference → latent z → joint setpointsCollect: every episode has its own physics ξ; a FIFO bufferFit the world model: 8 logged steps, match the simulatorImagine 32 steps in the model; back-propagate the returnA critic adds the return beyond step 32A velocity interface through the same model; what it costs
Objective
  • World model: $\mathcal{L}_{WM} = \mathbb{E}_{\mathcal{B}} \sum_{j=1}^{k} \big[\|\hat s_j - s_j\|^2 + (\hat r_j - r_j)^2 + \|\hat{\bar o}_j - \bar o_j\|^2 + \mathrm{BCE}(\hat d_j, d_j)\big]$, $\ k = 8$
  • Policy: $\hat J(\theta) = \mathbb{E}\big[\sum_{t=0}^{H-1} \gamma^t\,\hat c_t\,\hat r_{t+1} + \gamma^H \hat c_H\,V_\psi(\hat s_H, g_H, \xi)\big]$, $\ H = 32$, $\gamma = 0.99$
  • Reward: BeyondMimic tracking reward, no self-collision penalty; survival $\hat c_{t+1} = \hat c_t\,(1 - \hat d_{t+1})$
  • Critic: $\mathcal{L}_V = \mathbb{E}\sum_t (V_t - \hat R_t)^2$, $\ \hat R_t = \hat r_{t+1} + \gamma\,(1 - \hat d_{t+1})\big[(1-\lambda)\,V_{t+1} + \lambda \hat R_{t+1}\big]$, $\lambda = 0.95$
Training recipe
  1. Collect: roll out $\pi_\theta$ with action noise in 4096 envs; resets favor low-return 1 s motion bins
  2. Fit $f_\omega$: 10 batches of 2048 logged 8-step windows, unrolled autoregressively on logged actions
  3. Improve $\pi_\theta$ by BPTT over 4096 imagined 32-step rollouts (sensor noise re-added, $f_\omega$ fixed)
  4. Fit the critic to GAE targets of the same rollouts; loop for 320k iterations in a full run
3 h · 1 GPUto pass SONIC’s return (0.90)
> 1000×fewer GPU hours than ≈ 21k
264.6 hof motion (BONES-SEED)
25 → 0failed motions: 3 h, 1 GPU → 9.8 h, 2 GPUs
The detail that makes it work

No push perturbations in training: they create discontinuities a deterministic model cannot predict. Randomizing physics alone transfers.

Versus its predecessor

vs SONIC: the same tracking task with learned-model gradients instead of PPO. vs SuperTrack: adds a critic bootstrap and sim-context conditioning; runs on hardware.

BasicsImitatePriorsModelsScaleInteractReality
145
Results

Past SONIC’s return in three hours on one GPU

Both policies are scored on the same 129,514 motions, each played once from its first frame.
Authors’ film: 128 GPUs for 7 days, then 1 GPU for 3 hours
3 h · 1 GPU

to pass SONIC’s return. SONIC: 7 days on 128 GPUs, about 21,000 GPU-hours against 3: over 1000×.

Same motions: SONIC falls, SuperMimic completes them
179 → 0

failed motions, SONIC → SuperMimic after 9.8 h on 2 GPUs. Training motions for both: coverage, not generalization.

Authors’ film: a real Unitree G1 at 1× speed
zero-shot

on a real Unitree G1, without per-motion tuning: skipping jump, side roll, 90° flip, 360° flip on the spot.

Unlocked

The model-based road catches up: SONIC-level tracking, on real hardware, at a fraction of the compute.

Still open

If compute is no longer the limit, data is: where do motions with objects and scenes come from?

Liao, Shao, Long, Zakka, Horowitz, Abbeel, Sreenath. SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model. CoRL 2026 · clips: authors’ film
BasicsImitatePriorsModelsScaleInteractReality
146
Recap · Imitate, Priors, Models

The decade as a relay

Each paper’s limitation is the next paper’s title.
ImitateHow do we make it look human?2017 – 2019
2017EmergenceRL can move a humanoid, unnaturally
2018DeepMimicImitate one clip: human-quality motion
2019DReConKinematic planner + physics tracker
PriorsDo we need a reference at all?2021 – 2026
2021AMPA discriminator as the style reward
2022ASEPre-train a latent skill space, reuse it
2023CALMA latent per motion: directable skills
2023C·ASESkill labels fix coverage on big datasets
2023NCPDiscrete skill codes instead of a GAN
2023CompositeOne discriminator per body part
2023AdaptNetAdapt a trained policy in minutes
2026SMPA frozen diffusion model as the reward
ModelsCan we learn the simulator and differentiate it?2021 – 2024
2021SuperTrackTracking by supervised learning in a world model
2022ControlVAEA VAE skill space, trained through the model
2024MoConVQSkills as discrete tokens; LLM-ready
The baton goes on to Scale, Interact and Reality.
BasicsImitatePriorsModelsScaleInteractReality
147
Recap · Scale, Interact, Reality

The decade as a relay, continued

ScaleOne policy for all human motion, that we can direct?2023 – 2026
2023PHCOne tracker for 98.9 % of AMASS, with recovery
2024PULSEDistill the tracker into a universal latent
2024MaskedMimicMasked goals: one promptable controller
2025CLoSDA diffusion planner in the physics loop
2026GPCSkill tokens + a GPT, on 680 hours
InteractScenes, objects, sports, hands, and data beyond mocap?2023 – 2025
2023InterPhysScene-conditioned priors: sit, lie, carry
2024UniHSILanguage to contacts: an LLM plans the scene
2025TokenHSIOne transformer, many scene skills
2023Tennis · videoSkills from broadcast footage
2024SMPLOlympicsSports as a benchmark for motion priors
2024OmnigraspMotion latent as action space: grasping
2025InterMimicTeachers repair mocap: whole-body interaction
RealityCan the character leave the screen?2024 – 2026
2024OmniH2OTeacher-student tracking on a real humanoid
2025SONICTracking at foundation-model scale
2026ScaleBFMWhat to scale together: rollouts, motions, model
2026SuperMimicWorld model: >1000× fewer GPU hours than SONIC
BasicsImitatePriorsModelsScaleInteractReality
148
Takeaways

Three things the decade taught us

01

Data defines
“natural”

Hand-written rewards lost to mocap, and mocap imitation grew from one clip to 700 hours. The reward function got simpler every year.

Emergence  DeepMimic  PHC  SONIC
02

Tracking is the
pre-training task

Dense supervision, no reward engineering, scales with data. How to optimize it, PPO or a world model, is still a race.

PHC  PULSE  MaskedMimic · InterMimic
OmniH2O · SONIC, ScaleBFM vs SuperMimic
03

The interface
is the product

What changed most is what we feed the policy, one interface after another. The planner upstream stays kinematic.

DReCon  CLoSD  UniHSI  SONIC:
motion matching, diffusion, LLM, VLA
Graphics built the recipe on screen; robotics is now cashing it in. The two fields have become one pipeline.
BasicsImitatePriorsModelsScaleInteractReality
149
Outlook

What is still open

Data

Beyond motion capture

700 hours is tiny next to text or video corpora, and almost none of it has objects. Video — captured or generated by world models — is the obvious next source.

Seed: Tennis · skills from broadcast video (2023)
Also: GRAIL · data from video world models
Contact

Long, contact-rich tasks

Fingers and whole-body contact, articulated and deformable objects, minutes rather than seconds. Today’s skills are short and mostly rigid.

Seed: TokenHSI · one long task: carry, climb, sit
Also: Omnigrasp · InterMimic · HumanX
Perception

Eyes in the loop

Most controllers here are blind or read privileged state. Closing the loop through egocentric vision, on hardware, with the planner.

Seed: SONIC · VLA-driven loco-manipulation (2×)
If the last decade asked “how do we imitate the data?”, the next one asks “where does the data come from?”
BasicsImitatePriorsModelsScaleInteractReality
150
References · Imitate, Priors, Models

Papers in this lecture

Imitate
Heess et al. Emergence of Locomotion Behaviours in Rich Environments. arXiv:1707.02286, 2017
Peng, Abbeel, Levine, van de Panne. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. SIGGRAPH 2018
Bergamin, Clavet, Holden, Forbes. DReCon: Data-Driven Responsive Control of Physics-Based Characters. SIGGRAPH Asia 2019
Priors
Peng, Ma, Abbeel, Levine, Kanazawa. AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. SIGGRAPH 2021
Peng, Guo, Halper, Levine, Fidler. ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters. SIGGRAPH 2022
Tessler, Kasten, Guo, Mannor, Chechik, Peng. CALM: Conditional Adversarial Latent Models for Directable Virtual Characters. SIGGRAPH 2023
Dou, Chen, Fan, Komura, Wang. C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters. SIGGRAPH Asia 2023
Priors continued
Zhu, Zhang, Lan, Han. Neural Categorical Priors for Physics-Based Character Control (NCP). SIGGRAPH Asia 2023
Xu, Shang, Zordan, Karamouzas. Composite Motion Learning with Task Control. SIGGRAPH 2023
Xu, Xie, Andrews, Kry, Neff, McGuire, Karamouzas, Zordan. AdaptNet: Policy Adaptation for Physics-Based Character Control. SIGGRAPH Asia 2023
Mu, Zhang, Shi, …, Peng. SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control. SIGGRAPH 2026
Models
Fussell, Bergamin, Holden. SuperTrack: Motion Tracking for Physically Simulated Characters using Supervised Learning. SIGGRAPH Asia 2021
Yao, Song, Chen, Liu. ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters. SIGGRAPH Asia 2022
Yao, Song, Zhou, Ao, Chen, Liu. MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations. SIGGRAPH 2024
BasicsImitatePriorsModelsScaleInteractReality
151
References · Scale, Interact, Reality

Papers in this lecture, continued

Scale
Luo, Cao, Winkler, Kitani, Xu. Perpetual Humanoid Control for Real-time Simulated Avatars (PHC). ICCV 2023
Luo, Cao, Merel, Winkler, Huang, Kitani, Xu. Universal Humanoid Motion Representations for Physics-Based Control (PULSE). ICLR 2024
Tessler, Guo, Nabati, Chechik, Peng. MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting. SIGGRAPH Asia 2024
Tevet, Raab, Cohan, Reda, Luo, Peng, Bermano, van de Panne. CLoSD: Closing the Loop between Simulation and Diffusion for Multi-Task Character Control. ICLR 2025
Shi, Jiang, Tessler, Peng. GPC: Large-Scale Generative Pretraining for Transferable Motor Control. SIGGRAPH 2026
Interact
Hassan, Guo, Wang, Black, Fidler, Peng. Synthesizing Physical Character-Scene Interactions (InterPhys). SIGGRAPH 2023
Xiao, Wang, Wang, Cao, Zhang, Dai, Lin, Pang. Unified Human-Scene Interaction via Prompted Chain-of-Contacts (UniHSI). ICLR 2024
Pan, Yang, Dou, Wang, Huang, Dai, Komura, Wang. TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization. CVPR 2025
Interact continued
Zhang, Yuan, Makoviychuk, Guo, Fidler, Peng, Fatahalian. Learning Physically Simulated Tennis Skills from Broadcast Videos. SIGGRAPH 2023
Luo, Wang, Liu, Zhang, Tessler, Wang, Yuan, Cao, Lin, Wang, Hodgins, Kitani. SMPLOlympics: Sports Environments for Physically Simulated Humanoids. arXiv:2407.00187, 2024
Luo, Cao, Christen, Winkler, Kitani, Xu. Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. NeurIPS 2024
Xu, Ling, Wang, Gui. InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions. CVPR 2025
Reality
He, Luo, He, Xiao, Zhang, Zhang, Kitani, Liu, Shi. OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning. CoRL 2024
Luo, Yuan, Wang, et al. SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control. Science Robotics 2026 · arXiv 2025
Zeng, Yin, Niu, …, Pang, Wang. Scaling Behavior Foundation Model for Humanoid Robots (ScaleBFM). arXiv:2607.15163, 2026
Liao, Shao, Long, Zakka, Horowitz, Abbeel, Sreenath. SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model. CoRL 2026
Also shown or mentioned: Merel et al. 2017 · Xu & Karamouzas 2021 (ICCGAN) · Makoviychuk et al. 2021 (Isaac Gym) · He et al. 2024 (H2O) · Liao et al. 2025 (BeyondMimic) · Zhang et al. 2026 (LATENT) · Wang et al. 2026 (HumanX) · GRAIL (CoRL 2026). All clips and figures come from the authors’ public project pages, blogs and videos.
BasicsImitatePriorsModelsScaleInteractReality
152
Thank you
From one backflip
to one policy for everything.
Questions?