Helix4D: Complex 4D Mesh Generation (NeurIPS 2026 Oral)

From Trellis 3D to Helix 4D and to Code Video Model

Gordon Qian (钱国成) · Researcher @ ByteDance San Jose

guochengqian.github.io

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)01 / 12 · 0:20
BACKGROUND · PRIOR VIDEO-TO-4D METHODS

Video-to-4D: Progress and Remaining Gaps

Strong on rigid and simple objects; complex geometry and appearance remain challenging.

Reference Input video
Input video: fish in glass example

Monocular, object-centric video

Separate mesh + motion Mesh4D
Mesh4D: fish in glass example

Canonical mesh + dense deformation

Joint 4D generation SS4D
SS4D: fish in glass example

Structured spacetime latents

Remaining challenges

  • Complex topology variations
  • Material modeling
  • Transparency
  • Inner-surface reconstruction
Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)02 / 12 · 0:35
MOTIVATION · FROM STRONG 3D PRIORS TO 4D

A Stronger 3D Starting Point: TRELLIS.2

Geometry and material priors for thin, non-watertight and complex surfaces.

Image-to-3D results · static assets shown as turntables

Transparency + inner surfaces
TRELLIS.2: Transparency + inner surfaces

Transparent enclosure and interior geometry; cutaway view reveals the contents.

Thin + open structures
TRELLIS.2: Thin + open structures

Fine ropes, railings and sails without requiring a watertight surface.

Rich PBR materials
TRELLIS.2: Rich PBR materials

Detailed appearance with base color, roughness, metallic and opacity.

TRELLIS.2 · image → 3DHelix4D · video → 4D

Our idea: inherit TRELLIS.2’s expressive representation and pretrained strengths for video-conditioned dynamic mesh generation.

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)03 / 12 · 0:45
METHOD · FOUR CHALLENGES, FOUR DESIGN CHOICES

From TRELLIS.2 to Helix4D

Enable temporal reasoning while preserving pretrained 3D strengths.

01

Efficient cross-frame interaction

Share information at sparse-token scale.

Original per-frame attention has only diagonal frame blocks; cross-frame sliding-window attention adds interactions between nearby frames.

Sliding-window cross-frame attention

Local temporal context · efficient cross-frame interaction

02

Rare geometry, limited 4D data

Retain transparency and inner surfaces.

Sliding-window attention augmented with a clean first-frame anchor: every frame can attend to the first-frame column.

Sliding window + first-frame anchor

Clean TRELLIS.2 reference · inherited 3D capabilities

03

Time in a spatial-only encoding

Add time without disrupting pretrained Q/K phases.

Lowest-frequency spatial RoPE pairs reassigned to time; head dimension unchanged

Parameter-free 4D ReRoPE extension

Repurpose low-frequency spatial bands · fixed head dimension

04

Preserve pretrained quality

Keep strong geometry and material priors.

Reuse pretrained weights, update self-attention and freeze the other layers

Pretrained weights + minimal tuning

Reuse initialization · fine-tune self-attention only

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)04 / 12 · 1:00
METHOD · HELIX4D

Helix4D: Overview

Video-conditioned sparse structure → geometry → material

Original Figure 2, left: video-conditioned sparse-structure, geometry and material transformers turn noisy latents into a dynamic 4D asset.

Each stage generates 16 frames jointly using cross-frame attention, a first-frame anchor and 4D ReRoPE.

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)05 / 12 · 0:50
METHOD · CROSS-FRAME INTERACTION

Sliding-window Attention with Anchor

Local motion context + a clean global first-frame reference

Original Figure 3: full, causal, sliding-window, spatial, and sliding-window plus first-frame anchor attention.
Orange: allowed · gray: masked · red: first-frame anchor.

Sliding window

Nearby frames share motion and geometry information efficiently.

Window size 5 · up to 100K tokens per 4D sample.

First-frame anchor

Every frame can access clean frame-0 tokens for a stable shape and material reference.

First-frame conditioning

Train: ground-truth frame-0 latent, zero noise timestep; loss only on later frames.

Infer: generate the anchor with frozen TRELLIS.2.

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)06 / 12 · 1:00
METHOD · (b) 4D ROTARY

RoPE Frequency-band Analysis

Why can low-frequency spatial bands be repurposed for time?

Frozen TRELLIS.2 · 32 held-out objects · retain α of the spatial frequency pairs; replace the remaining low-frequency rotations with identity.

Original Figure 4: low-frequency spatial RoPE rotations are replaced with identity. Geometry deteriorates at retained ratios 0.0 and 0.2; results at 0.4, 0.8 and 1.0 are visually similar. The middle matrix reassigns the available slots to temporal rotations.
Retained ratio α 0.0 0.2 0.4 0.6 0.8 1.0
CD-3D vs full RoPE ↓ 0.0181 0.0098 0.0079 0.0079 0.0079 0 · reference

Observation

Retaining ≥40% gives visually similar geometry in this probe.

Band sensitivity

CD plateaus at 0.0079 for α = 0.4, 0.6 and 0.8.

4D allocation: α = 0.75

75% spatial pairs · 25% temporal pairs · same head width.

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking) 07 / 12 · 1:00
EXPERIMENTS · ALL BASELINES

Experiments

52-video Helix4DBench + ActionBench + held-out TexVerse

ULIP-2 ↑

+5.7%

vs Mesh4D · strongest alternative

ULIP-2 ↑: all reported baselines. Highlight +5.7%, vs Mesh4D · strongest alternative. Absolute bars start at zero.

Uni3D ↑

+7.8%

vs Motion 3-to-4 · strongest alternative

Uni3D ↑: all reported baselines. Highlight +7.8%, vs Motion 3-to-4 · strongest alternative. Absolute bars start at zero.

ActionBench CD-3D ↓

−3.8%

vs ActionMesh · strongest alternative

ActionBench CD-3D ↓: all reported baselines. Highlight −3.8%, vs ActionMesh · strongest alternative. Absolute bars start at zero.

User preference ↑

67.9%

vs ShapeGen4D · most competitive

User preference ↑: all reported baselines. Highlight 67.9%, vs ShapeGen4D · most competitive. Absolute bars start at zero.
Helix4DStrongest alternative for each metric
Method Helix4DBench ActionBench TexVerse
CLIP ↑ CLIP-N ↑ ULIP-2 ↑ Uni3D ↑ DreamSim ↓ FVD ↓ Ours wins ↑ CD-3D ↓ CD-4D ↓ CD-3D ↓ CD-4D ↓
SS4D 0.8468 — 0.4191 0.3739 0.2745 1046 71.2% 0.105 0.120 0.052 0.103
ShapeGen4D — 0.6876 0.4169 0.3687 — — 67.9% 0.056 0.170 0.075 0.129
Mesh4D 0.8533 0.7399 0.4353 0.3691 0.3025 979 79.8% 0.096 0.141 0.058 0.100
Motion 3-to-4 0.8674 0.7442 0.4331 0.3768 0.2697 955 80.7% 0.068 0.114 0.056 0.108
ActionMesh — 0.7156 0.4226 0.3761 — — 89.8% 0.053 0.081 0.056 0.111
Helix4D 0.8723 0.7569 0.4600 0.4063 0.2593 950 Ref. 0.051 0.093 0.045 0.097

ActionMesh leads ActionBench CD-4D (0.081 vs 0.093). Helix4D leads the other reported geometry metrics.

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)08 / 12 · 1:00
ABLATIONS · COMPONENTS AND ATTENTION

Ablations: What Matters?

First-frame conditioning, 4D ReRoPE and cross-frame attention

4D rotary

Original Figure 6: clean geometry with our 4D rotary; rough back and extra arm with stacked temporal RoPE.
“w/o 4D rotary” uses SS4D-style stacked temporal RoPE in the same backbone.

First-frame conditioning · CD-3D ↓

Table 3 first-frame-conditioning ablation: without a clean first-frame reference CD-3D is 0.0488; full Helix4D is 0.0464. Bars start at zero.
Clean frame-0 reference improves geometry.

ULIP-2: 0.2809 → 0.2881 · Uni3D: 0.2554 → 0.2589

Component ablation

Variant CD-3D ↓ ULIP-2 ↑ Uni3D ↑
Without first-frame conditioning 0.0488 0.2809 0.2554
Stacked temporal RoPE 0.0478 0.2819 0.2584
Full self-attention 0.0467 0.2819 0.2530
Full Helix4D 0.0464 0.2881 0.2589

“Full self-attention” replaces the proposed window + anchor pattern.

Attention pattern

Pattern CD-3D ↓ CD-4D ↓ ULIP-2 ↑ Uni3D ↑ Time ↓
Full 0.0470 0.1009 0.2835 0.2447 2.3×
Causal 0.0491 0.1038 0.2838 0.2564 1.5×
Window 0.0468 0.1085 0.2870 0.2538 0.9×
Spatial 0.0471 0.1011 0.2791 0.2548 0.4×
Window + anchor 0.0454 0.0970 0.2879 0.2576 1.0×

Window + anchor: 2.3× faster than full attention, with better reported quality.

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)09 / 12 · 0:50
CONCLUSION · HELIX4D

Helix4D Conclusion: From Pretrained 3D to Dynamic 4D

We presented Helix4D, an approach that adapts a pretrained image-to-3D generator into a video-to-4D mesh generator. We introduce a lightweight conversion of Trellis2 to dynamic generation through:

Temporal ReRoPE

Original Figure 4 left: spatial bands repurposed for temporal rotary encoding.

Encode time without adding parameters. Repurpose low-frequency spatial RoPE slots.

Sliding-window attention
with an anchor frame

Original Figure 3 right: our sliding-window attention plus first-frame anchor.

Share information efficiently. Combine local motion context with a global reference.

First-frame conditioning

A clean first-frame reference generated by frozen TRELLIS.2 conditions noisy later-frame latents, retaining geometry and material priors.

Carry strong 3D priors into 4D. Condition on a clean TRELLIS.2 first frame.

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)10 / 12 · 0:40
Code Video Model demo: What if a video started as code?
0:00 2:18

Thank you

Helix4D: Complex 4D Mesh Generation (NeurIPS 2026 Oral)

From Trellis 3D to Helix 4D and to Code Video Model

Gordon Qian (钱国成) · Researcher @ ByteDance San Jose

guochengqian.github.io

Gordon Qian (钱国成) · Helix4D · NeurIPS 2026 Oral talk (Peking)12 / 12 · 0:20