MAGNet: One Model for All Multi-Agent Interactions

Humans coordinate in groups naturally—whether we’re performing synchronized dance routines, sparring in a boxing ring, or simply navigating a crowded sidewalk. For AI systems, replicating this natural ability has been a longstanding challenge. But we’re excited to share a breakthrough that tackles this problem head-on.

In our new work, we introduce MAGNet (Multi-Agent Generative Network)—a unified framework that successfully handles the full complexity of multi-person coordination. Rather than building specialized models for different interaction types or group sizes, MAGNet provides a single, powerful architecture that generates remarkably realistic motion across diverse multi-person scenarios.

A Unified Solution to Fragmented Problems

The existing landscape of multi-person motion generation has been fragmented. Most methods are narrowly specialized: one model predicts how a partner reacts to your movements, another forecasts joint futures, a third handles specific activities like dancing or boxing. Even more limiting, these approaches typically handle only dyadic (two-person) interactions.

We built MAGNet to fundamentally change this paradigm. A single trained model now supports the full spectrum of multi-agent generation tasks:

The Secret Sauce: How It Works

MAGNet’s versatility comes from three tightly integrated design choices:

1. Relative Coordinate Representation

Rather than representing each person’s motion in absolute world coordinates, we encode everything as relative transformations between agents. Each person maintains their own local coordinate frame, and we represent how they’re positioned and oriented relative to every other agent in the scene.

This elegant approach makes the model completely agnostic to where the interaction happens in physical space. A group dancing in the center of a room has the same relational structure as one dancing in the corner—the model only needs to learn interaction patterns, not every possible absolute configuration. This dramatically reduces complexity and yields impressive generalization.

2. Discrete Motion Tokenization via VQ-VAE

Building on our successful approach from the couple dancing work, we use a Vector Quantized Variational Autoencoder (VQ-VAE) to learn a discrete latent representation of human motion. This converts continuous pose sequences into discrete tokens—essentially creating a learned vocabulary of body movements.

Working with discrete tokens rather than continuous joint positions provides a remarkably clean signal for the transformer. It eliminates high-frequency noise, captures meaningful motion primitives, and enables the use of classification-based training rather than regression.

But we’ve pushed this further than standard VQ-VAE approaches. Each motion token in MAGNet is multi-component, encoding not just an agent’s own pose but also their relative transforms to all other agents. This explicit representation of inter-agent relationships is the key to our success in capturing realistic coordination.

3. Diffusion Forcing Transformer

This is where the architecture gets particularly powerful. We adopt Diffusion Forcing, an innovative hybrid approach that applies different noise levels to different positions in a sequence during training. Each token (representing a specific agent at a specific timestep) receives an independently sampled noise level.

This elegant modification enables remarkably flexible conditioning at inference time. The model seamlessly handles any combination of known and unknown motion—whether you’re conditioning on clean past motion, inpainting specific agents, or gradually denoising entire future trajectories.

For multi-agent settings, our key innovation is that each agent’s tokens are independently noised. The transformer learns to intelligently modulate how much it relies on an agent’s own motion history versus their partners’ states—essentially learning when to trust individual dynamics versus inter-agent coupling. The results are stunning.

Ultra-Long Motion Generation

One of MAGNet’s most impressive capabilities is generating ultra-long motion sequences—spanning hundreds of timesteps while maintaining coherent coordination between agents. Where most motion models degrade after a few seconds, MAGNet autoregressively extends generation far beyond available ground truth, sustaining realistic dynamics and timing throughout.

Temporal Denoising Schedules

A key insight of Diffusion Forcing is that different sampling strategies simply correspond to different noise schedules applied across the token sequence. By choosing which tokens start clean (conditioned) versus noisy (to be generated), MAGNet handles all its capabilities within a single framework. The animated Gaussian textures below show remaining noise intensity at each denoising step.

Real-World Applications

The implications are genuinely exciting across multiple domains:

• Human-Robot Interaction: For robots working alongside humans—whether in manufacturing, healthcare, or domestic settings—MAGNet enables natural, real-time coordination. Robots can now generate their own motion trajectories while smoothly adapting to observed human movements, unlocking truly collaborative robotics.
• Virtual Reality and Gaming: Groups of NPCs can now coordinate dynamically, creating immersive experiences where digital characters move with unprecedented realism. The ability to generate long sequences is particularly valuable for open-world games.
• Animation and Digital Art: Artists finally have a tool that provides high-level control (specify keyframes, desired interaction type) while automating the complex details of physically plausible coordinated motion.
• Sports Analytics: MAGNet enables compelling counterfactual scenarios (“what if player X had moved differently?”) and realistic predictions of opponent strategies based on partial observations.
• Crowd Simulation: From evacuation planning to architectural design, MAGNet provides powerful tools for modeling how groups of people move through spaces—handling both tight coordination and looser interaction.

The Future: Toward Truly Social AI

Looking forward, we see transformative possibilities:

• Text-Conditioned Generation: Natural language control—“Three people greeting each other warmly” or “A basketball team executing a pick-and-roll play”—will make MAGNet accessible to artists, designers, and researchers without technical expertise.
• Interactive Real-Time Systems: Closing the loop from observation to generation to action will create true interactive multi-agent experiences where users control one agent while MAGNet generates realistic responses from others.
• Swarm Intelligence: Scaling to 100s of agents will enable emergent group dynamics with applications from realistic crowd generation to coordinated multi-robot teams.
• Cross-Modal Integration: Incorporating audio, objects, and environmental constraints will create a comprehensive model of human interaction.
• Deployed Robotics: Using MAGNet-generated motion as targets for robot controllers that adapt based on real-world physics will bridge the sim-to-real gap.

Our vision was ambitious but achievable: a generative model that captures the full richness of human social coordination—from intimate two-person interactions to complex group dynamics—in a unified framework that’s fast enough for real-time use and flexible enough for diverse applications. MAGNet represents a major leap toward that goal.