Learn PromptingMulti-Fusion Retrieval Augmented Generation (MoRAG)
- MoRAG utilizes a retrieval-augmented approach to create diverse and realistic motions from text.
- This is achieved through part-specific retrieval, where motions for individual body parts are retrieved from a database.
- The retrieved motions are then fused together to form a cohesive full-body motion sequence.
What is MoRAG?
Traditional models often struggle with generating realistic motion for complex or unfamiliar prompts, but MoRAG solves this by retrieving motion sequences for specific body parts, then fusing them into a cohesive full-body sequence. MoRAG (Multi-Fusion Retrieval Augmented Generation) is a framework designed to improve text-based human motion generation by enhancing motion diffusion models with a retrieval-augmented approach.
MoRAG breaks down prompts into motion sequences specific to body parts (like the torso, hands, and legs), and retrieves motions tailored to each part. These parts are then combined to generate a realistic, diverse motion sequence from natural language descriptions.
Example
If a user inputs:
Prompt
A person doing yoga.
MoRAG retrieves specific motions for the torso, arms, and legs from its database, and combines them to generate a realistic yoga pose sequence.
How to Use MoRAG
MoRAG enhances motion generation by combining retrieval-based techniques with diffusion models. The process involves:
- Input: A prompt (e.g., "A person dancing with raised hands").
- Part-Specific Retrieval: The system breaks down the motion into body parts (e.g., torso, hands, legs) and retrieves relevant motions from a motion database.
- Fusion: These part-specific motions are fused into a complete motion sequence.
- Generation: The fused motion is used as additional guidance in the motion diffusion model to create diverse, high-quality sequences.
Benefits of MoRAG
- Better generalization to unseen text descriptions.
- Increased diversity in generated motions.
- Higher alignment between text and motion.
MoRAG can be used in animation, virtual reality, and gaming for generating complex, realistic human motions from simple text descriptions.
To explore MoRAG further, check out the code and models released by the authors.
Valeriia Kuka
Valeriia Kuka, Head of Content at Learn Prompting, is passionate about making AI and ML accessible. Valeriia previously grew a 60K+ follower AI-focused social media account, earning reposts from Stanford NLP, Amazon Research, Hugging Face, and AI researchers. She has also worked with AI/ML newsletters and global communities with 100K+ members and authored clear and concise explainers and historical articles.
Footnotes
-
Shashank, K. S., Maheshwari, S., & Sarvadevabhatla, R. K. (2024). MoRAG – Multi-Fusion Retrieval Augmented Generation for Human Motion. https://arxiv.org/abs/2409.12140 ↩