Building an open-source AI pipeline for speech-driven 3D character animation. Looking for honest feedback before I invest time
artificial
Building an open-source AI pipeline for speech-driven 3D character animation. Looking for honest feedback before I invest time
Hi everyone, I'm a third-year engineering student working on an AI project that converts speech into a fully animated 3D character. Right now, my prototype works like this: Audio ↓ EMAGE (Speech → Motion) ↓ SMPL Motion (Research Prototype) ↓ Motion...
artificial

Building an open-source AI pipeline for speech-driven 3D character animation. Looking for honest feedback before I invest time

Hi everyone,

I'm a third-year engineering student working on an AI project that converts speech into a fully animated 3D character.

Right now, my prototype works like this:

Audio ↓ EMAGE (Speech → Motion) ↓ SMPL Motion (Research Prototype) ↓ Motion Retargeting (planned) ↓ Commercial Character Rig ↓ Blender Rendering ↓ MP4 Animation

The current implementation uses SMPL only as an internal research representation. Since SMPL isn't intended for unrestricted commercial use, my long-term goal is to retarget the generated motion to commercially usable character rigs (VRM, custom Blender rigs, etc.).

So I'm not trying to sell SMPL models—I'm trying to build a pipeline that generates expressive 3D character animations from speech.

Current Progress

  • Audio → body motion generation using EMAGE
  • Successfully animated the generated motion in Blender
  • Automatic rendering pipeline
  • Basic lighting and camera setup

Planned Features

  • Facial animation (lip sync, blinking, expressions)
  • Finger animation
  • Motion retargeting to commercial character rigs
  • Multiple avatars/characters
  • Automatic camera movement
  • Better lighting and rendering
  • Web interface/API
  • One-click video generation

Why I'm Posting

Before I spend the next few months building this, I'd like honest feedback from people who actually create content, games, films, AI tools, or use character animation in their workflow.

Questions

  1. Would you actually use or pay for something like this?
  2. Who do you think would benefit the most from this kind of tool?
  3. What features would make it valuable enough to replace or complement existing tools?
  4. If you've used HeyGen, Synthesia, NVIDIA Audio2Face, Character Creator, or similar tools, what do you feel they're missing?
  5. If you were building this project, what would you prioritize next?
  6. Am I solving a real problem, or should I pivot in another direction while I'm still early?

A Couple of Constraints

  • I'm building everything on a Ryzen 5 7000 laptop with 24 GB RAM and an RTX 3050 (6 GB VRAM), so my focus is on integrating existing open-source models rather than training large foundation models from scratch.
  • My goal is to build a useful product that people would actually use or pay for, not just reproduce research papers or publish another academic project.

I'm looking for brutally honest feedback. If you think this has potential, I'd love to know what direction you'd take it. If you think it's a dead end or there's a better opportunity in the 3D AI space, I'd really appreciate hearing that too.

Thanks in advance!

submitted by /u/Blesh011
[link] [comments]