NVIDIA and Hugging Face Team Up to Scale Diffusion Model Fine-Tuning with NeMo Automodel
· 4 min read ·
Diffusion models have driven some of the most notable open-source releases in generative AI over the past two years, including FLUX.1-dev for text-to-image generation and Wan 2.1 and HunyuanVideo for text-to-video. The Hugging Face Diffusers library has emerged as the central hub for these models, offering a unified interface for inference, adaptation, and pipeline composition. Now, a collaboration between NVIDIA and Hugging Face aims to make large-scale training and fine-tuning of these models significantly more accessible.
What NeMo Automodel Brings to the Table
NVIDIA NeMo Automodel is an open-source, PyTorch DTensor-native training library that is part of the broader NVIDIA NeMo framework. The integration with Diffusers is designed around two core principles. First, it is Hugging Face native: users can point a pretrained model path at any Diffusers model ID on the Hub and begin training immediately. NeMo Automodel loads models using standard Diffusers classes and pipelines, and checkpoints round-trip cleanly back into the Diffusers ecosystem for inference or sharing.
Second, the library follows a "one program, any scale" philosophy. Parallelism is treated as a configuration choice rather than a code rewrite. Users can switch between FSDP2, tensor parallel, expert parallel, context parallel, and pipeline parallel simply by declaring settings — not by rewriting model code. This makes it feasible to scale training from a single GPU to large multi-node clusters.
Currently, NeMo Automodel supports flow-matching models. Under the hood, it uses flow matching as the training objective, along with latent-space training via pre-encoded VAE outputs and multiresolution bucketed data loading to maximize throughput.
Practical Benefits for Diffusers Users
One of the most significant advantages of the integration is the elimination of checkpoint conversion. Pretrained weights from the Hugging Face Hub work out of the box, and fine-tuned checkpoints load directly into a DiffusionPipeline for inference. Downstream tools such as quantization, compilation, LoRA adapters, and custom samplers continue to function without modification.
When a new diffusion model is added to Diffusers, enabling it in NeMo Automodel requires only a small, contained code addition — a data preprocessing handler and a model adapter — rather than an entirely new training script. The rest of the recipe stack, including FSDP2, bucketed data loading, checkpointing, and generation, carries over unchanged through the same YAML-driven workflow.
The integration supports both full fine-tuning and LoRA-style parameter-efficient fine-tuning (PEFT). This means users can pursue maximum quality with full fine-tuning on a large cluster or maximum efficiency with LoRA on a single node, all within the same recipe structure.
Scalability extends well beyond what built-in Diffusers scripts offer. NeMo Automodel adds sharding schemes such as FSDP2, tensor, context, and pipeline parallelisms, along with multi-node orchestration via SLURM (with Kubernetes support planned). These capabilities make it possible to train larger models such as FLUX.1-dev at 12 billion parameters and HunyuanVideo at 13 billion parameters.
Fine-Tuning in Action
The collaboration ships with ready-to-use fine-tuning recipes for several open diffusion models. The recommended installation method is the NeMo Automodel Docker container, which comes with PyTorch, TransformerEngine, and other CUDA-compiled dependencies pre-built. Users can also install via pip or from source.
One documented example walks through a full-transformer fine-tune of FLUX.1-dev on a 78-card Rider–Waite tarot dataset. The workflow involves streaming images directly from Hugging Face, distributing preprocessing across all visible GPUs, and consuming cached VAE latents and text embeddings during training rather than encoding source images at every step. The process uses checked-in YAML configs with command-line overrides, eliminating the need for new configuration files.
After 200 training steps, generated images prompted with a tarot trigger token displayed a distinct vintage aesthetic — cream, red, and black color palettes, heavy ink contours, flat color fields, and allegorical card compositions. The same prompts without the trigger token retained a photographic style, confirming that the learned effect was associated with the trigger rather than globally replacing the base model's behavior.
Performance measurements were conducted on a single node with eight NVIDIA H100 80GB GPUs, fully NVLink-connected. Benchmark datasets included an image set of 256 cached samples and a video set of 112 cached samples, with full batches enforced and checkpoint writes disabled during timing.
Showcase examples further demonstrated the library's versatility. Fine-tuning the Wan 2.1 model on a Ghibli video dataset successfully adapted the output style, while applying a LoRA adapter to the same model produced characteristic Ghibli-style highlights, particularly in characters' eyes. Additional examples for FLUX.2 confirmed that both full fine-tuning and LoRA-style PEFT can effectively tailor outputs to specific stylistic domains.
What Comes Next
While YAML configuration is well-suited for reproducible, reviewable workflows, many teams also need a programmatic interface. An upcoming NeMo Automodel release plans to surface the diffusion recipes through a fully typed Pythonic API. This will allow users to compose model, data, optimizer, PEFT, parallelism, checkpointing, and generation components directly from Python, making the recipes easier to integrate into existing training code, notebooks, and experiment workflows. The Pythonic path will complement the existing YAML quick-start approach rather than replace it.
The entire integration is documented in the Diffusers training guide and is fully open source under the Apache 2.0 license. As diffusion models continue to grow in size and complexity, tools like NeMo Automodel could play a pivotal role in democratizing large-scale fine-tuning. If you found this breakdown useful, share it with your network and let others explore what distributed diffusion training can unlock.
