Notes

5. Training Pipeline

Let’s assume we fine-tune Stable Diffusion v1.5 with LoRA using Hugging Face Diffusers.

5.1. Install Dependencies
pip install diffusers transformers accelerate datasets safetensors
pip install xformers # Faster training

5.2. Prepare Dataset with Captions
from datasets import load_dataset

dataset = load_dataset("imagefolder", data_dir="dataset")
print(dataset)

5.3. Set Up Training Script (LoRA)
from diffusers import StableDiffusionPipeline
from peft import LoraConfig
from transformers import AutoTokenizer
from accelerate import Accelerator

model_name = "runwayml/stable-diffusion-v1-5"
tokenizer = AutoTokenizer.from_pretrained(model_name)

lora_config = LoraConfig(
r=16, # Rank of low-rank matrices
lora_alpha=32, # Scaling factor
target_modules=["CrossAttention"],
lora_dropout=0.05
)

from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained(model_name, torch_dtype="float16").to("cuda")

5.4. Train the Model
accelerate launch train_text_to_image_lora.py
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5"
--dataset_name="dataset"
--resolution=512
--train_batch_size=2
--gradient_accumulation_steps=4
--learning_rate=1e-4
--lr_scheduler="constant"
--max_train_steps=5000
--output_dir="lora-trained-model"

5.5. Use the Fine-Tuned Model
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype="float16"
)
pipe.unet.load_attn_procs("lora-trained-model")
pipe.to("cuda")

prompt = "A futuristic red sports car on Mars, ultra realistic, 8K"
image = pipe(prompt).images[0]
image.save("generated.png")

6. Best Practices

Use LoRA for quick, efficient fine-tuning.

Use DreamBooth if you want personalized objects.

Use gradient checkpointing to save VRAM.

Use mixed-precision (fp16/bf16) to speed up training.

Regularly evaluate outputs during training.

7. Tools to Simplify Fine-Tuning

Hugging Face Diffusers → Best for SD & LoRA

Kohya’s GUI → No-code LoRA & DreamBooth trainer

Automatic1111 WebUI → Popular for testing LoRAs

ComfyUI → Advanced control & pipeline customization