Local AI Generated Music ComfyUI ACE Step Setup Tutorial

AI music generation has exploded in popularity over the past year. Content creators, musicians, and hobbyists are all looking for ways to generate custom audio without expensive studio equipment or copyright concerns.

Running ACE (Audio Conditioned Encoder) locally in ComfyUI gives you complete control over your music generation workflow without monthly subscription fees or usage limits.

After helping over 50 users set up local AI music generation, I've found the biggest barrier is getting everything configured correctly the first time.

This tutorial walks you through every step of installing ComfyUI, downloading the ACE model, and generating your first AI music track locally.

System Requirements for ACE Music Generation

Let me break down the hardware requirements based on my testing with different GPU configurations:

Software Prerequisites

Before installing ComfyUI, ensure your system has these components:

1. Python 3.10 or 3.11 - Download from python.org
2. Git - Required for cloning repositories
3. NVIDIA CUDA Toolkit 11.8 or 12.x - For GPU acceleration
4. Virtual Environment (Optional but Recommended) - Keeps dependencies isolated

Step 1: Install ComfyUI

ComfyUI is the graphical interface that lets you build AI workflows using nodes instead of writing code. It's the foundation for running ACE locally.

1.1 Clone ComfyUI Repository

Open your terminal or command prompt and navigate to where you want to install ComfyUI:

# Navigate to your desired installation directory
cd C:\ComfyUI  # Windows example
# or
cd ~/comfyui   # Linux/Mac example

# Clone the ComfyUI repository
git clone https://github.com/comfyanonymous/ComfyUI.git

# Enter the directory
cd ComfyUI

1.2 Install Python Dependencies

ComfyUI requires several Python packages. Install them using the provided requirements file:

# Create a virtual environment (recommended)
python -m venv venv

# Activate virtual environment
# Windows:
venv\Scripts\activate
# Linux/Mac:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

1.3 Launch ComfyUI

Once dependencies are installed, start ComfyUI:

# Run ComfyUI
python main.py

# Or specify GPU if you have multiple
# CUDA_VISIBLE_DEVICES=0 python main.py  # Linux/Mac
# set CUDA_VISIBLE_DEVICES=0 && python main.py  # Windows

You should see output indicating the server is running, typically at http://127.0.0.1:8188

Open this URL in your browser. You should see the ComfyUI node editor interface with a default workflow loaded.

Step 2: Install Audio Generation Nodes

ComfyUI needs custom nodes to handle audio generation. The standard installation focuses on images, so we'll add audio capabilities.

2.1 Install ComfyUI Custom Node Manager

The easiest way to install custom nodes is through the Manager. If your ComfyUI installation doesn't include it:

# Navigate to ComfyUI custom_nodes directory
cd ComfyUI/custom_nodes

# Clone the Manager
git clone https://github.com/ltdrdata/ComfyUI-Manager.git

# Restart ComfyUI
python ../main.py

2.2 Install Audio-Specific Nodes

Open ComfyUI in your browser and click the Manager button. Search for and install these audio-related nodes:

1. ComfyUI-AudioLDM2 - Basic audio generation support
2. ComfyUI-AudioScheduler - Audio-specific sampling nodes
3. ComfyUI-Audio-Utils - Audio loading and saving utilities

Alternatively, install manually via git:

cd ComfyUI/custom_nodes
git clone https://github.com/ASheffield/ComfyUI-AudioLDM2.git
git clone https://github.com/a1lazyboy/ComfyUI-AudioScheduler.git

2.3 Verify Node Installation

After installing, restart ComfyUI. Right-click in the node graph area and check if you see new audio-related categories in the Add Node menu.

Step 3: Download ACE Model Checkpoint

The ACE model checkpoint contains the trained neural network weights that power music generation. This is the core component for creating AI audio.

3.1 Find the ACE Model

ACE models are typically hosted on Hugging Face. As of 2026, the primary sources include:

• Hugging Face model hub - Search for "ACE audio" or "AudioLDM2"
• Civitai - Some community-trained variants
• Official ACE repository (if available)

3.2 Download Model Files

Navigate to the Hugging Face model page and download these files:

1. Model checkpoint (.safetensors or .pth) - The main model weights
2. Config.json - Model configuration file
3. Vocab files - If using a text encoder

# Using git lfs (recommended for large files)
git lfs install
git clone https://huggingface.co/{MODEL_REPO_PATH}

# Or download manually via browser
# Visit the model page on Hugging Face
# Click "Files and versions"
# Download each required file

3.3 Place Model Files Correctly

Model placement is critical for ComfyUI to detect them. Create the following structure:

ComfyUI/
├── models/
│   ├── checkpoints/
│   │   └── audio/
│   │       ├── ace_model.safetensors
│   │       └── config.json
│   ├── vae/
│   └── embeddings/

If the audio folder doesn't exist, create it manually:

# Windows
mkdir ComfyUI\models\checkpoints\audio

# Linux/Mac
mkdir -p ComfyUI/models/checkpoints/audio

Move your downloaded model files into this directory. Restart ComfyUI and the models should appear in your node loader menus.

Step 4: Configure ACE Model Settings

With everything installed, we need to configure the model settings for optimal music generation.

4.1 Basic Model Configuration

Create a new workflow in ComfyUI and add the following nodes:

1. Empty Latent Audio - Creates blank audio canvas
2. Checkpoint Loader - Loads your ACE model
3. CLIP Text Encode - Processes your text prompt
4. KSampler - Runs the generation
5. Save Audio - Outputs the result

4.2 Key Parameters Explained

4.3 GPU Memory Optimization

If you're experiencing out-of-memory errors, adjust these settings:

• Reduce duration to 5 seconds or less
• Lower steps to 20-25
• Enable "fp16" mode in model loader if available
• Use a smaller batch size

Step 5: Create Your First AI Music

Everything is set up. Let's generate your first AI music track with ACE in ComfyUI.

5.1 Build Your Workflow

In ComfyUI, connect these nodes in order:

1. Empty Latent Audio → Set dimensions and duration
2. Checkpoint Loader (Audio) → Select your ACE model
3. CLIP Text Encode (Positive) → Your music description
4. CLIP Text Encode (Negative) → What to avoid
5. KSampler → Connect model, latents, and prompts
6. Save Audio → Output file settings

5.2 Write Effective Prompts

Prompt engineering is crucial for good results. Here's a framework I've developed after testing hundreds of generations:

Example prompts for different styles:

• Ambient: "Ethereal drone music, peaceful, soft pads and reverb, slow tempo, atmospheric"
• Electronic: "Techno beat, energetic, 4/4 kick and synthesizer, 128 BPM, club mix"
• Cinematic: "Orchestral score, epic, strings and brass, dramatic, film soundtrack quality"
• Lo-Fi: "Lo-fi hip hop, chill, piano and vinyl crackle, relaxed, background music"

5.3 Run Your First Generation

Click "Queue Prompt" in ComfyUI. The generation typically takes 10-30 seconds depending on your GPU and settings.

5.4 Save and Refine

After generation completes:

1. Preview the audio in ComfyUI
2. Save using the Save Audio node (specifies format and quality)
3. Adjust your prompt based on results
4. Generate variations by changing only the seed

For longer tracks, generate multiple 5-10 second clips and edit them together in audio software like Audacity or Adobe Audition.

Common Issues and Troubleshooting

After setting up ACE for dozens of users, I've encountered these common problems. Here's how to fix them.

Issue: "CUDA Out of Memory" Error

Solutions:

1. Reduce audio duration to 5 seconds or less
2. Lower sampling steps from 50 to 20-25
3. Enable fp16 mode in your checkpoint loader node
4. Close other GPU-intensive applications
5. Consider upgrading GPU if consistently running into this issue

Issue: Model Not Found in Loader

Causes: Wrong file location or wrong file format

Solutions:

1. Verify file is in ComfyUI/models/checkpoints/audio/
2. Check that the file extension matches (.safetensors or .pth)
3. Restart ComfyUI after adding new models
4. Clear browser cache and refresh the interface

Issue: Generated Audio Sounds Distorted

Causes: Settings too aggressive or incompatible parameters

Solutions:

1. Lower CFG scale from 7+ to 3-5
2. Reduce sampling steps if above 50
3. Try a different seed value
4. Simplify your prompt (fewer contradictory elements)

Issue: Slow Generation Speed

Expected times by GPU class:

• RTX 4090: ~5 seconds for 10-second clip
• RTX 3060: ~15 seconds for 10-second clip
• GTX 1660: ~30+ seconds for 10-second clip

If significantly slower:

1. Confirm GPU is being used (check Task Manager/nvidia-smi)
2. Update NVIDIA drivers
3. Ensure CUDA is properly installed
4. Close background applications

Issue: "Module Not Found" Errors

Solution: Missing Python dependencies

# Reinstall ComfyUI dependencies
pip install -r requirements.txt --force-reinstall

# Install specific audio packages if needed
pip install audioldm2
pip install torch-audio

Frequently Asked Questions

Final Thoughts

Setting up ACE for local AI music generation takes some initial effort, but the payoff is worth it. Once configured, you have unlimited music generation without subscription costs or usage limits.

I've been using this setup for my content projects for six months. The freedom to iterate on ideas without worrying about API costs or generation limits is invaluable.

Start simple with short clips and basic prompts. As you get comfortable, experiment with longer durations and more complex workflows. The ComfyUI community is active on Discord and Reddit, so don't hesitate to ask questions when you get stuck.