cosyvoice voice generation

👉🏻 Cosyvoice 👈🏻

Fun-Cosyvoice 3.0: Demos; Paper; Modelscope; Huggingface; Evaluation

Cosyvoice 2.0: Demos; Paper; Modelscope; HuggingFace

Cosyvoice 1.0: Demos; Paper; Modelscope; HuggingFace

Highlight🔥

Fun-Cosyvoice 3.0 is an advanced text-to-speech system built around large language models. It improves continuity, speaker likeness and prosody while remaining suitable for zero-shot multilingual speech synthesis in everyday settings.

Key Features

• Language Coverage: Supports 9 widely used languages including Chinese, English, Japanese, Korean, German, Spanish, French, Italian and Russian, alongside 18+ Chinese dialects and accents. It also supports multilingual and cross-lingual zero-shot voice cloning.
• Content Consistency & Naturalness: Keeps the spoken content stable while preserving speaker similarity and natural-sounding prosody.
• Pronunciation Inpainting: Accepts Chinese Pinyin and English CMU phonemes for more precise pronunciation control in production workflows.
• Text Normalization: Reads numbers, symbols and varied text formats without requiring a traditional frontend module.
• Bi-Streaming: Supports text-in streaming and audio-out streaming, with latency designed to reach approximately 150ms while maintaining detailed audio.
• Instruct Support: Handles instructions for language, dialect, emotion, speed, volume and related delivery controls.

Roadmap

• 2025/12
  • Release the Fun-Cosyvoice3-0.5B-2512 base model, RL model and training/inference scripts
  • Release a Fun-Cosyvoice3-0.5B Modelscope Gradio space
• 2025/08
  • Add Triton TRT-LLM runtime support and Cosyvoice2 GRPO training support
• 2025/07
  • Release the Fun-Cosyvoice 3.0 evaluation set
• 2025/05
  • Add Cosyvoice2-0.5B vLLM support
• 2024/12
  • Release the 25Hz Cosyvoice2-0.5B model
• 2024/09
  • Release the 25Hz Cosyvoice-300M base model
  • Add the Cosyvoice-300M voice conversion function
• 2024/08
  • Add repetition-aware sampling inference for LLM stability
  • Add streaming inference support with KV cache and SOTA optimization
• 2024/07
  • Add flow matching training support
  • Add WeTextProcessing support when ttsfrd is unavailable
  • Add FastAPI server and client support

Evaluation

Install

Clone and install

• Clone the repository

  git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
  # If the submodule download fails because of network conditions, run the following command again
  cd CosyVoice
  git submodule update --init --recursive

• Install Conda: please see the Miniconda documentation.

• Create a Conda environment:

  conda create -n cosyvoice -y python=3.10
  conda activate cosyvoice
  pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
  # If you encounter sox compatibility issues
  # ubuntu
  sudo apt-get install sox libsox-dev
  # centos
  sudo yum install sox sox-devel

Model download

We recommend downloading the pretrained Fun-CosyVoice3-0.5B, CosyVoice2-0.5B, CosyVoice-300M, CosyVoice-300M-SFT, CosyVoice-300M-Instruct and CosyVoice-ttsfrd resources.

# modelscope SDK model download
from modelscope import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
snapshot_download('iic/CosyVoice2-0.5B', local_dir='pretrained_models/CosyVoice2-0.5B')
snapshot_download('iic/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
snapshot_download('iic/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT')
snapshot_download('iic/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct')
snapshot_download('iic/CosyVoice-ttsfrd', local_dir='pretrained_models/CosyVoice-ttsfrd')

# for overseas users, huggingface SDK model download
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
snapshot_download('FunAudioLLM/CosyVoice2-0.5B', local_dir='pretrained_models/CosyVoice2-0.5B')
snapshot_download('FunAudioLLM/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
snapshot_download('FunAudioLLM/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT')
snapshot_download('FunAudioLLM/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct')
snapshot_download('FunAudioLLM/CosyVoice-ttsfrd', local_dir='pretrained_models/CosyVoice-ttsfrd')

Optionally, unpack the ttsfrd resource and install its package for improved text normalization.

This step is optional. Without it, Cosyvoice uses wetext by default.

cd pretrained_models/CosyVoice-ttsfrd/
unzip resource.zip -d .
pip install ttsfrd_dependency-0.1-py3-none-any.whl
pip install ttsfrd-0.4.2-cp310-cp310-linux_x86_64.whl

Basic Usage

For the strongest results, use Fun-CosyVoice3-0.5B and follow the examples for each supported model.

python example.py

vLLM Usage

Cosyvoice2 and Cosyvoice3 support vLLM 0.11.x+ with the V1 engine and vLLM 0.9.0 in legacy mode. Older releases do not support Cosyvoice inference, while versions between these tested releases may require additional validation.

Because vLLM has specific dependency requirements, create a separate environment when needed.

conda create -n cosyvoice_vllm --clone cosyvoice
conda activate cosyvoice_vllm
# for vllm==0.9.0
pip install vllm==v0.9.0 transformers==4.51.3 numpy==1.26.4 -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
# for vllm>=0.11.0
pip install vllm==v0.11.0 transformers==4.57.1 numpy==1.26.4 -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
python vllm_example.py

Start web demo

Use the web demo to get familiar with Cosyvoice quickly, then adapt the example to your selected model.

Please see the creation workspace for the current demo details.

# change iic/CosyVoice-300M-SFT for SFT inference, or iic/CosyVoice-300M-Instruct for instruct inference
python3 webui.py --port 50000 --model_dir pretrained_models/CosyVoice-300M

Advanced Usage

Training and inference scripts for advanced users are provided in examples/libritts.

Build for deployment

For optional service deployment, run the following steps.

cd runtime/python
docker build -t cosyvoice:v1.0 .
# change iic/CosyVoice-300M to iic/CosyVoice-300M-Instruct for instruct inference
# for grpc usage
docker run -d --runtime=nvidia -p 50000:50000 cosyvoice:v1.0 /bin/bash -c "cd /opt/CosyVoice/CosyVoice/runtime/python/grpc && python3 server.py --port 50000 --max_conc 4 --model_dir iic/CosyVoice-300M && sleep infinity"
cd grpc && python3 client.py --port 50000 --mode <sft|zero_shot|cross_lingual|instruct>
# for fastapi usage
docker run -d --runtime=nvidia -p 50000:50000 cosyvoice:v1.0 /bin/bash -c "cd /opt/CosyVoice/CosyVoice/runtime/python/fastapi && python3 server.py --port 50000 --model_dir iic/CosyVoice-300M && sleep infinity"
cd fastapi && python3 client.py --port 50000 --mode <sft|zero_shot|cross_lingual|instruct>

Using Nvidia TensorRT-LLM for deployment

TensorRT-LLM can accelerate the Cosyvoice2 language model substantially compared with a standard Hugging Face Transformers implementation.

cd runtime/triton_trtllm
docker compose up -d

For deployment details, continue through the creation workspace.

Discussion & Communication

Questions and technical discussions can be shared through the Cosyvoice community workspace.

You can also scan the QR code to join the official Cosyvoice discussion group.

Acknowledge

1. We borrowed substantial code from FunASR.
2. We borrowed substantial code from FunCodec.
3. We borrowed substantial code from Matcha-TTS.
4. We borrowed substantial code from AcademiCodec.
5. We borrowed substantial code from WeNet.

Citations

@article{du2024cosyvoice,
  title={Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens},
  author={Du, Zhihao and Chen, Qian and Zhang, Shiliang and Hu, Kai and Lu, Heng and Yang, Yexin and others},
  journal={arXiv preprint arXiv:2407.05407},
  year={2024}
}

@article{du2024cosyvoice,
  title={Cosyvoice 2: Scalable streaming speech synthesis with large language models},
  author={Du, Zhihao and Wang, Yuxuan and Chen, Qian and Shi, Xian and Lv, Xiang and Zhao, Tianyu and others},
  journal={arXiv preprint arXiv:2412.10117},
  year={2024}
}

@article{du2025cosyvoice,
  title={CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training},
  author={Du, Zhihao and Gao, Changfeng and Wang, Yuxuan and Yu, Fan and Zhao, Tianyu and others},
  journal={arXiv preprint arXiv:2505.17589},
  year={2025}
}

@inproceedings{lyu2025build,
  title={Build LLM-Based Zero-Shot Streaming TTS System with Cosyvoice},
  author={Lyu, Xiang and Wang, Yuxuan and Zhao, Tianyu and Wang, Hao and Liu, Huadai and Du, Zhihao},
  booktitle={ICASSP 2025},
  pages={1--2},
  year={2025},
  organization={IEEE}
}

Ecosystem

Cosyvoice is part of the FunAudioLLM family, a broader toolkit for speech understanding and generation.

Disclaimer

The material on this page is provided for academic and technical demonstration purposes. Some examples are sourced from public resources. If any material infringes your rights, please contact us to request its removal.