vLLM is a fast and easy-to-use library for LLM inference and serving.

Running vLLM#

Installing Dependencies#

vLLM is an optional dependency in DB-GPT, and you can manually install it using the following command:

pip install -e ".[vllm]"

Modifying the Configuration File#

Next, you can directly modify your .env file to enable vllm.


You can view the models supported by vLLM here

Then you can run it according to Run.