vLLM is a fast and easy-to-use library for LLM inference and serving.
vLLM is an optional dependency in DB-GPT, and you can manually install it using the following command:
pip install -e ".[vllm]"
Modifying the Configuration File#
Next, you can directly modify your
.env file to enable vllm.
You can view the models supported by vLLM here
Then you can run it according to Run.