Standalone Deployment#

Install Prepare#

git clone https://github.com/eosphoros-ai/DB-GPT.git

cd DB-GPT

Create conda environment#

conda create -n dbgpt_env python=3.10

conda activate dbgpt_env

Install Default Requirements#

# Install Default Requirements
pip install -e ".[default]"

Download and Prepare LLM Model and Embedding Model#

Tip

If you don’t have high performance hardware server

you can use openai api, tongyi api , bard api, etc.

mkdir models && cd models

# download embedding model, eg: text2vec-large-chinese
git clone https://huggingface.co/GanymedeNil/text2vec-large-chinese

set proxy api in .env

#set LLM_MODEL TYPE
LLM_MODEL=proxyllm
#set your Proxy Api key and Proxy Server url
PROXY_API_KEY={your-openai-sk}
PROXY_SERVER_URL=https://api.openai.com/v1/chat/completions

Tip

If you have high performance hardware server

mkdir models && cd models

# # download embedding model, eg: vicuna-13b-v1.5 or  
git clone https://huggingface.co/lmsys/vicuna-13b-v1.5

# download embedding model, eg: text2vec-large-chinese
git clone https://huggingface.co/GanymedeNil/text2vec-large-chinese

popd

Start all services with a single command.#

LLM_MODEL=vicuna-13b-v1.5 
dbgpt start webserver --port 6006

By default, the “dbgpt start webserver” command will start the Webserver, Model Controller, and Model Worker in a single Python process. Here, we specify the service to be started on port 6006.

View and validate the model service in the command line, you can use the following commands#

1.list the started model services and deployed Model Workers, you can use the following command#

dbgpt model list

output is:

+-----------------+------------+------------+------+---------+---------+-----------------+----------------------------+
|    Model Name   | Model Type |    Host    | Port | Healthy | Enabled | Prompt Template |       Last Heartbeat       |
+-----------------+------------+------------+------+---------+---------+-----------------+----------------------------+
| vicuna-13b-v1.5 |    llm     | 172.17.0.9 | 6006 |   True  |   True  |                 | 2023-10-16T19:49:59.201313 |
|  WorkerManager  |  service   | 172.17.0.9 | 6006 |   True  |   True  |                 | 2023-10-16T19:49:59.246756 |
+-----------------+------------+------------+------+---------+---------+-----------------+----------------------------+

The WorkerManager is the management process for Model Workers

validate the deployed model in the command line, you can use the following command#

dbgpt model chat --model_name vicuna-13b-v1.5

Then an interactive page will be launched where you can have a conversation with the deployed LLM in the terminal.

Chatbot started with model vicuna-13b-v1.5. Type 'exit' to leave the chat.


You: Hello
Bot: Hello! How can I assist you today?

You: