一、大模型推理
https://github.com/hiyouga/LLaMA-Factory/blob/main/README_zh.md
https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/README_zh.md
- 安装 v7.1
https://github.com/hiyouga/LLaMA-Factory/releases/tag/v0.7.1
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .[torch,metrics]
- 推理
- 以 Llama-3-8B-Instruct 为例，这里全部用 vllm 作为后端，不写就是 transformers 后端
- 使用命令行接口
- llama3_vllm.yaml
model_name_or_path: eta-Llama-3-8B-Instruct
template: llama3
infer_backend: vllm
vllm_enforce_eager: true
CUDA_VISIBLE_DEVICES=0 llamafactory-cli chat examples/inference/llama3_vllm.yaml
- 使用浏览器界面
CUDA_VISIBLE_DEVICES=0 llamafactory-cli webchat examples/inference/llama3_vllm.yaml
- 启动 OpenAI 风格 API
CUDA_VISIBLE_DEVICES=0 llamafactory-cli api examples/inference/llama3_vllm.yaml
- 模拟发起请求
curl http://0.0.0.0:8000/v1/chat/completions \-H "Content-Type: application/json" \-H "Authorization: Bearer " \-d '{"model": "Meta-Llama-3-8B-Instruct","messages": [{"role": "system","content": "You are a helpful assistant."},{"role": "user","content": "Who won the world series in 2020?"},{"role": "assistant","content": "The Los Angeles Dodgers won the World Series in 2020."},{"role": "user","content": "Where was it played?"}]}'