llama.cpp
https://github.com/echonoshy/cgft-llm
cgft-llm/llama-cpp/README.md at master · echonoshy/cgft-llm (github.com)
【大模型量化】- Llama.cpp轻量化模型部署及量化_哔哩哔哩_bilibili
Release模式是直接运行，Debug模式是调试模型。
github.com/ggerganov/llama.cpp
export HF_ENDPOINT=https://hf-mirror.comhuggingface-cli download shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-8bit --local-dir /root/autodl-tmp/models/Llama3-8B-Chinese-Chat-GGUF
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build_cpu
cmake --build build_cpu --config Releasecmake -B build_cuda -DLLAMA_CUDA=ON
cmake --build build_cuda --config Release -j 12
cd ~/code/llama.cpp/build_cuda/bin
./quantize --allow-requantize /root/autodl-tmp/models/Llama3-8B-Chinese-Chat-GGUF/Llama3-8B-Chinese-Chat-q8_0-v2_1.gguf /root/autodl-tmp/models/Llama3-8B-Chinese-Chat-GGUF/Llama3-8B-Chinese-Chat-q4_1-v1.gguf Q4_1
python convert-hf-to-gguf.py /root/autodl-tmp/models/Llama3-8B-Chinese-Chat --outfile /root/autodl-tmp/models/Llama3-8B-Chinese-Chat-GGUF/Llama3-8B-Chinese-Chat-q8_0-v1.gguf --outtype q8_0
