Ovis1.6-9B视觉大模型环境搭建推理

引子

前阵子，阿里Qwen2-VL刚刚闪亮登场，感兴趣的小伙伴可以移步Qwen2-VL环境搭建&推理测试-CSDN博客。这第一的宝座还没坐多久，自家兄弟Ovis1.6版本就来了，20240919阿里国际AI团队开源多模态大模型Ovis1.6。在多模态权威综合评测基准OpenCompass上，Ovis1.6-Gemma2-9B版本综合得分超越Qwen2VL-7B、InternVL2-26B和MiniCPM-V-2.6等主流开源模型，在300亿以下参数开源模型中位居第一。

一、模型介绍

根据OpenCompass评测基准，Ovis1.6-Gemma2-9B超过了Qwen2-VL-7B、MiniCPM-V-2.6等一众相同参数量级的知名多模态模型。在数学等推理任务中，甚至有媲美70B参数模型的表现。Ovis1.6的幻觉现象和错误率也低于同级别模型，展现了更高的文本质量和准确率。阿里国际AI团队的核心思路是：从结构上对齐视觉和文本嵌入。当前，多数开源多模态大语言模型（MLLM）并非从头训练整个模型，而是通过像多层感知机（MLP）这样的连接器，将预训练的大语言模型（LLM）和视觉Transformer集成起来，给LLM装上“眼睛”。这样一来，就导致了一个问题：MLLM的文本和视觉模块采用不同的嵌入策略，使得视觉和文本信息没办法无缝融合，限制了模型性能的进一步提升。针对这个问题，Ovis采用了视觉tokenizer+视觉嵌入表+大语言模型的架构。

二、环境搭建

1、模型下载

https://huggingface.co/AIDC-AI/Ovis1.6-Gemma2-9B/tree/main

2、环境安装

docker run -it --rm --gpus=all -v /datas/work/zzq:/workspace pytorch/pytorch:2.2.2-cuda12.1-cudnn8-devel bash

git clone https://github.com/AIDC-AI/Ovis.git

cd /workspace/Ovis/Ovis-main

pip install -r requirements.txt -i Simple Index

pip install -e .

三、推理测试

1、修改代码

from dataclasses import field, dataclass
from typing import Optional, Union, Listimport torch
from PIL import Imagefrom ovis.model.modeling_ovis import Ovis
from ovis.util.constants import IMAGE_TOKEN@dataclass
class RunnerArguments:model_path: strmax_new_tokens: int = field(default=512)do_sample: bool = field(default=False)top_p: Optional[float] = field(default=None)top_k: Optional[int] = field(default=None)temperature: Optional[float] = field(default=None)max_partition: int = field(default=9)class OvisRunner:def __init__(self, args: RunnerArguments):self.model_path = args.model_path# self.dtype = torch.bfloat16self.device = torch.cuda.current_device()# self.dtype = torch.bfloat16self.dtype = torch.float16self.model = Ovis.from_pretrained(self.model_path, torch_dtype=self.dtype, multimodal_max_length=8192)self.model = self.model.eval().to(device=self.device)self.eos_token_id = self.model.generation_config.eos_token_idself.text_tokenizer = self.model.get_text_tokenizer()self.pad_token_id = self.text_tokenizer.pad_token_idself.visual_tokenizer = self.model.get_visual_tokenizer()self.conversation_formatter = self.model.get_conversation_formatter()self.image_placeholder = IMAGE_TOKENself.max_partition = args.max_partitionself.gen_kwargs = dict(max_new_tokens=args.max_new_tokens,do_sample=args.do_sample,top_p=args.top_p,top_k=args.top_k,temperature=args.temperature,repetition_penalty=None,eos_token_id=self.eos_token_id,pad_token_id=self.pad_token_id,use_cache=True)def preprocess(self, inputs: List[Union[Image.Image, str]]):# for single image and single text inputs, ensure image aheadif len(inputs) == 2 and isinstance(inputs[0], str) and isinstance(inputs[1], Image.Image):inputs = reversed(inputs)# build queryquery = ''images = []for data in inputs:if isinstance(data, Image.Image):query += self.image_placeholder + '\n'images.append(data)elif isinstance(data, str):query += data.replace(self.image_placeholder, '')elif data is not None:raise RuntimeError(f'Invalid input type, expected `PIL.Image.Image` or `str`, but got {type(data)}')# format conversationprompt, input_ids, pixel_values = self.model.preprocess_inputs(query, images, max_partition=self.max_partition)attention_mask = torch.ne(input_ids, self.text_tokenizer.pad_token_id)input_ids = input_ids.unsqueeze(0).to(device=self.device)attention_mask = attention_mask.unsqueeze(0).to(device=self.device)if pixel_values is not None:pixel_values = [pixel_values.to(device=self.device, dtype=self.dtype)]else:pixel_values = [None]return prompt, input_ids, attention_mask, pixel_valuesdef run(self, inputs: List[Union[Image.Image, str]]):prompt, input_ids, attention_mask, pixel_values = self.preprocess(inputs)output_ids = self.model.generate(input_ids,pixel_values=pixel_values,attention_mask=attention_mask,**self.gen_kwargs)output = self.text_tokenizer.decode(output_ids[0], skip_special_tokens=True)input_token_len = input_ids.shape[1]output_token_len = output_ids.shape[1]response = dict(prompt=prompt,output=output,prompt_tokens=input_token_len,total_tokens=input_token_len + output_token_len)return responseif __name__ == '__main__':# runner_args = RunnerArguments(model_path='<model_path>')runner_args = RunnerArguments(model_path='/workspace/Ovis/Ovis-main/models')runner = OvisRunner(runner_args)# image = Image.open('<image_path>')image = Image.open('/workspace/Ovis/Ovis-main/test.png')# text = '<prompt>'text = 'solve the question in this image'response = runner.run([image, text])print(response['output'])

python ovis/serve/runner.py

好吧，显存不够