【HuggingFace Transformers】Bert Model的应用

在自然语言处理领域中，BERT 模型因其强大的文本理解能力被广泛应用于各种分类任务。具体而言，BERT 在分类任务中可以分为两种主要应用：Sequence 分类和 Token 分类。下面将分别介绍这两种应用及其实现方式。

1. Sequence分类

1.1 主要模块

• BERT 模型：提供输入序列的编码表示，尤其是 [CLS] 标记的全局表示。
• 分类器：一个简单的线性层，将编码后的表示转换为类别分布。
• 损失函数：根据任务类型动态调整损失函数类型，用于训练过程中优化模型参数。

1.2 应用场景

BertForSequenceClassification 类被设计用于各种序列分类任务，广泛应用于自然语言处理（NLP）的多个领域，如情感分析、文本分类、语义文本相似度等。通过微调 BERT 模型，可以使模型学习到与特定任务相关的特征，从而提高模型在该任务上的表现。

1.3 BertForSequenceClassification 源码解析

# -*- coding: utf-8 -*-
# @time: 2024/8/26 16:39
from typing import Optional, Tuple, Unionimport torch
from torch import nn
from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
from transformers import BertModel, BertPreTrainedModel
from transformers.modeling_outputs import SequenceClassifierOutput
from transformers.models.bert.modeling_bert import BERT_INPUTS_DOCSTRING, _CHECKPOINT_FOR_SEQUENCE_CLASSIFICATION, _CONFIG_FOR_DOC, _SEQ_CLASS_EXPECTED_OUTPUT, _SEQ_CLASS_EXPECTED_LOSS, BERT_START_DOCSTRING
from transformers.utils import add_start_docstrings, add_start_docstrings_to_model_forward, add_code_sample_docstrings@add_start_docstrings("""Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooledoutput) e.g. for GLUE tasks.""",BERT_START_DOCSTRING,
)
class BertForSequenceClassification(BertPreTrainedModel):def __init__(self, config):super().__init__(config)self.num_labels = config.num_labelsself.config = config# 加载并初始化 BERT 模型，BERT 模型的主要功能是对输入文本序列进行编码，输出包含丰富上下文信息的表示。self.bert = BertModel(config)classifier_dropout = (config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob)# 添加 Dropout 层，用于在训练时随机丢弃部分神经元，帮助防止过拟合。self.dropout = nn.Dropout(classifier_dropout)# 定义线性分类器，将 BERT 输出的 hidden_size 大小的向量转换为 num_labels 大小的类别分布。self.classifier = nn.Linear(config.hidden_size, config.num_labels)# Initialize weights and apply final processingself.post_init()@add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))@add_code_sample_docstrings(checkpoint=_CHECKPOINT_FOR_SEQUENCE_CLASSIFICATION,output_type=SequenceClassifierOutput,config_class=_CONFIG_FOR_DOC,expected_output=_SEQ_CLASS_EXPECTED_OUTPUT,expected_loss=_SEQ_CLASS_EXPECTED_LOSS,)def forward(self,input_ids: Optional[torch.Tensor] = None,attention_mask: Optional[torch.Tensor] = None,token_type_ids: Optional[torch.Tensor] = None,position_ids: Optional[torch.Tensor] = None,head_mask: Optional[torch.Tensor] = None,inputs_embeds: Optional[torch.Tensor] = None,labels: Optional[torch.Tensor] = None,output_attentions: Optional[bool] = None,output_hidden_states: Optional[bool] = None,return_dict: Optional[bool] = None,) -> Union[Tuple[torch.Tensor], SequenceClassifierOutput]:r"""labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If`config.num_labels > 1` a classification loss is computed (Cross-Entropy)."""return_dict = return_dict if return_dict is not None else self.config.use_return_dict# BertModel 对文本序列的编码输出outputs = self.bert(input_ids,attention_mask=attention_mask,token_type_ids=token_type_ids,position_ids=position_ids,head_mask=head_mask,inputs_embeds=inputs_embeds,output_attentions=output_attentions,output_hidden_states=output_hidden_states,return_dict=return_dict,)# outputs[1] 是 pooled_output，即 [CLS] 标记对应的隐藏状态。这个输出被视为整个序列的全局表示。pooled_output = outputs[1]pooled_output = self.dropout(pooled_output)logits = self.classifier(pooled_output)# 损失计算：据任务类型（回归、单标签分类、多标签分类）计算不同的损失loss = Noneif labels is not None:if self.config.problem_type is None:if self.num_labels == 1:self.config.problem_type = "regression"elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):self.config.problem_type = "single_label_classification"else:self.config.problem_type = "multi_label_classification"if self.config.problem_type == "regression":  # 回归任务: 使用均方误差损失（MSELoss）loss_fct = MSELoss()if self.num_labels == 1:loss = loss_fct(logits.squeeze(), labels.squeeze())else:loss = loss_fct(logits, labels)elif self.config.problem_type == "single_label_classification":  # 单标签分类任务: 使用交叉熵损失（CrossEntropyLoss）loss_fct = CrossEntropyLoss()loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))elif self.config.problem_type == "multi_label_classification":  # 多标签分类任务: 使用带有 Sigmoid 激活的二元交叉熵损失（BCEWithLogitsLoss）loss_fct = BCEWithLogitsLoss()loss = loss_fct(logits, labels)# 输出if not return_dict:output = (logits,) + outputs[2:]return ((loss,) + output) if loss is not None else outputreturn SequenceClassifierOutput(loss=loss,logits=logits,hidden_states=outputs.hidden_states,attentions=outputs.attentions,)

2. Token分类

2.1 主要模块

• BERT 模型：生成每个 token 的隐藏状态。
• 分类器：通过线性层对每个 token 的隐藏状态进行分类预测，得到每个 token 的类别得分。
• 损失函数：对于分类任务，通常使用交叉熵损失（CrossEntropyLoss）。

2.2 应用场景

BertForTokenClassification 主要用于标注任务（Token Classification Tasks），如命名实体识别（NER）、词性标注（POS tagging）以及其他需要对每个词或子词进行分类的任务。

• 命名实体识别 (NER)：将每个单词或子词分类为不同的实体类别（如人名、地名、组织等）
• 词性标注 (POS Tagging)：标注句子中的每个单词的词性类别（如名词、动词、形容词等）

2.3 BertForTokenClassification 源码解析

# -*- coding: utf-8 -*-
# @time: 2024/8/26 16:45
from typing import Optional, Tuple, Unionimport torch
from torch import nn
from torch.nn import CrossEntropyLoss
from transformers import BertModel, BertPreTrainedModel
from transformers.modeling_outputs import TokenClassifierOutput
from transformers.models.bert.modeling_bert import BERT_INPUTS_DOCSTRING, _CONFIG_FOR_DOC, BERT_START_DOCSTRING, _CHECKPOINT_FOR_TOKEN_CLASSIFICATION, _TOKEN_CLASS_EXPECTED_OUTPUT, _TOKEN_CLASS_EXPECTED_LOSS
from transformers.utils import add_start_docstrings, add_start_docstrings_to_model_forward, add_code_sample_docstrings@add_start_docstrings("""Bert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. forNamed-Entity-Recognition (NER) tasks.""",BERT_START_DOCSTRING,
)
class BertForTokenClassification(BertPreTrainedModel):def __init__(self, config):super().__init__(config)self.num_labels = config.num_labels# 加载并初始化 BERT 模型，BERT 模型的主要功能是对输入文本序列进行编码，输出包含丰富上下文信息的表示。self.bert = BertModel(config, add_pooling_layer=False)classifier_dropout = (config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob)# 添加 Dropout 层，用于在训练时随机丢弃部分神经元，帮助防止过拟合。self.dropout = nn.Dropout(classifier_dropout)# 定义线性分类器，将 BERT 输出的 hidden_size 大小的向量转换为 num_labels 大小的类别分布。self.classifier = nn.Linear(config.hidden_size, config.num_labels)# Initialize weights and apply final processingself.post_init()@add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))@add_code_sample_docstrings(checkpoint=_CHECKPOINT_FOR_TOKEN_CLASSIFICATION,output_type=TokenClassifierOutput,config_class=_CONFIG_FOR_DOC,expected_output=_TOKEN_CLASS_EXPECTED_OUTPUT,expected_loss=_TOKEN_CLASS_EXPECTED_LOSS,)def forward(self,input_ids: Optional[torch.Tensor] = None,attention_mask: Optional[torch.Tensor] = None,token_type_ids: Optional[torch.Tensor] = None,position_ids: Optional[torch.Tensor] = None,head_mask: Optional[torch.Tensor] = None,inputs_embeds: Optional[torch.Tensor] = None,labels: Optional[torch.Tensor] = None,output_attentions: Optional[bool] = None,output_hidden_states: Optional[bool] = None,return_dict: Optional[bool] = None,) -> Union[Tuple[torch.Tensor], TokenClassifierOutput]:r"""labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`."""return_dict = return_dict if return_dict is not None else self.config.use_return_dict# BertModel 对文本序列的编码输出outputs = self.bert(input_ids,attention_mask=attention_mask,token_type_ids=token_type_ids,position_ids=position_ids,head_mask=head_mask,inputs_embeds=inputs_embeds,output_attentions=output_attentions,output_hidden_states=output_hidden_states,return_dict=return_dict,)# outputs[0] 包含输入序列中所有token的隐藏状态sequence_output = outputs[0]sequence_output = self.dropout(sequence_output)logits = self.classifier(sequence_output)# 损失计算loss = Noneif labels is not None:loss_fct = CrossEntropyLoss()loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))# 输出if not return_dict:output = (logits,) + outputs[2:]return ((loss,) + output) if loss is not None else outputreturn TokenClassifierOutput(loss=loss,logits=logits,hidden_states=outputs.hidden_states,attentions=outputs.attentions,)

3. 两种分类对比

• 任务类型：BertForSequenceClassification 适用于句子级别的分类任务，而 BertForTokenClassification 适用于词级别的标注任务。
• 输出维度：BertForSequenceClassification 的输出是整个序列的一个分类结果，而 BertForTokenClassification 的输出是序列中每个 token 的分类结果。
• 应用场景：BertForSequenceClassification 用于文本分类任务，而 BertForTokenClassification 则用于序列标注任务。

源码地址：transformers/src/transformers/models/bert/modeling_bert.py