Sentence-BERT Sentence Embeddings using Siamese BERT-Networks

paper: https://arxiv.org/abs/1908.10084

giit: https://github.com/UKPLab/sentence-transformers/tree/master/examples/applications

1.贡献

基于bert利用孪生结构或者三胞胎结构训练，使得产生在低维空间可用的句子Embedding。对于文本匹配任务，可以离线计算句子Embedding，然后基于句子Embedding在线匹配，可实现快速高精度的匹配。

2.结构

文章提出三种结构和目标函数，三胞胎结构作者没有画图

1.Classification Objective Function

$$loss=cross-entropy(softmax(W_t(u,v,|u-v|)),y_{true})$$

2.Regression Objective Function

$$loss=MSE(cosine-sim(u, v),y_{true})$$

3.Triplet Objective Function

$$loss=max(||s_a-s_p||-||s_a-s_n||+\sigma,0)$$

$||.||$计算向量距离，$s_a$为样本本身，$s_p$为正样本，$s_n$为负样本，$\sigma$使得正样本至少比负样本距离样本近$\sigma$。

对于pooling，文章提出三种策略

1.Using the output of the CLS-token
2.computing the mean of all output vectors (MEAN_strategy)
3.computing a max-over-time of the output vectors (MAX_strategy). The default configuration is MEAN.

3.实验结果

3.1 Unsupervised STS

3.2 Supervised STS

3.3 Argument Facet Similarity

3.4 Wikipedia Sections Distinction

We use the Triplet Objective

4.代码

from sentence_bert.sentence_transformers import SentenceTransformer, util

###load model
model = SentenceTransformer(model_path)

# Single list of sentences
sentences = ['The cat sits outside',
             'A man is playing guitar',
             'I love pasta',
             'The new movie is awesome',
             'The cat plays in the garden',
             'A woman watches TV',
             'The new movie is so great',
             'Do you like pizza?']

#Compute embeddings
embeddings = model.encode(sentences, convert_to_tensor=True)

#Compute cosine-similarities for each sentence with each other sentence
cosine_scores = util.pytorch_cos_sim(embeddings, embeddings)

#Find the pairs with the highest cosine similarity scores
pairs = []
for i in range(len(cosine_scores)-1):
    for j in range(i+1, len(cosine_scores)):
        pairs.append({'index': [i, j], 'score': cosine_scores[i][j]})

#Sort scores in decreasing order
pairs = sorted(pairs, key=lambda x: x['score'], reverse=True)

for pair in pairs[0:10]:
    i, j = pair['index']
    print("{} \t\t {} \t\t Score: {:.4f}".format(sentences[i], sentences[j], pair['score']))

The new movie is awesome 		 The new movie is so great 		 Score: 0.9283
The cat sits outside 		 The cat plays in the garden 		 Score: 0.6855
I love pasta 		 Do you like pizza? 		 Score: 0.5420
I love pasta 		 The new movie is awesome 		 Score: 0.2629
I love pasta 		 The new movie is so great 		 Score: 0.2268
The new movie is awesome 		 Do you like pizza? 		 Score: 0.1885
A man is playing guitar 		 A woman watches TV 		 Score: 0.1759
The new movie is so great 		 Do you like pizza? 		 Score: 0.1615
The cat plays in the garden 		 A woman watches TV 		 Score: 0.1521
The cat sits outside 		 The new movie is awesome 		 Score: 0.1475