我正在嘗試SentenceTransformer來自SBERT.net的模型，我想知道它如何處理物體名稱。它們是否被標記為未知 - 它們是否被標記分解等。我想確保它們在比較中被使用。
但是，要做到這一點，我需要查看它為查詢構建的詞匯——甚至可能將嵌入轉換為文本。
查看 api - 我不知道如何做到這一點。
這是他們檔案中的一個簡單示例：
embedder = SentenceTransformer("all-MiniLM-L6-v2")
corpus = [
"A man is eating food.",
"A man is eating a piece of bread.",
"The girl is carrying a baby."
]
corpus_embeddings = embedder.encode(corpus, convert_to_tensor=True)
# Query sentences:
queries = [
"A man is eating pasta."
]
top_k = min(5, len(corpus))
for query in queries:
query_embedding = embedder.encode(query, convert_to_tensor=True)
...
uj5u.com熱心網友回復：
您的SentenceTransformer模型實際上正在打包并使用 Hugging Facetransformers庫中的標記器。您可以將其作為.tokenizer模型的屬性進行訪問。這種記號的典型行為是分解詞片記號中的未知記號。在這一點上，我們可以繼續檢查它是否確實是它所做的，因為它相對簡單：
embedder = SentenceTransformer("all-MiniLM-L6-v2")
corpus = [
"A man is eating food.",
"A man is eating a piece of bread.",
"The girl is carrying a baby."
]
# the tokenizer is just here:
tokenizer = embedder.tokenizer # BertTokenizerFast
# and the vocabulary itself is there, if needed:
vocab = tokenizer.vocab # dict of length 30522
# get the split of sentences according to the vocab, for example:
inputs = tokenizer(corpus, padding='longest', truncation=True)
tokens = [e.tokens for e in inputs.encodings]
# tokens contains:
# [
# ['[CLS]', 'a', 'man', 'is', 'eating', 'food', '.', '[SEP]', '[PAD]', '[PAD]', '[PAD]']
# ['[CLS]', 'a', 'man', 'is', 'eating', 'a', 'piece', 'of', 'bread', '.', '[SEP]']
# ['[CLS]', 'the', 'girl', 'is', 'carrying', 'a', 'baby', '.', '[SEP]', '[PAD]', '[PAD]']
# ]
# now let's try with some unknown tokens and see what it does
queries = [
"Edv Beq is eating pasta."
]
q_inputs = tokenizer(queries, padding='longest', truncation=True)
q_tokens = [e.tokens for e in q_inputs.encodings]
# q_tokens contains:
# [
# ['[CLS]', 'ed', '##v', 'be', '##q', 'is', 'eating', 'pasta', '.', '[SEP]']
# ]
轉載請註明出處，本文鏈接：https://www.uj5u.com/net/491855.html
下一篇：在Makefile中決議數字