All Questions
Tagged with bert-language-model word-embedding
74
questions
31
votes
6
answers
40k
views
How to cluster similar sentences using BERT
For ElMo, FastText and Word2Vec, I'm averaging the word embeddings within a sentence and using HDBSCAN/KMeans clustering to group similar sentences.
A good example of the implementation can be seen ...
- 1,123
16
votes
2
answers
31k
views
Download pre-trained sentence-transformers model locally
I am using the SentenceTransformers library (here: https://pypi.org/project/sentence-transformers/#pretrained-models) for creating embeddings of sentences using the pre-trained model bert-base-nli-...
- 311
12
votes
1
answer
8k
views
What is the difference between Sentence Encodings and Contextualized Word Embeddings?
I have seen both terms used while reading papers about BERT and ELMo so I wonder if there is a difference between them.
- 133
9
votes
2
answers
12k
views
How to find the closest word to a vector using BERT
I am trying to get textual representation(or the closest word) of given word embedding using BERT. Basically I am trying to get similar functionality as in gensim:
>>> your_word_vector = ...
- 2,134
9
votes
1
answer
9k
views
BERT document embedding
I am trying to do document embedding using BERT. The code I use is a combination of two sources. I use BERT Document Classification Tutorial with Code, and BERT Word Embeddings Tutorial. Below is the ...
- 1,159
8
votes
1
answer
14k
views
How to store Word vector Embeddings?
I am using BERT Word Embeddings for sentence classification task with 3 labels. I am using Google Colab for coding. My problem is, since I will have to execute the embedding part every time I restart ...
- 1,107
6
votes
2
answers
7k
views
Latest Pre-trained Multilingual Word Embedding
Are there any latest pre-trained multilingual word embeddings (multiple languages are jointly mapped to a same vector space)?
I have looked at the following but they don't fit my needs:
FastText / ...
- 423
5
votes
1
answer
12k
views
How to get cosine similarity of word embedding from BERT model
I was interesting in how to get the similarity of word embedding in different sentences from BERT model (actually, that means words have different meanings in different scenarios).
For example:
sent1 =...
- 143
4
votes
1
answer
602
views
Training SVM classifier (word embeddings vs. sentence embeddings)
I want to experiment with different embeddings such Word2Vec, ELMo, and BERT but I'm a little confused about whether to use the word embeddings or sentence embeddings, and why. I'm using the ...
- 125
3
votes
1
answer
9k
views
resize_token_embeddings on the a pertrained model with different embedding size
I would like to ask about the way to change the embedding size of the trained model.
I have a trained model models/BERT-pretrain-1-step-5000.pkl.
Now I am adding a new token [TRA]to the tokeniser and ...
- 61
3
votes
1
answer
3k
views
How to combine embeddins vectors of bert with other features?
I am working on a classification task with 3 labels (0,1,2 = neg, pos, neu). Data are sentences. So to produce vectors/embeddings of sentences, I use a Bert encoder to get embeddings for each sentence ...
- 323
3
votes
1
answer
3k
views
How to use BERT pretrain embeddings with my own new dataset?
My dataset and NLP task is very different from the large corpus what authors have pre-trained their model (https://github.com/google-research/bert#pre-training-with-bert), so I can't directly fine-...
- 31
3
votes
1
answer
5k
views
How to save sentence-Bert output vectors to a file?
I am using Bert to get similarity between multi term words.here is my code that I used for embedding :
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('bert-large-...
- 314
3
votes
1
answer
1k
views
How do you get single embedding vector for each word (token) from RoBERTa?
As you may know, RoBERTa (BERT, etc.) has its own tokenizer and sometimes you get pieces of given word as tokens, e.g. embeddings » embed, #dings
Since the nature of the task I am working on, I need a ...
- 71
3
votes
0
answers
710
views
Google BERT and antonym detection
I recently learned about the following phenomenon: Google BERT word embeddings of well-known state-of-the-art models seem to ignore the measure of semantical contrast between antonyms in terms of the ...
- 555
3
votes
0
answers
150
views
What are some techniques to improve contextual accuracy of semantic search engine using BERT?
I am implementing a semantic search engine using BERT (using cosine distance) To a certain extend the method is able to find out sentences in a high level context. However when it comes narrowed down ...
- 187
2
votes
1
answer
752
views
ReadError: file could not be opened successfully. But I am not sure where the tar file is stored to resolve this
I am using biobert-embeddings==0.1.2 and torch==1.2.0 versions to embed some documents. But, I get the following error when I try to load the model by
from biobert_embedding.embedding import ...
- 21
2
votes
1
answer
4k
views
Compare cosine similarity of word with BERT model
Hi I am looking to generate similar words for a word using BERT model, the same approach we use in gensim to generate most_similar word, I found the approach as:
from transformers import BertTokenizer,...
- 467
2
votes
0
answers
95
views
use BERT word to vector embedding only on word, not sentence
How to use BERT word to vector embedding only on word, not sentence?
I have list of nouns and I need vector version of these words using BERT. I researched a lot on how to do it, but I could only ...
- 21
2
votes
0
answers
140
views
Bert Embedding element #308
I've been getting some practice with Bert Embedding. Specifically, I'm using BERT-Base Uncased model with the PyTorch libraries
import torch
from pytorch_pretrained_bert import BertTokenizer, ...
- 1,848
1
vote
1
answer
8k
views
Fine tuning of Bert word embeddings
I would like to load a pre-trained Bert model and to fine-tune it and particularly the word embeddings of the model using a custom dataset.
The task is to use the word embeddings of chosen words for ...
- 2,067
1
vote
2
answers
1k
views
Bert model output interpretation
I searched a lot for this but havent still got a clear idea so I hope you can help me out:
I am trying to translate german texts to english! I udes this code:
tokenizer = AutoTokenizer....
- 535
1
vote
1
answer
1k
views
Accuracy of fine-tuning BERT varied significantly based on epochs for intent classification task
I used Bert base uncased as embedding and doing simple cosine similarity for intent classification in my dataset (around 400 classes and 2200 utterances, train:test=80:20). The base BERT model ...
- 79
1
vote
1
answer
2k
views
why nn.Embedding layer is used for positional encoding in bert?
In the huggingface implementation of bert model, for positional embedding nn.Embedding is used.
Why it is used instead of traditional sin/cos positional embedding described in the transformer paper? ...
- 25
1
vote
1
answer
650
views
Calculate cosine similarity between 2 words using BERT
I am trying to calculate the cosine similarity beteween two given words using BERT, but I am getting an error which says:
IndexError: Dimension out of range (expected to be in range of [-1, 0], but ...
- 29
1
vote
1
answer
3k
views
How to combine different embedding's generated from different algorithms like from Word2vec, GLOVE , BERT?
I want to know about the best way to combine the different embeddings that I generated from different algorithms like word2vec, GLOVE, or BERT to generate the final one.
1
vote
1
answer
235
views
How to use my own corpus on word embedding model BERT
I am trying to create a question-answering model with the word embedding model BERT from google. I am new to this and would really want to use my own corpus for the training. At first I used an ...
- 421
1
vote
1
answer
270
views
What does the embedding elements stand for in huggingFace bert model?
Prior to passing my tokens through encoder in BERT model, I would like to perform some processing on their embeddings. I extracted the embedding weight using:
from transformers import TFBertModel
# ...
- 13
1
vote
0
answers
496
views
BERT embeddings + LSTM for NER
I am working with the Conll-2003 dataset for Named Entity Recognition. What I want to do is to use the BERT embeddings as an input to a simple LSTM. Here's the code:
class Model(nn.Module):
def ...
- 31
1
vote
0
answers
3k
views
How to pop elements from a tensor in Pytorch?
I want to drop/pop elements from a tensor in Pytorch, something similar to pop operation in python. In the following code , if the condition is met, it removes two elements from the array, current and ...
- 145
1
vote
0
answers
884
views
How to calculate word similarity based on transformer?
I know I can train word embedding in Tensorflow or Gensim, then I can retrieve top N most similar words for a target word. Given that transformer is now the main stream model for text representation, ...
- 6,847
1
vote
0
answers
278
views
BERT: how to batch vectorize efficiently?
I am trying to convert the sentence into vector using BERT.
def bert_embedding(text):
# text: list of strings(sentences)
vector = []
for sentence in tqdm(text):
e = bert_tokenizer....
- 819
1
vote
0
answers
143
views
Is there an embeddings technique to represent multilingual paragraphs?
I have a dataset that includes English, Spanish and German documents. I want to represent them using document embeddings techniques to compute their similarities. However, as the documents are in ...
- 635
1
vote
0
answers
128
views
Tensorflow input for a series of (1, 512) tensors
I have a pandas dataset with a column of tensors of shape TensorShape([1, 512]) which are a result of tf.hub Bert embeddings. I know I can use an embedding layer directly in tensorflow, but is there a ...
- 11
1
vote
1
answer
557
views
'list' object has no attribute 'shape
I am passing an embedding matrix to the embedding layer in Keras
model = Sequential()
model.add(Embedding(max_words, 30, input_length=max_len, weights=[all]))
model.add(BatchNormalization())
model.add(...
- 1,107
1
vote
0
answers
103
views
BERT word embbeding on a simple dataset
I want to learn how can I use BERt for word embbeding. I tried this code:
from bert_serving.client import BertClient
bc = BertClient()
But the problem is that it took a lot and I finally couldn't run ...
- 122
1
vote
0
answers
325
views
Using BERT embeddings for Seq2Seq model building
Earlier I've used Glove embedding to build the seq2seq model for text summarization, Now I want to change the Glove with BERT to see the performance of the model. For this, I used the bert-as-service ...
1
vote
1
answer
3k
views
Comparison among ELMo, BERT, and GloVe
What are the differences among ELMo, BERT, and GloVe in word representation? How differently do they perform word embedding tasks? Which one is better and what advantages and disadvantages does each ...
user14251114
0
votes
1
answer
838
views
Extracting word features from BERT model
So as you know, we can extract BERT features of word in a sentence. My question is, can we also extract word features that are not included in a sentence? For example, bert features of single words ...
- 1,471
0
votes
1
answer
1k
views
How to improve code to speed up word embedding with transformer models?
I need to compute words embeddings for a bunch of documents with different language models.
No problem with that, the script is doing fine, except I'm working on a notebook, without GPU and each text ...
- 890
0
votes
1
answer
702
views
BERT without positional embeddings
I am trying to build a pipeline in HuggingFace which will not use the positional embeddings in BERT, in order to study the role of the embeddings for a particular use case. I have looked through the ...
- 65
0
votes
1
answer
294
views
BERT embeddings for entire sentences vs. verbs
First off, I am drawing upon assumption that majority of the semantic value of the sentence is mediated by verbs that connect the subject and the object of said verb. I am aware that I simplify a bit ...
- 488
0
votes
1
answer
868
views
Calculating semantic coherence in a given speech transcript
I am trying to calculate the semantic coherence in a given paragraph/transcript, ie. if somebody goes off track while talking about a thing or topic - more specifically describing a picture (the ...
- 252
0
votes
1
answer
13
views
the key did not present in Word2vec
I am confronted some problem when I use the pretrained_model: w2v_512.model.
the error is "Key 'xxx' is not present"
I think this may the word of 'xxx' can not be convert to embedding from ...
- 1
0
votes
0
answers
69
views
Train new Word Embedding for mBART
TL;DR: I want to train a (set of) new word embedding(s) for mBART instead of training it for BERT—how do I do that?
Background:
I found an interesting code here: https://github.com/tai314159/PWIBM-...
- 226
0
votes
1
answer
311
views
How to get Attentions Part from the output of a Bert model?
I am using Bert-Model for Query Expansion and I am trying to extract the keywords from the Document I have
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel....
0
votes
1
answer
64
views
Why does model.fit() function give error?
I generated embedding matrix from BERT embeddings as below:
# Load pre-trained model tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
model = BertModel....
- 909
0
votes
1
answer
720
views
BERT word embeddings
I'm trying to use BERT in a static word embeddings kind of way to compare to Word2Vec and show the differences and how BERT is not really meant to be used in a contextless manner.
This is how (based ...
- 181
0
votes
1
answer
365
views
Does NLP Transformer has backpropagation and how BERT has its word embedding?
I was reading Attention all you need papers and i have not get any idea how the weights are updated in the Transformer base architecture is there any Backpropagation ? normally yes for the model to ...
- 11
0
votes
1
answer
36
views
How to model with NLP when the token is not relevant (by itself) but its type is?
I would like to build an NLP classification model.
My input is a paragraph or a sentence. Ideally, my output is a score or probability (between 0 and 1).
I have defined specific entities ex-ante, each ...
- 25