All Questions
Tagged with bert-language-model embedding
42
questions
9
votes
1
answer
9k
views
BERT document embedding
I am trying to do document embedding using BERT. The code I use is a combination of two sources. I use BERT Document Classification Tutorial with Code, and BERT Word Embeddings Tutorial. Below is the ...
- 1,159
6
votes
1
answer
9k
views
Using BERT Embeddings in Keras Embedding layer
I want to use the BERT Word Vector Embeddings in the Embeddings layer of LSTM instead of the usual default embedding layer. Is there any way I can do it?
- 1,107
5
votes
3
answers
1k
views
BERT token vs. embedding
I understand that WordPiece is used to break text into tokens. And I understand that, somewhere in BERT, the model maps tokens into token embeddings that represent the meaning of the tokens. But ...
- 61
3
votes
2
answers
2k
views
Where can I get the pretrained word embeddinngs for BERT?
I know that BERT has total vocabulary size of 30522 which contains some words and subwords. I want to get the initial input embeddings of BERT. So, my requirement is to get the table of size [30522, ...
- 346
3
votes
1
answer
2k
views
Using Sentence-Bert with other features in scikit-learn
I have a dataset, one feature is text and 4 more features. Sentence-Bert vectorizer transforms text data into tensors. I can use these sparse matrices directly with a machine learning classifier. Can ...
- 45
3
votes
1
answer
5k
views
How to save sentence-Bert output vectors to a file?
I am using Bert to get similarity between multi term words.here is my code that I used for embedding :
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('bert-large-...
- 314
3
votes
0
answers
3k
views
NLP - Best document embedding library
Good day, fellow humans (?).
I have a methodological question that is confused by a deep research in a tiny amount of time.
The question arises from the following problem(s): I need to apply semi-...
- 159
3
votes
0
answers
372
views
Does ktrain package combine input embedding with bert embedding when used for test classification?
I am running the code given in the link below. What embeddings does the ktrain package of python use for bert text classification. I believe the code is using a pre-trained model of Bert. In that is ...
- 31
2
votes
2
answers
1k
views
keras LSTM get hidden-state (converting sentece-sequence to document context vectors)
Im trying to create document context vectors from sentence-vectors via LSTM using keras (so each document consist of a sequence of sentence vectors).
My goal is to replicate the following blog post ...
- 323
2
votes
1
answer
2k
views
cuda out of memory problem in gpu in google colab
I am trying to run code to get stacked embedding from flair and bert and I am getting the following error. one of the suggestion was to reduce the batch size, but how to pass the data in batches? here ...
- 39
2
votes
0
answers
195
views
Can we use bert on an image with embeddings created with the help of CNN architecture?
I am trying to use bert for the images, the following steps I'm considering to do this approach:
Create an embedding of an image using VggNet (extracting avgpool layer from the network).
Using PCA ...
- 167
1
vote
1
answer
259
views
Understanding the results of Vespa BERT embeddings
I am copying parts of the Simple Semantic Search sample application at https://github.com/vespa-engine/sample-apps/tree/master/simple-semantic-search to get started with dense vector search.
I have ...
- 101
1
vote
1
answer
3k
views
Max position embedding in BERT
I'm studying BERT right now.
I thought BERT limits position embedding as 512 because of the memory problem.
However, when I look up the BERT code in hugging face I found this parameter on config.
...
- 11
1
vote
1
answer
931
views
Why can Bert's three embeddings be added?
I already know the meaning of Token Embedding, Segment Embedding, and Position Embedding. But why can these three vectors be added together? The Size and direction of vectors will change after the ...
- 13
1
vote
2
answers
93
views
Calculating embedding overload problems with BERT
I'm trying to calculate the embedding of a sentence using BERT. After I input the sentence into BERT, I calculate the Mean-pooling, which is used as the embedding of the sentence.
Problem
My code can ...
- 31
1
vote
1
answer
514
views
Is it normal that model output slightly different on different platforms?
I am using Huggingface to generate bert embedding for text, but they are slightly different for the same text on my Mac and Linux platforms. For examples, one pair of results:
Mac
[0.9832047820091248, ...
- 6,847
1
vote
1
answer
292
views
How to store Bert embeddings in cassandra
I want to use Cassandra as feature store to store precomputed Bert embedding,
Each row would consist of roughly 800 integers (ex. -0.18294132) Should I store all 800 in one large string column or 800 ...
- 371
1
vote
2
answers
2k
views
Get feature vectors from BertForSequenceClassification
I have successfully build a sentiment analysis tool with BertForSequenceClassification from huggingface/transformers to classify $tsla tweets as positive or negative.
However, I can't find out how I ...
- 19
1
vote
1
answer
623
views
Do BERT word embeddings change depending on context?
Before answering "yes, of course", let me clarify what I mean:
After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
1
vote
1
answer
127
views
How to know which contextual embedding to use at test time
Models like BERT generate contextual embeddings for words with different contextual meanings, like 'bank', 'left'.
I don't understand which contextual embedding the model chooses to use at test time? ...
- 11
1
vote
1
answer
545
views
UnparsedFlagAccessError: Trying to access flag
I'm a beginner with BERT and i'm trying to run the code that the developers put as example. Unfortunately, when i'm in the cell that use the bert tokenizer i had this error
----------------------------...
- 101
1
vote
0
answers
121
views
BadZipFile File is not a zip file
In the following code, i get bad zip file error.
ctx = mx.gpu(0)
bert = BertEmbedding(ctx=ctx)
The output of the code is
Downloading C:\Users\USER\.mxnet\models\...
- 91
1
vote
0
answers
759
views
how to resolve errors in bert embedding installation
import mxnet as mx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\USER\anaconda3\envs\gpu3\lib\site-packages\mxnet\__init__.py", ...
- 91
1
vote
1
answer
624
views
Sentence embeddings BERT
I need an info.
I used this: https://towardsdatascience.com/improving-sentence-embeddings-with-bert-and-representation-learning-dfba6b444f6b to extract features but I got word embeddings.
If I want ...
- 39
1
vote
0
answers
276
views
How to generate embeddings using Bert
I started using the following Kaggle kernel:
https://www.kaggle.com/taindow/bert-a-fine-tuning-example
After, I have used the following code which is getting close:
bert_config = modeling....
- 2,171
0
votes
1
answer
3k
views
How to extract Sentence Embedding Using BERT model from [CLS] token
I am following this link:
BERT document embedding
I want to extract sentence-embedding using BERT model using CLS token. Here is the code:
import torch
from keras.preprocessing.sequence import ...
- 1,455
0
votes
1
answer
4k
views
Pytorch Loss Function for making embeddings similar
I am working on an embedding model, where there is a BERT model, which takes in text inputs and output a multidimensional vector. The goal of the model is to find similar embeddings (high cosine ...
- 23
0
votes
1
answer
2k
views
About Bert embedding (input_ids, input_mask)
As far as I understand it, in Bert's operating logic, he changes 50% of his sentences that he takes as input. It doesn't touch the rest.
1-) Is the changed part the transaction made with tokenizer....
- 47
0
votes
0
answers
27
views
How to evaluate the performance of sentence embedding models against benchmark dataset
I am relatively new to this field and would like guidance on how to effectively test an embedding model using a benchmark dataset. Specifically, I have acquired a few embedding models related to ...
0
votes
1
answer
58
views
Can Kmeans Clustering using cosine distance in sklearn?
I want to clustering my document using BERT embedding from Sentence Transoformer especially bert-base-nli-mean tokens, and i want to cluster that embedding with kmeans clustering but i have a problem, ...
- 1
0
votes
0
answers
84
views
Memory-efficient BERT Text Embedding for Large Dataset Preprocessing in TensorFlow
I'm working with a dataset containing approximately 920,614 rows and multiple columns, including "orig_item_title," "sub_item_title," "is_brand_same," and "...
0
votes
0
answers
112
views
KeyError: 'bert.embeddings.LayerNorm.weight'
Im running this code in Jetson Nano developer kit. Im getting following error.
Error log:
File "intentc.py", line 89, in
model = BertClassifier(h_size_bert=768, h_size_classifier=50, ...
0
votes
0
answers
23
views
Getting the Embedding of Text in a Dataframe
So I have a dataframe which contains 5 articles. Each article has been tokenized sentences using nltk. Here's an example of the article.
Example Article
I want to get the embedding of each sentence in ...
0
votes
0
answers
140
views
Using Bert for long sentence
I have a document which consist 200 rows(sentences). Each sentences has more than 512 tokens. I want to turn my whole document into a vector using Bert. But I am getting errors.
I have tried this,
...
- 1
0
votes
0
answers
90
views
Where did the Transformer embedding numbers come from?
I'm a student studying Transformer. I want to ask, when I will vectorize words with Transformer BERT and get 768 vector dimensions for each word, I'm confused about where these numbers come from, is ...
0
votes
1
answer
626
views
Can I feed categorical data in Keras embedding layer without encoding the data?
I am trying to feed multicolumn categorical data into Keras embedding layer. Can I feed categorical data in Keras embedding layer without encoding ?
If not then which encoding method is preferable to ...
0
votes
1
answer
621
views
Combining BERT and other types of embeddings
The flair model can give a representation of any word (it can handle the OOV problem), while the BERT model splits the unknown word into several sub-words.
For example, the word "hjik" will ...
0
votes
1
answer
657
views
How to concatenate new vectors into existing Bert vector?
For a sentence,I may extract a few entities and each of the entities is embedded with 256 dimension vectors. Then I compute an average for these entities to be a single vector to represent these ...
- 6,847
0
votes
1
answer
344
views
How can I train a bert model for representational learning task that is domain specific?
I am trying to generate good sentence embeddings for some specific type od texts, using sentence transformer models while testing the the similarity and clustering using kmeans doesnt give good ...
- 1
0
votes
0
answers
186
views
problem in training RNN using bert embeddings
I have been working with bert embedding using a neural network model for the sentiment classification task. during model fit it's giving indices error, and I am still new to this so could not able to ...
- 39
-1
votes
1
answer
793
views
Should feature embeddings be taken before or after dropout layer in neural network?
I am training a binary text classification model using BERT as follows:
def create_model():
text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
preprocessed_text = ...
- 3,247
-2
votes
1
answer
348
views
extract_features sentence embedding BERT
I'm using this code to get the embeddings of sentences that are in my dataset(I'm using my pretrained model).
`python extract_features.py \
--input_file=/tmp/input.txt \
--output_file=/tmp/...
- 39