All Questions
Tagged with bert-language-model python
807
questions
50
votes
10
answers
125k
views
CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)
I got the following error when I ran my PyTorch deep learning model in Google Colab
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
1370 ret = ...
- 971
31
votes
6
answers
40k
views
How to cluster similar sentences using BERT
For ElMo, FastText and Word2Vec, I'm averaging the word embeddings within a sentence and using HDBSCAN/KMeans clustering to group similar sentences.
A good example of the implementation can be seen ...
- 1,123
24
votes
1
answer
49k
views
How does max_length, padding and truncation arguments work in HuggingFace' BertTokenizerFast.from_pretrained('bert-base-uncased')?
I am working with Text Classification problem where I want to use the BERT model as the base followed by Dense layers. I want to know how does the 3 arguments work? For example, if I have 3 sentences ...
- 3,872
24
votes
1
answer
62k
views
PyTorch BERT TypeError: forward() got an unexpected keyword argument 'labels'
Training a BERT model using PyTorch transformers (following the tutorial here).
Following statement in the tutorial
loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=...
- 1,826
23
votes
3
answers
21k
views
Cased VS uncased BERT models in spacy and train data
I want to use spacy's pretrained BERT model for text classification but I'm a little confused about cased/uncased models. I read somewhere that cased models should only be used when there is a chance ...
- 1,039
22
votes
6
answers
27k
views
AttributeError: module 'torch' has no attribute '_six'. Bert model in Pytorch
I tried to load pre-trained model by using BertModel class in pytorch.
I have _six.py under torch, but it still shows module 'torch' has no attribute '_six'
import torch
from pytorch_pretrained_bert ...
- 221
21
votes
1
answer
30k
views
PyTorch: RuntimeError: Input, output and indices must be on the current device
I am running a BERT model on torch. It's a multi-class sentiment classification task with about 30,000 rows. I have already put everything on cuda, but not sure why I'm getting the following run time ...
- 984
19
votes
5
answers
68k
views
Pytorch: IndexError: index out of range in self. How to solve?
This training code is based on the run_glue.py script found here:
# Set the seed value all over the place to make this reproducible.
seed_val = 42
random.seed(seed_val)
np.random.seed(seed_val)
torch....
- 243
18
votes
1
answer
12k
views
BertForSequenceClassification vs. BertForMultipleChoice for sentence multi-class classification
I'm working on a text classification problem (e.g. sentiment analysis), where I need to classify a text string into one of five classes.
I just started using the Huggingface Transformer package and ...
- 39.8k
17
votes
2
answers
33k
views
The size of tensor a (707) must match the size of tensor b (512) at non-singleton dimension 1
I am trying to do text classification using pretrained BERT model. I trained the model on my dataset, and in the phase of testing; I know that BERT can only take to 512 tokens, so I wrote if condition ...
- 1,561
17
votes
5
answers
67k
views
Transformer: Error importing packages. "ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler'"
I am working on a machine learning project on Google Colab, it seems recently there is an issue when trying to import packages from transformers. The error message says:
ImportError: cannot import ...
- 231
15
votes
3
answers
35k
views
Python: BERT Error - Some weights of the model checkpoint at were not used when initializing BertModel
I am creating an entity extraction model in PyTorch using bert-base-uncased but when I try to run the model I get this error:
Error:
Some weights of the model checkpoint at D:\Transformers\bert-entity-...
- 917
15
votes
6
answers
40k
views
With BERT Text Classification, ValueError: too many dimensions 'str' error occuring
Trying to make a classifier for sentiments of texts with BERT model but getting ValueError : too many dimensions 'str'
That is the DataFrame for values of train data; so they are train_labels
0 notr
...
- 151
14
votes
1
answer
14k
views
PyTorch torch.no_grad() versus requires_grad=False
I'm following a PyTorch tutorial which uses the BERT NLP model (feature extractor) from the Huggingface Transformers library. There are two pieces of interrelated code for gradient updates that I don'...
- 39.8k
12
votes
3
answers
37k
views
OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index']
When I load the BERT pretrained model online I get this error OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] found in directory uncased_L-12_H-768_A-12 or '...
- 189
12
votes
2
answers
5k
views
Get probability of multi-token word in MASK position
It is relatively easy to get a token's probability according to a language model, as the snippet below shows. You can get the output of a model, restrict yourself to the output of the masked token, ...
- 27.7k
11
votes
3
answers
15k
views
Transformers pretrained model with dropout setting
I'm trying to use transformer's huggingface pretrained model bert-base-uncased, but I want to increace dropout. There isn't any mention to this in from_pretrained method, but colab ran the object ...
- 685
11
votes
2
answers
3k
views
Removing SEP token in Bert for text classification
Given a sentiment classification dataset, I want to fine-tune Bert.
As you know that BERT created to predict the next sentence given the current sentence. Thus, to make the network aware of this, ...
- 5,327
10
votes
3
answers
12k
views
BertTokenizer - when encoding and decoding sequences extra spaces appear
When using Transformers from HuggingFace I am facing a problem with the encoding and decoding method.
I have a the following string:
test_string = 'text with percentage%'
Then I am running the ...
- 1,058
9
votes
1
answer
23k
views
RuntimeError: The size of tensor a (4000) must match the size of tensor b (512) at non-singleton dimension 1
I'm trying to build a model for document classification. I'm using BERT with PyTorch.
I got the bert model with below code.
bert = AutoModel.from_pretrained('bert-base-uncased')
This is the code for ...
9
votes
1
answer
24k
views
BERT tokenizer & model download
I`m beginner.. I'm working with Bert. However, due to the security of the company network, the following code does not receive the bert model directly.
tokenizer = BertTokenizer.from_pretrained('bert-...
- 575
9
votes
2
answers
12k
views
Outputting attention for bert-base-uncased with huggingface/transformers (torch)
I was following a paper on BERT-based lexical substitution (specifically trying to implement equation (2) - if someone has already implemented the whole paper that would also be great). Thus, I wanted ...
- 674
9
votes
4
answers
32k
views
How to resolve ERROR: Could not build wheels for hdbscan, which is required to install pyproject.toml-based projects
I am trying to install bertopic and I got this error:
pip install bertopic
Collecting bertopic
> Using cached bertopic-0.11.0-py2.py3-none-any.whl (76 kB)
> Collecting ...
- 97
9
votes
1
answer
7k
views
Clause extraction / long sentence segmentation in python
I'm currently working on a project involving sentence vectors (from a RoBERTa pretrained model). These vectors are lower quality when sentences are long, and my corpus contains many long sentences ...
- 483
9
votes
1
answer
4k
views
BERT embedding for semantic similarity
I earlier posted this question. I wanted to get embedding similar to this youtube video, time 33 minutes onward.
1) I dont think that the embedding that i am getting from CLS token are similar to ...
- 6,258
8
votes
4
answers
16k
views
Error importing BERT: module 'tensorflow._api.v2.train' has no attribute 'Optimizer'
I tried to use bert-tensorflow in Google Colab, but I got the following error:
--------------------------------------------------------------------------- AttributeError ...
- 615
8
votes
1
answer
3k
views
HuggingFace BERT `inputs_embeds` giving unexpected result
The HuggingFace BERT TensorFlow implementation allows us to feed in a precomputed embedding in place of the embedding lookup that is native to BERT. This is done using the model's call method's ...
- 1,174
7
votes
1
answer
5k
views
How exactly should the input file be formatted for the language model finetuning (BERT through Huggingface Transformers)?
I wanted to employ the examples/run_lm_finetuning.py from the Huggingface Transformers repository on a pretrained Bert model. However, from following the documentation it is not evident how a corpus ...
- 79
7
votes
3
answers
19k
views
Why can't I import functions in bert after pip install bert
I am a beginner for bert, and I am trying to use files of bert given on the GitHub:https://github.com/google-research/bert
However I cannot import files(such as run_classifier, optimisation and so on)...
- 352
7
votes
1
answer
2k
views
Fine-tune Bert for specific domain (unsupervised)
I want to fine-tune BERT on texts that are related to a specific domain (in my case related to engineering). The training should be unsupervised since I don't have any labels or anything. Is this ...
- 1,036
7
votes
1
answer
8k
views
Mismatched size on BertForSequenceClassification from Transformers and multiclass problem
I just trained a BERT model on a Dataset composed by products and labels (departments) for an e-commerce website. It's a multiclass problem. I used BertForSequenceClassification to predict the ...
7
votes
1
answer
2k
views
Use BERT under spaCy to get sentence embeddings
I am trying to use BERT to get sentence embeddings. Here is how I am doing it:
import spacy
nlp = spacy.load("en_core_web_trf")
nlp("The quick brown fox jumps over the lazy dog")....
- 1,065
7
votes
1
answer
8k
views
How to specify a proxy in transformers pipeline
I am using sentiment-analysis pipeline as described here.
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
It's failing with a connection error message
ValueError: ...
- 194
6
votes
2
answers
9k
views
How to untokenize BERT tokens?
I have a sentence and I need to return the text corresponding to N BERT tokens to the left and right of a specific word.
from transformers import BertTokenizer
tz = BertTokenizer.from_pretrained("...
- 183
6
votes
2
answers
11k
views
BERT get sentence embedding
I am replicating code from this page. I have downloaded the BERT model to my local system and getting sentence embedding.
I have around 500,000 sentences for which I need sentence embedding and it is ...
- 6,258
6
votes
2
answers
3k
views
ImportError when from transformers import BertTokenizer
My code is:
import torch
from transformers import BertTokenizer
from IPython.display import clear_output
I got error in line from transformers import BertTokenizer:
ImportError: /lib/x86_64-linux-gnu/...
- 61
6
votes
3
answers
5k
views
How to stop BERT from breaking apart specific words into word-piece
I am using a pre-trained BERT model to tokenize a text into meaningful tokens. However, the text has many specific words and I don't want BERT model to break them into word-pieces. Is there any ...
- 517
6
votes
2
answers
3k
views
How to test masked language model after training it?
I have followed this tutorial for masked language modelling from Hugging Face using BERT, but I am unsure how to actually deploy the model.
Tutorial: https://github.com/huggingface/notebooks/blob/...
user14946125
6
votes
2
answers
3k
views
How to fix random seed for BERTopic?
I'd like to fix the random seed from BERTopic library to get reproducible results. Looking at the code of BERTopic I see it uses numpy. Will using np.random.seed(123) be enough? or do I also need to ...
- 998
6
votes
1
answer
34k
views
Pytorch expects each tensor to be equal size
When running this code: embedding_matrix = torch.stack(embeddings)
I got this error:
RuntimeError: stack expects each tensor to be equal size, but got [7, 768] at entry 0 and [8, 768] at entry 1
I'm ...
- 79
6
votes
1
answer
3k
views
Bert Embedding Layer raises `Type Error: unsupported operand type(s) for +: 'None Type' and 'int'` with BiLSTM
I've problems integrating Bert Embedding Layer in a BiLSTM model for word sense disambiguation task,
Windows 10
Python 3.6.4
TenorFlow 1.12
Keras 2.2.4
No virtual environments were used
PyCharm ...
- 319
6
votes
1
answer
5k
views
Using BERT to generate similar word or synonyms through word embeddings
As we all know the capability of BERT model for word embedding, it is probably better than the word2vec and any other models.
I want to create a model on BERT word embedding to generate synonyms or ...
- 467
6
votes
1
answer
939
views
How to predict the probability of an empty string using BERT
Suppose we have a template sentence like this:
"The ____ house is our meeting place."
and we have a list of adjectives to fill in the blank, e.g.:
"yellow"
"large"
&...
- 1,474
6
votes
3
answers
5k
views
Why aren't transformers imported in Python?
I want to import transformers in jupyter notebook but I get the following error. What is the reason for this error? My Python version is 3.8
ImportError: cannot import name 'TypeAlias' from '...
- 101
6
votes
3
answers
4k
views
TypeError: Layer input_spec must be an instance of InputSpec. Got: InputSpec(shape=(None, 128, 768), ndim=3)
I am trying to use a BERT pretrained model to do a multiclass classification (of 3 classes). Here's my function to use the model and also added some extra functionalities:
def create_model(max_seq_len,...
6
votes
0
answers
6k
views
How to add index to python FAISS incrementally
I am using Faiss to index my huge dataset embeddings, embedding generated from bert model. I want to add the embeddings incrementally, it is working fine if I only add it with faiss.IndexFlatL2 , but ...
- 467
6
votes
0
answers
2k
views
How to slice string depending on length of tokens
When I use (with a long test_text and short question):
from transformers import BertTokenizer
import torch
from transformers import BertForQuestionAnswering
tokenizer = BertTokenizer.from_pretrained('...
user12975267
5
votes
1
answer
12k
views
How to get cosine similarity of word embedding from BERT model
I was interesting in how to get the similarity of word embedding in different sentences from BERT model (actually, that means words have different meanings in different scenarios).
For example:
sent1 =...
- 143
5
votes
3
answers
6k
views
AttributeError: 'str' object has no attribute 'dim' in pytorch
I got the following error output in the PyTorch when sent model predictions into the model. Does anyone know what's going on?
Following are the architecture model that I created, in the error output, ...
- 71
5
votes
1
answer
13k
views
TypeError: linear(): argument 'input' (position 1) must be Tensor, not str
so ive been trying to work on some example of bert that i found on github as its the first time im trying to use bert and see how it works. The respiratory im working with is the following: https://...
user14388704