All Questions
Tagged with bert-language-model huggingface-transformers
551
questions
46
votes
5
answers
58k
views
ValueError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]] - Tokenizing BERT / Distilbert Error
def split_data(path):
df = pd.read_csv(path)
return train_test_split(df , test_size=0.1, random_state=100)
train, test = split_data(DATA_DIR)
train_texts, train_labels = train['text'].to_list(), ...
- 736
24
votes
1
answer
62k
views
PyTorch BERT TypeError: forward() got an unexpected keyword argument 'labels'
Training a BERT model using PyTorch transformers (following the tutorial here).
Following statement in the tutorial
loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=...
- 1,826
18
votes
1
answer
12k
views
BertForSequenceClassification vs. BertForMultipleChoice for sentence multi-class classification
I'm working on a text classification problem (e.g. sentiment analysis), where I need to classify a text string into one of five classes.
I just started using the Huggingface Transformer package and ...
- 39.8k
17
votes
5
answers
67k
views
Transformer: Error importing packages. "ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler'"
I am working on a machine learning project on Google Colab, it seems recently there is an issue when trying to import packages from transformers. The error message says:
ImportError: cannot import ...
- 231
17
votes
2
answers
11k
views
Difficulty in understanding the tokenizer used in Roberta model
from transformers import AutoModel, AutoTokenizer
tokenizer1 = AutoTokenizer.from_pretrained("roberta-base")
tokenizer2 = AutoTokenizer.from_pretrained("bert-base-cased")
sequence = "A Titan RTX has ...
- 971
16
votes
3
answers
23k
views
How to understand hidden_states of the returns in BertModel?(huggingface-transformers)
Returns last_hidden_state (torch.FloatTensor of shape (batch_size,
sequence_length, hidden_size)): Sequence of hidden-states at the
output of the last layer of the model.
pooler_output (torch....
- 211
15
votes
3
answers
21k
views
BERT sentence embeddings from transformers
I'm trying to get sentence vectors from hidden states in a BERT model. Looking at the huggingface BertModel instructions here, which say:
from transformers import BertTokenizer, BertModel
tokenizer = ...
- 19.2k
15
votes
3
answers
35k
views
Python: BERT Error - Some weights of the model checkpoint at were not used when initializing BertModel
I am creating an entity extraction model in PyTorch using bert-base-uncased but when I try to run the model I get this error:
Error:
Some weights of the model checkpoint at D:\Transformers\bert-entity-...
- 917
15
votes
2
answers
10k
views
BertModel transformers outputs string instead of tensor
I'm following this tutorial that codes a sentiment analysis classifier using BERT with the huggingface library and I'm having a very odd behavior. When trying the BERT model with a sample text I get a ...
- 2,922
14
votes
1
answer
14k
views
PyTorch torch.no_grad() versus requires_grad=False
I'm following a PyTorch tutorial which uses the BERT NLP model (feature extractor) from the Huggingface Transformers library. There are two pieces of interrelated code for gradient updates that I don'...
- 39.8k
13
votes
4
answers
8k
views
How to fine tune BERT on unlabeled data?
I want to fine tune BERT on a specific domain. I have texts of that domain in text files. How can I use these to fine tune BERT?
I am looking here currently.
My main objective is to get sentence ...
- 541
12
votes
2
answers
12k
views
How to train BERT from scratch on a new domain for both MLM and NSP?
I’m trying to train BERT model from scratch using my own dataset using HuggingFace library. I would like to train the model in a way that it has the exact architecture of the original BERT model.
In ...
- 379
12
votes
8
answers
37k
views
SSLError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /dslim/bert-base-NER/resolve/main/tokenizer_config.json
I am facing below issue while loading the pretrained BERT model from HuggingFace due to SSL certificate error.
Error:
SSLError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries ...
- 627
12
votes
4
answers
11k
views
Training TFBertForSequenceClassification with custom X and Y data
I am working on a TextClassification problem, for which I am trying to traing my model on TFBertForSequenceClassification given in huggingface-transformers library.
I followed the example given on ...
- 872
12
votes
2
answers
5k
views
Get probability of multi-token word in MASK position
It is relatively easy to get a token's probability according to a language model, as the snippet below shows. You can get the output of a model, restrict yourself to the output of the masked token, ...
- 27.7k
11
votes
2
answers
14k
views
Continual pre-training vs. Fine-tuning a language model with MLM
I have some custom data I want to use to further pre-train the BERT model. I’ve tried the two following approaches so far:
Starting with a pre-trained BERT checkpoint and continuing the pre-training ...
- 2,531
11
votes
3
answers
15k
views
Transformers pretrained model with dropout setting
I'm trying to use transformer's huggingface pretrained model bert-base-uncased, but I want to increace dropout. There isn't any mention to this in from_pretrained method, but colab ran the object ...
- 685
11
votes
1
answer
6k
views
what is so special about special tokens?
what exactly is the difference between "token" and a "special token"?
I understand the following:
what is a typical token
what is a typical special token: MASK, UNK, SEP, etc
when ...
- 123
10
votes
1
answer
14k
views
How to get intermediate layers' output of pre-trained BERT model in HuggingFace Transformers library?
(I'm following this pytorch tutorial about BERT word embeddings, and in the tutorial the author is access the intermediate layers of the BERT model.)
What I want is to access the last, lets say, 4 ...
- 1,262
10
votes
1
answer
3k
views
How to use existing huggingface-transformers model into spacy?
I'm here to ask you guys if it is possible to use an existing trained huggingface-transformers model with spacy.
My first naive attempt was to load it via spacy.load('bert-base-uncased'), it didn't ...
- 253
9
votes
1
answer
23k
views
RuntimeError: The size of tensor a (4000) must match the size of tensor b (512) at non-singleton dimension 1
I'm trying to build a model for document classification. I'm using BERT with PyTorch.
I got the bert model with below code.
bert = AutoModel.from_pretrained('bert-base-uncased')
This is the code for ...
9
votes
1
answer
24k
views
BERT tokenizer & model download
I`m beginner.. I'm working with Bert. However, due to the security of the company network, the following code does not receive the bert model directly.
tokenizer = BertTokenizer.from_pretrained('bert-...
- 575
9
votes
2
answers
12k
views
Outputting attention for bert-base-uncased with huggingface/transformers (torch)
I was following a paper on BERT-based lexical substitution (specifically trying to implement equation (2) - if someone has already implemented the whole paper that would also be great). Thus, I wanted ...
- 674
9
votes
1
answer
8k
views
How do I use BertForMaskedLM or BertModel to calculate perplexity of a sentence?
I want to use BertForMaskedLM or BertModel to calculate perplexity of a sentence, so I write code like this:
import numpy as np
import torch
import torch.nn as nn
from transformers import ...
- 113
8
votes
1
answer
9k
views
How to calculate perplexity of a sentence using huggingface masked language models?
I have several masked language models (mainly Bert, Roberta, Albert, Electra). I also have a dataset of sentences. How can I get the perplexity of each sentence?
From the huggingface documentation ...
- 2,148
8
votes
3
answers
5k
views
How to compute mean/max of HuggingFace Transformers BERT token embeddings with attention mask?
I'm using the HuggingFace Transformers BERT model, and I want to compute a summary vector (a.k.a. embedding) over the tokens in a sentence, using either the mean or max function. The complication is ...
- 39.8k
8
votes
6
answers
6k
views
Problem with inputs when building a model with TFBertModel and AutoTokenizer from HuggingFace's transformers
I'm trying to build the model illustrated in this picture:
I obtained a pre-trained BERT and respective tokenizer from HuggingFace's transformers in the following way:
from transformers import ...
- 1,672
8
votes
1
answer
3k
views
HuggingFace BERT `inputs_embeds` giving unexpected result
The HuggingFace BERT TensorFlow implementation allows us to feed in a precomputed embedding in place of the embedding lookup that is native to BERT. This is done using the model's call method's ...
- 1,174
7
votes
1
answer
5k
views
How exactly should the input file be formatted for the language model finetuning (BERT through Huggingface Transformers)?
I wanted to employ the examples/run_lm_finetuning.py from the Huggingface Transformers repository on a pretrained Bert model. However, from following the documentation it is not evident how a corpus ...
- 79
7
votes
2
answers
14k
views
The model did not return a loss from the inputs - LabSE error
I want to fine tune LabSE for Question answering using squad dataset. and i got this error:
ValueError: The model did not return a loss from the inputs, only the following keys: last_hidden_state,...
7
votes
1
answer
8k
views
max_seq_length for transformer (Sentence-BERT)
I'm using sentence-BERT from Huggingface in the following way:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
model.max_seq_length = 512
model....
- 779
7
votes
1
answer
6k
views
Passing multiple sentences to BERT?
I have a dataset with paragraphs that I need to classify into two classes. These paragraphs are usually 3-5 sentences long. The overwhelming majority of them are less than 500 words long. I would like ...
- 107
7
votes
1
answer
14k
views
How padding in huggingface tokenizer works?
I tried following tokenization example:
tokenizer = BertTokenizer.from_pretrained(MODEL_TYPE, do_lower_case=True)
sent = "I hate this. Not that.",
_tokenized = tokenizer(sent, ...
- 2,829
7
votes
1
answer
8k
views
Mismatched size on BertForSequenceClassification from Transformers and multiclass problem
I just trained a BERT model on a Dataset composed by products and labels (departments) for an e-commerce website. It's a multiclass problem. I used BertForSequenceClassification to predict the ...
7
votes
2
answers
4k
views
Pretraining a language model on a small custom corpus
I was curious if it is possible to use transfer learning in text generation, and re-train/pre-train it on a specific kind of text.
For example, having a pre-trained BERT model and a small corpus ...
- 477
7
votes
1
answer
8k
views
How to specify a proxy in transformers pipeline
I am using sentiment-analysis pipeline as described here.
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
It's failing with a connection error message
ValueError: ...
- 194
7
votes
1
answer
5k
views
Token indices sequence length error when using encode_plus method
I got a strange error when trying to encode question-answer pairs for BERT using the encode_plus method provided in the Transformers library.
I am using data from this Kaggle competition. Given a ...
- 1,191
6
votes
2
answers
9k
views
How to untokenize BERT tokens?
I have a sentence and I need to return the text corresponding to N BERT tokens to the left and right of a specific word.
from transformers import BertTokenizer
tz = BertTokenizer.from_pretrained("...
- 183
6
votes
2
answers
11k
views
BERT get sentence embedding
I am replicating code from this page. I have downloaded the BERT model to my local system and getting sentence embedding.
I have around 500,000 sentences for which I need sentence embedding and it is ...
- 6,258
6
votes
1
answer
11k
views
BertWordPieceTokenizer vs BertTokenizer from HuggingFace
I have the following pieces of code and trying to understand the difference between BertWordPieceTokenizer and BertTokenizer.
BertWordPieceTokenizer (Rust based)
from tokenizers import ...
- 3,413
6
votes
1
answer
7k
views
"You have to specify either input_ids or inputs_embeds", but I did specify the input_ids
I trained a BERT based encoder decoder model (EncoderDecoderModel) named ed_model with HuggingFace's transformers module.
I used the BertTokenizer named as input_tokenizer
I tokenized the input with:
...
- 13.6k
6
votes
1
answer
8k
views
huggingface bert showing poor accuracy / f1 score [pytorch]
I am trying BertForSequenceClassification for a simple article classification task.
No matter how I train it (freeze all layers but the classification layer, all layers trainable, last k layers ...
- 11k
6
votes
2
answers
3k
views
How to test masked language model after training it?
I have followed this tutorial for masked language modelling from Hugging Face using BERT, but I am unsure how to actually deploy the model.
Tutorial: https://github.com/huggingface/notebooks/blob/...
user14946125
6
votes
3
answers
9k
views
Huggingface BERT Tokenizer add new token
I am using Huggingface BERT for an NLP task. My texts contain names of companies which are split up into subwords.
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
tokenizer....
- 111
6
votes
1
answer
8k
views
Sliding window for long text in BERT for Question Answering
I've read post which explains how the sliding window works but I cannot find any information on how it is actually implemented.
From what I understand if the input are too long, sliding window can be ...
- 63
6
votes
1
answer
939
views
How to predict the probability of an empty string using BERT
Suppose we have a template sentence like this:
"The ____ house is our meeting place."
and we have a list of adjectives to fill in the blank, e.g.:
"yellow"
"large"
&...
- 1,474
6
votes
3
answers
5k
views
Why aren't transformers imported in Python?
I want to import transformers in jupyter notebook but I get the following error. What is the reason for this error? My Python version is 3.8
ImportError: cannot import name 'TypeAlias' from '...
- 101
6
votes
0
answers
2k
views
How to slice string depending on length of tokens
When I use (with a long test_text and short question):
from transformers import BertTokenizer
import torch
from transformers import BertForQuestionAnswering
tokenizer = BertTokenizer.from_pretrained('...
user12975267
5
votes
2
answers
38k
views
No module named 'transformers.models' while trying to import BertTokenizer
I am trying to import BertTokenizer from the transformers library as follows:
import transformers
from transformers import BertTokenizer
from transformers.modeling_bert import BertModel, ...
- 93
5
votes
1
answer
11k
views
Overfitting when fine-tuning BERT sentiment analysis
I am newbie to Machine Learning in general. I am currently trying to follow a tutorial on sentiment analysis using BERT and Transformers https://curiousily.com/posts/sentiment-analysis-with-bert-and-...
- 121