Questions tagged [bert-language-model]
BERT, or Bidirectional Encoder Representations from Transformers, is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. BERT uses Transformers (an attention mechanism that learns contextual relations between words or sub words in a text) to generate a language model.
1,796
questions
3
votes
5
answers
3k
views
bert-serving-start giving error TypeError: cannot unpack non-iterable NoneType object - tried multiple paths to the model
I am trying to use BERT with bert-serving-start in python3.8 but it does not initialise and throws error:
TypeError: cannot unpack non-iterable NoneType object
This may have something to do with the ...
- 1,375
3
votes
1
answer
4k
views
Fast and slow tokenizers yield different results
Using HuggingFace's pipeline tool, I was surprised to find that there was a significant difference in output when using the fast vs slow tokenizer.
Specifically, when I run the fill-mask pipeline, ...
- 153
3
votes
1
answer
2k
views
Bert pre-trained model giving random output each time
I was trying to add an additional layer after huggingface bert transformer, so I used BertForSequenceClassification inside my nn.Module Network. But, I see the model is giving me random outputs when ...
user13445425
3
votes
2
answers
6k
views
ImportError: cannot import name 'warmup_linear'
While trying to import warmup_linear, I'm getting this error
ImportError: cannot import name 'warmup_linear'
Import -
from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear
...
- 2,147
3
votes
1
answer
3k
views
ValueError: Unknown layer: TFBertModel. Please ensure this object is passed to the `custom_objects` argument
Here I training the bert model. below code i used to train, when i load the saved model for predict, it's shows this error. can anyone please help me out?
import tensorflow as tf
import logging
from ...
- 71
3
votes
1
answer
9k
views
resize_token_embeddings on the a pertrained model with different embedding size
I would like to ask about the way to change the embedding size of the trained model.
I have a trained model models/BERT-pretrain-1-step-5000.pkl.
Now I am adding a new token [TRA]to the tokeniser and ...
- 61
3
votes
1
answer
2k
views
How to calculate per document probabilities under respective topics with BERTopics?
I am trying to use BERTopic to analyze the topic distribution of documents, after BERTopic is performed, I would like to calculate the probabilities under respective topics per document, how should I ...
- 33
3
votes
1
answer
1k
views
BERT Heads Count
From the literature I read,
Bert Base has 12 encoder layers and 12 attention heads. Bert Large has 24 encoder layers and 16 attention heads.
Why is Bert large having 16 attentions heads ?
- 2,115
3
votes
1
answer
4k
views
How to map token indices from the SQuAD data to tokens from BERT tokenizer?
I am using the SQuaD dataset for answer span selection. After using the BertTokenizer to tokenize the passages, for some samples, the start and end indices of the answer don't match the real answer ...
- 145
3
votes
2
answers
1k
views
Multilingual Bert sentence vector captures language used more than meaning - working as interned?
Playing around with BERT, I downloaded the Huggingface Multilingual Bert and entered three sentences, saving their sentence vectors (the embedding of [CLS]), then translated them via Google Translate, ...
- 718
3
votes
2
answers
2k
views
How bert is a bidirectional?
Bert encoder takes the input and goes for the multi-head attention model. But how do they maintain sequence? Since current words don't take sequence of previous words. Besides, why is it bidirectional?...
- 155
3
votes
1
answer
2k
views
Using Sentence-Bert with other features in scikit-learn
I have a dataset, one feature is text and 4 more features. Sentence-Bert vectorizer transforms text data into tensors. I can use these sparse matrices directly with a machine learning classifier. Can ...
- 45
3
votes
1
answer
3k
views
How to combine embeddins vectors of bert with other features?
I am working on a classification task with 3 labels (0,1,2 = neg, pos, neu). Data are sentences. So to produce vectors/embeddings of sentences, I use a Bert encoder to get embeddings for each sentence ...
- 323
3
votes
3
answers
2k
views
Transformers pipeline model directory
I'm using the Huggingface's Transformers pipeline function to download the model and the tokenizer, my Windows PC downloaded them but I don't know where they are stored on my PC. Can you please help ...
- 404
3
votes
1
answer
2k
views
huggingface transformers bert model without classification layer
I want to do a joint-embedding from vgg16 and bert for classification.
The thing with huggingface transformers bert is that it has the classification layer which has num_labels dimension.
But, I ...
- 11k
3
votes
2
answers
1k
views
Is possible multiples GPUs work as one with more memory?
I have a deep learning workstation where there are 4 GPUs with 6 GB of memory each. Would it be possible to make a docker container see the 4 GPUs as one but with 24 GB?
Thank you.
- 724
3
votes
2
answers
2k
views
Company name extraction with bert-base-ner: easy way to know which words relate to which?
Hi I'm trying to extract the full company name from a string description about the company with bert-base-ner. I am also open to trying other methods but I couldn't really find one. The issue is that ...
- 41
3
votes
1
answer
644
views
InternalError when using TPU for training Keras model
I am attempting to fine-tune a BERT model on Google Colab from the Tensorflow Hub using this link.
However, I run into the following error:
InternalError: RET_CHECK failure (third_party/tensorflow/...
- 43
3
votes
1
answer
2k
views
How to get probability of an answer using BERT model and is there a way to ask multiple questions for a context
I am new to AI models and currently experimenting with the QandA model. Particularly I am interested in following 2 models.
1. from transformers import BertForQuestionAnswering
2. from ...
- 235
3
votes
1
answer
3k
views
BERT classification on imbalanced or small dataset
I have a large corpus, no labels. I trained this corpus to get my BERT tokenizer.
Then I want to build a BertModel to do a binary classification on a labeled dataset. However, this dataset is highly ...
- 43
3
votes
1
answer
2k
views
Must the vocab size must math the vocab_size in bert_config.json exactly?
I am seeing someone other's BERT model, in which the vocab.txt's size is 22110, but the vocab_size parameter's value is 21128 in bert_config.json.
I understand that these two numbers must be exactly ...
- 6,847
3
votes
1
answer
559
views
Why doesn't BertForMaskedLM generate right masked tokens?
I am testing this piece of code:
from transformers import BertTokenizer, BertModel, BertForMaskedLM
tokenizer = BertTokenizer.from_pretrained("hfl/chinese-roberta-wwm-ext")
model = ...
- 6,847
3
votes
1
answer
4k
views
What does 'output_dir' mean in transformers.TrainingArguments?
On the huggingface site documentation, it says 'The output directory where the model predictions and checkpoints will be written'. I don't quite understand what it means. Do I have to create any file ...
3
votes
2
answers
5k
views
About BertForMaskedLM
I have recently read about Bert and want to use BertForMaskedLM for fill_mask task. I know about Bert architecture. Also, as far as I know, BertForMaskedLM is built from Bert with a language modeling ...
- 31
3
votes
1
answer
1k
views
BERT - Is that needed to add new tokens to be trained in a domain specific environment?
My question here is no how to add new tokens, or how to train using a domain-specific corpus, I'm already doing that.
The thing is, am I supposed to add the domain-specific tokens before the MLM ...
- 253
3
votes
1
answer
3k
views
Unable to find the word that I added to the Huggingface Bert tokenizer vocabulary
I tried to add new words to the Bert tokenizer vocab. I see that the length of the vocab is increasing, however I can't find the newly added word in the vocab.
tokenizer.add_tokens(['covid', 'wuhan'])
...
3
votes
2
answers
3k
views
How can i get all outputs of the last transformer encoder in bert pretrained model and not just the cls token output?
I'm using pytorch and this is the model from huggingface transformers link:
from transformers import BertTokenizerFast, BertForSequenceClassification
bert = BertForSequenceClassification....
- 101
3
votes
1
answer
3k
views
Which dimensionality reduction technique works well for BERT sentence embeddings?
I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
- 61
3
votes
1
answer
1k
views
TFBertMainLayer gets less accuracy compared to TFBertModel
I had a problem with saving weights of TFBertModel wrapped in Keras. the problem is described here in GitHub issue and here in Stack Overflow.The solution proposed in both cases is to use
config = ...
- 2,720
3
votes
1
answer
3k
views
Transformers PreTrainedTokenizer add_tokens Functionality
Referring to the documentation of the awesome Transformers library from Huggingface, I came across the add_tokens functions.
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = ...
- 14.2k
3
votes
1
answer
3k
views
How to use BERT pretrain embeddings with my own new dataset?
My dataset and NLP task is very different from the large corpus what authors have pre-trained their model (https://github.com/google-research/bert#pre-training-with-bert), so I can't directly fine-...
- 31
3
votes
1
answer
331
views
Semantic search with pretrained BERT models giving irrelevant results with high similarity
I'm trying to create a Semantic search system and have experimented with multiple pretrained models from the SentenceTransformers library: LaBSE, MS-MARCO etc. The system is working well in returning ...
- 43
3
votes
2
answers
2k
views
Sentence-Transformer Training and Validation Loss
I am using the Sentence-Transformers model to Fine Tune(using PyTorch) it on a custom dataset which is the same as the Semantic Text Similarity (STS) Dataset.
I am unable to get(or print) the training ...
- 37
3
votes
1
answer
888
views
How to get topic-probs matrix in bertopic modeling
I ran BERTopic to get topics for 3,500 documents. How could I get the topic-probs matrix for each document and export them to csv? When I export them, I want to export the identifier of each document ...
- 31
3
votes
1
answer
3k
views
Save a Bert model with custom forward function and heads on Hugginface
I have created my own BertClassifier model, starting from a pretrained and then added my own classification heads composed by different layers. After the fine-tuning, I want to save the model using ...
- 380
3
votes
1
answer
545
views
how can we get the attention scores of multimodal models via hugging face library?
I was wondering if we could get the attention scores of any multimodal model using the api provided by the hugging face library, as it's relatively easier to get such scores of normal language bert ...
- 167
3
votes
2
answers
4k
views
Bert Model Compile Error - TypeError: Invalid keyword argument(s) in `compile`: {'steps_per_execution'}
I have been using bert and trying to compile the model using the below line of code.
model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased')
optimizer = tf.keras.optimizers.Adam(...
- 31
3
votes
1
answer
2k
views
How to find the similarity of sentences in 2 columns of a dataframe using spacy
I pulled this code from https://spacy.io/universe/project/spacy-sentence-bert
import spacy_sentence_bert
# load one of the models listed at https://github.com/MartinoMensio/spacy-sentence-bert/
nlp = ...
- 43
3
votes
1
answer
2k
views
Using Hugging-face transformer with arguments in pipeline
I am working on using a transformer. Pipeline to get BERT embeddings to my input. using this without a pipeline i am able to get constant outputs but not with pipeline since I was not able to pass ...
- 548
3
votes
1
answer
5k
views
How to save sentence-Bert output vectors to a file?
I am using Bert to get similarity between multi term words.here is my code that I used for embedding :
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('bert-large-...
- 314
3
votes
1
answer
793
views
Retrieve the "relevant tokens" with a BERT model (already fine-tuned)
I already fine-tuned a BERT model ( with the huggingface library) for a classification task to predict a post category in two types (1 and 0, for example). But, I would need to retrieve the "...
3
votes
3
answers
2k
views
List index out of range when saving finetuned Tensorflow model
I'm trying to fine-tune a pre-trained BERT model from Huggingface using Tensorflow. Everything runs smoothly and the model builds and trains without error. But when I try to save the model it stops ...
- 47
3
votes
1
answer
939
views
How to find the (Most important) responsible Words/ Tokens/ embeddings responsible for the label result of a text classification model in PyTorch
Let us suppose I have a model like:
class BERT_Subject_Classifier(nn.Module):
def __init__(self,out_classes,hidden1=128,hidden2=32,dropout_val=0.2):
super(BERT_Subject_Classifier, self)....
- 3,872
3
votes
1
answer
2k
views
Error "version" not found after adding bert as a submodule to my git repo
After adding BERT as a submodule, cannot use it, the version info is missing in the config file. These are the main steps:
1- I've used the git submodule add https://huggingface.co/bert-base-...
- 615
3
votes
2
answers
2k
views
Why is there no pooler layer in huggingfaces' FlauBERT model?
BERT model for Language Model and Sequence classification includes an extra projection layer between the last transformer and the classification layer (it contains a linear layer of size hidden_dim x ...
- 93
3
votes
1
answer
829
views
NLP : Get 5 best candidates from QuestionAnsweringPipeline
I am working on a French Question-Answering model using huggingface transformers library. I'm using a pre-trained CamemBERT model which is very similar to RoBERTa but is adapted to french.
Currently, ...
- 33
3
votes
1
answer
1k
views
Loading a GPU trained BERTopic model on CPU?
I trained a BERTopic model on a GPU, and now for visualization purposes I want to load it on a CPU.
But when I tried to do that I got:
RuntimeError: Attempting to deserialize object on a CUDA device ...
- 31
3
votes
1
answer
807
views
How to set output_shape of BERT preprocessing layer from tensorflow hub?
I am building a simple BERT model for text classification, using the tensorflow hub.
import tensorflow as tf
import tensorflow_hub as tf_hub
bert_preprocess = tf_hub.KerasLayer("https://tfhub....
- 1,219
3
votes
1
answer
2k
views
Can we calculate feature importance in Huggingface Bert?
We can fit a LinearRegression model on the regression dataset and retrieve the coeff_ property that contains the coefficients found for each input variable. These coefficients can provide the basis ...
- 43
3
votes
1
answer
373
views
Result of TPU and GPU are different
I'm trying to fine tuning using bert model.
I'm using pre trained bert model and pytorch.
The problem is that the result of GPU and the result of TPU are slightly different.
(accuracy is different ...
- 31