All Questions
Tagged with bert-language-model text-classification
76
questions
94
votes
10
answers
95k
views
How to use Bert for long text classification?
We know that BERT has a max length limit of tokens = 512, So if an article has a length of much bigger than 512, such as 10000 tokens in text
How can BERT be used?
- 1,221
15
votes
6
answers
40k
views
With BERT Text Classification, ValueError: too many dimensions 'str' error occuring
Trying to make a classifier for sentiments of texts with BERT model but getting ValueError : too many dimensions 'str'
That is the DataFrame for values of train data; so they are train_labels
0 notr
...
- 151
10
votes
4
answers
14k
views
Is it necessary to do stopwords removal ,Stemming/Lemmatization for text classification while using Spacy,Bert?
Is stopwords removal ,Stemming and Lemmatization necessary for text classification while using Spacy,Bert or other advanced NLP models for getting the vector embedding of the text ?
text="The ...
- 254
9
votes
2
answers
7k
views
How to get all documents per topic in bertopic modeling
I have a dataset and trying to convert it to topics using berTopic modeling but the problem is, i cant get all the docoments of a topic. berTopic is only return 3 docoments per topic.
topic_model = ...
- 91
7
votes
1
answer
6k
views
Passing multiple sentences to BERT?
I have a dataset with paragraphs that I need to classify into two classes. These paragraphs are usually 3-5 sentences long. The overwhelming majority of them are less than 500 words long. I would like ...
- 107
6
votes
1
answer
8k
views
Sliding window for long text in BERT for Question Answering
I've read post which explains how the sliding window works but I cannot find any information on how it is actually implemented.
From what I understand if the input are too long, sliding window can be ...
- 63
5
votes
1
answer
2k
views
Extend BERT or any transformer model using manual features
I have been doing a thesis in my citation classifications. I just implemented Bert model for the classification of citations. I have 4 output classes and I give an input sentence and my model returns ...
- 73
4
votes
1
answer
2k
views
Finetuning BERT on custom data
I want to train a 21 class text classification model using Bert. But I have very little training data, so a downloaded a similar dataset with 5 classes with 2 million samples.t
And finetuned ...
- 644
4
votes
0
answers
285
views
How to handle text classification model that gives few results with higher confidence to wrong category?
I had a dataset of 15k records. I trained the model using a k-train package and 'bert' model with 5k samples. The train-test split is 70-30% and test results gave me accuracy and f1 scores as 93-94%. ...
- 41
3
votes
3
answers
5k
views
what is the difference between pooled output and sequence output in bert layer?
everyone! I was reading about Bert and wanted to do text classification with its word embeddings. I came across this line of code:
pooled_output, sequence_output = self.bert_layer([input_word_ids, ...
- 453
3
votes
2
answers
3k
views
How can i get all outputs of the last transformer encoder in bert pretrained model and not just the cls token output?
I'm using pytorch and this is the model from huggingface transformers link:
from transformers import BertTokenizerFast, BertForSequenceClassification
bert = BertForSequenceClassification....
- 101
3
votes
0
answers
372
views
Does ktrain package combine input embedding with bert embedding when used for test classification?
I am running the code given in the link below. What embeddings does the ktrain package of python use for bert text classification. I believe the code is using a pre-trained model of Bert. In that is ...
- 31
2
votes
1
answer
2k
views
What is the simplest way to continue training a pre-trained BERT model, on a specific domain?
I want to use a pre-trained BERT model in order to use it on a text classification task (I'm using Huggingface library). However, the pre-trained model was trained on domains that are different than ...
- 25
2
votes
1
answer
4k
views
Text classification using BERT - how to handle misspelled words
I am not sure if this is the best place to submit that kind of question, perhaps CrossValdation would be a better place.
I am working on a text multiclass classification problem.
I built a model ...
- 647
2
votes
1
answer
751
views
Bertopic assign topics to data frame
I have build a topic model with Bertopic.
After getting topic how could I assign them to dataset.
My main aim is to convert unsupervised topic modelling to supervised multi label classification ...
- 21
2
votes
1
answer
570
views
reporting other metrics during training evaluation simpletransformers
I am training a text classification model over a large set of data and I am using bert classifier (bert-base-uncased) of simpletransformer library. Simpletransformer retports by default mcc and ...
- 787
2
votes
1
answer
881
views
Spacy's BERT model doesn't learn
I've been trying to use spaCy's pretrained BERT model de_trf_bertbasecased_lg to increase accuracy in my classification project. I used to build a model from scratch using de_core_news_sm and ...
- 1,039
2
votes
1
answer
1k
views
ValueError: The model did not return a loss from the inputs, only the following keys: logits
I want to make a metaphor detector model.
As a pretrained model, I use a DistilBert model that I have, previously, fine-tuned with masking (this is the model I use to make the new metaphor detection ...
2
votes
0
answers
412
views
BERT (Huggingface Transformer) - Get important features for class
My goal is to get the most important features for each class in a text classification task.
I created the model, learner and predictor like this:
t = text.Transformer(model_name, maxlen=MAX_SEQ_LENGTH,...
- 29
2
votes
0
answers
1k
views
Python "Can't pickle local object" exception during BertModel training
I am using simpletransformers.classification to train a Bert moder to classify some text inputs. Here is my code.
from simpletransformers.classification import ClassificationModel
import torch
...
- 374
2
votes
1
answer
1k
views
How to retrieve attention weight alignment for tokens using transformer (BERT) model?
I am working on text classification with transformer models (PyTorch, Huggingface, running on GPU).
I have already my model and my training loop and it works fine but to better understand wrong ...
- 21
1
vote
1
answer
1k
views
BERT binary Textclassification get different results every run
I do binary text classification with BERT from the Simpletransformer.
I work in Colab with GPU runtime type.
I have generated train and test set with the sklearn StratifiedKFold Method. I have two ...
- 389
1
vote
1
answer
1k
views
Is splitting a long document of a dataset for BERT considered bad practice?
I am fine-tuning a BERT model on a labeled dataset with many documents longer than the 512 token limit set by the tokenizer.
Since truncating would lose a lot of data I would rather use, I started ...
- 341
1
vote
1
answer
2k
views
Using BETO - Spanish version BERT
I'm trying to build a Spanish text classifier based on BERT. So I selected a model called BETO https://github.com/scruz03/beto. I am working on Google Colab and trying to replicate the steps indicated ...
- 109
1
vote
1
answer
645
views
Customize the encode module in huggingface bert model
I am working on a text classification project using Huggingface transformers module. The encode_plus function provides the users with a convenient way of generating the input ids, attention masks, ...
- 301
1
vote
1
answer
634
views
Error : Target size (torch.Size([8])) must be the same as input size (torch.Size([8, 2])) while training a binary classifier deepset/gbert-base
I am aware of most of the solutions which are discussed here previously regarding the same problem but still I had no luck with those solutions.
I’m trying to implement a binary classifier. I’m using ...
1
vote
1
answer
4k
views
huggingface transformers convert logit scores to probability
I'm a beginner to this field and am stuck. I am following this tutorial (https://towardsdatascience.com/multi-label-multi-class-text-classification-with-bert-transformer-and-keras-c6355eccb63a) to ...
- 130
1
vote
1
answer
835
views
Can I add a layer of meta data in a text classification model?
I am trying to create a multiclass classifier to identify topics of Facebook posts from a group of parliament members.
I'm using SimpleTransformers to put together an XML-RoBERTa-based classification ...
- 73
1
vote
1
answer
886
views
How to make BERT model converge?
I am trying to use BERT for sentiment analysis but I suspect I am doing something wrong. In my code I am fine tuning bert using bert-for-tf2 but after 1 epoch I am getting an accuracy of 42% when a ...
- 933
1
vote
0
answers
164
views
Sequence to sequence classification (predicting sequence of labels) using Transformers
Im looking for a way to feed a transformer (HuggingFace trained model) a sequence of sentences(introducing context) in order to predict a sequence of labels. The goal is to predict each sentence by ...
- 11
1
vote
0
answers
68
views
Tensorflow NLP with BERT Preprocessing data
so this is a specific question involving two Tensorflow text classification tutorials on tensorflow.org. Sorry if this is the wrong place to ask.
Basically, there are two tutorials, one is "...
1
vote
0
answers
40
views
how do i improve my nlp model to classify 4 different mental illness?
I have a dataset in csv containing 2 columns: 1 is the label which determines the type of mental illness of the patient and the other is the corresponding reddit posts from a certain time period of ...
1
vote
1
answer
2k
views
BERT text classification with K fold cross validation returns "Target 3 is out of bounds" error
I am working on training a dataset of sentences to be grouped into one of three categories. I am trying to do this based off of code found in this very helpful medium article (https://medium.com/swlh/...
- 11
1
vote
1
answer
2k
views
Bert Text Classification Loss is Nan
I'm try to make an model that classify the text in 3 categories.(Negative,Neural,Positive)
I have csv file that contain comments on different apps with their rating.
First I import all the necessary ...
user5244504
1
vote
0
answers
707
views
What can be the cause of the validation loss increasing and the accuracy remaining constant to zero while the train loss decreases?
I am trying to solve a multiclass text classification problem. Due to specific requirements from my project I am trying to use skorch (https://skorch.readthedocs.io/en/stable/index.html) to wrap ...
- 11
1
vote
0
answers
537
views
Fine-Tuning Pretrained BERT with CNN. How to Disable Masking
Has anyone used CNN in Keras to fine-tune a pre-trained BERT?
I have been trying to design this but the pre-trained model comes with Masks (i think in the embedding layer) and when the fine-tuning ...
- 379
1
vote
2
answers
4k
views
Implementing HuggingFace BERT using tensorflow fro sentence classification
I am trying to train a model for real disaster tweets prediction(Kaggle Competition) using the Hugging face bert model for classification of the tweets.
I have followed many tutorials and have used ...
- 11
0
votes
1
answer
720
views
Multilabel text classification with BERT and highly imbalanced training data
I'm trying to train a multilabel text classification model using BERT. Each piece of text can belong to 0 or more of a total of 485 classes. My model consists of a dropout layer and a linear layer ...
0
votes
3
answers
2k
views
Does Bert model need text?
Does Bert models need pre-processed text (Like removing special characters, stopwords, etc.) or I can directly pass my text as it is to Bert models. (HuggigFace libraries).
note: Follow up question to:...
0
votes
1
answer
287
views
How does TfBertForSequenceClassification work?
The main job of bert is two - one is to predict the masked word and second is to predict the if sentence b is related to sentence a or not , then how TfBertForSequenceClassification is helping me for ...
- 225
0
votes
1
answer
54
views
Having trouble understanding the predictions array in classification model evaluation
I'm working on a sarcasm detector with the BERT model (binary classification). Currently, I'm having trouble with the model evaluation as I don't really understand the predictions array. The model ...
- 23
0
votes
1
answer
14k
views
IndexError: Target is out of bounds
I am currently trying to replicate the article
https://towardsdatascience.com/text-classification-with-bert-in-pytorch-887965e5820f
to get an introduction to PyTorch and BERT.
I used some own sample ...
- 37
0
votes
1
answer
530
views
HuggingFace Transformers model for German news classification
I've been trying to find a suitable model for my project (multiclass German text classification) but got a little confused with the models offered here. There are models with text-classification tag, ...
- 1,039
0
votes
1
answer
1k
views
How to optimize fine-tuned BERT's model size in TensorFlow 2.0?
After fine tuning BERT for classification the model size is ~1.3GB while the pre-trained model size was ~400MB. This happens due to additional variables related to Adam saved with the model and can be ...
- 639
0
votes
0
answers
18
views
Getting different probability scores for same text when passed in batches at the time of prediction for custom tuned BERT in text classification
BERT (BASE UNCASED) custom model is trained on 1.2 million texts for text classification task for 97 categories. Validation and Test data sets are around 250k. Since predicting entire test data doesn'...
0
votes
1
answer
282
views
Shap value for binary classification using Pre-Train Bert: How to extract summary graph?
I used pre-train bert model for binary classification. After training my model with my small data, I wanted to extract summary graph like this the graph I want. However, I want to replace these ...
- 11
0
votes
0
answers
42
views
Creating Embedding Matrix for LSTM Model with BERT Feature Representations on Arabic Dataset
I'm working on implementing an LSTM model for an Arabic dataset using BERT feature representations. I've utilized the 'asafaya/bert-base-arabic' model for this purpose:
bert_model = ...
0
votes
0
answers
22
views
BERT classifier adding previous and next row as context
I'm working on a classifier in order to sort the text of a page into utterances and other textual information. I've annotated some training data and wondering if there is a way to pass this contextual,...
- 81
0
votes
0
answers
48
views
Can bert uncased predict text classification on foreign data?
I am trying to do the fake news/real news classification and used a pre-trained bert uncased model as transfer learning and it gave a solid 81% accuracy. But the problem is while doing sanity checks, ...
0
votes
1
answer
192
views
How to get the word on which the text classification has been made?
I am doing a multi-label text classification using a pre-trained model of BERT. Here is an example of the prediction that has been made for one sentence-
pred_image
I want to get those words from the ...
