All Questions
Tagged with bert-language-model keras
104
questions
43
votes
2
answers
27k
views
Why Bert transformer uses [CLS] token for classification instead of average over all tokens?
I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
- 12.3k
10
votes
1
answer
14k
views
How to get intermediate layers' output of pre-trained BERT model in HuggingFace Transformers library?
(I'm following this pytorch tutorial about BERT word embeddings, and in the tutorial the author is access the intermediate layers of the BERT model.)
What I want is to access the last, lets say, 4 ...
- 1,262
9
votes
1
answer
4k
views
BERT embedding for semantic similarity
I earlier posted this question. I wanted to get embedding similar to this youtube video, time 33 minutes onward.
1) I dont think that the embedding that i am getting from CLS token are similar to ...
- 6,258
8
votes
1
answer
14k
views
How to store Word vector Embeddings?
I am using BERT Word Embeddings for sentence classification task with 3 labels. I am using Google Colab for coding. My problem is, since I will have to execute the embedding part every time I restart ...
- 1,107
8
votes
6
answers
6k
views
Problem with inputs when building a model with TFBertModel and AutoTokenizer from HuggingFace's transformers
I'm trying to build the model illustrated in this picture:
I obtained a pre-trained BERT and respective tokenizer from HuggingFace's transformers in the following way:
from transformers import ...
- 1,672
6
votes
1
answer
3k
views
Bert Embedding Layer raises `Type Error: unsupported operand type(s) for +: 'None Type' and 'int'` with BiLSTM
I've problems integrating Bert Embedding Layer in a BiLSTM model for word sense disambiguation task,
Windows 10
Python 3.6.4
TenorFlow 1.12
Keras 2.2.4
No virtual environments were used
PyCharm ...
- 319
6
votes
1
answer
9k
views
Using BERT Embeddings in Keras Embedding layer
I want to use the BERT Word Vector Embeddings in the Embeddings layer of LSTM instead of the usual default embedding layer. Is there any way I can do it?
- 1,107
6
votes
3
answers
4k
views
TypeError: Layer input_spec must be an instance of InputSpec. Got: InputSpec(shape=(None, 128, 768), ndim=3)
I am trying to use a BERT pretrained model to do a multiclass classification (of 3 classes). Here's my function to use the model and also added some extra functionalities:
def create_model(max_seq_len,...
6
votes
1
answer
9k
views
How to feed Bert embeddings to LSTM
I am working on a Bert + MLP model for text classification problem. Essentially, I am trying to replace the MLP model with a basic LSTM model.
Is it possible to create a LSTM with embedding? Or, is ...
- 685
4
votes
2
answers
2k
views
BERT get sentence level embedding after fine tuning
I came across this page
1) I would like to get sentence level embedding (embedding given by [CLS] token) after the fine tuning is done. How could I do it?
2) I also noticed that the code on that ...
- 6,258
4
votes
2
answers
3k
views
Loading tf.keras model, ValueError: The two structures don't have the same nested structure
I created a tf.keras model that has BERT and I want to train and save it for further use.
Loading this model is a big issue cause I keep getting error: ValueError: The two structures don't have the ...
- 43
3
votes
1
answer
3k
views
ValueError: Unknown layer: TFBertModel. Please ensure this object is passed to the `custom_objects` argument
Here I training the bert model. below code i used to train, when i load the saved model for predict, it's shows this error. can anyone please help me out?
import tensorflow as tf
import logging
from ...
- 71
3
votes
1
answer
1k
views
TFBertMainLayer gets less accuracy compared to TFBertModel
I had a problem with saving weights of TFBertModel wrapped in Keras. the problem is described here in GitHub issue and here in Stack Overflow.The solution proposed in both cases is to use
config = ...
- 2,720
3
votes
1
answer
807
views
How to set output_shape of BERT preprocessing layer from tensorflow hub?
I am building a simple BERT model for text classification, using the tensorflow hub.
import tensorflow as tf
import tensorflow_hub as tf_hub
bert_preprocess = tf_hub.KerasLayer("https://tfhub....
- 1,219
3
votes
2
answers
2k
views
how to save and load custom siamese bert model
I am following this tutorial on how to train a siamese bert network:
https://keras.io/examples/nlp/semantic_similarity_with_bert/
all good, but I am not sure what is the best way to save the model ...
- 916
3
votes
0
answers
372
views
Does ktrain package combine input embedding with bert embedding when used for test classification?
I am running the code given in the link below. What embeddings does the ktrain package of python use for bert text classification. I believe the code is using a pre-trained model of Bert. In that is ...
- 31
2
votes
2
answers
1k
views
keras LSTM get hidden-state (converting sentece-sequence to document context vectors)
Im trying to create document context vectors from sentence-vectors via LSTM using keras (so each document consist of a sequence of sentence vectors).
My goal is to replicate the following blog post ...
- 323
2
votes
1
answer
975
views
stacking LSTM layer on top of BERT encoder in Keras
I have been trying to stack a single LSTM layer on top of Bert embeddings, but whilst my model starts to train it fails on the last batch and throws the following error message:
Node: 'model/tf....
- 705
2
votes
1
answer
2k
views
Tensorflow2.4 NotFoundError: No algorithm worked! with Keras Conv1D Layer
I've been looking for a solution to this error for days and I can't find solutions for this:
NotFoundError: 3 root error(s) found.
(0) Not found: No algorithm worked!
[[node model/conv1d/conv1d (...
2
votes
1
answer
3k
views
How to freeze some layers of BERT in fine tuning in tf2.keras
I am trying to fine-tune 'bert-based-uncased' on a dataset for a text classification task. Here is the way I am downloading the model:
import tensorflow as tf
from transformers import ...
- 108
2
votes
1
answer
1k
views
Bert with Padding and Masked Token Predicton
I am Playing around with Bert Pretrained Models (bert-large-uncased-whole-word-masking)
I used Huggingface to try it I first Used this Piece of Code
m = TFBertLMHeadModel.from_pretrained("bert-...
2
votes
1
answer
715
views
BERT Text Classification
I am new to BERT and try to learn BERT Fine-Tuning for Text Classification via a coursera course https://www.coursera.org/projects/fine-tune-bert-tensorflow/
Based on the course, I would like to ...
- 35
2
votes
1
answer
2k
views
Cannot add CRF layer on top of BERT in keras for NER
I am facing an unknown issue while training my BERT-CRF model for NER. I am using keras.contrib for the CRF model.
Here are the imported libraries.
!pip install transformers
!pip install git+https://...
- 302
2
votes
0
answers
660
views
model.predict(dataset) give me "unsupported operand type(s) for *: 'int' and 'NoneType" error
Now I'm editing Movie Reviews with bert-for-tf2 on TPU.ipynb to run Korean samples.
https://colab.research.google.com/drive/1BeG1n9IJmoxBZ2FicKKqWTdQ11jfYxlC?usp=sharing
To test the trained model with ...
- 21
2
votes
0
answers
151
views
Saving and loading BERT model in R
I used this tutorial to partially train a model (pretrained BERT) on some text data. Except from data, all other code is the same.
I have problems saving and loading the trained model. If I use the ...
- 71
2
votes
1
answer
653
views
BERT embeddings for abstractive text summarisation in Keras using encoder-decoder model
I am working on a text summarization task using encoder-decoder architecture in Keras. I would like to test the model's performance using different word embeddings such as GloVe and BERT. I already ...
- 105
2
votes
1
answer
3k
views
How to implement network using Bert as a paragraph encoder in long text classification, in keras?
I am doing a long text classification task, which has more than 10000 words in doc, I am planing to use Bert as a paragraph encoder, then feed the embeddings of paragraph to BiLSTM step by step.
The ...
- 1,221
1
vote
1
answer
74
views
How to pass multiple embedded layers into TFDF using Keras?
In the Keras tutorial it shows how to embed a text field and pass it to TFDF in this tutorial.
sentence_encoder_layer = hub.KerasLayer(
"https://tfhub.dev/google/universal-sentence-encoder/4&...
- 13
1
vote
1
answer
721
views
NLP BERT in R with tensorflow/Keras setup
I am trying to get BERT to run in R.
I got other NLP tasks (e.g. word2vec) done with Keras, so the general setup should be ok.
I adapted the model code from here: https://towardsdatascience.com/...
- 23
1
vote
1
answer
414
views
How to increase the rank (ndim) of input of BERT keras hub layer for learning-to-rank
I am trying to implement a learning-to-rank model using a pre-trained BERT available on tensorflow hub. I am using a variation of ListNet loss function, which requires each training instance to be a ...
- 11
1
vote
1
answer
573
views
loss is NaN when using keras bert for classification
I'm using keras-bert for classification. On some datasets, it runs well and calculates the loss, while on others the loss is NaN.
The different datasets are similar in that they are augmented versions ...
- 379
1
vote
1
answer
3k
views
Getting an error(cannot import name 'BertPreTrainedModel') while importing classification model from simpletransformers
Getting the following error while trying to import the classificationmodel from simpletransformers.
ImportError Traceback (most recent call last)
<ipython-input-1-...
1
vote
1
answer
545
views
Providing tensor input using tf.keras.layers.Input
I am trying to perform a multiclass classification using a simple BERT model. I am referring to the following tutorial for help: tensorflow classification tutorial with bert.
I use the following code ...
1
vote
1
answer
849
views
Using roberta model cannot define the model .compile or summary
Using roberta model for sentiment analysis cannot define the model .compile or summary
from transformers import RobertaTokenizer, RobertaForSequenceClassification
from transformers import BertConfig
...
- 13
1
vote
1
answer
339
views
Why does BERT Model fail to find an option that matches my input positional arguments?
While attempting an NLP exercise, I tried to make use of BERT architecture to get a good training model. So I defined a function that builds and compiles the model using BERT as the layer. However, ...
- 135
1
vote
1
answer
927
views
Is there a way to use bert-large as a text classification tool without fine-tuning?
I'm currently have a task of converting a keras BERT-based model for any text classification problem to the .pb file. For this I already have a function, that takes in the keras model, but the point ...
- 95
1
vote
1
answer
1k
views
Issue when preprocessing text with Ktrain and DistilBERT
Following the example notebook here:
https://github.com/amaiya/ktrain/blob/master/examples/text/20newsgroup-distilbert.ipynb
At STEP 1: Preprocess Data, I run into the errors listed below. When I do ...
- 21
1
vote
1
answer
886
views
How to make BERT model converge?
I am trying to use BERT for sentiment analysis but I suspect I am doing something wrong. In my code I am fine tuning bert using bert-for-tf2 but after 1 epoch I am getting an accuracy of 42% when a ...
- 933
1
vote
0
answers
20
views
ResourceExhaustedError In Tensorflow BERT Classifier
I am trying to use the BertClassifier from the keras_nlp library but when I train the model I get this error:
2024-03-22 22:53:03.932926: W external/local_tsl/tsl/framework/bfc_allocator.cc:487] ...
- 21
1
vote
0
answers
25
views
Square brackets at the end of TFBertModel call method
I'm trying to understand how to use bert-base-cased pretrained model in my code, so I was reviewing this code:
input_ids = Input(shape=(max_len,), dtype=tf.int32, name="input_ids")
...
- 73
1
vote
0
answers
50
views
AttributeError: 'Functional' object has no attribute '_jit_compile'
I'm trying to run bert use keras. But facing this problem. Can anyone help me? very thanks QQ
AttributeError: in user code:
File "C:\Users\user.conda\envs\bertdio\lib\site-packages\keras\engine\...
- 11
1
vote
0
answers
91
views
How to calculate the accuracy of this Transfer learning bert model
I have found this Git Hub code for intent classification. Could someone please tell me how to check the model accuracy of this?
https://colab.research.google.com/github/ShawonAshraf/nlu-jointbert-...
1
vote
0
answers
215
views
Can't save model in saved_model format when finetune bert model
When training the bert model, the weights are saved well, but the entire model is not saved.
After model.fit,
save model as model.save_weights('bert_xxx.h5') and load_weights works fine,
but since ...
- 71
1
vote
0
answers
132
views
'Functional' object has no attribute 'predict_classes' error when we use cross_val_predict
I want to use Bert through Tensorflow.
My plan is to use Bert with 5-fold cross validation, batch_size = 64, epoch = 100, and early stopping.
I have 5 classes in my data.
Therefore, I wrote the ...
- 99
1
vote
1
answer
55
views
Is there any TF implementation of the Original BERT other than Google and HuggingFace?
Trying to find any Tensorflow/Keras implementation of the original BERT model trained using MLM/NSP. The official google and HuggingFace implementations are very complex and has so much of added ...
- 4,304
1
vote
0
answers
104
views
How to get the index of predicted wrong data using BERT embedder with Keras?
I have a dataset that contain three columns, first column is DOI of different articles ,second column is abstract of related articles and third column is label of each articles(0 or 1).
I used the ...
- 13
1
vote
0
answers
557
views
Explaining a model built with BERT using SHAP with RepeatDataset and BatchDataset type objects
I have built a bit complex model using pretrained BERT weights. The model structure is as follows:
Model: "model_1"
...
- 420
1
vote
1
answer
278
views
How to use bert layer for Multiple instance learning using TimeDistributed Layer?
I want to perform Multiple Instance Learning Using Bert. A bag of instances contain 40 sentences. Each Sentence should output a label, and the final label should be average of all the labels.
I have ...
- 384
1
vote
1
answer
2k
views
Cannot import name 'network' from 'tensorflow.python.keras.engine'
When trying to load BERT QA I get the following ImportError:
"Cannot import name 'network' from 'tensorflow.python.keras.engine'"
The full error log follows below
Following this post,
...
- 41
1
vote
1
answer
557
views
'list' object has no attribute 'shape
I am passing an embedding matrix to the embedding layer in Keras
model = Sequential()
model.add(Embedding(max_words, 30, input_length=max_len, weights=[all]))
model.add(BatchNormalization())
model.add(...
- 1,107