01/09/2019 at 10:25 pm

Hello, Can you show how to do this without GPU? Thanks

01/16/2019 at 5:10 pm

A summary of the resource usage/speed/accuracy of the various approaches might be of interest to your base. It is still relatively expensive to use a GPU/TPU.

01/16/2019 at 10:36 pm

Thanks for this great post. I have a question. Did you try adding bilstm-crf model ahead of the bert model during fine-tuning? Have you thought of that? Because I have implemented the same thing using tensorflow and I found out the performance is worser by adding bilstm-crf model. I don’t know whether it is true or not. Thanks!

01/22/2019 at 5:35 pm

Hi thanks for the code. I take it I save at the end of the training with
torch.save(model.state_dict(), ‘ner_model’)

then:
Load as:
model.load_state_dict(torch.load(‘ner_model’))

But do I need to do this:
model = BertForTokenClassification.from_pretrained(“bert-base-uncased”, num_labels=len(tag2idx))

+ FULL_FINETUNING = True ?

The validation accuracy when loading the model is vastly different to that outputted from training, making me think I didn’t load it properly.

Thanks

01/22/2019 at 5:42 pm

Hi, thanks for the tutorial,

what’s the method to load the saved weights? This:
model.load_state_dict(torch.load(‘PATH))

doesn’t replicate the accuracy from training. Making me think I’ve loaded it wrongly.

01/25/2019 at 4:03 am

Hi, thanks for sharing,

Have you ever tried to apply FP16_Optimizer to your codes ? I used your source-code and applied fp16 follow the logic given by “https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py”
, so I would use FP16_Optimizer and optimizer.backward(loss) as well. But found the error in optimizer.step():
…
File “/usr/local/lib/python3.6/dist-packages/apex-0.1-py3.6-linux-x86_64.egg/apex/optimizers/fp16_optimizer.py”, line 147, in step
grads_groups_flat.append(_flatten_dense_tensors([p.grad for p in group]))
File “/usr/local/lib/python3.6/dist-packages/torch/_utils.py”, line 194, in _flatten_dense_tensors
flat = torch.cat([t.contiguous().view(-1) for t in tensors], dim=0)
File “/usr/local/lib/python3.6/dist-packages/torch/_utils.py”, line 194, in
flat = torch.cat([t.contiguous().view(-1) for t in tensors], dim=0)
AttributeError: ‘NoneType’ object has no attribute ‘contiguous’

I went as deeply as I could, but still couldn’t fix. Could you please try and give me the ideas ?.

Thanks lots.

01/30/2019 at 3:26 pm

I noticed in the BERT paper (and the figure from that paper that you display in this article) there is an extra ‘[CLS]’ token at the beginning of an input sequence. Is this something we need to add manually?

02/12/2019 at 6:16 pm

Hi Tobias,
I used the same dataset and parameters as you. But the performance is very poor. I want to know if here’s anything I’m missing.
The following are the result from my training:
Epoch: 0%| | 0/5 [00:00<?, ?it/s]Train loss: 0.14929957927830842
Validation loss: 0.1385537704328696
Validation Accuracy: 0.9221349206349209

Epoch: 20%|██ | 1/5 [23:02<1:32:09, 1382.35s/it]F1-Score: 0.41764101560589006
Train loss: 0.11814392791347118
Validation loss: 0.12850322524706523
Validation Accuracy: 0.9272111111111115

Epoch: 40%|████ | 2/5 [46:05<1:09:07, 1382.64s/it]F1-Score: 0.4333100558659218
Train loss: 0.09879241194479371
Validation loss: 0.12466114262739818
Validation Accuracy: 0.9192996031746031

Epoch: 60%|██████ | 3/5 [1:09:05<46:03, 1381.72s/it]F1-Score: 0.4177507882751372
Train loss: 0.08402714759161112
Validation loss: 0.12018664911389351
Validation Accuracy: 0.9205309523809526

Epoch: 80%|████████ | 4/5 [1:32:07<23:01, 1381.80s/it]F1-Score: 0.4267321539268514
Train loss: 0.07209687749665365
Validation loss: 0.1262998983512322
Validation Accuracy: 0.9167210317460324

Epoch: 100%|██████████| 5/5 [1:55:09<00:00, 1382.00s/it]
F1-Score: 0.4178773530430766

Validation loss: 0.1262998983512322
Validation Accuracy: 0.9167210317460324
Validation F1-Score: 0.4178773530430766

02/13/2019 at 12:30 pm

Hi Tobias,
I just copy and paste your code. I didn’t change anything.
Every step was the same before the performance part.

-Juan

02/17/2019 at 11:28 am

Hi,
Thanks for the great post, it’s really clear and easy to read.
I am too facing similar issues – the NER performance of the model is poor even on the dataset used in your blog…
My code is similar to yours, expect for the following modifications:
1. According to its documentation the f1_score function receives two input params – ground truth (y_true) and prediction (y_pred) at that order, and not as demonstrated in the blog.
2. After each epoch I’ve printed the classification report to allow label-based analysis

Notice the result for 5 epochs:

# epoch 1
Train loss: 2.1515635643844253
Validation loss: 1.5441952099402745
Validation Accuracy: 0.9520636574074074
F1-Score: 0
precision recall f1-score support

org 0.00 0.00 0.00 379
per 0.00 0.00 0.00 272
tim 0.00 0.00 0.00 352
geo 0.00 0.00 0.00 595
gpe 0.00 0.00 0.00 289
nat 0.00 0.00 0.00 3
art 0.00 0.00 0.00 8
eve 0.00 0.00 0.00 8

avg / total 0.00 0.00 0.00 1906

———————————————-
# epoch 2

Train loss: 0.7686076947936306
Validation loss: 0.5796454784770807
Validation Accuracy: 0.9522557870370368
F1-Score: 0
precision recall f1-score support

org 0.00 0.00 0.00 379
per 0.00 0.00 0.00 272
tim 0.00 0.00 0.00 352
geo 0.00 0.00 0.00 595
gpe 0.00 0.00 0.00 289
nat 0.00 0.00 0.00 3
art 0.00 0.00 0.00 8
eve 0.00 0.00 0.00 8

avg / total 0.00 0.00 0.00 1906

———————————————-
# epoch 3
Train loss: 1.2535134771907772
Validation loss: 0.8845297023653984
Validation Accuracy: 0.9522557870370368
F1-Score: 0
precision recall f1-score support

org 0.00 0.00 0.00 379
per 0.00 0.00 0.00 272
tim 0.00 0.00 0.00 352
geo 0.00 0.00 0.00 595
gpe 0.00 0.00 0.00 289
nat 0.00 0.00 0.00 3
art 0.00 0.00 0.00 8
eve 0.00 0.00 0.00 8

avg / total 0.00 0.00 0.00 1906

———————————————-
# epoch 4
Train loss: 0.7686076947936306
Validation loss: 0.5796454784770807
Validation Accuracy: 0.9522557870370368
F1-Score: 0
precision recall f1-score support

org 0.00 0.00 0.00 379
per 0.00 0.00 0.00 272
tim 0.00 0.00 0.00 352
geo 0.00 0.00 0.00 595
gpe 0.00 0.00 0.00 289
nat 0.00 0.00 0.00 3
art 0.00 0.00 0.00 8
eve 0.00 0.00 0.00 8

avg / total 0.00 0.00 0.00 1906

———————————————-
# epoch 5
Train loss: 0.5286654621638633
Validation loss: 0.44472165405750275
Validation Accuracy: 0.9522557870370368
F1-Score: 0
precision recall f1-score support

org 0.00 0.00 0.00 379
per 0.00 0.00 0.00 272
tim 0.00 0.00 0.00 352
geo 0.00 0.00 0.00 595
gpe 0.00 0.00 0.00 289
nat 0.00 0.00 0.00 3
art 0.00 0.00 0.00 8
eve 0.00 0.00 0.00 8

avg / total 0.00 0.00 0.00 1906

Notice that model did not classify tokens to one of the target classes.
I suspect that the validation accuracy has (very slightly) as the other rates improved over time as the model just classified everything as “O”.

I than tried training the model for additional 30 epochs and noticed that the performance per class slightly improves… It seems as if it learns from scratch… which contradicts Bert’s promise to generate a semi-SOTA model in only a few epochs.

Please advise,
Danny

02/25/2019 at 7:58 pm

I have been using the PyTorch implementation of Google’s [BERT][1] by [HuggingFace][2] for the MADE 1.0 dataset for quite some time now. Up until last time (11-Feb), I had been using the library and getting an **F-Score** of **0.81** for my Named Entity Recognition task by Fine Tuning the model. But this week when I ran the exact same code which had compiled and run earlier, it threw an error when executing this statement:

input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts], maxlen=MAX_LEN, dtype=”long”, truncating=”post”, padding=”post”)

> ValueError: Token indices sequence length is longer than the specified
> maximum sequence length for this BERT model (632 > 512). Running this
> sequence through BERT will result in indexing errors

The full code is available in this [colab notebook][3].

To get around this error I modified the above statement to the one below by taking the first 512 tokens of any sequence and made the necessary changes to add the index of [SEP] to the end of the truncated/padded sequence as required by BERT.

input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt[:512]) for txt in tokenized_texts], maxlen=MAX_LEN, dtype=”long”, truncating=”post”, padding=”post”)

The result shouldn’t have changed because I am only considering the first 512 tokens in the sequence and later truncating to 75 as my (MAX_LEN=75) but my **F-Score** has dropped to **0.40** and my **precision** to **0.27** while the **Recall** remains the same **(0.85)**. I am unable to share the dataset as I have signed a confidentiality clause but I can assure all the preprocessing as required by BERT has been done and all extended tokens like (Johanson –> Johan ##son) have been tagged with X and replaced later after the prediction as said in the [BERT Paper][4].

Has anyone else faced a similar issue or can elaborate on what might be the issue or what changes the PyTorch (Huggingface) people have done on their end recently?

[1]: https://github.com/google-research/bert#fine-tuning-with-bert
[2]: https://github.com/huggingface/pytorch-pretrained-BERT
[3]: https://colab.research.google.com/drive/1JxWdw1BjXZCFC2a8IwtZxvvq4rFGcxas
[4]: https://arxiv.org/abs/1810.04805

03/01/2019 at 5:42 pm

Awesome post man, thanks a lot for sharing that, really great work!
I noticed that the code in your post run fine w/ good F1 Score if you run w/ pytorch-pretrained-bert==0.4.0. It should be somethine related either w/ tokenizer or BertForTokenClassification.

When I run w/ newer versions of pytorch-pretrained-bert (e.g. pytorch-pretrained-bert==0.6.1), F1 Score goes down as people is reporting.

**pytorch-pretrained-bert==0.4.0**
– F1-Score: 0.7806423229212494
– Train loss: 0.02669612845037273
– Validation loss: 0.039759145801266035
– Validation Accuracy: 0.9882277777777783

**pytorch-pretrained-bert==0.6.1**
– F1-Score: 0.4406215316315205
– Train loss: 0.09970482976470636
– Validation loss: 0.12539973077674707
– Validation Accuracy: 0.9194174603174605

03/08/2019 at 9:21 am

Hi Tobias,

i tried to use your code for conll-2003 dataset, but unfortunately i’m facing very much low accuracy.

why did you choose “finding evaluation accuracy for each batch and then finally computing overall accuracy w.r.t total batch size” over “from seqeval.metrics import accuracy_score” ?

because for me accuracy_score is coming more for seqeval compare to sum(flat_accuracy)/total_batch.

any reason ?

for more details (https://stackoverflow.com/questions/55058998/what-is-causing-f1-score-high-but-accuracy-low-in-a-deep-learning-model)

03/11/2019 at 3:36 am

Hi, Does BERT create word embeddings as part of model architecture? As I see we convert tokens to ids as input.

03/13/2019 at 5:45 pm

Hi

Thanks for sharing your work.

I am able to get the predictions just fine.
But i am finding it difficult to map the input batch to the original word form, after getting the predictions.

Any suggestions?

Thanks in advance.

03/19/2019 at 7:33 am

Replicated results from BERT paper on CoNLL-2003 dataset
https://github.com/kamalkraj/BERT-NER

03/21/2019 at 2:23 pm

Hi Tobias,
Thanks for the wonderful tutorials and great work in BERT NER, just curious to know the training time took for each epoch, tried in local with 16 GB RAM in i5 but didnt see anything after 1 hour and 15 minutes so have to kill the process manually. Tried in google colab with GPU acceleration, saw the first epoch completing after 45 minutes and showing the metrics. This is normal behavior? Please let me know if I need to improvise anything.

03/22/2019 at 4:50 pm

Hi and thanks for the post.
I followed the tutorial on this data-set and got very promising results (f1 score 80 %) . However, when trying to run train the NER on another model, I get much lower f1 scores (45 % is my best result) . I noticed something strange about the tokenizer and labeling and how to distribute the labels after the tokenization of the text.
I don’t understand how this example here handles post-tokens such as hyperdimesia = hyper + ##dimen + ##sia ( .
In the Bert paper. they suggest that post-fixes starting with ## must be labelled as X and we do not train or preform predictions on them. I’m wondering on how to do that and if that may be the reason my model is failing to train properly, considering my other data-set has a large number of words that are unknown to Bert and hence will be split up.
Thanks for your help

03/25/2019 at 8:26 am

Which one (of the 8) will be the best to try first and then tune the model, if the training dataset is very small [with ~1000 labeled senetnces and ~10 tags]?

03/28/2019 at 3:28 pm

Hello, Is there a way to get this to run on the 1060 with 6GB of memory? I tried and it gave the following error

RuntimeError: CUDA out of memory. Tried to allocate 7.13 MiB (GPU 0; 6.00 GiB total capacity; 4.61 GiB already allocated; 140.80 KiB free; 58.77 MiB cached)

Thanks

04/07/2019 at 11:05 am

How to perform prediction of the bert model like on elmo embedding
like model.predict(test_data)

04/08/2019 at 5:33 am

Even I wanted some help regarding how to use the trained BERT model to do the prediction. If someone could help with this it would be very useful. Thanks in advance.

04/08/2019 at 2:11 pm

I’m experiencing strange “non deterministic” effect. Running BertForTokenClassification for the same sentence several times I’ve got different results.
Any thoughts?

04/08/2019 at 2:33 pm

Hi Tobias, Thank you for article.
I’ve got some issues.
– running classifier several times I’ve got different results.
– how to clean context between documents?

04/28/2019 at 2:35 pm

Hi! Great tutorial! Just to add:
While I was creating pad_sequences I was getting an error that translated to argument given to pad_sequences is empty. on further inspection, I found that this line:

[[tag2idx.get(l) for l in lab] for lab in labels]

for breaking tokens to character level: B-GEO became ‘b’,’-‘,’g’,’e’,’o’. and thus it’s tag2idx was None everywhere.

To solve for this I used the code from your previous LSTM tutorial:

[[tag2idx[w[2]] for w in s] for s in sentences]

Making the code this:

tags = pad_sequences([[tag2idx[w[2]] for w in s] for s in sentences],
maxlen=MAX_LEN, value=tag2idx[“O”], padding=”post”,
dtype=”long”, truncating=”post”)

It works fine now. Thanks again for this blog post!

04/29/2019 at 10:00 am

Hey! Great post! I’ve a question on the max_norm parameter in clipping gradients. I see that it’s set to 1.0, how do you decide the right parameter to set here and what does it exactly mean to clip parameters to 1.0?
Would really appreciate your help here, thanks again!

05/01/2019 at 9:32 am

After bert_tokenizer length of tokenized_sen and thier respective labels are not matching.
so now ground truth is not valid.

05/06/2019 at 8:12 am

I wonder does this approach change weights of pretrained BERT or just classification layer?

According to the code below
no_decay = [‘bias’, ‘gamma’, ‘beta’]
optimizer regulates other layers.

05/07/2019 at 5:46 am

Hi Tobias,

Great Post! I want to know if there is a way to train custom entities using BERT. I have large amount of unsupervised text corpus with a limited number of labelled data-set (40-50 examples usually) and i want my custom entities to be trained using that data-set.
How can i achieve this using BERT ? I’ve been looking for this but unable to find an appropriate solution for this.
Any help is highly appreciated!

Thanks

05/11/2019 at 2:56 pm

Could you fix the identation for In [24]: and 23? or at least clarify?
Thanks for the excellent post

05/14/2019 at 9:16 am

Hi Tobias,

How can we calculate training accuracy? In the below code is there any way to get the training accuracy along with the loss?

loss = model(b_input_ids, token_type_ids=None,
attention_mask=b_input_mask, labels=b_labels)

05/19/2019 at 6:48 pm

Hi, I’m trying to run your code. But in the training section (specifically at this line:

loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)

I get the following error:

RuntimeError: Expected tensor for argument #1 ‘indices’ to have scalar type Long; but got CUDAType instead (while checking arguments for embedding)

If I don’t use CUDA I’ll get the same error but it finds CPUType instead.

05/29/2019 at 3:57 pm

Very nice work! Unbelievable!

Can I ask you if there’s a problem of indentation in the last snippet of code? I was trying to copy and paste but of course there’s a problem of indentation, I tried to do it by myself, but given that I’m a noob I made some mistakes for sure (even if the code is running).

05/31/2019 at 12:38 pm

Thanks for sharing your work.

I am able to get the predictions just fine.
But i am finding it difficult to map the input batch to the original word form, after getting the predictions.
can you please give any suggestions ,because i am working on different dataset

07/21/2019 at 1:10 pm

how to save the trained model? and also can you please share the test code for this testing this model..
Thanks in advance

07/25/2019 at 6:48 am

i want to make a model which find the entity(company name, name, address, phone number, email etc.) from business card but where to start it. we facing some problem like- our business card text is unstructure and person name and company name my be same. So how to start it. thanks

07/30/2019 at 7:52 pm

A cool and informative tutorial and equally enlightening discussions.

Won’t it be a problem treating B- and I- separately ? How do we interpret the result if the sequence of labels predicted by the model is “O I- B-” or “O I- I- O”.

Does demarcating the beginning and intermediate tokens of a mention span add any advantage ?

08/01/2019 at 2:11 pm

Hi Tobias,

Very nice work !
I have a question about the tokenization.
I know BERT tokenizer uses wordpiece tokenization, but when it splits a word to a prefixe and suffixe, what happens to the tag of the word ?

For example :
The word indian is tagged with B-gpe, let’s say it is tokenized as “in” and “##dian”.
Then what is the label corresponding to “in” and what is the label corresponding to “##dian” ?

08/18/2019 at 5:58 am

Hi,

When you tokenize a word and the BERT tokenizer splits a given word into more than one token (e.g. “aardvark” to [‘aa’, ‘##rd’, ‘##var’, ‘##k’]), then wouldn’t you need to extend the tags/labels accordingly? Otherwise you may be mapping other unrelated labels to the sub-tokens ‘##rd’, ‘##var’, ‘##k’.

Best,

Alvaro

09/11/2019 at 10:23 am

Hello,

thank you for the code and the explanation,

in my case I have a multi word targets, how can I tag multi word target after word tokenization ?
For a example for the sentence “I’m playing video-games”, after tokenization I will have “video” “-” “games”, wile my target is “video-games”.

Thank you in advance.

10/04/2019 at 7:08 am

Please let me know how to test with any input

10/11/2019 at 7:00 am

I am facing this issue. Can anybody help?

Can’t find a vocabulary file at path ‘models/bert-base-cased/vocab.txt’. To load the vocabulary from a Google pretrained model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`

10/20/2019 at 1:36 pm

Hi, why do you use RandomSampler during training? Will this add anything, or can be omitted?

11/11/2019 at 2:33 pm

Hi Tobias Sterbak,

Thanks for such a detailed post. I am trying this out for my project.
I am using it for Entity Detection hence using B-ENT,I-ENT, O format.

The only issue that I am now facing is while retrieving back the word text and labels.

I am using something like this :

val_inputtext = [tokenizer.convert_ids_to_tokens(ids) for ids in val_inputs.to(‘cpu’).numpy()]

Once I get back the words from this I am seeing that a word like ecg has now split into two words like
ec ,
##g
This certainly is an issue due to the masks that we have applied. I dont think this was our intended behaviour . Did you also face this ?Could you please look into this?

11/17/2019 at 9:30 am

Could you update the tutorial for the new version of the library?

11/17/2019 at 9:35 am

I have a dataset with BIO labels for each word, but where sentences constitute essays. And maybe there are long term dependencies. Do you think that I could benefit, if I used an essay (4-5 sentences each) as a sentence? First of all is this practically possible? I was thinking to give to the network 1) the sentences, as you do AND also 2) the essays (4-5 sentences). This means that the sentences will be seen two times from the network. But maybe there are issues with the gpu memory. What do you think? Could you provide some feedback with your thoughts?

12/03/2019 at 10:43 am

Google is now working more towards quality content, and easily search-able content and I think BERT update will enforce the voice optimization, even more.

12/04/2019 at 4:28 pm

Nice explanation, however you should use `bert_base_cased` instead of the uncased model. Unless your training data are lower-cased.
The performance will reach 91 F1 for CoNLL once you fix it.

12/08/2019 at 5:50 pm

Hi, I wonder about these wordpieces and their labels, example: Indian => Ind + ##ian. What happend with the output label? What will be happen to their labels(B-gpe)? We have to convert the label to B-gpe( Ind) I-gpe(##ian)?. Hope see your answer! Thanks so much!!

12/23/2019 at 11:33 pm

Dear Tobias, thank you! You are perfect in explaining materials in the clearest way.
I was searching to resolve the problem with tokenization. As discussed in the comments, BERT cut a word into pieces, such as
converting “Johanson” into “Johan ##son”. This paper discussed the possible solution in part 5.4:
https://arxiv.org/pdf/1906.02437.pdf
For ones who face the same issue like me the following comments also may be helpful:
https://github.com/google-research/bert/issues/646