Train NER with POS and DEP annotations – Error [E024]

[michel-ds]

Hi,

I am trying to add POS and DEP annotations as input to train a new NER model (and updating the tok2vec component from 'en_core_web_lg' vectors).

I'm receiving an error when trying to train the pipeline after the first epoch:
ValueError: [E024] Could not find an optimal move to supervise the parser.
My config:

[paths]
train = null
dev = null
vectors = "en_core_web_lg"
init_tok2vec = null
source = "en_core_web_lg"

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tagger","parser","tok2vec","ner"]
batch_size = 2000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "tok2vec"

[components.tok2vec]
source = ${paths.source}

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","NORM","PREFIX","SUFFIX","SHAPE", "DEP"]
rows = [5000,2500, 2500,2500,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.tagger]
source = ${paths.source}
replace_listeners = ["model.tok2vec"]

[components.tagger.model]
@architectures = "spacy.Tagger.v1"
nO = null

[components.tagger.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}

[components.tagger.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v1"
attrs = ["NORM","DEP"]
rows = [5000,2500]
include_static_vectors = false

[components.parser]
source = ${paths.source}
replace_listeners = ["model.tok2vec"]

[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = true
nO = null

[components.parser.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.35
accumulate_gradient = 1
patience = 2000
max_epochs = 0
max_steps = 30000
eval_frequency = 200
frozen_components = ["tagger","parser"]
annotating_components = ["tagger","parser"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.0001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null
dep_uas = null
dep_las = null
sents_f = null
tag_acc = null

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

The curious thing is that if I comment out the replace_listener line for the parser above like so:

[components.parser]
source = ${paths.source}
#replace_listeners = ["model.tok2vec"]

The error dissappears and all I get is a warning [W087] Component 'tok2vec' will be (re)trained, but the component 'parser' depends on it... which is expected given the parser now listenes to a component that is being re-trained.

Any ideas why this error occurs in the first scenario?

spacy debug config doesn't return any errors in either scenario.
spacy debug data does not complain about any whitespace issues.

========================== Named Entity Recognition ==========================
ℹ 3 label(s)
0 missing value(s) (tokens with '-' label)
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace

and I checked that gold_train_data["ws_ents"] is indeed 0.

Any pointers would be greatly appreciated!

spaCy version    3.1.4
Platform         Linux-5.4.0-1059-aws-x86_64-with-glibc2.17
Python version   3.8.12
Pipelines        en_core_web_trf (3.1.0), en_core_web_lg (3.1.0)

[adrianeboyd]

Before getting into the details: I wouldn't expect adding TAG or DEP as features to help the ner performance much, but there's obviously nothing preventing you from trying it out. We'd be curious about the results!

Since tagger and parser both listen to the same tok2vec, replacing the listeners will make the model larger and slower without much other benefit. And you can either source the components or train them from scratch with different features, not both, so you can't do this:

[components.tagger]
source = ${paths.source}

...

[components.tagger.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v1"
attrs = ["NORM","DEP"]

You can't change the attrs for a sourced component (it's already been initialized and trained with different attrs), and in addition source currently discards all these settings when it loads in the component. (We'd like to have the option to override some features where it makes sense, but this is still under development.)

Instead, for en_core_web_lg, I would recommend:

• source tok2vec, tagger, parser from en_core_web_lg with only source = and no other settings
• put them all in frozen_components and annotating_components (as I see you've already done above!)
• use an internal tok2vec instead of a listener with ner (look at the ner config for en_core_web_lg -- you basically just move the tok2vec config under ner.model.tok2vec; alternatively give the second tok2vec a custom name and change upstream for ner)

With a trf model, you'd need the original transformer for tagger + parser, plus a new transformer for the ner component for this to work well. (The final model will be twice as large and twice as slow.) Internal/inline transformer components should work better as of spacy-transformers v1.1, but if you run into problems (we've seen a few bug reports that we haven't been able to replicate), you can start out with a separate second transformer with a custom name+upstream.

[panamantis]

Great suggestions -- but it would be great if there was a specific example for such an optimum ner specific config

[michel-ds]

Hi @adrianeboyd,

Thanks for your quick reply. I have adapted my training config according to your suggestions and it is working now. 👍

For background information, I am training an ner component that identifies sources of statements and quotes so I was hoping that part-of-speech tagging and dependency parsing would help as sources tend to have the same gramatical roles in the sentences (quite often sentence/subordinate subjects).

That said, I only see a 1-2% performance increase, which is well within the error marging of indiidual training runs so that's dissapointing.

The reason for using a 'large' model rather than the RoBERTa-based transformer model en_core_web_trf, is that quotation marks (and other special characters like dashes) don't seem to have a representation in the model (following this article to derive a token vector equivalent). Thus, we see many more false positives than with the large model.