b1g-fragment02 is only partially annotated in VUAMC — 9,287 unannotated all-POS training tokens carry gold label `0`

[PiPi-ineq]

Summary

While auditing the VUA-18 / VUA-20 training data I found that one of the 117 texts, b1g-fragment02, is annotated for metaphor only in part in the source corpus, but is present in full in the distributed VUAMC.xml. Because TEI encoding marks metaphor by the presence of a <seg function="mrw"> element, absence of annotation and a negative judgement look identical to a converter. As a result, vua_xml_parser.py assigned gold label 0 to roughly 9,287 all-POS training tokens that were never annotated by anyone.

This is a format/conversion issue, not an annotation error, and not a mistake in the shared task design. The corpus creators documented the coded extent correctly; the information simply is not machine-readable from the XML. I am reporting it here because the affected artefacts are the ones distributed from this repository, and because they are still widely used.

The test partition is unaffected — b1g-fragment02 falls entirely inside the training split (0 tokens in all_pos_tokens_test.csv and verb_tokens_test.csv). Published test scores remain valid; the effect is on what systems could learn, not on how they were measured.

Affected artefacts

Artefact	Release	What is affected
all_pos_tokens.csv (72,611 lines)	naacl_flp_train_gold_labels.zip, v1.0	10,925 tokens from b1g-fragment02; 9,287 of them (85.0%) are unannotated but labelled 0
verb_tokens.csv (17,240 lines)	naacl_flp_train_gold_labels.zip, v1.0	1,717 tokens from b1g-fragment02; 1,459 unannotated but labelled 0
vuamc_corpus_train.csv	generated by vua_xml_parser.py, naacl_flp_starter_kit.zip, v1.0	the same sentences appear without M_ prefixes

The annotated part of b1g-fragment02 is confined to two contiguous sentence bands, sentences 738–765 and 1485–1584. Everything outside those two bands is unannotated.

Evidence

Each of the following is independent of the others, and none of them uses a model or any training:

1. The corpus creators' own documentation. Steen et al. (2010), A Method for Linguistic Metaphor Identification, Appendix A, lists 3,006 lexical units as coded for b1g, whereas the distributed section runs to about 20,170 words. Herrmann (2013), Appendix A1, gives the same figure independently.

2. The distribution of the markup in the XML. Of the 637 mrw segments in b1g-fragment02, 634 fall inside the two bands and 3 outside. The boundary is sharp — annotation does not thin out gradually, it stops.

3. A second, independent annotation layer. The corresp attributes (a different annotator task from metaphor marking) draw the same boundary: 13 of 16 pairs lie inside the bands.

4. The published register statistics. Counting the unannotated part as annotated makes the academic register about 17,238 lexical units larger than the published corpus description.

5. The published metaphor rate. Excluding the unannotated part reproduces the published academic figure (18.46% against a published 18.5%); including it does not.

6. A corpus-wide audit. All 117 documents were tested with the same procedure for a "two bands plus a sharp boundary" pattern. Exactly one document shows it: b1g-fragment02. This is a single, localised defect, not a diffuse property of the corpus.

7. A model-free prior test. A leave-one-document-out lemma prior, computed only from the gold labels themselves, predicts about 1,634.9 metaphor positives in the unannotated region. The observed number is 3 (z = −53.66). In the Verbs band the unannotated region contains 0 positives out of 1,459 target tokens.

8. The shared task's own published anomaly. Table 2 of Leong et al. (2018) reports an academic-register metaphor rate of 14.41% in training against 24.14% in test — a roughly ten-point gap that has, to my knowledge, never been explained. Removing the unannotated region accounts for it.

Why the conversion produces this

vua_xml_parser.py walks the TEI tree and emits M_ for every token inside a <seg function="mrw">. It has no way to ask whether a given section was in scope for annotation at all, because the XML does not record annotation scope — only annotation results. For a fully annotated document the two are equivalent. For a partially annotated one they are not, and the difference is silently converted into supervision.

Measured impact

Fine-tuning bert-base-uncased as a token classifier on the official split, three seeds per condition, identical recipe throughout:

Condition	All POS F1	Verbs F1
BASE — as distributed	0.7070	0.7368
TRIM — unannotated region given no supervision	0.7200	0.7575
RAND — size-matched control: the same number of valid tokens removed	0.7040	0.7345

• All POS: TRIM − BASE = +0.0131, 95% CI [+0.0082; +0.0194]; TRIM − RAND = +0.0161 [+0.0115; +0.0210]; RAND − BASE = −0.0030, CI includes zero.
• Verbs: TRIM − BASE = +0.0207, 95% CI [+0.0112; +0.0307]; TRIM − RAND = +0.0230 [+0.0133; +0.0344]; RAND − BASE = −0.0023, CI includes zero.
• Document-level cross-validation over all 117 texts, evaluated on the token set common to both corpora: ΔF1 = +0.0081 [+0.0031; +0.0140], with a built-in null control (the fold containing b1g-fragment02, where both corpora train on identical data) measuring +0.0001.
• Confidence intervals come from a document-level percentile bootstrap (4,000 resamples), so they reflect document sampling rather than token-level pseudo-precision.

For scale: on the corrected data an unmodified bert-base reaches the organisers' official BERT baseline (0.718; Leong et al., 2020) with no architectural change. The correction is not decisive for the ranking — I have not re-run any submitted system — but it is the same order of magnitude as differences usually reported as modelling progress.

What I am not claiming

• Not that the corpus is wrong. The VUAMC annotation is what it says it is; the coded extent is documented.
• Not that the published rankings are invalid. Establishing that would require re-running every system on corrected data, which I have not done.
• Not that the test scores are affected. They are not.

Possible remedies

Roughly in order of effort, any of which would help:

1. A note in VUA-shared-task/README.md recording that b1g-fragment02 is only partially annotated in the source corpus, and that its unannotated tokens carry label 0 in the v1.0 training gold files.
2. A supplementary exclusion list — the token IDs of the unannotated region — distributed alongside the existing gold files, so that users can opt into the corrected supervision without breaking reproduction of earlier results.
3. A revised gold file in which the affected tokens are marked as unknown rather than negative (for example, a third label value, or simply omitted from the token list).

I would suggest against silently changing all_pos_tokens.csv and verb_tokens.csv in place: a great deal of published work trained on those exact files, and being able to reproduce the old numbers has value. An additive exclusion list keeps both paths open.

More generally: a conversion step of this kind needs to know the boundaries of the annotated range, and where it does not know them it should emit unknown, not zero.

Reproduction

Everything below is deterministic and needs no GPU:

• the exclusion list (token IDs of the unannotated region, for both bands),
• the two coded sentence-band boundaries,
• the corpus-wide audit script that finds the pattern in 1 of 117 documents,
• the lemma-prior test,
• the register-statistics recomputation,
• the full correction-experiment recipe, per condition and per seed, with SHA-256 manifests of inputs, code and outputs.

Deposited at: https://doi.org/10.5281/zenodo.22216961

Primary sources used: VUAMC.xml from the Oxford Text Archive (record 2541) and the v1.0 gold files from this repository.

Related

A manuscript describing the audit is in preparation. I am reporting the finding here first, ahead of submission, so that it can be discussed by the people who maintain the resource before it appears in print. I am happy to prepare a pull request for whichever remedy you consider appropriate, or to supply the exclusion list in whatever format is most useful.

I have sent a parallel note to the VU Amsterdam Metaphor Lab, since the question of the coded extent is ultimately theirs to confirm.

Thank you for maintaining this dataset — the fact that the conversion pipeline is public and reproducible is exactly what made this finding possible.

— Gergely Tóth, toth.gergely@kre.hu