EXLMR: Vocabulary-Extended XLM-R for Tigrinya Text Classification

---

Overview

Multilingual pretrained language models such as XLM-R suffer from high out-of-vocabulary rates and heavy subword fragmentation on Ge'ez-script languages like Tigrinya, which are underrepresented in their training data.

EXLMR ("Extended XLM-R") addresses this for Tigrinya by extending xlm-roberta-base's subword vocabulary with Tigrinya-specific tokens, then fine-tuning the resulting model as a binary text classifier over Tigrinya social-media-style text.

A note on the label names: the model's output labels are negative / positive, inherited from the original training script. Based on manual inspection of the training data, the negative class is dominated by offensive language, slurs, and hostile/hateful comments — not merely critical or unfavorable opinions — while positive covers compliments, well-wishes, and other non-offensive content. In practice this model behaves as an offensive-language / hate-speech-leaning classifier for Tigrinya rather than a general-purpose sentiment analyzer. See Model Behavior for a concrete illustration.

---

Results

Evaluated on a held-out Tigrinya test set of 4,000 examples. Because load_best_model_at_end=True with metric_for_best_model="f1", the checkpoint shipped in this repository is the best of 3 epochs by weighted F1 (reached at epoch 1), not the final epoch:

Metric	Value
Accuracy	80.4%
F1 (weighted)	0.8024
Precision (weighted)	0.8160
Recall (weighted)	0.8043
Eval loss	0.6558

Per-epoch weighted F1 across the full run: epoch 1 — 0.8024 (best, shipped), epoch 2 — 0.7968, epoch 3 — 0.8020.

---

Supported Languages

Language	Script	ISO Code
Tigrinya	Ge'ez (Ethiopic)	tir

Tigrinya-distinctive grammatical markers (e.g. ካብ, እንታይ, ኣሎ, ድዩ, ኣይኮነን) appear thousands of times in the training data, while Amharic-distinctive equivalents are effectively absent. This model has not been evaluated on Amharic or other Ge'ez-script languages and should not be assumed to generalize to them despite the shared script and multilingual base model.

---

Model Architecture

XLM-R (xlm-roberta-base)
    └── Vocabulary Expansion (+30,145 Tigrinya subword tokens)
            └── Mixed embedding initialization (random + mean of source embeddings)
                    └── Fine-tuning: binary sequence classification

• Base model: xlm-roberta-base
• Tokenizer: original XLM-R SentencePiece vocabulary, extended with Tigrinya-specific tokens
• Vocabulary extension: 30,145 new tokens actually added (of 30,522 candidates; the rest already existed in XLM-R's vocabulary) → final vocabulary size 280,147 (verified to match the model's embedding matrix)
• Embedding initialization: each new token's embedding is the average of a random-normal vector and the mean of all pretrained token embeddings
• Training framework: Hugging Face Transformers, PyTorch

---

Installation

pip install transformers torch sentencepiece

---

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "Hailay/EXLMR"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = "እንቋዕ ብደሓን መጻእካ፣ ጽቡቕ ስራሕ ኢዩ"
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

prediction = outputs.logits.argmax(-1)
print(model.config.id2label[prediction.item()])

---

Model Behavior

Input: Tigrinya text written in Ethiopic script. Output: one of negative or positive.

Verified examples (actual model output):

Input:  እንቋዕ ብደሓን መጻእካ፣ ጽቡቕ ስራሕ ኢዩ        (a well-wish / compliment)
Output: positive   (confidence 0.999)

Input:  ደንቆሮ ኢኻ ትም በል                        (a mild insult)
Output: negative   (confidence 0.994)

This next example illustrates the labeling nuance above — a critical-but-not-offensive opinion is classified positive (i.e. non-offensive), not negative:

Input:  እዚ ኣገልግሎት ኣዝዩ ደስ ዘይብል ነበረ           ("This service was very unpleasant")
Output: positive   (confidence 0.995)

---

Training Details

Parameter	Value
Base model	xlm-roberta-base
Training examples	49,374 (label 0: 25,031 · label 1: 24,343)
Test examples	4,000
Added vocabulary tokens	30,145
Final vocabulary size	280,147
Epochs	3
Train batch size	16
Eval batch size	8
Learning rate	1e-5
Weight decay	0.01
Max sequence length	128
Model selection	best checkpoint by weighted F1

Training data source and annotation methodology are not documented elsewhere in this project — treat the offensive vs. non-offensive framing above as an empirical observation from inspecting label samples, not a confirmed annotation guideline.

---

Limitations

• Binary classification only — no neutral class or intensity levels.
• Functions as an offensive/hate-speech-leaning detector rather than a general sentiment analyzer: plain negative opinions without hostile or offensive language may be misclassified as positive (see the third example above).
• Trained and evaluated on Tigrinya only; no verified Amharic or other Ethiopic-script language coverage.
• Training data appears to be informal, social-media-style text; performance on formal, domain-specific, code-mixed, Romanized/Latin-transliterated, or dialectal Tigrinya has not been evaluated.
• Training data source and annotation guidelines are undocumented in the source project — verify the model's behavior against your own definitions before deployment in hate-speech-sensitive applications.

---

Related Work & Acknowledgements

• Base model: XLM-R (Conneau et al., 2020)

---

License

Not specified in the source project. Contact the repository owner for licensing terms before reuse.