Amharic / Afaan Oromo News Topic Classifier

Type a sentence in Amharic or Afaan Oromo and get its news topic. The language is detected from the script and routed to the matching model, so there is nothing to select.

TF-IDF over character n-grams plus logistic regression, trained on MasakhaNEWS. No transformers, no GPU — the whole thing trains in about ten minutes on a laptop.

Macro-F1 of 0.92 on Amharic and 0.86 on Afaan Oromo, both on held-out test splits, where always guessing the most common topic would score 0.27 and 0.31 accuracy respectively.

---

Why these languages

Amharic and Afaan Oromo each have tens of millions of speakers and almost no public NLP tooling. Afaan Oromo in particular is badly served: the obvious sentiment corpus for African languages, AfriSenti, ships an Afaan Oromo training split containing zero rows — 396 validation and 2,096 test examples and nothing whatsoever to fit on. MasakhaNEWS was chosen precisely because it gives both languages a real training split and lets Afaan Oromo be a first-class half of the project rather than an afterthought.

Results

Both models were selected on validation and scored once on test.

Amharic — 4 topics, 373 test documents

Topic	F1 (full article)	F1 (headline only)	Test support
sports	0.962	0.968	95
politics	0.926	0.902	100
business	0.902	0.823	81
health	0.897	0.851	97
macro average	0.922	0.886	373

Afaan Oromo — 5 topics, 325 test documents

Topic	F1 (full article)	F1 (headline only)	Test support
sports	0.982	0.929	81
health	0.912	0.910	97
politics	0.902	0.893	100
technology	0.857	0.815	15
entertainment	0.645	0.656	32
macro average	0.860	0.841	325

The "headline only" column matters more than it looks. Training articles run to a median of 324 words in Amharic and 341 in Afaan Oromo against a 9-word median headline, but a visitor to the demo types a sentence. Headline-only accuracy is the number that honestly predicts what the demo does, so it is reported alongside the flattering one.

Three things the data turned out to demand

The two languages do not share a label set. Amharic has business and no entertainment or technology; Afaan Oromo is the reverse. Only politics, health and sports are common to both. So this is two models with two label spaces behind one interface, not one multilingual model — which is also the more truthful description of two different news landscapes.

The published Amharic splits leak. 34 articles appear in both train and test, and another 10 in both validation and test, so an untreated model can memorise about 9% of its own test set. Cross-split duplicates are dropped from the splits we fit on rather than from test, so the evaluation set stays comparable to the published benchmark. Together with 68 articles repeated inside the Amharic train split itself, cleaning costs 124 of the 1,311 published Amharic training rows and buys a number worth quoting.

Short input had to be trained for, not hoped for. Every training document is used twice — once as headline plus body, once as headline alone. This doubles the training set and, more to the point, puts short text inside the training distribution. The drop from article to headline ended up at 3.6 points for Amharic and 1.9 for Afaan Oromo, far smaller than the 10–15 expected.

Character n-grams beat word n-grams, enormously for Afaan Oromo

Both languages are agglutinative, so whitespace tokens are poor units. Best validation mean macro-F1 per configuration:

Vectoriser	Amharic	Afaan Oromo
word 1–2 grams	0.852	0.729
char_wb 2–5 grams	0.864	0.848
word + char union	0.849	0.845

The Amharic gap is modest, but Afaan Oromo gains 12 points from character n-grams. That is the opposite of what I expected going in — I assumed Ge'ez script would be the harder case for word tokenisation. Oromo's productive suffixing appears to fragment the word-level vocabulary badly enough that a 1,126-document training set never sees most surface forms twice, while character n-grams recover the shared stems.

What the models actually learned

Top-weighted n-grams per topic, which are legible even if you do not read the languages:

Topic	Amharic	Afaan Oromo
politics	ምርጫ election, ጦር war	waraana war, hidha dam
health	ጤና health, ክትባት vaccine, ኮቪድ covid	covid, fayyaa health
sports	ኳስ ball, ዋንጫ trophy	taph- play
business	ዋጋ price, ንግድ trade, ብር birr	—
entertainment	—	suura picture, aartii art
technology	—	app, interneeta

These are the right words, which is the cheapest available evidence that the model is not exploiting some artefact of the corpus.

Limitations

• entertainment is the weak class, at 0.65 F1. Interestingly it is not the smallest — technology has less than half the training data (50 rows versus 110) and scores 0.86. Technology news carries distinctive loanword vocabulary; entertainment is topically diffuse and, per the confusion matrix, scatters into health and politics. Volume was not the binding constraint here.
• Training sets are small, 1,187 and 1,126 documents after deduplication.
• News domain only. The model predicts what topic a news story is about. Conversational or technical text is out of distribution.
• Test set touched once, in evaluate.py. All tuning used validation.

Running it

python -m venv .venv && .venv/Scripts/activate   # source .venv/bin/activate on Unix
pip install -r requirements.txt

python src/data.py       # download, normalise, de-duplicate, cache
python src/train.py      # vectoriser comparison, fit, save models
python src/evaluate.py   # test metrics and confusion matrices
python app.py            # demo at http://127.0.0.1:7860

src/predict.py holds language detection and inference and imports no UI code, so the demo and the evaluation script call exactly the same path — what is demonstrated is what was measured.

Dataset

MasakhaNEWS (Adelani et al., 2023), CC-BY-4.0. Amharic and Afaan Oromo configs.

@article{adelani2023masakhanews,
  title   = {MasakhaNEWS: News Topic Classification for African languages},
  author  = {David Ifeoluwa Adelani and others},
  journal = {arXiv preprint arXiv:2304.09972},
  year    = {2023}
}