Lost Between Scripts: Recovering Arabic Words from Noisy Learner Orthographies

Code and data-preparation pipeline for the EMNLP 2026 paper “Lost Between Scripts: Recovering Arabic Words from Noisy Learner Orthographies.”

This repository contains no ArabScribe data, enriched data, model predictions, or evaluation results. Obtain ArabScribe from the CAMeL Lab ArabScribe page.

Overview

People often search for a word while remembering only an approximate spelling and part of its meaning. This work studies that problem for noisy Arabic and Roman-script learner transcriptions. We formulate gloss-grounded intended-word recovery: given a noisy transcription, an English meaning gloss, or both, a model returns ten ranked Arabic word candidates.

The experiments show that combining form and meaning is consistently stronger than using either cue alone. The benefit is particularly clear for distorted and Roman-script input. Expert annotators show the same qualitative benefit from glosses, while rare targets, misleading spellings, and exact-form confusions remain difficult.

This repository supports the paper's:

• Word, Gloss, and Word+Gloss conditions;
• Arabic- and Roman-script evaluation;
• 0-, 5-, and 20-shot prompting;
• OpenAI Responses, Gemini, and OpenAI-compatible endpoints;
• Acc@1, Acc@10, and MRR under exact, undiacritized, and normalized matching;
• deterministic few-shot sampling, resumable runs, and cached gloss-only queries.

The paper benchmarks two API models, GPT-5.4 and Gemini-3.1-Pro, and two open-weight models, Llama-4-Scout and Fanar-2. The paper-run command runs the same two scripts, three cues, and three shot counts for any one model.

Repository structure

.
├── data/                           # instructions for local data preparation
├── docs/                           # schemas and run instructions
├── src/arabscribe_benchmark/       # reusable benchmark package and CLI
└── tests/                          # unit and integration tests

Installation

Python 3.10 or newer is required.

cd arabscribe-benchmark
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

This installs the benchmark CLI, NumPy for paper-compatible shot selection, openpyxl for gloss workbook preparation, and the openai SDK for OpenAI and OpenAI-compatible endpoints. The Gemini adapter uses Python's standard-library HTTPS client.

Data preparation

This project uses the ArabScribe dataset introduced by Zhang et al. (2017) and enriches its target words with curated English glosses derived from SAMA. After obtaining ArabScribe from the CAMeL Lab site and the separate gloss workbook under their respective terms, run:

arabscribe prepare \
  --arabscribe-dir ../Arabscribe \
  --gloss-workbook ../Word\ Lists.xlsx \
  --output-dir data/processed

If you have the corresponding gloss lexicon as a CSV with word and gloss_raw columns, use --gloss-csv path/to/lexicon.csv in place of --gloss-workbook. ArabScribe alone does not contain the English glosses needed for the Gloss and Word+Gloss conditions.

The command creates:

• train.csv and test.csv, containing tidy item-level transcriptions and raw and POS-stripped glosses;
• lexicon.csv, containing the 500 target words and gloss provenance;
• provenance.json, containing source hashes, counts, and validation results; and
• ARABSCRIBE_LICENSE.txt, copied from the source distribution.

With the paper's source files, all 500 target words receive a unique gloss mapping. The converter stops on missing or conflicting glosses, malformed user/transcription pairs, and unknown user IDs.

The original data and generated derivatives are intentionally excluded from Git. Review Data access and licensing before preparing or publishing them.

Run an experiment

Reproduce the paper conditions

The paper uses these model IDs:

Profile	Model ID	Access
gpt	gpt-5.4	OpenAI API
gemini	gemini-3.1-pro-preview	Gemini API
llama	meta-llama/Llama-4-Scout-17B-16E-Instruct	Local vLLM server
fanar	QCRI/Fanar-2-27B-Instruct	Local vLLM server

From the repo root, inspect the 18 cells for one model without making API calls:

arabscribe paper-run --model-profile gpt --dry-run

Set OPENAI_API_KEY and GEMINI_API_KEY for the hosted models, then run:

arabscribe paper-run --model-profile gpt
arabscribe paper-run --model-profile gemini

For Llama and Fanar, install and start vLLM in a separate environment on a machine with sufficient GPU memory. Serve one model at a time. In the server terminal:

vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --port 8000

In a second terminal with this repo's environment active:

arabscribe paper-run --model-profile llama --base-url http://localhost:8000/v1

Then stop that server and serve QCRI/Fanar-2-27B-Instruct at the same endpoint before running:

arabscribe paper-run --model-profile fanar --base-url http://localhost:8000/v1

Use --base-url with your server's host when it runs on another machine. The runner uses the vLLM OpenAI-compatible Chat API; vLLM and model weights are separate from requirements.txt.

Each invocation writes 18 prediction JSONL files and one metrics CSV under the ignored results/local/paper/ directory. Across the four profiles, this is the paper's 72 model/script/cue/shot cells. Use --limit 2 for a small local check, and rerun the same command to resume incomplete cells. The primary paper table uses the 0-shot rows with normalization=normalized.

After all four model runs, combine their prediction files into the full metrics table:

arabscribe evaluate \
  --data data/processed/test.csv \
  --predictions results/local/paper/*.jsonl \
  --output results/local/paper/all_metrics.csv

Run one condition

Each command runs one script, cue, and shot condition. For example:

arabscribe run \
  --provider openai \
  --model gpt-5.4 \
  --data data/processed/test.csv \
  --train-data data/processed/train.csv \
  --script roman \
  --cue word+gloss \
  --shots 0 \
  --request-diacritics \
  --output results/local/gpt-5.4_roman_word-gloss_0shot.jsonl

API keys are read from OPENAI_API_KEY, GEMINI_API_KEY, or OPENAI_COMPATIBLE_API_KEY; they are not written to result files. Runs resume by default, and completed item IDs are skipped. Use --overwrite to restart or --limit for a smoke test.

Provider-specific examples and the complete option reference are in docs/RUNNING_EXPERIMENTS.md.

Evaluate predictions

arabscribe evaluate \
  --data data/processed/test.csv \
  --predictions results/local/gpt-5.4_roman_word-gloss_0shot.jsonl \
  --output results/local/gpt-5.4_roman_word-gloss_0shot.metrics.csv

The evaluator reports Acc@1, Acc@10, MRR, coverage, malformed outputs, and duplicate candidates under all three matching regimes. Candidates are scored at their returned ranks and are not silently reordered or deduplicated.

Reproducibility notes

• The primary paper setting uses 0-shot prompting and normalized matching.
• Run arabscribe evaluate locally to calculate exact, undiacritized, and normalized scores for your own predictions.
• Few-shot demonstrations are deterministic for a fixed source order and random seed. The selected archived Gemini and local-model run seeds were not recorded, so exact 5- and 20-shot prompts cannot be guaranteed. New API runs may also differ because hosted models and serving systems can change.

Tests

PYTHONPATH=src python -m unittest discover -s tests -v

Data access and licensing

ArabScribe's bundled license permits internal research and evaluation use but does not grant redistribution or modification rights. Consequently, this repository does not track ArabScribe, its enriched derivatives, the gloss workbook, or identifying participant metadata. Obtain the resources through an authorized source and confirm that your permissions cover the intended use. See THIRD_PARTY_DATA.md for details.

Citation

If you use this code or benchmark setup, please cite the EMNLP 2026 paper:

@inproceedings{adel-etal-2026-lost,
  title     = {Lost Between Scripts: Recovering Arabic Words from Noisy Learner Orthographies},
  author    = {Adel, Mohamed and Shahid, Nisa and Habash, Nizar},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  address   = {Budapest, Hungary},
  year      = {2026}
}

If you use ArabScribe, please also cite the paper requested by the dataset's license:

@inproceedings{zhang-etal-2017-robust,
  title     = {Robust Dictionary Lookup in Multiple Noisy Orthographies},
  author    = {Zhang, Lingliang and Habash, Nizar and Toussaint, Godfried},
  booktitle = {Proceedings of the Third Arabic Natural Language Processing Workshop},
  address   = {Valencia, Spain},
  publisher = {Association for Computational Linguistics},
  pages     = {119--129},
  year      = {2017},
  doi       = {10.18653/v1/W17-1315},
  url       = {https://aclanthology.org/W17-1315/}
}

Both entries are also available in CITATION.bib, while CITATION.cff provides GitHub-compatible citation metadata.