A team I worked with had an Arabic support assistant scoring beautifully in evaluation and getting complaints in production. The eval set was a few hundred well-formed questions in Modern Standard Arabic. Their users were Saudi, writing the way Saudi people actually write to a support channel: Gulf dialect, English loanwords, occasional Arabizi, and no interest in formal grammar.
The model was fine. The evaluation was measuring a language variety that almost none of the users were speaking.
This is the most common failure in Arabic AI products, and it is entirely fixable.
MSA competence is not dialect competence
Modern Standard Arabic is the language of formal writing, news and documentation. It is what most Arabic training data over-represents and what most benchmark suites test.
Spoken and informally written Arabic diverges substantially by region. Gulf Arabic is meaningfully different from Egyptian and from Levantine — in vocabulary, in common verb forms, and in what a native speaker considers natural phrasing. A model that produces flawless MSA can still generate output that reads to a Saudi user as stilted, foreign, or subtly like a translation.
That perception matters more than accuracy in customer-facing flows. Users forgive an assistant that does not know something. They do not forgive one that sounds like it was built somewhere else.
Build the eval set from real input
Not from your imagination of how users write, and not from a translation of your English eval set — translated prompts carry English sentence structure and produce artificially easy inputs.
Sources that work:
-
Historical support tickets and chat logs, sampled across time rather than cherry-picked.
-
App store reviews in Arabic — an underrated source of authentically informal writing.
-
Search queries from your own product, which are shorter and messier than anything you would invent.
Sample deliberately across the messy dimensions:
-
Dialect intensity, from near-MSA through to strongly colloquial.
-
Code-switching, where English terms appear mid-sentence — extremely common in technical and commercial contexts.
-
Arabizi, Arabic written in Latin characters with numerals for sounds. Users do it; if your product does not handle it, they experience that as a broken feature.
-
Orthographic variation — hamza and alif variants, taa marbuta versus haa, missing diacritics — all of which are normal in informal writing and all of which can break naive matching.
-
Mixed-direction content, since real messages contain URLs, order numbers and prices.
A few hundred genuinely representative items outperform several thousand synthetic ones, every time.
Score what your users actually notice
Exact-match and BLEU-style metrics are close to useless here, because Arabic permits many correct phrasings of the same meaning and heavily penalizes surface comparison.
Score on dimensions that map to complaints:
-
Task success. Did it do the thing? Binary, judged by a native speaker.
-
Register appropriateness. Is the formality level right for the context? Overly formal MSA in a casual support chat is a real failure mode, not a stylistic preference.
-
Dialect naturalness. Would a Saudi speaker phrase it this way? This is the dimension that separates products users like from products they tolerate.
-
Factual grounding. Especially where the answer comes from your own documentation — hallucination in Arabic is harder for a non-Arabic-reading team to spot in review.
-
Safety and refusal behaviour, tested in dialect rather than only in MSA. Guardrails frequently degrade outside the language variety they were tuned on, which is a genuine risk worth measuring.
Use native reviewers for a scored sample and an LLM judge for scale, but calibrate the judge against the human scores first. An uncalibrated judge in a language your team cannot read is a confidence generator, not a quality signal.
Keep the harness provider-independent
Two practical reasons, one of which is specific to this region.
First, Arabic model quality is moving quickly, including from regional providers. You want to be able to re-run last week's eval against a new model tomorrow and get a comparable number.
Second, and more binding: for products serving Saudi Arabia, where inference happens can become a compliance question. Sending user content to a model API outside the Kingdom is a cross-border processing decision under PDPL, and sector regulators may constrain it further. If your quality benchmarks are entangled with one vendor's behaviour, you cannot actually move even when the interface allows it.
So: provider calls behind your own interface, prompts and eval sets versioned in your repository, and scores stored per model version so regressions are visible rather than anecdotal.
The last mile is human
Whatever the harness says, a native Arabic speaker with product judgment should review real outputs before every release, and there should be a defined path for what happens when the model is confidently wrong in a customer-facing flow — degrade to a deterministic response, escalate to a human, or decline. Silently shipping a fluent wrong answer in Arabic to a Saudi user costs more trust than an honest "I can't help with that."
The full guide — SAR costs, PDPL and residency, local integrations, and how to vet a development partner — is here: Mobile App Development Company in Saudi Arabia: The 2026 Buyer's Guide. Our approach is on the AI development services page.
Frequently Asked Questions
Is Modern Standard Arabic enough for a consumer product?
Rarely. Users write in dialect with code-switching and informal orthography. A model evaluated only on MSA can score well and still read as foreign or stilted to Saudi users, which is the perception that drives churn in customer-facing flows.
Where should Arabic evaluation data come from?
Real user input — support tickets, chat logs, app store reviews and your own search queries. Translated English eval sets carry English sentence structure and produce artificially easy inputs that overstate quality.
What is Arabizi and does it need handling?
Arabic written in Latin characters with numerals substituting for certain sounds. It is common in informal messaging, and if your product cannot parse it, users experience that as a broken feature rather than a language limitation.
Which metrics work for Arabic generation quality?
Not exact-match or BLEU. Score task success, register appropriateness, dialect naturalness, factual grounding and safety behaviour — with native reviewers on a sample and an LLM judge calibrated against those human scores for scale.
Do safety guardrails work equally well in dialect?
Often not. Refusal and safety behaviour can degrade outside the language variety a model was tuned on, so test guardrails in dialect explicitly rather than assuming MSA results transfer.
Why keep the evaluation harness provider-independent?
Because Arabic model quality is improving quickly, including from regional providers, and because inference location can become a PDPL compliance question. Benchmarks entangled with one vendor make switching impossible in practice even when the code permits it.