Behind the Build: Why the Assessment Scores Fluency Higher Than Accuracy (And the Story of the Decision)

When I built the fluency assessment, I made one design decision that everything else flowed from. When fluency and accuracy disagree on a sample, which one wins?

Most assessments weight accuracy higher. They were built for academic certification, where grammatical correctness is the proxy for "knows the language." We weight fluency higher. The assessment was built for working professionals, where the proxy is "can use the language to lead a meeting."

This is the story of that decision. The two weeks of arguing it out with myself, the three pieces of data that settled it, what the decision costs us, and the kind of user we under-serve as a result.

The fork in the road

A speaking assessment that uses AI scoring has to weight four sub-scores into one overall CEFR level: fluency, lexical range, grammatical range, and coherence. The four sub-scores rarely agree on a single sample. A speaker who produces grammatically perfect sentences slowly, with frequent self-corrections and long pauses, will score high on grammar and low on fluency. A speaker who produces continuous, structured, persuasive speech with two or three small grammar errors will score high on fluency and lower on grammar.

The weighting decides which speaker is rated higher overall. The decision is not technical. It is editorial. It depends on what the assessment is for.

For an academic exam (Cambridge, IELTS, Trinity), accuracy weights higher, because the use case is "can the candidate produce correct English under controlled conditions?" For a workplace assessment, the use case is different. The question is "can this candidate function in an English-speaking professional environment?" And in that environment, fluency wins.

I knew that. The question was how strongly to weight it.

The data that decided it

Three data points settled the question.

First, the correlation in our own learner cohort. I pulled the post-programme career outcome surveys from learners who had completed at least six months on the platform. The fluency sub-score from their initial assessment correlated with reported career outcomes (promotion, role change, senior meeting confidence) at roughly twice the strength of the grammar sub-score. The lexical range sub-score correlated about as strongly as fluency. The coherence sub-score correlated about as strongly as grammar.

That alone would have been enough to weight fluency higher than grammar. It would not have told me by how much.

Second, the workplace literature. The finding that fluency predicts perceived workplace competence more strongly than accuracy past a B1 threshold is not new. It is in the SLA literature going back decades. What is new is the willingness to act on that finding in an assessment design. Most academic assessments have not, because their use case is academic. Mine could, because the use case was not.

Third, the recruiter conversation. I spent two weeks calling recruiters and hiring managers who routinely assess non-native candidates for senior roles. I asked each of them, in different words, the same question: when you decide whether a candidate's English is good enough for a senior role, what are you actually listening for? Without exception, the answer was about flow, response speed, comfort under pressure, and ability to take the floor, not about grammar. The recruiters had been operating with an internal fluency-first model the whole time. I was just designing the assessment to match it.

The weighting

Fluency and lexical range together carry more weight than grammatical range and coherence combined. Fluency is the single highest-weighted sub-score. The two sub-scores that academic exams typically place at the top of the hierarchy (grammatical range and coherence in its academic sense) sit lower in the EFO weighting on purpose.

This is the single most consequential editorial decision in the entire platform. Everything downstream (the practice recommendations, the coaching focus, the Sophie session structure) is calibrated against this weighting. A learner who tests at B2 on our assessment is a learner who has demonstrated working-professional fluency, not academic precision. That is the user we know how to serve.

What this costs us

The decision has two real costs, and I think about both of them often.

The first is disagreement with other assessments. Our assessment will sometimes return a higher CEFR level for a fluent-but-slightly-inaccurate speaker than they would receive on a traditional academic exam. This confuses users who have been graded elsewhere. We explain the weighting in the result page, but a user who needs an "official" CEFR for a visa or a tender knows our score is not that score. We are honest about this. The free assessment is not a certificate. It is a diagnostic.

The second cost is a specific kind of user we under-rate. The very careful, very accurate, very slow speaker (often academic, often older, often from a culture where slow careful speech is the formal register) will test lower with us than they would on a traditional exam. Their grammar is excellent. Their lexical range is rich. Their fluency, by our criteria, is constrained, because the criteria are calibrated for working English where speed-of-response matters.

For most of these users, our score is the more useful one in their actual professional context. But for a minority, especially academic professionals whose role does not actually require fast conversational English, our score under-rates them. We accept this trade-off because the alternative is calibrating the entire assessment for a population we are not the right platform for.

The decision I would not change

Every six months, I look at the weighting again. The data has only ever moved one direction: toward higher fluency weighting, not lower. Each cohort of learners reinforces that fluency is the sub-score that predicts the career outcomes the learners actually want.

If I were starting again, the weighting might be 35-30-20-15, leaning even harder on fluency. The reason I have not made that move yet is that the current weighting is the one against which two years of recommendations are calibrated. Shifting the weighting would shift every recommendation downstream of it. That recalibration is more expensive than the marginal improvement would be worth.

But the direction is clear. The assessment is fluency-first, and the further into operation we run, the more confident I am that it should be.

TL;DR

The fluency assessment weights fluency and lexical range above grammatical range and coherence, with fluency the single highest-weighted sub-score. The weighting was decided by three data points: the correlation between fluency sub-score and learner career outcomes in our own cohort; the workplace SLA literature on fluency vs accuracy past B1; and two weeks of recruiter interviews that all converged on "we are listening for flow, not grammar." The cost is disagreement with traditional CEFR-aligned exams and under-rating of the careful-accurate-slow speaker. The benefit is that the score tracks real-world professional outcomes more closely than weightings that treat fluency and accuracy as equal. The direction of every re-evaluation has been toward higher fluency weighting, not lower.

More from the Behind the Build series