Code-switching is normal speech because bilingual people draw on the words, rhythm and context that best carry the thought. AI struggles because it needs examples of those switches in real speech, paired with reliable text and meaning, and those datasets remain scarce for many language communities.
At 6:40 p.m. in Kumasi, fictional Ama is holding a half-chopped onion over a bowl while her younger brother sends a voice note about a payment. He starts in English, switches into Twi to explain who the money is for, then returns to English for the amount. Ama needs the name before she replies. A wrong answer could send the money to the wrong person, and the transfer window is closing.
The hard part is not that either language appears in the message. The hard part sits in the joins: the proper name after a Twi phrase, the English financial term inside an otherwise Twi explanation, the meaning carried by emphasis rather than spelling. That is how people talk when both languages are available to them.
A language switch often carries meaning
People code-switch for practical reasons. One word may be more familiar in English. A phrase may feel warmer in Twi. A speaker may use both because the listener does too.
The switch can also signal context. “Medaase, but the payment no…” has a tone and relationship inside it. Forcing the whole sentence into one language can make the exchange feel stiff, or change what the speaker meant.
Speech makes the challenge deeper. Spoken language arrives with pauses, clipped words, background noise, overlapping voices and names that are hard to spell from sound alone. A system has to work out what was said, where one language ends and another begins, and what the combined sentence means.
Ama’s brother repeats the name once. This time, the assistant needs to recognise that the correction belongs to the name, rather than treat it as a fresh request. If it gets that wrong, Ama may lose the chance to fix the transfer before it goes through.
Scarce data leaves important speech underrepresented
AI learns patterns from examples. For code-switched speech, useful examples need more than a recording in Twi or English. They need natural speech where the languages mix, accurate transcripts that preserve the switches, and dependable links between what was spoken and what it means in another language.
That combination is difficult to collect. Everyday conversations are private. Recording them raises consent questions. Transcribing them demands people who understand both languages and the local ways speakers mix them. Translation adds another layer, because a literal rendering can erase tone, intent or a key word choice.
The result is a familiar gap. Systems may perform better on cleaner, more heavily documented language settings, then become uncertain when a speaker talks as they do with family, colleagues or friends. The failure can look small on screen, a missed word, an odd pause, an answer that ignores the correction. In a real exchange, it can break trust.
That is why visible errors matter. An assistant that does not understand should show the problem rather than quietly invent a confident response. It gives the person speaking a chance to correct it.
What CoSTA points toward
CoSTA, short for Code-Switched Speech Translation using Aligned Speech-Text Interleaving, puts attention on a central research need: speech and text have to be connected carefully when languages mix. Code-switched speech translation cannot treat audio, transcription and translation as separate chores that happen to share a sentence.
Alignment matters because timing and sequence matter. A word in English may clarify a Twi phrase that came immediately before it. A correction might arrive after a pause. A speaker may repeat a phrase with different emphasis. Research approaches that preserve those connections create a better basis for evaluating what a system heard and how it interpreted the full utterance.
For multilingual societies, the lesson reaches beyond translation. Building responsibly starts with the speech people actually use. That means collecting data with clear consent, including natural code-switching rather than filtering it out, and testing whether the system handles corrections, names and mixed-language turns.
It also means being precise about limits. A broad label such as “Twi support” tells someone very little about whether an assistant can follow the conversation they are about to have. A Twi interface alone does not guarantee a useful Ghanaian conversation.
Building for the speaker in front of you
When Ama tries again, she does not need a performance about artificial intelligence. She needs the assistant to keep up with the way her brother speaks, let her interrupt when the reply is too long, and make clear when it cannot confidently help.
That is a practical standard for products built around multilingual speech. Make it easy to speak naturally. Keep the person in control of what reaches the cloud. Let them decide whether history stays on their device. Give them a clear path to export or delete their data.
For builders, the work begins before a model answers. Ask whose speech appears in the dataset, whose switches were preserved, who checked the transcript, and what happens when the system is uncertain. For speakers, choose tools that welcome the mix of languages you already use and explain what happens to your words after you speak.
Ama finally hears the name clearly enough to confirm it. The onion is still on the board. Supper can continue.
Comments
No comments yet.