Indigenous languages are under threat. Some 3 000 – three-quarters of the total – could disappear before the end of the century. That’s one every two weeks, according to UNESCO.
As part of a movement to protect such languages, New Zealand’s Te Hiku Media, a broadcaster focused on the Māori people’s indigenous language known as te reo, is using trustworthy AI to help preserve and revitalize the tongue.
Using ethical, transparent methods of speech data collection and analysis to maintain data sovereignty for the Māori people, Te Hiku Media is developing automatic speech recognition (ASR) models for te reo, which is a Polynesian language.
Built using the open-source Nvidia NeMo toolkit for ASR and Nvidia A100 Tensor Core GPUs, the speech-to-text models transcribe te reo with 92% accuracy. It can also transcribe bilingual speech using English and te reo with 82% accuracy. They’re pivotal tools, made by and for the Māori people, that are helping preserve and amplify their stories.
“There’s immense value in using Nvidia’s open-source technologies to build the tools we need to ultimately achieve our mission, which is the preservation, promotion and revitalization of te reo Māori,” says Keoni Mahelona, chief technology officer at Te Hiku Media, who leads a team of data scientists and developers, as well as Māori language experts and data curators, working on the project.
“We’re also helping guide the industry on ethical ways of using data and technologies to ensure they’re used for the empowerment of marginalised communities,” adds Mahelona, a Native Hawaiian now living in New Zealand.
Te Hiku Media began more than three decades ago as a radio station aiming to ensure te reo had space on the airwaves. Over the years, the organisation incorporated television broadcasting and, with the rise of the internet, it convened a meeting in 2013 with the community’s elders to form a strategy for sharing content in the digital era.
“The elders agreed that we should make the stories accessible online for our community members – rather than just keeping our archives on cassettes in boxes – but once we had that objective, the challenge was how to do this correctly, in alignment with our strong roots in valuing sovereignty,” Mahelona says.
Instead of uploading its video and audio sources to popular, global platforms – which, in their terms and conditions of use, require signing over certain rights related to the content – Te Hiku Media decided to build its own content distribution platform.
Called Whare Kōrero – meaning “house of speech” – the platform now holds more than 30 years’ worth of digitised, archival material featuring about 1 000 hours of te reo native speakers, some of whom were born in the late 19th century, as well as more recent content from second-language learners and bilingual Māori people.
Now, around 20 Māori radio stations use and upload their content to Whare Kōrero. Community members can access the content through an app.
“It’s an invaluable resource of acoustic data,” Mahelona says.
Such a trove held incredible value for those working to revitalise the language, the Te Hiku Media team quickly realized, but manual transcription required pulling lots of time and effort from limited resources. So began the organisation’s trustworthy AI efforts, in 2016, to accelerate its work using ASR.
“No one would have a clue that there are eight Nvidia A100 GPUs in our derelict, rundown, musky-smelling building in the far north of New Zealand – training and building Māori language models,” Mahelona says. “But the work has been game-changing for us.”
To collect speech data in a transparent, ethically compliant, community-oriented way, Te Hiku Media began by explaining its cause to elders, garnering their support and asking them to come to the station to read phrases aloud.
“It was really important that we had the support of the elders and that we recorded their voices, because that’s the sort of content we want to transcribe,” Mahelona says. “But eventually these efforts didn’t scale — we needed second-language learners, kids, middle-aged people and a lot more speech data in general.”
So the organisation ran a crowdsourcing campaign, Kōrero Māori, to collect highly labeled speech samples according to the Kaitiakitanga license, which ensures Te Hiku Media uses the data only for the benefit of the Māori people.
In just 10 days, more than 2 500 signed up to read more than 200 000 phrases, providing over 300 hours of labeled speech data, which was used to build and train the te reo Māori ASR models.
In addition to other open-source trustworthy AI tools, Te Hiku Media now uses the Nvidia NeMo toolkit’s ASR module for speech AI throughout its entire pipeline. The NeMo toolkit comprises building blocks called neural modules and includes pretrained models for language model development.
“It’s been absolutely amazing – Nvidia’s open-source NeMo enabled our ASR models to be bilingual and added automatic punctuation to our transcriptions,” Mahelona says.
Te Hiku Media’s ASR models are the engines running behind Kaituhi, a te reo Māori transcription service now available online.
The efforts have spurred similar ASR projects now underway by Native Hawaiians and the Mohawk people in southeastern Canada.
“It’s indigenous-led work in trustworthy AI that’s inspiring other indigenous groups to think: ‘If they can do it, we can do it, too,'” Mahelona says.