Entertainment

Spotting AI writing: how reliable are the detectors?

Published: 

Author Thélyson Orélien is shown in this undated handout photo. THE CANADIAN PRESS/Handout — Francois Couture (Mandatory Credit)

Claims that Canadian-Haitian writer Thelyson Orelien used AI tools to write his best-selling debut novel are rocking the French literary world and posing questions about how AI can be detected.

An X account with a handful of followers sparked the controversy by claiming that the AI-detecting program Pangram had concluded the tale of a Haitian migrant travelling to Canada had been largely written by a text-generator such as Claude or ChatGPT.

Orelien, 38, denies the allegations and has said he will prove that he wrote the book, which has been sold to publishers in more than 20 countries.

“I didn’t write this novel with my head, but with my guts and my heart,” he told AFP recently.

AFP ran excerpts of the book through various AI detection tools and came up with a variety of results -- from it being “very likely” to have been written by a human to “100 per cent” AI generated.

Thelyson Orelien 'C'était ça ou mourir,' 'It Was Either That or Die' cover The cover of Thelyson Orelien's novel 'C'était ça ou mourir,' 'It Was Either That or Die' in English. (Credit: Indigo.ca)

How do AI detectors work?

Dozens of tools are now available online to detect AI generated text -- many of them designed for use in education or publishing.

They generally have large troves of AI-generated text alongside content written by humans -- and compare the two.

Their performance has improved significantly in recent years.

In a paper published in October 2025 called “Artificial Writing and Automated Detection,” researchers at the University of Chicago found Pangram “achieving near-zero error rates” on medium-to-long extracts.

AQR’s Asness, Liew Donate $60 Million to University of Chicago The University of Chicago campus. Photographer: Beata Zawrzel/NurPhoto/Getty Images

What are the clues?

Thierry Poibeau, a researcher at France’s national CNRS research institute and specialist in natural language processing, said the programs look for “linguistic tics.”

“The classic example is triadic phrasing (using three clauses), or em dashes,” Poibeau explained.

These telltale signs used to be common in generated texts because the tools had been trained on American scientific writing or American novels, he said.

Pangram claims it is able to “detect AI text by analyzing and understanding subtle cues in the writing.”

Some words or phrases show up regularly, while monotonous sentence structures are also a warning sign.

Carleton University humanities A student writes in a book at Carleton University in Ottawa. Sept. 17, 2026. (Peter Szperling/CTV News Ottawa)

Are the tools reliable?

Many tools publish percentage figures about their accuracy, but the figures are impossible to verify and do not take into account the risk of a false positive.

Pangram CEO and co-founder Max Spero stressed to the American outlet The Atlantic that the tool should not be the ultimate arbiter, but rather a starting point for a more in-depth investigation.

According to Thierry Poibeau, these detection systems also run up against a “moving target problem” because AI programs are “constantly evolving.”

The expert also points to a potential problem when it comes to detecting the use of AI in novels.

“The tool may pick up something particular in the writing, something repetitive, but that might be exactly what constitutes a writer’s style,” he said.

Orelien claimed this week his style was based on Haitian and Caribbean traditions, telling Liberation newspaper the software was “used to the French literary tradition and classical authors.”

Thélyson Orélien Haitian-born Quebecois writer, poet, and novelist Thélyson Orélien, poses during a photo session in Paris on Sept. 8, 2026. (JOEL SAGET / AFP via Getty Images)

What do the experts say?

Benoit Raphael, an entrepreneur and writer specialized in AI, published his own research using Pangram to test Orelien’s work -- and the results were also positive for AI.

“The fact that it rated at around 100 per cent AI, or even 90 per cent, is a pretty telling sign,” he told Nouvel Obs magazine.

And although the book was widely praised by critics and was flying off shelves in France before the scandal, some doubts had been raised.

Lea Bory on her podcast Torchon was possibly the first to say publicly earlier this month that she thought it had been written by AI because of the repetitive nature of the sentences and the over-use of analogies that are hallmarks of AI.

Writing in newspaper Canard Enchaine, Fabrice Colin also criticized the formulaic sentence structure and said he “vainly looked for a unique voice” in the prose.