So...how can you prove your work's not AI-generated?

LLMs aren't going anywhere, and neither are the accusations. Let's discuss.

Last week, amid a frenzy of fascinating takes on and coverage of SHY GIRL’s cancellation, I published an op-ed in the New York Times about how generative AI threatens to erode trust between writers, readers, and publishers. Here’s a gift link, and here’s the upshot:

…as A.I. models continue to improve, I’m concerned that it will become difficult to distinguish between something written by a human versus a bot. As more A.I.-generated writing is put out in the world, more readers will question whether the text they are poring over was penned by a human. We’re barreling toward a rapid erosion of trust between authors and readers, and the publishing industry is unprepared to deal with the consequences.

Readers still feel confident in their intuitive, internal slop detectors, noting (sometimes gleefully) the telltale word choices, rhythms, and patterns of popular LLMs. But this won’t be the case for long as each subsequent AI model is far more advanced than the last.

I tried to cover a lot of ground in this opinion piece, but I want to return to this one paragraph:

Several made-by-human certifications exist for written works. But many of these programs operate on the honor system, so the badge lacks teeth. In theory, the author of “Shy Girl” could’ve slapped the Authors Guild’s “Human Authored” stamp on its cover (it doesn’t require detection tools). Book contracts typically state that authors’ work must be original. But again, enforcement options are effectively limited to fallible A.I. checkers that often have a high rate of false positives — and vibes.

My first draft contained a bit more nuance here, and as is common in the editing process, my words got whittled down. (I also regret using the word “slapped” above—the truth is more complicated, as you’ll see below.)

The same day my op-ed came out, New York Magazine published a piece about the exact fear I expressed: The People Getting Falsely Accused of Using AI to Write. It’s already happening. And you can imagine the nightmare, Red Scare, worst-case scenario: Individuals accusing anyone they don’t like of AI plagiarism, leaving the defendant with few tactics to prove their work’s authenticity. Spiteful writers could ruin their nemeses; political figures could attempt to get unflattering books cancelled before they even hit shelves. Is this the future we want?

Let’s set aside some of the existential stuff for now and focus on the question: Can writers prove their writing is human-made? Below, I’m sharing (with permission) some fascinating exchanges I’ve had with experts on the topic as well as lightbulb-moment observations shared by y’all. Let’s dig in!

---

1. Some AI detectors are scams

Remember this Note I shared?

Many of you shared that some free online detectors are a cash-grab ploy to force you to pay for the “humanize” function. So they'll always give you a “high AI” score to start. This is…alarming, to say the least, considering some official-ish channels might think these checkers are legit.

Kostiantyn Dmytriiev’s comment, in particular, haunts me (here and throughout, I’ve added emphasis, line breaks for readability, and light edits for typos):

I somewhere read a story, not sure if true. Of how a university adopted an AI-checking tool. It marked most of the work as AI-generated. The catch was that the same company, like on the screen, was proposing a way to humanize the text. So, AI is marking something as an AI-generated, and then AI makes a text that it, itself will mark as a human text. This particular story was about writing a thesis, which was marked also as 80%+ AI. The author revised it, on his own, got 90%+ AI. Then dumbed it down to impossible, got 18%, but got rejected at the university, because the writing quality was so poor. So, the only solution was to buy an access on the platform to be marked as human text on the platform.

---

2. Some AI detectors are better than others

After my op-ed was published, an email came through to my assistant:

Outstanding and important article from Andrea in the Times this morning. I agree with her 99%. Maybe 100%.

But I wonder if it’s possible to seek a correction or clarification.

I’m Derek Newton (also a writer) and founder of Verify My Writing, which Andrea linked to in her piece, and which is great. We provide certifications that written work is human-written, not AI junk.

The problem is that where Andrea links to us, she writes, immediately after our link, “But many of these programs operate on the honor system, so the badge lacks teeth.” And this point is indeed a problem. She is right. But it does not apply to us. We are the only provider that checks, scores, and certifies. We are the only provider with teeth. And, as you can see, putting our company next to the line about the “honor system” and “badge lacks teeth” is, at best, not related to us.

As a writer, I get that it may be hard or complicated to try to correct or clarify that. But as it sits, readers would get the impression that Verify My Writing is an honor system solution when we are the opposite of that. It’s a crucial distinction for us.

Derek was very gracious (the NYT editors discussed and declined to update the story), and I appreciated the context. To test out Verify My Writing, I pasted in my same (organic) writing sample: 100% human-generated. Then I pasted in ChatGPT’s Uncanny Valley version of my writing: 100% AI. Unlike the online tool I randomly used, Verify My Writing got it right!

This gave me some relief, but I told the founder/CEO I’d love to talk to him more about what’s going on under the hood, since this is going to be an arms race with ever-improving LLMs. Think how much better they’ve gotten since that initial release, right? Just last year, referencing a still-unreleased update, OpenAI CEO Sam Altman tweeted, “we trained a new model that is good at creative writing…this is the first time i have been really struck by something written by AI.” Gird your loins.

And keep an eye out because Derek and I are going to do an interview once I’m no longer on deadline! Please let me know if there are questions you’d like me to ask.

Edited on 4/4/26 to add: This Wall Street Journal article flags limitations and inconsistencies in the performance of Pangram, the technology underlying Verify My Writing:

My first step was to run the articles through Pangram, whose website allows four free scans a day. To my surprise, the same tool applied to the same articles produced different results. Mr. Blum’s piece came up 100% human; Mr. Spencer’s, 44% AI and 56% human, which would make it “mixed,” not “AI-generated.” Only Dr. Saphier’s article was still labeled 100% AI-generated. Complicating matters further, when I checked the researchers’ database, it labeled the Blum and Saphier articles “mixed” and only the Spencer one “AI.” To my mind, these wildly inconsistent results are enough to discredit any accusation based on a Pangram analysis.

Vauhini Vara reported in the Atlantic last week that a Washington Post editor conducted the same exercise with similar results. Pangram has updated its software since the study was conducted, according to Ms. Vara. Mr. Spero told her, in her paraphrase, that “the current iteration of Pangram . . . was designed to be more conservative . . . in flagging material as AI-generated, partly for fear of spreading false accusations.” Even so, “when he and Russell reran their data set of opinion articles through the current version, the underlying assessments were similar to those in the earlier iteration.”

I’m adding this here because I don’t want to convey the message that Pangram is perfect, like, yay, we’ve found a solution. We haven’t! Which is kind of the point; to me, even the sense of “I have no idea what really happened here” (which is how I feel about SHY GIRL right now) is going to erode reader trust. Like, “if we can’t be sure it’s human-written, why bother?” The same experience we’re having with news articles and likely-deepfake photos and videos right now. It’s a lot!

Edited again on 5/28/26 to add: Substacker Ron Charles shared a brief, nifty item about Verify My Writing and “certified human” seals, then added this coda:

P.S. The entire item you’ve just read was written by AI. To create it, I submitted a transcript of my phone interview with Derek Newton to ChatGPT and told it to generate a 500-word story in my voice. I then sent that AI slop to VerifyMyWriting.com and received an Authenticity Certificate for “100% Human Based.” The whole process — including my payment of $10 — took 90 seconds, or about the same amount of time it took for American journalism to die.

So that’s…distressing. I can’t repeat his experiment because I won’t use ChatGPT under any circumstances, but I’m back to believing that unless a written piece has an AI prompt left in it, or the author admitted to using an LLM, you simply cannot be sure. That’s why I think it’s more important than ever to communicate your own stance on AI usage.

I’m leaving my interview with Derek published for transparency. Again, if anything, this proves my point: AI is leading to massive distrust between creators, patrons, and their industries, and that is very, very bad.

---

3. The Authors Guild’s Human Authored certification has legal teeth

Mary Rasenberger, a lawyer and the CEO of the Authors Guild (whom, full disclosure, I worked alongside in Bartz v. Anthropic), reached out after reading my op-ed:

Hi Andrea, I loved your guest essay in the NYT and the way you explained the reader writer connection. It is an excellent exploration of the issues.

I do want to point out though that you can’t just slap on a human authored certificate.

You have to sign a license agreement where you represent in warrant that all of the text is human authored, except for a de minimis. As we say on our website and in the license agreement, the consequences for falsely stating that is that you are liable for contract reach, trademark infringement and consumer fraud. And we will be enforcing the trademark, meaning bringing actions against people who use it fraudulently. I just don’t think there are very many people who knowingly used AI to write text that are going to sign an agreement that states in bold what their liability is. But I could be wrong and that could change if readers start expecting it.

We are however considering whether to use a service like pangram—either to spot check, to use if there are reports, to get an additional certification or even to use it before certifying any work. Of course, we have the concern that there are false positives.

Here’s more info on the certification. I thanked Mary for her email and replied, in part: “As you mentioned, I do still fear that some authors—seeing the backlash to SHY GIRL’s AI usage, or even noting how James Frey’s latest didn’t “work” despite obvious publisher investment—could decide it’s worth the risk to pass AI-written work off as their own, and even to state as a marketing tactic that it’s ‘human-written.’ Some will do it knowingly and others will find ways to justify it to themselves: I totally rewrote what it generated, I was just using it to get the juices flowing, etc.” She replied:

Unlike the other certifiers, the Authors Guild is a not for profit whose mission is to support and protect authors, and we are not trying to make money from our certification, but only to give authors something that they want and that is useful to them. So , it is very helpful and important to us to get feedback from writers on what is most helpful or not, and we’ll make adjustments as necessary.

I do think many authors today are tempted to use AI and not disclose it, but the vast majority will not go out of their way to get a human authored certifcation for partially AI generated text and and risk liability.

We want the Human Authored certification to mean something to writers and readers, though, so if people still do not trust the certification without something other than certification to back it up, we may use AI detection. We are looking into spot checking for now using the best AI detection systems there are, with a robust appeals process, as well as partnering with services like Verify My Writing for additonal tested certifications – all of which will add costs.

---

4. For now, documenting the process might be our best bet

In practical terms, I draft my thrillers in Google Docs, which automatically saves my revision history, and I’ve seen authors encouraging others to save every draft of their work-in-progress in a separate, dated file, so they can document its painstaking creation and evolution. (And, yes, I see the irony in using a Google product when Gemini is part of the problem.)

For shorter assignments or quick-hit writing, though, I don’t see an obvious solution. Recently, when I posted an admittedly pat writing tip to Substack Notes, a stranger deemed it “infomercial slop.” The word choice felt intentional—and how could I prove the words were mine?

Vera Kurian, a genius and one of my favorite authors to follow on Substack, posted the following Note:

I was thinking today about AI detection and about how people are concerned that there is no reliable way of telling if text is written by AI. I think there is probably a not-too-difficult way of doing AI detection (please if you are a tech person, take this idea from me, for the love of god.)

The problem is that we should not be looking at the CONTENT of what people write (which is what these imperfect programs do), but HOW people write. If I am writing a 3k Substack article, I could show you a screenshot of my versions history where you could see multiple different versions saved over time with slow programs and small changes between them—this is a not perfect but reasonable way of showing you it was not written by AI.

When I write, I type about 50-60 words per minute, but make tons of typos. I write a few sentences, then pause to think. I write dialogue rapidly, but then I go back up two paragraphs and realize that I wrote “their” instead of “there.” I delete an entire paragraph when I realize it is stupid. In other words, there could be a signature of HOW I write that is inherently human.

AI, on the other hand— when I’ve seen Gemini or Claude write, you ask it a question and it “thinks” for a while, then it spits out a large amount of text. The text appears more rapidly than I could type. It does not do strange things like go back two pages and change something, or pause to look up “how fast can you kill someone by garroting” or consistently spell “bureaucracy” wrong.

If you were using AI within the writing platform, the behaviors of using AI to write text would not look human. It would also not look human if you copy pasted from a large amount of text from another location into Substack’s editor, for example. But of course many people write on word processing software.

But maybe this is something they can offer, the first improvement that might actually be interesting in years? What if Word actually worked on some form of authenticity thumbprint rather than that shit Copilot which no one uses? A thumbprint would be a better means of authentification than what the Authors Guild is proposing, which is the honor system.

Turns out this is actually becoming a reality…for example, Puddin’ “verif[ies] authorship at the moment of creation using behavioral signals and cryptographic proof.” (Thanks to Mary from the Authors Guild for the tip!) Their website says: “Behavioral writing signals are captured during composition and transformed into cryptographic evidence that can be independently verified.”

I’m interested to learn about other companies doing the same (and would love for Office to integrate this technology, as Vera mentioned). To continue my arms-race handwringing, though: Would this be easy for AI to fake? Will LLMs learn this is how they can circumvent AI detectors? Only time (and those who know more than I do) can tell. (Rest assured, I’ll keep looking into it!)

---

I do wish we had a WGA-like authors’ union that would (a) give members clear guidelines for LLM usage and (b) spell out what publishers can and can’t do with regards to AI-written work. (Do they want to replace us with cheap, fast, non-complaining robots?! Let’s hope not.) But alas, that’s a huge topic for another day.

So let’s give Benjamin Dreyer the last word, shall we?

Friends, I’d love to hear your thoughts! This is a huge and emotionally charged topic, so let’s keep it respectful and focused on the issue of proving the authenticity of our own work.

---

Further reading:

Why I sued Anthropic