Here is a readable version of this StoryScope article, which describes an attempt to sort out AI-generated stories not on stylistic grounds, but on higher-level characteristics, such as “discourse-level narrative choices such as character agency and chronological discontinuity.” This is really interesting, and many thanks to commenter MJ for drawing my attention to this experiment.
This article begins this way: AI fiction is already under our noses. In March 2026, Hachette, a major publishing house, pulled the horror novel Shy Girl after it was flagged as 78% AI-generated, the first commercially published novel canceled over AI allegations. Nearly 20% of a sample of 14,000 self-published Amazon novels were flagged by Pangram as largely AI-generated, a figure that jumped 41% year-over-year. Overall, readers are increasingly being misled into purchasing AI-generated books attributed to human authors. If authors are unwilling to self-disclose AI usage, how can we address this issue?
And here are the main points, drawn from the abstract of the article, which I’m presenting here in bulleted form because that’s so much easier to read than a brick of text.
Can AI-generated fiction be distinguished from human-written fiction based on narrative choices rather than writing style? Previous research has focused primarily on stylistic signals, such as word choice and sentence structure. StoryScope instead examines narrative features, such as character agency, plot structure, and chronology.
Dataset:
- 10,272 writing prompts, each used to produce one human-written story and five AI-generated stories.
- Five AI models: Claude, DeepSeek, Gemini, GPT, and Kimi.
- 61,608 stories total, approximately 5,000 words each.
The researchers started with 10,272 existing human-written short stories, then used Gemini to reverse-engineer a premise from each story. They gave those reconstructed prompts to the five AI models and generated a whole whopping lot more stories, and then they examined the original human-written stories compared to the generated stories. I’m pretty sure that’s right.
I’m pausing here to say: someone REALLY needs to look at NOVELS, which are qualitatively different from short stories, especially quite short stories of 5000 words, which is about 15 pages, so I mean, these are short. I immediately conclude the researchers are (a) not novelists, and (b) did not ask a novelist for opinions about their study design. Or, I mean, if they did, the novelists they asked did not say, as I would have: The distinction between short stories and longer works is much more important than I think you may have allowed for in your design. You are essentially certain to find that AI-generation is much more suited to the generation of short stories, the shorter the better, rather than novels. For example, and this is just one of infinite possible examples, a novelist can set up information in chapter two that then pays off in a thoroughly unexpected way in chapter twenty. This happens all the time! The author has to set up what the reader knows, what each character knows, what each character believes other characters know, and how all those states change over time — and then create a satisfying payoff based on setup the reader noticed, but that wasn’t so blatant the reader exclaimed, “Hello, Mr. Chekov!” when they saw it. This is a TREMENDOUSLY more demanding task than anything you’re going to see in a short story, and novels are FILLED TO THE BRIM with this kind of complicated stuff. I’m sure we could all sit here for an hour explaining the many ways that novels are far more complicated than fifteen-page short stories.
I grant, short stories are easier to work with, which does not mean they are the best items to look at. I think nonfiction essays (Substack style) and novels are both much more relevant to discussions about generated prose, because both are just more numerous and (I’m pretty sure) have a lot more money involved than short stories. So, I see why these researchers chose to work with short stories, but honestly, I would rather see a smaller dataset of novels than a giant dataset of short stories.
However, here’s what they found:
Detection accuracy:
- Using narrative features alone, StoryScope achieved >90% success when distinguishing human-written from AI-generated stories.
- It achieved nearly 70% success when identifying which of the six sources (five AI models or a human) produced a story.
- Narrative features were sufficient to accomplish classification, even when stylistic features were excluded from the analysis.
- Thirty core narrative features captured much of the difference between human and AI fiction.
Ah, thirty features! I bet the statistics looks pretty complicated. Oh, these percentages are “macro-F1 scores,” not simple percentages? All right, I looked it up, and evidently a “macro-F1 score” combines two measures of classification performance, then averages the results across categories. The point of using this statistic is to prevent the much larger number of AI-generated stories from dominating the score. However, what matters here is, these scores mean the researchers achieved clear, strong separation between human and AI stories in their dataset. That’s enough for me; I’m not doing any kind of deep dive into macro-F1 vs F1 vs anything else.
Anyway, what were some of those narrative features?
- Themes are explained more explicitly. The narrator tends to comment explicitly on the theme, the dialogue delivers the theme, the resolution emphasizes understanding or acceptance.
- Plots tend to be tidy, straightforward, and focused on a single narrative track, with emotional and thematic conflict explicitly resolved.
- Stories tend to occupy a relatively narrow range of narrative possibilities, with the protagonist generally driving the resolution.
These stories were only 5000 words long! How many narrative tracks do you expect to see in a story that short? And who else should drive the resolution, if not the protagonist? However, no, apparently in generated stories, the protagonist drove the resolution nearly 70% of the time, and in human-authored stories, this was true less than 50% of the time.
Common characteristics of human-written stories:
- Protagonists’ choices are presented with greater moral ambiguity.
- Stories exhibit greater temporal complexity, including flashbacks and nonlinear structures.
- Narrative choices are more diverse overall.
- More frequently “unusual” — showing features unusual compared to “neighboring” stories.
All right, so more stories featured complicated narrative structures than I would have expected. Also, the StoryScope study does include factors I consider stylistic, such as use of all five senses, and “show don’t tell” hammering heartbeats and clammy palms and all the rest of the overdone show-show-show physicality. ChatGPT tells me chatbots get this from all the bad show-don’t-tell writing advice online, and I would bet people who “trained” AI chatbots don’t understand how stupid the hammering heartbeats get, or why and when it’s better to say, “He was afraid.” Also, human-written stories contain a whole lot more specific cultural references, as one would expect. Again, I’d call that stylistic, not structural. This conflation of narrative features and stylistic features weakens the central premise of the study, imo.
Do all AI Chatbots tend to handle fiction the same way?
- Claude: Relatively flat escalation of events.
- GPT: Frequent use of gossip as a plot mechanism.
- Gemini: A tendency toward external descriptions of characters.
- Gemini, DeepSeek, and Kimi tend to produce stories roughly half as long as the human originals (even when directed to aim for equivalent lengths).
Overall finding: AI-generated stories cluster together in a shared region of narrative possibilities, while human-written stories exhibit greater narrative diversity. Main conclusion: Differences in how stories are constructed—not merely differences in prose style—can help distinguish human-written fiction from AI-generated fiction.
Also, there’s no reason to think that this analysis would generalize successfully to commercial fiction, future AI models, different prompting practices, or human authors using AI tools as, say, a writing prosthetic.
Also, this study has very little to say about NOVELS, seriously, that is a major problem for generalizing most of the conclusions reached by this study. If someone tried to generate a novel, as I’m sure many people are, I would not be surprised if it turned out to be 100,000 words of repetitive description, dialogue, and emotional beats sprinkled through loosely connected or unconnected or pointlessly drawn-out events. A whole bunch of repetitive words would be a lot easier to generate than an actual novel. And it does seem to me that LLMs always want to reach pat resolutions as fast as possible, so maybe also a repeated tendency to try to do that, even though there are 96,000 words to go, so resolving everything right now at the 4000-word mark is a bad idea.
How about looking at features such as persistence of character style and transformation of character motivations, the plausible development of relationships, the successful integration of the plot arc with the relationship arc, long-term causality — I mean plot setup leading to payoff — and the existence and number of apparently unrelated narrative threads that eventually converge. Those are all narrative features rather than stylistic features, so how about those?
StoryScope is asking, “Do AI-generated short stories make different narrative choices than human writers make when writing short stories?” This is an interesting question, but it’s not quite the same as asking, “Can AI successfully use and manage the kinds of narrative choices that make novels work?” I vote for “No, not yet,” but I don’t think anybody is looking at that. Or maybe they are and I haven’t heard about it! I hadn’t heard of this StoryScope thing until commenter MJ mentioned it. If someone knows about a study looking at actual generated novels, please mention it. I know of at least one person who has generated lots of (short!) novels and put them on Amazon, and since I have figured out where to find examples, I should post about that some time.
Meanwhile, you can click through and read the whole thing about StoryScope yourself if you wish.
I enjoyed reading your analysis! More than just studying AI-generated novels, I do wish the StoryScope study had generated those, rather than the shorts (though I still think it’s useful for anyone who wants to take a gander at AI prose). A lot of the discourse surrounding AI writing also seems to focus on short stories when it comes to attempts to muddy the differences between human vs AI-written.
I do get the sense though that the sentiment that AI for creative writing is atrocious is growing–at least on Reddit. Instagram and Facebook seem to like telling others the lines are too blurry. Goodreads is a toss-up.
The one study I did find that looks at novels is this one: https://arxiv.org/pdf/2607.20349
I haven’t had a chance to fully look through it yet, but it’s the one responsible for recent waves that AI books make up over 40% of Amazon’s offerings, and the Daggermouth scandal (but like Shy Girl, many readers were already wondering if the book had AI content before the Atlantic blew it up).
— As an aside on that, I think it’s pertinent to note that the names Elara, Kael, and Lira appear in Daggermouth. If you’re not familiar, they are three of the most common names found in AI-generated… well, everything. And AI books especially. Search for “Elara and Kael”, and you’ll find an endless deluge of AI content where they show up. Lira is usually spelled Lyra, but maybe she’s feeling particularly stealthy here.
That said, I’m surprised this point isn’t brought up at all when the book’s controversy is discussed–but I think most people doing so in the first place haven’t even touched it or read any of it, and gloss over that a 60% AI detected result by Pangram over the course of an entire human-written novel is ridiculously difficult, and 60% doesn’t mean it’s a detection failure–it means editors and editing exist. —
You are right that a novel is far more extensive work than a short story. If you look into AI writing communities, or self-proclaimed gurus of using AI to write, you’ll find some will go through such extensive extents to extend their creative extensions to an LLM, that they look more like a project manager than anything. Thousands of words & pages upon pages of prompts + multiple LLMs are sometimes used by those who feel queasy at the thought of pecking down their own words. It still does you no good when you wrestle with the chatbot’s inclination to go for the most average, probable output it leans on at every turn, and it’s worse if you don’t know how to write or what actually makes AI writing stand out in the first place.
There is a “fun” exercise you can try if you have Pangram access beyond its free credits: Take one of the many Storyscope stories. Plug it in and see the AI score. Now try editing or rewriting every single sentence in your own words, keeping its original meaning and intent. Check the score now. Edit it some more and cut sentences here and there to try and lower the score. Check again. Then try writing it all on your own, the way you would write it. Check the score on that.
I tried this myself, and Pangram accurately detected when it was pure AI and when it was edited AI. Writing it in my own way flagged, unsurprisingly, 100% human. Now consider the effort it took to make the Storyscope short story flag human before we even go into anything else and apply that to a novel length AI word dump. Maybe imagine you’re a fledgling new writer without years of experience. Oh dear.
A second fun exercise is writing in such a way to trick Pangram into flagging a false positive, and it needs to be at least several thousands of words. This is difficult in its own way. I’ve tried this also, and let me tell you that it was exhausting. It was a constant fight to resist my own way of writing. I’m considering posting my results from these experiments at some point.
I’m digressing a bit… but what I’m trying to say is that other writers should try these exercises too. Not just to gain familiarity with AI writing and see why it’s damn hard to write that way consistently, but to see where the accuracy of the most reputable AI text detector stands for themselves–and why some folks have a lot to lose if it gains wider acceptance. I think that’s why some will post a tiny excerpt and claim it was falsely flagged AI by Pangram (or simply ask you to believe them anecdotally), which is like showing someone a piece of a jigsaw puzzle and expecting them to know what image the other 499 pieces represent.
Now granted, I’ve only tried this with _creative_ writing, so I can’t speak for Pangram outside of that, but the study’s mention of a 25% or higher AI-generated text detection in a novel being considered AI-written makes a lot of sense to me, because that does suggest AI was involved at a level more than just a flirty interaction at a bar as you pass through its city.
I could go on and on about this–and have–because it’s hard to stop once I start. I’ll end it with again, I enjoyed your analysis, and hopefully you find the study about the novels enlightening if you haven’t already seen it before. :)
I should pay for Pangram just so I can try experiments like this.
Also, I doubt very much that any AI will ever generate a sentence like, That does suggest AI was involved at a level more than just a flirty interaction at a bar as you pass through its city, which is literally made me laugh out loud.
I hadn’t seen that study about novels, and I’m very interested — thank you for dropping it into the comment!
$20 a month gets you more than enough… but the first week is free, and if you try to cancel a day or two before, they offer the first month at $11 (at least for now). A second cancellation attempt towards the end of that also offers another discount.
If you tried to get AI to create a sentence like that, I imagine it’d come up with something like:
“Whispers of AI’s scaffolding in the controversial text landed with mechanical weight, like a dive bar in a city that tried to seduce your dignity and kiss its ears.”
Glad to be of help. :)