The Word Winnower

Students are embracing the magical abilities of AI tools in ever-increasing numbers. Theological faculties and seminaries are scrambling to respond with appropriate ethical policies. Developers are forever finding new ways to foil AI detection. Unsurprisingly, specialist theological editors and proofreaders (such as yours truly) are now facing the same fate as the dodo. And isn’t only from self-interest that I consider this a bad thing.

Disturbing Discovery # 1

In an article promisingly titled “Dangers of Artificial Intelligence to Theology,” published on a conservative Protestant website, Aaron Isaacs explores the “epistemological challenges” of dealing with AI-generated theological text. These challenges include, among others, difficulty detecting the subtle biases, distortions, and doctrinal errors that result from AI’s unreflective automated sweep of all available web-based sources.

The other main theme of Isaac’s article is the way AI undermines the orthodox understanding of the human being made in the image of God. When AI performs the tasks once carried out by human beings, creativity is destroyed, ethical reflection is rendered unnecessary, and relationality and community (“grounded in the image of a relational God)” are eviscerated.

Oh with what premature smug satisfaction I read all this. So many valid points inviting deeper investigation and reflection. Isaacs certainly seemed to know his stuff.

Or did he?

Did Isaacs know anything at all?

 For alongside a photograph of the author, the words: “Aidan Isaacs (AI) is not a real person. His profile picture was generated using an Artificial Intelligence image generator, and this article was written using ChatGPT.”

Disturbing Discovery # 2

Once recovered from the shock of finding out just how easily I had been duped, I performed my own version of the experiment. I chose for the purpose an AI website that promised instant essays in theology (or indeed in any other subject).

My obliging AI assistant, who I shall call TheolBot, popped up at once to ask what it could do for me. “Come up with an argument against the Roman Catholic doctrines of priesthood and papacy from a Protestant perspective,” I said. (I chose this topic because it is something I know a fair bit about.)

TheolBot “thought” for no more than five seconds before presenting me with a perfectly respectable, logically-ordered piece covering all the main points of the Reformed argument. It was a little lifeless, granted; it didn’t seem to fully grasp the concept of the “priesthood of all believers”;  and the references were terrible (Wikipedia and popular theology websites), but overall it wasn’t bad, especially as it would have taken me several hours and considerable thought to produce something similar.

TheolBot was exceedingly proud of itself. “I have written a scholarly argument with a formal academic tone,” it said. Then, with more of that disturbing use of the first person: “Would you like me to modify any aspect of the argument or adjust the tone in any way?”

“The references are rubbish,” I replied.

 TheolBot agreed with unctuous servility:

“You’re absolutely right. Let me search specifically for more scholarly sources to strengthen the academic rigor of this essay.”

And so it did. In three more seconds it had added various points and references from the abstracts of theological articles it poached from sites such as JStor and Wiley.

One of many disturbing reflections from this encounter with TheolBot was the conviction that even if I were to rewrite the piece quite extensively and thus make it something vaguely resembling my own, the Bot would have already done the heavy lifting. Without requiring the slightest thought, research, creativity, or inspiration on my part, it had handed me a sound structure. This is not an insignificant point, for structure is often and arguably the most difficult (and important) element in a piece of writing.

Disturbing Discovery # 3

 I could not help feeling concerned about the temptations of this magic-wand technology, especially, although not exclusively, for struggling students. Just how easy would it be to get away with what can only be described as cheating? Clearly the next step was to run TheolBot’s essay through a couple of AI detection tools.

From Grammarly’s detection tool TheolBot’s work returned with the inaccurate verdict of “61% AI generated.” The second tool, Quillbot, was less easily deceived and rightly deemed the text 100% AI. This was a little reassuring, but only a little, for I had now become aware of the existence of many supplementary AI “humanization” tools.

Disturbing Discovery # 4

So I ran TheolBot’s piece through its “humanizing” process to see how it did. Not great, in fact. Indeed, almost laughably badly, as in the following examples:

TheolBot’s opening sentence:

The Protestant Reformation marked a decisive break with the Roman Catholic understanding of ecclesiastical authority, particularly regarding the nature of priesthood and papal supremacy.

TheolBot’s humanized version (in a style reminiscent of the Good News Bible minus the childlike line illustrations):

Let’s talk about one of the biggest breakups in history. No, not Taylor Swift and Jake Gyllenhaal – I’m talking about when the Protestants decided they’d had enough of the Catholic Church’s way of doing things.

Theolbot’s original text:

The Catholic practice of maintaining an unmarried priesthood has also faced significant Protestant criticism. Historical records indicate that this requirement, formalized at the Second Lateran Council in 1139, represents a human tradition rather than biblical mandate. This practice contradicts the apostolic example, as several of Christ’s apostles, including Peter himself, were married.

TheolBot’s humanized version:

Here’s something interesting: The Catholic Church required (and still requires) priests to stay single. But here’s the plot twist – many of Jesus’s original apostles were married, including Peter, who Catholics consider the first pope. It’s like requiring all teachers to be childless when the founder of your school system had kids. Makes you think, doesn’t it?

Indeed it does. All this makes me think very hard, although that is not an activity AI encourages. For laughably bad though this version was, TheolBot again offered me the chance to alter the tone and make further refinements. I had had enough of TheolBot by then, so I declined.

Disturbing Discovery # 5

All that remained was to see how the newly humanized version fared with the two AI detectors used earlier.

Grammarly’s tool approved of all the nonsense about Taylor Swift’s breakup, and declared the new text 100 percent human-generated. Quillbot was also fooled, only not so completely, judging the new version 65% AI, and 35% the work of human hands.  

This result is alarming, for matters are only going to get worse. There can be little doubt that in the coming years AI concealment will become increasingly effective and accurate detection correspondingly more difficult.

A Partial Solution

All of which returns me to the question with which I opened this piece: How can theological institutions maintain academic integrity in face of the increasing sophistication of AI tools?

The question is perhaps more relevant in theology and related disciplines than in other fields. For theology is relational in ways other disciplines are not, being born out of the relationship between the Creator and the created. Theology, as St Anselm famously said, is “faith seeking understanding.”

AI has neither faith nor understanding. As Andrew Davison concludes in an amusing (yet disturbing) recent article in the Church Times, in which he puts the recently-launched Roman Catholic virtual priest “Father Justin” through his paces: “Artificial intelligence cannot understand anything, but it is easier than ever to be fooled into thinking that it can.”

It is precisely for this reason that theological editors and proofreaders, often highly educated in theology, and generally deeply engaged in the writing processes of their clients, are as important as before in maintaining the academic integrity of theological institutions. Because unlike AI, which understands nothing, the competent theological editor has the discernment to solve problems that are more than just grammatical: being conceptual, doctrinal, ethical, and even aesthetical.