Like many of my contemporaries, I’ve been experimenting with AI, and one of the bigger challenges I’ve run into isn’t around output quality, hallucinations, or other issues. No, the biggest issue for me has been the overconfidence AI tends to instill in the user.

Now that I think of it, South Park had an episode on the topic, called “Sickofancy”. In it, an AI assistant was overly encouraging to Randy’s obviously terrible ideas. Another source, winther’s essay on the pitfalls of AI-assisted writing briefly touched the topic, too.

More than once I tried to use AI for brainstorming, and AI convinced me of terrible ideas instead of offering a human’s healthy scepticism. I tried using various models for help with writing, and each time AI convinced me the output is wonderful, and each time I showed the output to my wife she said something to the tune of “This doesn’t sound like you at all, it reads more like a timeshare advertisement”. And she’s right every single time, because I’ve really struggled with getting meaningful critique from today’s chat bots.

This is an unsurprising finding, but I think it’s worth noting. Working with AI tooling today is like having a writing partner who’s read every book out there, but never experienced a single emotion in life and doesn’t know how to contextualize the ideas. I wonder why’s that?

Even when prompting the models to not be overly agreeable or requesting pushback - the models lack judgement. You ask them to evaluate your idea - they’ll spit out that it’s the best idea ever conceived. You ask for criticism, you’ll get told that it’s a terrible idea. Even rubber ducking - using an inanimate object as a sounding board that is - yields better results in my experience, since at least I get to utilize critical thinking.

This isn’t really an anti-AI rant or anything. The technology is here, you can’t put it back in the box, and there are real use cases out there (hey, I just saved myself a few minutes of fiddling with a spreadsheet formula by getting Gemini to do it) - but human overconfidence supported by AI is a real problem we’ll have to be mindful about.