I thought I’d fork off a discussion of AI that appeared in another thread: Question for folks coming to the 2026 Rendezvous by boat
Some thoughts triggered by Rob’s comment below, and the discussion which followed it in that other thread:
My background is in AI and computer science. In addition to having done R&D on the topic since 1972, I also did three different stints in the U.S. government where I funded a lot of others in the field.
Computers do not make math errors. When Claude says it did, it’s effectively lying (even though, in fact, it doesn’t know even that it’s lying). Lemme explain why I say it’s lying but doesn’t know it.
This particular exchange is a good illustration of how Large Language Models (the tech behind all of these chatbots) actually work.
What they do doesn’t actually involve understanding your inputs or its own outputs at all. Instead, they use immense collections of data about text scraped from the internet. They “simply” treat what a user has said to them as a prompt to generate the most statistically likely string of words that might be produced in response.
There’s actually a lot more to it, but this is a good enough description for the purposes of explaining what it can and can’t do. Obviously, that’s not simple. The data analyses that go into being able to perform that feat are why data centers consume amounts of electricity which exceed that used by a number of entire countries.
But my point is, chatbots are telling you not what they know but what people are likely to say.
So, when Claude said it made a math error, it was simply because that’s the response most likely to follow being told of a math error. (Which I guess speaks well of mankind, since they produced the data showing that words admitting error are more likely.)
Anyway, because of the incredible quantity of data they’re using, these tools do remarkably well. However, to borrow a phrase from another context: deep down, they’re shallow.
They also have programmed-in biases. Why did Claude say, “Ha — great catch, you’re absolutely right…”? Because most chatbots have been tweaked to give greater than actual statistical weight to phrasings that appeared in longer interactions, and thus favor customer engagement.
Bringing this all back to using these tools for things like trip planning or other nautical decision making: the dictum of never relying on a single source of navigational information applies in spades to these tools.
Never ask just one chatbot, use multiple and compare. Brian Godfrey’s advice about setting them up to emphasize accuracy is helpful. A former student of mine who’s currently writing a textbook that covers these tools also recommends always asking multiple chatbots to critique plans, regardless of whether he generated them or another chatbot did.
If that seems like a lot of trouble, you can always ask yourself how critical the correctness of the answer is to you. Since chatbots are giving you the statistically likely answer, think of it as giving you an average answer for the context you’ve provided them. For a lot of things, that’s good enough.
But the devil will always be in the details. If a trustworthy answer’s critical, be very careful out there.
Consequently, while AI chatbots are likely to displace people who know the answers, they’ll be creating a huge need instead for people who know how to ask the right questions.
That’s a quick (?) technical rant. For a pithy analysis of the ethical side of this new technology, there’s an entertaining video of a show John Oliver did entirely on AI chatbots: Last Week Tonight with John Oliver (AI Chatbots)
-- Bob
