The other day Claude refused to do a meeting summary "I'm sorry, but this appears to be a private conversation between two individuals". Safety team/product team have conflicting goals, reminds me of tension between risk management and portfolio management arms in investment companies, Nassim Taleb wrote about working in risk management being one of the hardest things for career because your coworkers hate you
From personal experience, product teams can also be concerned about their products' safety. For moral and reputational reasons, but also because if there's a safety incident related to your product, you're the one who's getting a call in the middle of the night.
I was on a panel for UW-Madison undergrads a couple of years ago where a lot of the discussion was about academic dishonesty, and one of the students asked how worried I was about it, and I said, "I'm somewhat worried about you guys using AI instead of doing your homework but I'm much more worried about you guys using AI instead of having friends." I'm a superforecaster too!
I think mostly the narratives were set because the arc was known and it’s hard to change the culture even if their fundamental assumptions don’t apply. Makes for wild spectating.
I agree with Jessica's essay, but yours is written in a weirdly aggressive style that seemingly cannot resist the urge to sprinkle in a dunk every sentence. Are you really this mad, or are you trying to affect such a spirituality ugly communication style because you think it will give you views?
We recently published a scathing critique of the HHH framework which will hopefully help to bring down the allure of the BS alignment science behind which most of these claims are sold: https://link.springer.com/article/10.1007/s10676-025-09837-2
Completely missing the point. The only reason people put so much faith into these bots is because “AI safety” grifters have told them how powerful, godlike and powerful they are. The problem is created by “safety”people so they can get a place within the industry and high pay packages.
In the same way that removing friction has revolutionized transport, increased machine efficiency and has enabled to us to achieve our goals; friction is still necessary to maintain control. E.g. without friction you would be unable to walk, you would just keep sliding in the same direction.
It's a matter of calibration.
Likewise, chatbots remove friction in knowledge acquisition which is important in unlocking new frontiers and extending the capabilities of humanity. However, friction is necessary to prevent situations exactly like this - removing it everwhere would only lead to chaos
Would be interested to hear about different ways that friction can be imposed in a healthy way
+1 in the chat for all the long time Jessica Dai believers out there
HAHA
The other day Claude refused to do a meeting summary "I'm sorry, but this appears to be a private conversation between two individuals". Safety team/product team have conflicting goals, reminds me of tension between risk management and portfolio management arms in investment companies, Nassim Taleb wrote about working in risk management being one of the hardest things for career because your coworkers hate you
From personal experience, product teams can also be concerned about their products' safety. For moral and reputational reasons, but also because if there's a safety incident related to your product, you're the one who's getting a call in the middle of the night.
I was on a panel for UW-Madison undergrads a couple of years ago where a lot of the discussion was about academic dishonesty, and one of the students asked how worried I was about it, and I said, "I'm somewhat worried about you guys using AI instead of doing your homework but I'm much more worried about you guys using AI instead of having friends." I'm a superforecaster too!
Even worse with all the people hoping for continual learning / rl on the human users themselves.
Thanks for writing well on a challenging topic.
I think mostly the narratives were set because the arc was known and it’s hard to change the culture even if their fundamental assumptions don’t apply. Makes for wild spectating.
FWIW, it wasn't hard to get Claude to teach me how to grow Botulinum bacteria.
https://claude.ai/share/bb323c36-f1ef-403e-9f03-7d1cef17c349
We even got to discussing how to detect the bacteria. All in the name of food safety, of course.
I agree with Jessica's essay, but yours is written in a weirdly aggressive style that seemingly cannot resist the urge to sprinkle in a dunk every sentence. Are you really this mad, or are you trying to affect such a spirituality ugly communication style because you think it will give you views?
Thanks for the blunt and on point rant, Ben.
We recently published a scathing critique of the HHH framework which will hopefully help to bring down the allure of the BS alignment science behind which most of these claims are sold: https://link.springer.com/article/10.1007/s10676-025-09837-2
Completely missing the point. The only reason people put so much faith into these bots is because “AI safety” grifters have told them how powerful, godlike and powerful they are. The problem is created by “safety”people so they can get a place within the industry and high pay packages.
As you say, we've known these risks for decades, and social media's been around for decades. Why do you think we'll do anything about it now?
Friction is good
In the same way that removing friction has revolutionized transport, increased machine efficiency and has enabled to us to achieve our goals; friction is still necessary to maintain control. E.g. without friction you would be unable to walk, you would just keep sliding in the same direction.
It's a matter of calibration.
Likewise, chatbots remove friction in knowledge acquisition which is important in unlocking new frontiers and extending the capabilities of humanity. However, friction is necessary to prevent situations exactly like this - removing it everwhere would only lead to chaos
Would be interested to hear about different ways that friction can be imposed in a healthy way
That was a great post Ben. I think we are seeing the first of many such cases :(