What is the "human-machine" caution about anthropomorphising AI?

Machines are not humans: there is a gap between a machine's enormous performance and its complete lack of consciousness and (genuine) intelligence — and treating it as human-like obscures that gap.

Anthropomorphisation is the tendency to project human qualities — understanding, feeling, intention — onto systems that have none. Budelacci's warning is to keep the difference in view: a model can outperform us at narrow tasks while having no awareness, no understanding, no inner life. Because chatbots, robots and avatars are deliberately designed to seem person-like, they invite us to forget this. Deploying them therefore requires explicit reflection on the foundations and consequences of AI use, rather than sliding into talking to (and trusting) a machine as if it were a mind.

Go deeper:

• Wikipedia — ELIZA effect — how even a trivial 1960s chatbot led people to project understanding onto it.
• Wikipedia — Anthropomorphism — the general tendency to ascribe human traits to non-human things.