Skip to main content

Hey folks, let's have a little chat about construct validity --- this is the concept that if you're going to use a psychological test, there should be good reason to believe that a) the thing it's meant to be testing is real and b) results on the test reflect something about that thing.

spectrum.ieee.org/theory-of-mi…

>>


AI Outperforms Humans in Theory of Mind Tests

Large language models convincingly mimic the understanding of mental states
Eliza Strickland (IEEE Spectrum)
in reply to Prof. Emily M. Bender(she/her)

Prof. Emily M. Bender(she/her)
Prof. Emily M. Bender(she/her)
• •

So, even if a test has been established to have construct validity as a test relating to human cognition, you can't just throw it at a chatbot and take the results as meaningful.

What would it mean for a language model to have a theory of mind? How does string manipulation relate to that? Without answers to these questions, the tests are meaningless.

in reply to Prof. Emily M. Bender(she/her)

Dawn Ahukanna
Dawn Ahukanna
• •
I don’t know how you stay so calm and civil.
Spicy auto-suggest providing responses to a “rapid recall” prompt and eventually getting a passing grade does not mean understanding, of any kind, happened.
in reply to Prof. Emily M. Bender(she/her)

Shae
Shae
• •
Theory of mind is such a problematic concept & has done appalling harm to neurodivergent folks.
in reply to Prof. Emily M. Bender(she/her)

MylesRyden
MylesRyden
• •

I know I am preaching to the choir, but I can't even get past the headline. The ability to "convincingly mimic the understanding of mental states" is not the same as understanding mental states.

I can say the words that "mimic the understanding" of the Theory of Relativity, but that in no way makes me Einstein.

In fact I have said this many times to my SO...I can say the words about quantum mechanics (for example) but I have no actual idea what those words actually mean to an actual theoretical physicist.

in reply to Prof. Emily M. Bender(she/her)

tuban_muzuru
tuban_muzuru
• •

My old interrogation instructor in the Army taught me I would learn more from lies than truth.

Here's Google Gemini, obfuscating and lying, today. Paid version, advanced.

docs.google.com/document/d/1MZ…


Gemini does a Sgt. Schultz

Google Docs
in reply to Prof. Emily M. Bender(she/her)

silverwizard
silverwizard
•
@Prof. Emily M. Bender(she/her) I felt like their last paragraph was a little stretched out. They coulda just written "I didn't listen to a work Professor Bender said and shouldn't be reporting on this" and iit would have summed up their writing better >.<
@Prof. Emily M. Bender(she/her)
in reply to Prof. Emily M. Bender(she/her)

François Houste
François Houste
• •
“Tests show that Artificial Intelligence programs are twice as likely to develop neurosis than a human.”
in reply to Prof. Emily M. Bender(she/her)

Susan Kaye Quinn 🌱(she/her)
Susan Kaye Quinn 🌱(she/her)
• •
it's like constant propaganda to convince the world that AI is more than a plaigarism machine (and I think many of folks are getting high on their own AGI supply)
⇧