@risottobias when the system accidentally (I assume, that phrasing troubles me because it indicates the presence of will) generates a similar, but not accurate answer.
yeah more like a terrible name for fuzziness. The LLMs don’t know what words mean, or what true is. They just know how words go together, and as a result can write a ton of fiction interspersed with fact. The way they write doesn’t help, they’ve been tutored in writing in a confident, knowledgeable style... which given they can just write fiction is a bit of a poor choice.
LLMs simply don’t know what is factual and what isn’t, they merely statistically infer what is a highly probable thing to say next based on what it has received as input. Any correct output is mere coincidence or “lucky” in the sense that training data created very highly probable output for a given topic. Sometimes it simply statistically takes a wrong turn in large or small ways and mixes that into the reply. It is entirely made up but statistically plausible from a “ingested the entire internet into probabilities” perspective…
it is literally probabilistically generating the next word based on its neural net weightings. those weightings cannot be right every time for every context because, math, but also the LLMs are trained on the sum of written English accessible from the Internet.
similar to how the image detection systems are only 94% sure that's a face in the image. but tweak the input a bit and now it's an apple (adversarial machine learning). these systems are fragile and basically piles of numbers in matrices, without knowledge or logic encoded in them.
@theruran Joseph wrote an entire book trying to explain this effect and how computers aren’t actually intelligent after he invented Eliza and the entire computer science field immediately fell into the belief that it was, in fact, conscious and knowingly answering their questions.
I also find it confounding. I asked an LLM for the Metroid cheat code. It spit back “Justin Bailey.” Correct. Then I asked for some SimCity 2000 cheat codes and it spit back garbage. Like pure garbage made up fantasy cheat codes. Why? Surely both appear in the training corpus, probably many times over. Why the correct result for one but not the other?
building a response from an LLM involves following a sort of "pathway" through data. Like a very complex version of predictive text that takes a lot more variables into consideration when deciding how to interpret input and generate output.
You can think of a hallucinations as the algorithm sort of "taking a wrong turn" and going down a pathway that has factual errors or incorrect connections or what have you.
Yes, it's a lot more complex than that, but keeping it ELI5…
@theruran This is also why it consistently fabricates citations and books that don’t exist and assigns them to authors who didn’t write them. The best way to think about it is that it tries to statistically answer your prompt in the way that [a superposition of a random person on reddit|quora|stormfront] would be likely to respond to your prompt, one token at a time. So the responses sound “like” a reasonable response when they comply with your expectation bias, and they sound “like” a hallucination when they comply with your expectation bias but you also have the curse of knowledge to know it isn’t accurate.
@vortex_egg @theruran in short, LLMs are bullshit generators. With tuning, the bullshit can often have utility; sometimes it can even seem like it's not bullshit at all. But it's still actually just bullshit
Bullshitting this convincingly has some useful and practical applications, and it's an interesting advance in NLP and a couple other parts of AI research. But it's not even close to "thinking"
LLMs don't answer questions, they generate text that looks like what an answer to a question that looks like your question looks like. It's got enough text from the internet in it that something that looks like an answer is often the right answer because, well, your not as creative as you think and someone's probably had the same conversation you're having before and the LLM eavesdropped on it. Hallucinations are just when text that looks like an answer happens to not be right. It's not really a special case, it's just the other side of the same coin.
so let me get this straight... You're telling me, that some marketing guy wormed his way into the engineering team, and decided to anthropomorphize a pile of data by pretending it has a mind and a will?!?
Who would do such a thing? I find it irresponsible.
@montag I think you would be misreading the situation by denying that engineers are the ones driving this. If you go into the spaces of the people who are the biggest true believers that this stuff is *actually already sentient*, they are living and breathing linear algebra and python.
@vortex_egg @montag Not sure about this... not a single one of my engineering colleagues thinks this or is a fan of AI in general. It is seen as a new set of tools that might be useful for very specific applications instead.
@vortex_egg so, it's like (authors of) a TV show explaining something; it may sound plausible because they use terms from that field, but when you actually have knowledge of that field, you realise it quickly falls apart? @theruran @TheGibson
Stephen Wolfram explores the broader picture of what's going on inside ChatGPT and why it produces meaningful text. Discusses models, training neural nets, embeddings, tokens, transformers, language syntax.
My daughter, who has had a degree in computer science for 25 years, posted this observation about ChatGPT on Facebook. It's the best description I've seen:
{Insert Pasta Pun}
in reply to The Gibson 🅅 • • •you mean like, AI that intentionally does it? like daydream?
or when AI accidentally does it?
or when people hack into it?
The Gibson 🅅
in reply to {Insert Pasta Pun} • • •{Insert Pasta Pun}
in reply to The Gibson 🅅 • • •AI is built on a corpus of confident sounding responses.
it is trained to sound confident, even if there's no basis in reality.
the sentences that it tunes to are meant to seem realistic.
nothing about it is unit tested to whether it really works.
it writes fiction.
The Gibson 🅅
in reply to {Insert Pasta Pun} • • •@risottobias hmmm
Except it doesn't sometimes. When asked to do a specific function and then reverse that function, the answers don't always match, for example.
This is not a cute hallucination, but sort of befuddlingly useless.
silverwizard
in reply to The Gibson 🅅 • •@The_Gibson :hackers_town: @Risotto Bias I want to be clear that it *always* writes fiction.
ChatGPT can't run a function nor can it know what a function is. It just runs down a path in a tokenizer, it doesn't really run code or whatever .
Stefan Prandl
in reply to The Gibson 🅅 • • •The Gibson 🅅
in reply to Stefan Prandl • • •Stefan Prandl
in reply to The Gibson 🅅 • • •Johannes
in reply to The Gibson 🅅 • • •theruran 🌐🏴
in reply to The Gibson 🅅 • • •it is literally probabilistically generating the next word based on its neural net weightings. those weightings cannot be right every time for every context because, math, but also the LLMs are trained on the sum of written English accessible from the Internet.
similar to how the image detection systems are only 94% sure that's a face in the image. but tweak the input a bit and now it's an apple (adversarial machine learning). these systems are fragile and basically piles of numbers in matrices, without knowledge or logic encoded in them.
The Gibson 🅅
Unknown parent • • •@vortex_egg @theruran
Right, so it really is just RACTER with a bigger library.
Scott, Drowning in Information
in reply to The Gibson 🅅 • • •theruran 🌐🏴
in reply to The Gibson 🅅 • • •@vortex_egg it falls apart quickly when you prompt it do basic math. It's like it dug up some answers on Quora and Reddit and mixed them together.
I've found it's more helpful to prompt me, as a feedback mechanism to check my own knowledge. because its answers are frequently slightly wrong
Craig Linton
in reply to The Gibson 🅅 • • •calcifer
in reply to The Gibson 🅅 • • •building a response from an LLM involves following a sort of "pathway" through data. Like a very complex version of predictive text that takes a lot more variables into consideration when deciding how to interpret input and generate output.
You can think of a hallucinations as the algorithm sort of "taking a wrong turn" and going down a pathway that has factual errors or incorrect connections or what have you.
Yes, it's a lot more complex than that, but keeping it ELI5…
Scott, Drowning in Information
in reply to theruran 🌐🏴 • • •calcifer
in reply to Scott, Drowning in Information • • •@vortex_egg @theruran in short, LLMs are bullshit generators. With tuning, the bullshit can often have utility; sometimes it can even seem like it's not bullshit at all. But it's still actually just bullshit
Bullshitting this convincingly has some useful and practical applications, and it's an interesting advance in NLP and a couple other parts of AI research. But it's not even close to "thinking"
montag
in reply to The Gibson 🅅 • • •The Gibson 🅅
in reply to montag • • •so let me get this straight... You're telling me, that some marketing guy wormed his way into the engineering team, and decided to anthropomorphize a pile of data by pretending it has a mind and a will?!?
Who would do such a thing? I find it irresponsible.
I am Jack's Lost 404 reshared this.
Scott, Drowning in Information
in reply to The Gibson 🅅 • • •Kadsenchaos
in reply to Scott, Drowning in Information • • •Not sure about this... not a single one of my engineering colleagues thinks this or is a fan of AI in general.
It is seen as a new set of tools that might be useful for very specific applications instead.
The Gibson 🅅
in reply to Kadsenchaos • • •calcifer
Unknown parent • • •The Gibson 🅅
Unknown parent • • •@theruran @calcifer @vortex_egg
Seriously.
FiXato
in reply to Scott, Drowning in Information • • •@theruran @TheGibson
Sean Heber
in reply to The Gibson 🅅 • • •What Is ChatGPT Doing … and Why Does It Work?—Stephen Wolfram Writings
writings.stephenwolfram.comI am Jack's Lost 404
in reply to The Gibson 🅅 • • •I like this answer:
social.coop/@DrewKadel/1101540…
@DrewKadel
Drew Kadel
2023-04-06 21:43:08
theruran 🌐🏴
in reply to Scott, Drowning in Information • • •