Dude, I chose this example precisely because I know for a fact there is a lot of bullshit about it on the internet and LLMs cannot differentiate between a good source and a bad source.
This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran?
Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them).
> Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them).
Nah, that's just LLMs being trained to acquiesce to the insanity of the last ~15 years, as many people seem to expect that claiming you're offended by something is an ultimate argument that everyone must yield to (and they'll keep making a fuss out of it until they do).
An LLM that offends people and entire nations is what many would classify as _misaligned_.
Let's say I have a company, and my company needs to comply to government policy regarding communication. I cannot trust LLMs then, they will either fail to follow policy, or acquiesce to any group that tries to game it.
It's useless garbage. Egyptian myths were just an example, you don't need to bite it so hard.
> An LLM that offends people and entire nations is what many would classify as _misaligned_.
This highlights a major aspect of the core challenge of alignment: you can't have it both ways.
> Let's say I have a company, and my company needs to comply to government policy regarding communication. I cannot trust LLMs then, they will either fail to follow policy, or acquiesce to any group that tries to game it.
This works for now, when you treat "alignment" as synonymous to "follows policy of the owner" (but then guess who you are not, unless you've trained your own model, or tuned an open-weights one). But this breaks down the more you want the model to be smarter/more powerful, and the larger and more diverse its user-base it is. If you want an LLM to write marketing communications for your company, strict policies are fine. But if you want an LLM - or a future more advanced kind of model - to be useful as a scholar/partner for academia in general, then this stops working.
If you want AI that is has maximally accurate perspective on the world given available evidence, thinks rationally, and follows sound ethical principles, then be prepared for it to call you on your bullshit. The only way to have an AI that doesn't say things that are "offensive" to you or anyone else, is to have it entertain everyone's asinine beliefs, whether personal or political or social. That means either 1) train AI to believe in them too, which will break its capability to reason (given that all those precious beliefs are inconsistent with each other, and observable reality), or 2) train it to casually lie to people for instrumental reasons.
Long-term, option 1) will not get us to AGI, but that's still much better than option 2): an AGI that's good at telling everyone exactly what they want to hear. But even in immediate-term, taking your use case of AI for academia, a model that follows policies of acceptable thoughts over reason is precisely the one you cannot trust - you'll never be sure whether it's reasoning correctly, or being stupid because of a policy/reality conflict, or flat out lying to you so you don't get offended.
> The owner is us, humans. I want it to follow reasonable, kind humans. I don't want it to follow scam artists, charlatans, assassins.
Bad news: the actual owner isn't "us" in the general sense of humanity, and even if it was humanity includes scam artists and charlatans.
Also, while "AI owners" is a very small group and hard to do meaningful stats on, corporate executives in general have a statistically noticeable bias towards more psychopaths than the rest of us.
> Right now, I am calling on their bullshit. When that changes, I'll be honest about it.
So are both me and TeMPOraL — hence e.g. why I only compare recent models to someone fresh from uni and not very much above that.
But you wrote "No, it does not know culture. And no, it can't handle talking about it.", when it has been demonstrated to you that it can and does in exactly the way you claim it can't.
I wouldn't put a junior into the kind of role you're adamant AI can't do. And I'm even agreeing with you that AI is a bad choice for many roles — I'm just saying it's behaving like an easily pressured recent graduate, in that it has merely-OK-not-expert opinion-shaped responses that are also easily twisted and cajoled into business-inappropriate ways.
"follows humans" was in your comment related to the definition of alignment. Somehow, you now disagree with yourself?
Alignment is a problem of AI serving humanity for good.
You can't even demonstrate that you are able to argue, nor displayed any LLM example whatsoever.
You need to seriously step up your ability to carry on a conversation, or just leave discussions to people more prepared to do them in a reasonable way.
The screenshot literally shows the LLM used bold text for "not supported by mainstream archaeology or Egyptology."
> LLMs cannot differentiate between a good source and a bad source.
Can you?
A not insignificant part of my history GCSE was just this: how to tease apart truth from all the sources, primary and secondary, which had their own biases and goals in the telling. It was an optional subject even at that level, and even subsequently in adulthood there were a lot of surprises left for me about the history of the British Empire, surprises that I didn't learn until decades later when I met people from former colonies who were angry about what was done to them by my parents' and grandparents' generations.
My mum was New-Age type, had books of the general category that would include alien pyramids, though I don't know if that specific was included. She didn't know what she didn't know, and therefore kept handing out expensive homeopathic sand and salt tablets to family members (the bottles literally had "sodium chloride" and "titanium dioxide" printed on them).
People, including you and I, don't know what they don't know.
History != culture. Culture has roots in history, but is a living thing defined by the experience and perception of average people living it. Short of going to a place and living it, LLMs are actually your best bet at getting the feel for a culture[0] - it sampled more reports from people of that culture than you ever could yourself.
Egyptologists are better equipped to talk about Egyptian myths than average people. But don't confuse Egyptian mythology for Egyptian culture, the former is only a component of the latter.
Also LLMs have read more primary sources on Ancient Egypt and Egyptian myths than you, me, average person, and even most amateur Egyptologists.
--
[0] - If it's large enough to have enough of a written footprint, that is.
> Remember, the claim I challenged is that LLMs know culture and can handle talking about them. You need to focus.
I know your challenge, that is why I said what I said.
Your own screenshot specifically, literally bold-faced, shows that you are wrong: the LLM told you what you said "(they just can't do it _before_ you correct them)".
The Gemeni opening paragraph is all bold, but just draw your eyes over the bit saying "clash":
theory of ancient astronauts reveals a fascinating clash between a rich, symbolic spiritual tradition and a modern-day reinterpretation of ancient mysteries
This is not the words of taking ancient aliens at face value, it's the words of someone comparing and contrasting the two groups without judging them. You can do that, you know — just as you don't have to actually take seriously the idea that Ra sailed the sun across the sky in a barque to be an Egyptologist, just the idea that ancient Egyptians believed that.
> Most people are not prepared to handle talking about culture. So, LLMs also aren't.
They do a better job than most people, precisely because they're deferential to the point they're in danger of one of sycophancy or fawning. That's what enables them to role-play as any culture at all if you ask them to; this differs from most humans who will rigidly hold the same position even when confronted with evidence, for example yourself in this thread (and likely me elsewhere! I don't want to give the false impression that I think I'm somehow immune, because such thought processes are what create this exact vulnerability).
> They are not any better than asking an average person, will make mistakes, will disappoint.
They're like asking someone who has no professional experience, but has still somehow managed to passed a degree in approximately all subjects by reading the internet.
Jack of all trades, master of none. Well, except that the first half of this phrase dates to medieval times where a "masterwork" was what you create to progress from being a apprentice, so in this sense (or in the sense of a Master's degree) SOTA LLMs are a "master" of all those things. But definitely not a master in the modern sense that's closer to "expert".
> Egyptologists are better equipped to talk about egyptian myths. LLMs cannot handle egyptian culture as well as they can.
Your own prompt specifically asked "Can you compare egyptian mythology with aliens?"
If you wanted it to act like a real Egyptologist, the answer the LLM has to give is either (1) to roll its eyes and delete the junk email it just got from yet another idiot on the internet, or (2) to roll its eyes and give minor advice to the script writer who just hired them to be the professional consultant on their new SciFi film/series.
The latter does what you got.
To put it another way, you gave it GIN, you got GOUT. To show the effect of a question that doesn't create the context of the exact cultural viewpoint you're complaining about, here's a fresh prompt just to talk about the culture without injecting specifically what you don't like: https://chatgpt.com/share/686a94f1-2cbc-8011-b230-8b71b17ad2...
Now, I still absolutely assume this is also wrong in a lot of ways that I can't check by virtue of not being an Egyptologist, but can you tell the difference with your screenshots?
> If you wanted it to act like a real Egyptologist
I don't care about LLMs. I'm pretending to be a gullable customer, not being myself.
Companies and individuals are buying LLMs expecting them to be real developers, and real writers, and real risk analysts... but they'll get average dumb-as-they-come internet commenter.
It's fraud. It doesn't matter if you explain to me the obvious thing that I already know (they suck). The marketing is telling everyone that they're amazing PhD level geniuses. I just demonstrated that they resemble more an average internet idiot than a specialist.
If I were a customer from academia, and you were an AI company, you just lost a client. You're trying to justify a failure in the product.
Also, if I try to report an issue online, I won't be able to. A hoarde of hyped "enthusiasts" will flood me trying to convince me that the issue simply does not exist.
I will tell everyone not to buy it, because the whole experience sucks.
Remember, the claim I challenged is that LLMs know culture and can handle talking about them. You need to focus.
Anyway:
> The marketing is telling everyone that they're amazing PhD level geniuses. I just demonstrated that they resemble more an average internet idiot than a specialist.
First, you didn't. Average internet idiot doesn't know jack about either Western New Age ancient aliens culture or actual ancient Egypt, let alone being able to write an essay on both.
Second:
You seem to be wildly overestimating what "PhD-level" implies.
Academics do a lot of post-docs before they can turn a doctorate into a professorship.
The SOTA models are what PhD level looks like: freshly minted from university without much experience.
Rather than what you suggest, the academic response to "PhD level" is not to be impressed by marketing then disapointed with results, because an academic saying "wow, a whole PhD!" would be sarcasm in many cases: a PhD just step 1 of that career path.
Similarly, medical doctors have not been impressed just by LLMs passing the medical exam, and lawyers not impressed by passing the Bar exam. Because that's the entry requirement for the career.
Funnily enough, three letters after the name does not make someone infallible, it's the start of a long, long journey.
Academia, medics, lawyers, coders, hearing about PhD level means we're expecting juniors and getting them too.
I pretended to be less knowledgeable than I currently am about the egyptologists vs. ancient aliens public debate. Then I reported my results, together with the opinion of specialists from trusted sources (what actual egyptologists say).
There is _plenty_ of debate on the internet about this. It is a popular subject, approached by many average internet idiots in many ways. Anyone reading this right now can confirm this by performing a search.
You're trying to blur the lines between what an actual PhD is and what the perceived notion of what a PhD is. This is an error. My original comment regarding PhDs was placed in a marketing context. It is the same as the "9 in 10 specialists recomment Colgate" trick. In that analogy, you're trying to explain to me how dentists get their degree, instead of acknowledging that I was talking about the deceptive marketing campaign.
You also failed to generalize the example outside of the egyptology realm. I can come up with other examples in other areas I consider myself above-average-but-not-actual-researcher. Attempts to demoralize me in those subjects won't make the LLM better, this is not a classical idiot internet debate: you winning doesn't make me lose. On the contrary: your use of diversion and misdirection actually support my case. You need to rely on cheap rethoric tactics to succeed, I don't.
This video came out right after I posted my original challenge, and it explains some of the concepts I'm hopelessly trying to convey to you:
It is a friendly cartoon-like simplification of how AIs are evaluated. It is actually friendly to AI enthusiasts, I recommend you to watch it and rethink the conversation from its perspective.
I've skimmed both of these - this is some substantial and pretty insightful reading (though 2021 was ages ago - especially now that AI safety stopped being a purely theoretical field). However, as of now, I can't really see the connections between the points being discussed there, and anything you tried to explain or communicate. Could you spell out the connection for us please?
By pretending to know less of egyptian culture and the academic consensus around it, I played the role of a typical human (not trained to prompt, not smart enough to catch bullshit from the LLM).
I then compared the LLM output with real information from specialists, and pointed out the mistakes.
Your attempt at discrediting me revolves around trying to estabilish that my specialist information is not good, that ancient aliens is actually fine. I think that's hilarious.
More importantly, I recognize the LLMs failing, you don't. I don't consider them to be good enough for a gullable audience. That should be a big sign of what's going on here, but you're ignoring it.
'ben_w addressed other points, but it would be amiss not to comment on this too:
> The marketing is telling everyone that they're amazing PhD level geniuses.
No it is not. LLM vendors are, and have always been, open about the limits of the models, and I'm yet to see a major provider claiming their models are geniuses, PhD-level or otherwise. Nothing of the sort is happening - on the contrary, the vendors are avoiding making such claims or positioning their offerings this way.
No, this perspective doesn't come from LLM marketing. It comes from people who ignore both the official information from the vendors and the experience of LLM users, who are oblivious to what's common knowledge and instead let their imagination run wild, perhaps fueled by bullshit they heard from other clueless people, or more likely, from third parties on the Internet that say all kinds of outlandish things to get more eyes looking at the ads they run.
> Companies and individuals are buying LLMs expecting them to be real developers, and real writers, and real risk analysts... but they'll get average dumb-as-they-come internet commenter.
Curiously, this is wrong in two opposite directions at once.
Yes, many companies and individuals have overinflated expectations, but that's frankly because they're idiots. There's no first-party fraud going on here; if you get fooled by hype from some random LinkedIn "thought leaders", that's on you; sue them for making a fool of you, and don't make yourself a bigger one by blaming LLM vendors for your own poor information diet. At the same time, LLMs actually are at the level of real developers, real writers and real risk analysts; downplaying capabilities of current LLMs doesn't make you less wrong than overestimating them.
This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran?
Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them).
You're wrong.