The collapse of the current American management of global supply chains isnāt exactly an optimistic expectation, but I guess it beats social media continuing as it is into the future and maybe a better global order will develop in the aftermath.
No77e is correctly noting the discrepancy between the rationalist obsession with eugenics and the belief in an imminent (or even the next 40 years) technological singularity, but fails to realize that the general problem is the eugenics obsession of rationalists. It is kind of frustrating how close but far they are from realizing the problem.
Also, reminder of the time Eliezer claimed Genesmithās insane genetic engineering plan was one of the most important projects in the world (after AI obviously): https://www.lesswrong.com/posts/DfrSZaf3JC8vJdbZL/how-to-make-superbabies?commentId=fxnhSv3n4aRjPQDwQ Apparently Eliezerās plan if we arenāt all doomed by LLMs is to let the genetically engineered geniuses invent friendly AI instead.
Itās a good blog series.
But just to point it outā¦ note the author still buys the AI hype too much. This post is criticizing Microsoft for missing out because OpenAI made that $300 billion deal with Oracle (with the assumption that Microsoft could have a similar amount of revenue from OpenAI instead). Except neither OpenAI nor Oracle has the money or means to carry out that deal, Oracle is struggling to raise the capital to fulfill their end and an analysis of time to bring data centers online suggest they canāt meet their target goals even with the money, and OpenAI doesnāt have the money to pay for their end, the revenue just isnāt coming in unless they somehow become more ubiquitous and lucrative than the entire market for, for example, all streaming services put together (thanks to Ed Zitron for that fun comparison).
I had hoped that with the whole āagentā push that we would start seeing more sane usage, like having AI be a fuzzy logic step in a chain of formal logic and existing deterministic tools
I think this is the best you can expect out of LLMs, and the relatively more successful āagenticā AI efforts are probably doing exactly this, but their relative success is serving as hype fuel for the more impossible promises of LLMs. Also, if you have formal logic and deterministic tools wrapping and sanity checking the LLM bitsā¦ I think the value add of evaporating rivers and firing up jet turbines to train and serve ācutting edgeā models that only screw up 1% of the time isnāt there because you can run a open weight model 1/100th the size that screws up 10% of the time instead. (Note one important detail: training costs go up quadratically with model size, so a 100x size model is 10,000x training compute.) I think the frontier LLM companies should have pivoted to prioritizing smaller size, greater efficiency, and actually sustainable business practices 4 years ago. At the very latest, 2 years ago, with the release of 4o OpenAI should have realized pushing up model size was the wrong direction (as they should have realized training Chain-of-Thought was not going to be the magic bullet).
And to be clear I still think this is really generous to the use case of smaller LMs.
On a more productive note, this feels likely to be tied in with the usual issues of AI sycophancy re: false positive rate.
I suspect this is the real limit. Claude Mythos might find real vulnerabilities, but if they are buried among loads of false positives it wonāt be that useful to black or white hat hackers and the endless tide of slop PRs and bug reports will keep coming.
I tried looking through Anthropicās āpreviewā for a description of the false positive rateā¦ they sort of beat around the bush as to how many false positives they had to sort out to find the real vulnerabilities they reported (even obliquely addressing the issue was better than I expected but still well short of the standard for a good industry-standard security report from what I understand).
Theyāve got one class of bugs they can apparently verify efficiently?
Memory safety violations are particularly easy to verify. Tools like Address Sanitizer perfectly separate real bugs from hallucinations; as a result, when we tested Opus 4.6 and sent Firefox 112 bugs, every single one was confirmed to be a true positive.
Itās not clear from their preview if Claude was able to automatically use Address Sanitizer or not? Also not clear to me (Iāve programmed with Python for the past ten years and havenāt touched C since my undergraduate days), maybe someone could explain, how likely is it that these bugs are actually exploitable and/or show up for users?
Moving onā¦
This process means that we donāt flood maintainers with an unmanageable amount of new workābut the length of this process also means that fewer than 1% of the potential vulnerabilities weāve discovered so far have been fully patched by their maintainers.
So its good they arenāt just flooding maintainers with slop (and it means if they do publicly release mythos maintainers will get flooded with slop bug fixes), butā¦ this makes me expect they have a really high false positive rate (especially if you rule minor code issues that donāt actually cause bugs or vulnerabilities as false positives).
Iāve read speculation that in 30-50 years people will have an attitude towards social media that we have towards cigarettes now.
That would be really nice but that scenario feels pretty optimistic to me on a few points. For one, scientists doing research were able to overcome the lobbying influence and paid think tanks of cigarette companies. I am worried science as a public institution isnāt in good enough shape to do that nowadays. Likewise part of the push back against cigarettes included a variety of mandatory labeling and sin taxes on them, and it would take some pretty major shifts for the political will for that kind of action to be viable. Well maybe these things are viable in the EU, the US is pretty screwed.
Old Twitter was terrible for peopleās souls.
It almost makes me feel sorry for the way the rationalists are still so attached to it. But they literally have two different forums (lesswrong and the EA forum), so staying on twitter is entirely their choice, they have alternatives.
Fun fact! Over the past few years, Eliezer has deliberately cut his lesswrong posting in favor of posting on twitter, apparently (heās made a few comments about this choice) because lesswrong doesnāt uncritically accept his ideas and nitpicks them more than twitter does. (How bad do you have to be to not even listen to critique on a website that basically loves you and take your controversial foundational premises seriously?)
tldr; one of the MIRI aligned rationalist (Rob Bensinger) complained about how EA actually increased AI-risk long-run by promoting OpenAI and then Anthropic. Scott Alexander responded aggressively, basically saying they are entirely wrong and also they are bad at public communications! Various lesswrongers weigh in, seemingly blind to irony and hypocrisy!
Some highlights from the quotes of the original tweets and the lesswronger comments on them:
Scott Alexander tries blaming Eliezer for hyping up AI and thus contributing to OpenAI in the first place. Just a reminder, Scott is one of the AI 2027 authors, he really doesnāt have room to complain about rationalist creating crit-hype.
Scott Alexander tries claiming SBF was a unique one off in the rationalist/EA community! (Anthropicās leadership has been called out on the EA forums and lesswrong for a similar pattern of repeated lying)
Rob Bensinger is indirectly trying to claim Eliezer/MIRI has been serious forthright honest commentators on AI theory and policy, as opposed to Open-Phil/EA/Anthropic which have been āstrategicā with their public communication, to the point of dishonesty.
habryka is apparently on the verge of crashing out? I canāt tell if they are planning on just quitting twitter or quitting their attempts at leadership within the rationalist community. Quitting twitter is probably a good call no matter what.
Load of tediously long posts, mired with that long-winded rationalist way of talking, full of rationalist in-group jargon for conversations and conflict resolution
Disagreement on whether Ilya Sutskeverās $50 billion dollar startup is going to contribute to AI safety or just continue the race to AGI.
Arguments over who is with the EAs vs. Open Philanthropy vs. MIRI!
Argument over the definition of gaslighting!
To be clear, I agree with the complaints about EA and Anthropic, I just also think MIRI has its own similar set of problems. So they are both right, all of the rationalists are terrible at pursing their alleged nominal goals of stopping AI Doom.
I did sympathize with one lesswrongerās comment:
More than any other group Iāve been a part of, rationalists love to develop extremely long and complicated social grievances with each other, taking pages and pages of text to articulate. Maybe Iām just too stupid to understand the high level strategic nuances of whatās going on ā what are these people even arguing about? The exact flavor of comms presented over the last ten years?
Eliezer is trying to get around that with some weird conditions and game on the prediction market question:
This market resolves N/A on Jan 1st, 2027. All trades on this market will be rolled back on Jan 1st, 2027. However, up until that point, any profit or loss you make on this market will be reflected in your current wealth; which means that purely profit-interested traders can make temporary profits on this market, and use them to fund other permanent bets that may be profitable; via correctly anticipating future shifts in prices among people who do bet their beliefs on this important question, buying low from them and selling high to them.
I donāt think that actually helps. But Eliezer is committed to prediction markets being useful on a nearly ideological level, so he has to try to come up with weird complicated strategies to try to get around their fundamental limits.
A Korean developer named Sigrid Jināfeatured in the Wall Street Journal earlier this month for having consumed 25 billion Claude Code tokensāwoke up at 4 a.m. to the news. He sat down, ported the core architecture to Python from scratch using an AI orchestration tool called oh-my-codex, and pushed claw-code before sunrise. The repo hit 30,000 GitHub stars faster than any repository in history.
Considering how one of the major use cases of llm coding agents is laundering open source and copy left, this is some well deserved payback to Anthropic imho.
This article is where I was getting the 2018 date from: https://theweek.com/feature/1020838/jk-rowlings-transphobia-controversy-a-complete-timeline
And since in that 2018 incident Rowling was trying to backpedal/downplay it, I assume before that she was keeping the mask firmly on.
Yeah. And it is not like Eliezer usually holds himself back from throwing out hot takes or inserting himself into conversations he is tangentially relevant to, so the silence is conspicuous in this case.
Being the kind of writer I am, whenever this comes up I am tempted to suggest ways it could have been done better.
The premise kind of does work in a setting like Harry Potter, the wizarding world is insular enough that a clever kid could bring in some new ideas. The problem is Eliezer wanted to throw in too many shortcuts. Its not enough for creativity with transmutations to give the protagonist a small edge, transmutation is made into the ultimate all-purpose spell so the protagonist can exploit it easier. The protagonist isnāt just moderately better at Patronus with some muggle psychology, his patronus can kill dementors. And the philosopherās stone is changed into some ancient atlantean super-magic, because fuck wizards ever inventing anything, and also instead of some moderate rate of its typical mythological powers it is super transmutation.
But, first, I am not glazing the work of Rowling, even indirectly, no way, no how.
Rowling went mask off transphobe in 2018, HPMOR finished in 2015. So I wonāt blame Eliezer for not picking a different fandom at the time. Eliezer has actually said moderately supportive comments, including of using peopleās preferred pronouns (weāve mocked another lesswronger for writing long screeds complaining about this). In general, I think the average lesswrong attitude towards trans people is better than the average Americanās attitudeā¦ but that is because the bar is in hell. But yeah Iāve seen plenty of shitty takes towards trans people on lesswrong.
Second, HPMoR was cult shit all along, not meant to teach science but to sow distrust of scientists under the glossy sheen of being able to name the six quarks.
Yep. And it didnāt even stick to its premise of ātry to do science to magic and compare muggle scientifically gained knowledge to magicā and instead went into some Enderās game pastiche followed by Death Note style āI know you know I knowā plotting, then Harry gets handed all the magical power handed to him at the end of the story thanks to Dumbledore following some insane combination of prophecy.
The attitude that you can substitute a bunch of cheap tricks and hacks to get around fundamentally difficult problems reminds me of the techbro attitude that leads to stuff like pushing fundamentally non-viable technologies (like Theranos or the LLM boosters) and of DOGE trying to asking an LLM how to cut the DEI.
Your first point is trueā¦ with the key words being āshorn of contextā. When you look at how many ratfics go in that direction your second and fourth points become problems.
As to your fourth pointā¦ the techbro billionaires like Elon or Peter Thiel do like referencing fiction (often in hamfisted or ignorant ways that makes me think a bit of fandom gatekeeping actually is good sometimesā¦ i.e. naming your surveillance company palantir, or naming one of your kids a nonsensical WH40K reference). So I wouldnāt entirely neglect the possibility of rationalist managing a bit of inspiration to the billionaires in between the bootlicking. And although there may not be a magical ācoup the governmentā power in real life, the influence they are trying to focus on themselves and harness is still worrying.
A lesswronger asks are we rationalfic protagonists the baddies? https://www.lesswrong.com/posts/FuGfR3jL3sw6r8kB4/richard-ngo-s-shortform?commentId=uDuzmfMEvEqpyApLh
tldr; rationalfic has a very common trend of the protagonist gaining and using overwhelming power to radically reform the world. This is almost (with a few notable exceptions) portrayed as clearly unambiguously good thing.
My take: Donāt get me wrong, the Wizarding World (for example), as canonically portrayed needs some very strong reforms if not an entire revolution. But rationalfic almost never portrays the slow hard work of building support networks and alliances and developing a materialist theoretical understanding of how to reform society, as opposed to a lone (or small friend group) rationalist hero finding some overwhelming magical or technological advantage they can use to single-handedly take control and use their rationalist intellect to unilaterally fix everything. Part of it is the normal disconnect of fiction to the real world were it is more narrative satisfying (and easier to write) to have a central protagonist the solves the major problems or is at least directly involved with them, and rationalfic involves that protagonist gaining even more agency than they canonically do. The problem is that rationalist take this attitude back into real life, and so end up idolizing mythologized techbro billionaires or venture capitalist or the myth of the lone genius scientist/inventor.
Also, quality sneer in the replies, ārationalā teletubbies: https://tomasbjartur.bearblog.dev/rational-teletubbies/
Is switching to pdm as easy as installing it, then initializing pdm and remaking the pyproject.toml by adding all the dependencies with pdm? (If Iām understanding right, this basically seems like the same workflow as poetry?)
There is poetry for package management. Apparently uv is substantially faster at solving package dependencies although poetry is more feature rich. (Iāve only used poetry, so I know it is adequate, but I have had times Iāve sat there for minutes or even tens of minutes while it worked through installing all the right versions of all the right libraries.)
Yeah. When it comes down to it, the libs think the problem with Trump isnāt the fundamentals of what he is doing, it is that he is doing it without decorum or checking all the legal boxes or saying the usual lib pabulum to justify American imperialism. Skipping the legal checks and decorum is also bad, but in fact kids in cages was horrible when Obama was doing it the ārightā way.
Lesswrong is too centrist-brained to ever even hint at legitimizing (non-state-sanctioned) destruction of property as a means of protest or political action. But according to the orthodox lesswrong lore, Sam Altmanās actions are literally an existential threat to all humanity, so they canāt defend him either. So they are left with silence.
I actually kind of agree with the anarchy-libertarianās response? It is massively down voted.
Bingo. Dear leader Yudkowsky can ask to bomb the data centers, and as long as this action goes through the US political process, that violence is legitimate, regardless of how ill-behaved the US is or itās political processes degraded from actually functioning as a democracy.