Want to wade into the sandy surf of the abyss? Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid.
Welcome to the Stubsack, your first port of call for learning fresh Awful youāll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cutānāpaste it into its own post ā thereās no quota for posting and the bar really isnāt that high.
The post Xitter web has spawned so many āesotericā right wing freaks, but thereās no appropriate sneer-space for them. Iām talking redscare-ish, reality challenged āculture criticsā who write about everything but understand nothing. Iām talking about reply-guys who make the same 6 tweets about the same 3 subjects. Theyāre inescapable at this point, yet I donāt see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldnāt be surgeons because they didnāt believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I canāt escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this.)
The fuck is Mythos?
Is it their next model that tbey swear isnāt vaporware but no! It is too dangerous to release into the world because itāll find too much insecure code or whatever.
Ia ia Claude! Phānglui mglwānafh Claude Anthrālyeh wgahānagl fhtagn! Ia! Ia!
Okay but like is it materially different than whatever the current Claude thing is or did they just pump the size of the matrix?
Probably a markdown file telling it āyou are a l33t h4x0rā
Okay but thatās already in Claude
I still laugh every time I see that this is what qualifies as proper ātuningā and āsecurity controlsā for these things.
I had hoped that with the whole āagentā push that we would start seeing more sane usage, like having AI be a fuzzy logic step in a chain of formal logic and existing deterministic tools, but the cult still has people treating them like reliable second brains. Theyāre used as the baseline fucking orchestrator rather than anywhere they might make a bit of sense.
I think this is the best you can expect out of LLMs, and the relatively more successful āagenticā AI efforts are probably doing exactly this, but their relative success is serving as hype fuel for the more impossible promises of LLMs. Also, if you have formal logic and deterministic tools wrapping and sanity checking the LLM bitsā¦ I think the value add of evaporating rivers and firing up jet turbines to train and serve ācutting edgeā models that only screw up 1% of the time isnāt there because you can run a open weight model 1/100th the size that screws up 10% of the time instead. (Note one important detail: training costs go up quadratically with model size, so a 100x size model is 10,000x training compute.) I think the frontier LLM companies should have pivoted to prioritizing smaller size, greater efficiency, and actually sustainable business practices 4 years ago. At the very latest, 2 years ago, with the release of 4o OpenAI should have realized pushing up model size was the wrong direction (as they should have realized training Chain-of-Thought was not going to be the magic bullet).
And to be clear I still think this is really generous to the use case of smaller LMs.
they added more leetspeak and changed the vm they run it on to kali linux for extra h4xx0r power
Anthropicās latest model that they havenāt released to the public yet since theyāre worried its gonna fuck up cybersecurity this thread goes over it a bit
XCancel link for those of us sick of being badgered to sign up/in
On a more productive note, this feels likely to be tied in with the usual issues of AI sycophancy re: false positive rate. If you ask the model to tell you about security vulnerabilities, itās never going to tell you there arenāt any, any more than existing scanners will. When I worked for F5 it was not uncommon to have to go down a list of vulnerabilities that someoneās scanner turned out and figure out whether they were actually something that needed mitigation that could be applied on our box, something that needed to be configured somewhere else in the network (usually on their actual servers) or (most commonly) a false positive, e.g. āyour software version would be vulnerable here, which is why it flagged, but you donāt have the relevant module activated and if an attacker is able to modify your system to enable it youāre already compromised to a far greater degree than this would allow.ā That was with existing tools that werenāt trying to match a pattern and complete a prompt.* Given that weāve seen the shitshow that is Claude Code I think itās pretty clear theyāre getting high on their own supply and this announcement ought be catnip for black hats.
Wow, sounds like they just automated āshitty infosec teams that only forward scanner output without evaluating itā out of a job. Holy shit they were right that AI was coming for jobs!
True. I will say that the shitty infosec teams are probably being hit less hard than the SMEs they offloaded their jobs onto, because from their perspective it doesnāt actually matter whether itās f5 support engineer or a chatbot that tells them the answer; either way theyāve successfully offloaded the task of validating security onto another entity that can make up for their shortcomings with a combination of accuracy and authority. Nobody is going to get fired for not fixing a bug that the vendor SME told them wasnāt actually an issue for them, effectively. And when the org has been pushing AI as hard as so many of them have its pretty easy to throw the chatbot under the same bus and expect the bus to stop instead.
I suspect this is the real limit. Claude Mythos might find real vulnerabilities, but if they are buried among loads of false positives it wonāt be that useful to black or white hat hackers and the endless tide of slop PRs and bug reports will keep coming.
I tried looking through Anthropicās āpreviewā for a description of the false positive rateā¦ they sort of beat around the bush as to how many false positives they had to sort out to find the real vulnerabilities they reported (even obliquely addressing the issue was better than I expected but still well short of the standard for a good industry-standard security report from what I understand).
Theyāve got one class of bugs they can apparently verify efficiently?
Itās not clear from their preview if Claude was able to automatically use Address Sanitizer or not? Also not clear to me (Iāve programmed with Python for the past ten years and havenāt touched C since my undergraduate days), maybe someone could explain, how likely is it that these bugs are actually exploitable and/or show up for users?
Moving onā¦
So its good they arenāt just flooding maintainers with slop (and it means if they do publicly release mythos maintainers will get flooded with slop bug fixes), butā¦ this makes me expect they have a really high false positive rate (especially if you rule minor code issues that donāt actually cause bugs or vulnerabilities as false positives).