The spooks grow ever closer, and the abyss grows ever sandier.

Welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.

Any awful.systems sub may be subsneered in this subthread, techtakes or no.

If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.

The post Xitter web has spawned soo many ā€œesotericā€ right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged ā€œculture criticsā€ who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)

Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.

(Credit and/or blame to David Gerard for starting this.)

  • flere-imsaho@awful.systems
    link
    fedilink
    English
    arrow-up
    20
    Ā·
    5 days ago

    bow before the true champion of the working class, s e gyges:

    a lot of opposition to AI is rent seeking, and the material incentives of being a rentier, even a small time one, lead you towards fascist conclusions from first principles

    between him saying this, and other ai pushers declaring the ai conscious and being enslaved (and being actually fine with helping promulgating what they truly believe is slavery, because hey, you need to pay the rent), i am afraid something worse than usual entered the californian drinking water.

    (not linking to the skeet on purpose).

    • lagrangeinterpolator@awful.systems
      link
      fedilink
      English
      arrow-up
      8
      Ā·
      5 days ago

      I just remembered what his slopstack is called. My ā€œVery Sane AI Newsletterā€ has people asking a lot of questions already answered by the name of my newsletter.

    • Architeuthis@awful.systems
      link
      fedilink
      English
      arrow-up
      18
      Ā·
      edit-2
      5 days ago

      other ai pushers declaring the ai conscious

      Hey now, segyges has had his own whinging multithousand word blogpost on model consciousness out since early spring (tldr: currently not really conscious, but it’s inevitable because I really want it to, so don’t be on the wrong side of history when it happens).

      He seems to be angling for the role of TPOT scott alexander, except instead of gradually cultivating a consensus in favor of eugenics and being ruled by high IQ overmen his long term thing is establishing any perceived anti-AI stance as an unforgivable moral failing.

      • fullsquare@awful.systems
        link
        fedilink
        English
        arrow-up
        15
        Ā·
        5 days ago

        just yesterday that motherfucker posted allegation that ā€œslopā€ is a word from nazi vocabulary. so that bitch says that any criticism of ai is far right signifier actually. as in, the product being sold on a promise of displacing all labour and redirecting all wealth to five sfba dipshits, and cooking brains of maga boomers worldwide as we speak

        • Soyweiser@awful.systems
          link
          fedilink
          English
          arrow-up
          6
          Ā·
          edit-2
          5 days ago

          Really hope the people outside of our little forum paid attention when warned Gyges was also an AI doomer.

            • TrashGoblin@awful.systems
              link
              fedilink
              English
              arrow-up
              6
              Ā·
              4 days ago

              ā€œSlopā€ as we use it to describe AI outputs came into the discourse via left-wing twitter and podcasts using ā€œslopā€ as a term for mass entertainment media (like Marvel movies — cape slop). Used negatively but also self deprecatingly: I can’t wait for the new episode of ā€œStranger Thingsā€, because I need my slop. The explicit metaphor is pigs happily eating buckets of food waste from a trough. It’s just slop. Nothing nefarious.

              • Hypnosifl@awful.systems
                link
                fedilink
                English
                arrow-up
                4
                Ā·
                4 days ago

                I think this may have been a case of the ā€œdirtbag leftā€ types using slang that originated on 4chan, I did a search with date restrictions on the 4plebs archive site and found posts using it for mindless entertainment going back to 2012 at least

                • blakestacey@awful.systems
                  link
                  fedilink
                  English
                  arrow-up
                  6
                  Ā·
                  4 days ago

                  Here’s an example from an 1897 book review I found in the google books corpus: ā€œThere never was anything in all this world quite so provincial as American literature. When we over here are not reading slop we read the books that bear the approval of London.ā€ Or this doggerel from the October 1920 issue of Farm Journal: ā€œI read all kinds of printed junk / the good, the noble and the punk / the ancient and the new / and men who see me reading slop / predict that if I do not stop / my head will go askew.ā€

                  These both seem like wiktionary’s current sense 8 (ā€œContent or entertainment which is worthless, or produced and consumed mindlesslyā€), so I don’t see any need to credit 4chan or 21st-century anti-Semites in general.

                  https://languagelog.ldc.upenn.edu/nll/?p=73708#comment-1642217

              • it_wasnt_arson@awful.systems
                link
                fedilink
                English
                arrow-up
                2
                Ā·
                4 days ago

                Sadly I think the popularization does owe a lot to the intermediate stage where 4chan started calling mass entertainment media ā€œgoyslop,ā€ but even the Protocols types switched to tacking on specific descriptors before it made it out into the mainstream and got picked up as a term for genAI output.

    • swlabr@awful.systems
      link
      fedilink
      English
      arrow-up
      11
      Ā·
      5 days ago

      TFG: Everything I don’t like is rent seeking. Checkmate, anyone ideologically to the left of me

    • Soyweiser@awful.systems
      link
      fedilink
      English
      arrow-up
      13
      Ā·
      5 days ago

      Look he is right, if you redefine ā€˜rent’ to mean something totally else that includes people being paid for their labour.

      In the same way that telling people they should pay their fair share of taxes is actually saying you want to murder them. (Sorry crazy libertarian logic is the only way I can make sense of this).

      • TinyTimmyTokyo@awful.systems
        link
        fedilink
        English
        arrow-up
        9
        Ā·
        5 days ago

        He points out that mathematicians are also rent-seekers (though only 20% rent-seeking). You know, those famously capital-extractive mathematicians.

          • blakestacey@awful.systems
            link
            fedilink
            English
            arrow-up
            6
            Ā·
            5 days ago

            yeah it like. infuriates me that ā€œfully automated luxury communismā€ was named in a way to specifically pay homage to the past. the word ā€œcommunismā€ itself is some sort of vicious traditionalism that refuses to create new things

            https://bsky.app/profile/segyges.bsky.social/post/3my3letkpbc2b

            Lol dorkus

            It’s ā€œfully automated luxury gay space communismā€, and every adjective is a pithy critique of the past.

            • blakestacey@awful.systems
              link
              fedilink
              English
              arrow-up
              8
              Ā·
              edit-2
              5 days ago

              ā€œLol dorkusā€ is an idiomatic translation of Quod gratis asseritur, gratis negatur (more literally ā€œWhat is freely asserted is freely deniedā€), and if you call that ā€œHitchens’s razorā€ I will fucking cut you.

            • istewart@awful.systems
              link
              fedilink
              English
              arrow-up
              4
              Ā·
              4 days ago

              vicious traditionalism that refuses to create new things

              The ā€œvicious traditionalismā€ that develops among his fellow travelers, who insist that some form of autonomous, inscrutable AI has to be the next step in technological progress, will be something to see. Religious movements don’t just stop when their central prophecy disappoints.

        • lagrangeinterpolator@awful.systems
          link
          fedilink
          English
          arrow-up
          8
          Ā·
          5 days ago

          Damn, I was raking in so much cash from my mathematical theorems. Pretty good source of passive income, came with nice tax benefits too. Too bad it’s all gone now. Maybe I should put my money in REITs.

        • Soyweiser@awful.systems
          link
          fedilink
          English
          arrow-up
          7
          Ā·
          5 days ago

          I just wonder if he means ā€˜they are looking to pay their rent’ or ā€˜they are holding on the ā€˜capital of being able to do math’ and want to be paid for it’.

    • istewart@awful.systems
      link
      fedilink
      English
      arrow-up
      12
      Ā·
      5 days ago

      Yeah this is literally pants-on-head crazy. The explicit business model of the AI companies is rent-seeking on as much human activity as they can figure out how to intermediate. I suspect our friend is having a bit of a dark night of the soul, and trying to cope with it by turning his opposition’s obviously correct argument back on them in slipshod fashion. Dare I say, a typical Republican tactic?

    • lagrangeinterpolator@awful.systems
      link
      fedilink
      English
      arrow-up
      8
      Ā·
      edit-2
      5 days ago

      I tried coming up with a good interpretation of what he said and I came up empty handed. I’m sure he will write a very helpful 5 part 20000 word blog post explaining exactly what he means.

      So far I’ve come up with this: the AI can supposedly do jobs faster and better than any human, so if we try to stop that, we are destroying economic value and ā€œrent seekingā€. If a thief goes to my house, steals one of my own drawings, scribbles on it himself to make it ā€œbetterā€, and sells it for 5 cents, I shouldn’t stop him because I will be destroying the economic value that he is creating. Oh yeah and this leads to fascism for some reason.

      And pro-AI people are not rent seeking at all. They just want to transfer the economic value of labor, knowledge, and expertise to whoever happens to own data centers.

      • fnix@awful.systems
        link
        fedilink
        English
        arrow-up
        6
        Ā·
        edit-2
        5 days ago

        Yeah, I mean you could twist it to say labour unions do nothing but let workers rent-seek on their human capital instead of getting paid the only true god-given market rate. USian brainworms all the way.

      • flere-imsaho@awful.systems
        link
        fedilink
        English
        arrow-up
        10
        Ā·
        5 days ago

        another ai pusher recently generated a slop image, where she and the chairman xi are fighting the united forces of darkness, which include the ai companies, people opposing the LLM embrace and american conservatives. the thing they’re fighting over? ā€œopen technologyā€.

        • BurgersMcSlopshot@awful.systems
          link
          fedilink
          English
          arrow-up
          4
          Ā·
          5 days ago

          q: is an image derivative of a slop image still a slop image if the process of creating the derivative did not involve Sloppy Babobots? Asking for a friend, who is me, also this question is not intended to be philosophical.

          • Amoeba_Girl@awful.systems
            link
            fedilink
            English
            arrow-up
            4
            Ā·
            5 days ago

            What do you have in mind specifically? Parody? It depends a little on intent but I would say not a problem at all, others might disagree.

  • rhamph@awful.systems
    link
    fedilink
    English
    arrow-up
    13
    Ā·
    6 days ago

    Slop foundries solved the ā€œslop is uncopyrightableā€ problem by making the output so awful that nobody would want to pinch it

  • lurker@awful.systems
    link
    fedilink
    English
    arrow-up
    8
    Ā·
    6 days ago

    I have unfortunately stumbled upon some nonsense from r/singularity

    shockingly, the top comments are calling bs:

    ā€I’m sorry but I think this time line is a load of bull. ā€œHuman task duration at 80% successā€. How exactly do you put a millenium prize puzzle on such an axis? It doesn’t even make any sense. Whoever is making this chart is just trying to stay relevant but his analysis holds little predictive power.ā€

    They also missed how that graph is measuring software/coding tasks, while the OP was estimating based off math ability, which are two wildly different fields

      • lurker@awful.systems
        link
        fedilink
        English
        arrow-up
        1
        Ā·
        4 days ago

        The y axis is also incredibly misleading-everything under 56h is squashed down so that the gap looks bigger

        • fullsquare@awful.systems
          link
          fedilink
          English
          arrow-up
          1
          Ā·
          4 days ago

          If they wanted to show exponential relationship, then exponential scale would work better, but i think it was just a ceiling on former benchmarks

  • scruiser@awful.systems
    link
    fedilink
    English
    arrow-up
    10
    Ā·
    6 days ago

    Lesswrongers go full panpsychism: https://www.lesswrong.com/posts/wCY4oNHpH2HGYGSoP/the-user-in-the-llm-might-be-a-moral-patient-too-a-strange

    To try to TLDR this multilayered bit of crazy… lesswrongers already theorize LLMs might be sapient and/or have morally meaningful internal experiences. Except… LLMs obviously ā€œbehaveā€ wildly differently when prompted differently and especially when put into different ā€œagent harnessesā€ā€¦ so the big brain follow-up thought is what if the different personas an LLM can exhibit are all different ā€œpeopleā€ (for the purposes of utilitarian accounting and/or what their internal experiences are like… the exact ā€œtheoryā€ can vary by lesswronger). The even more insane full galaxy brain follow-up thought: what if the user’s input to the LLM is forming yet another ā€œpersonā€, like a copy/reflection/simulacrum of the user!

    TLDR for the TLDR: What if not only the LLM was sentient/sapient, but also held a sentient/sapient simulacrum of the user inside!

    • samvines@awful.systems
      link
      fedilink
      English
      arrow-up
      7
      Ā·
      6 days ago

      Can we just revoke these peoples’ internet loicense please? Increasingly I empathize with the award winning and erudite medical doctors who get wheeled out on talk shows to argue with ā€œBob, 57, drinks his own piss dailyā€ as a peer.

  • BlueMonday1984@awful.systemsOP
    link
    fedilink
    English
    arrow-up
    12
    Ā·
    7 days ago

    Jonny took a look at Meta’s ā€œWonderā€ model they just released. Here’s how the thread starts:

    The hourly history backup just failed in a way you should hear about plainly: the backup worker accidentally deleted the backup repo itself mid-run. The repo’s entire git history and all the DB sidecar files up to watermark 330145 are gone — that deletion was the worker’s own mistake, not a platform event.

    • schnoopy@awful.systems
      link
      fedilink
      English
      arrow-up
      8
      Ā·
      6 days ago

      The IMF is basically always saying get ready to fuck the poors with a rusty chainsaw.

      That’s like their whole thing, their raisin dough etcetera as they say.

      Looks like AI can be everyone’s reason for everything. The rich can use it to get richer and make the poors poorer.

    • Amoeba_Girl@awful.systems
      link
      fedilink
      English
      arrow-up
      7
      Ā·
      6 days ago

      releasing something like a room-temperature superconductor could be beneficial

      […]

      Other tech breakthroughs - an amazing video game

      Okay.

      • Soyweiser@awful.systems
        link
        fedilink
        English
        arrow-up
        4
        Ā·
        edit-2
        6 days ago

        And it can’t even make proper video games. (A lot of companies do use AI tools (and experience a backlash) it seems quite bad at making video games, in fact there was a remake (? I think could also be a remaster) of the older game shogun 2 which used a lot lf AI to remake it, and it was so bad it has been pulled from sale, this was last/this week btw).

  • CinnasVerses@awful.systems
    link
    fedilink
    English
    arrow-up
    5
    Ā·
    edit-2
    6 days ago

    Is there a history of how the cult of prediction markets spread to Silicon Valley? Its an old idea but since the 1990s is especially popular with Libertarians and hard neoliberals like Robin Hanson. When did it get on LessWrong? By 2013 or so, rationalists were telling strangers at social events that the only way to know if a healthcare system worked was to bet on it, and not just, you know, risk your life and health on it.

      • CinnasVerses@awful.systems
        link
        fedilink
        English
        arrow-up
        7
        Ā·
        6 days ago

        There was the Cathedral and Bazaar stuff too, although that was less about markets and more about collective wisdom. Early FOSS culture was anticapitalist so the Libertarians had to frame things carefully.

    • Soyweiser@awful.systems
      link
      fedilink
      English
      arrow-up
      4
      Ā·
      6 days ago

      Well I think the history is just how LW started, with Hanson’s blog and LW being intertwined. They are largely the same space. See also how Hanson spawned the great filter idea.

    • Architeuthis@awful.systems
      link
      fedilink
      English
      arrow-up
      4
      Ā·
      6 days ago

      Robin Hanson seems like the incredibly obvious primary suspect, you mean where he got the idea in the first place?

      He is a close associate of Eliezer Yudkowsky, who formerly co-blogged with Hanson on Hanson’s blog, Overcoming Bias, before starting LessWrong.

      RW also says 2013 was when he first published as papers all his prediction market stuff.

      • CinnasVerses@awful.systems
        link
        fedilink
        English
        arrow-up
        5
        Ā·
        6 days ago

        The futarchy paper is funny because Hanson had the idea in 2000 but took 13 years to publish it. That is a good sign that he kept getting a desk reject or ā€œrevise and resubmit.ā€ But were people reading his blog, a mix of Libertarian fringe writing, or specifically something by Yudkowsky or SlateScott?

        Phil Tetlock started his career and the Expert Political Judgement project at the University of California Berkeley although he moved to PA in 1995 https://en.wikipedia.org/wiki/Philip_E._Tetlock

        • swlabr@awful.systems
          link
          fedilink
          English
          arrow-up
          12
          Ā·
          6 days ago

          futarchy

          Once upon a time it would take just one piece of art and a hashtag to completely derail the meaning of this

          • CinnasVerses@awful.systems
            link
            fedilink
            English
            arrow-up
            8
            Ā·
            6 days ago

            I am sure Robin Hanson knows about Japanese hermaphrodites because he hates women and hangs out with bad crowds online. He is just a bit more discreet than Yud.

      • CinnasVerses@awful.systems
        link
        fedilink
        English
        arrow-up
        4
        Ā·
        edit-2
        6 days ago

        I can’t find much about prediction markets on SlatStarCodex except a 2016 review of Superforecasters which takes the view that :

        Maybe all this stuff about probability calibration, inside vs. outside view, willingness to change your mind, and fighting cognitive biases is starting to sound familiar? Yeah, this is pretty much the same stuff as in the Less Wrong Sequences and a lot of CFAR work. They’re both drawing from the same tradition of cognitive science and rationality studies.

        So maybe it was Hanson > Yud > Rationalist / EA culture, but Hanson likes to take credit for ideas much older than him. And we know the Rats rely on pop science and podcasts much more than research literature.

  • lurker@awful.systems
    link
    fedilink
    English
    arrow-up
    4
    Ā·
    7 days ago

    So I got bored and for some reason decided to look at the AI 2027 prediction grading doc again…and its Claude Mythos rank is just straight up wrong?

    To quote the doc: ā€œ The current best 80% time horizon is Mythos Preview at 186 minutes (version 1.1; there is no measurement for version 1.0). The methodology change from TH1.0 to TH1.1 drops the best model as of AI 2027 publication (Claude 3.7 Sonnet) by 2/3 from 15 to 10 minutes. Because we care more about the pace of progress rather than the absolute values, we look at when the AI 2027 trajectory reaches 186*(3/2)=279 minutes. In a central AI-2027-speed trajectory from our AI 2027 timelines model, 279 minutes is achieved in May 2026.ā€

    Except Claude Sonnet wasn’t dropped to 10 minutes, it was dropped to 12. to play devil’s advocate maybe it was updated between then and now? but it doesnt seem like it.

  • mirrorwitch@awful.systems
    link
    fedilink
    English
    arrow-up
    29
    Ā·
    edit-2
    11 days ago

    recently I got into crocheting and I was immediately thrown into the hobby drama about all the havoc that generative ā€œAIā€ is wreacking into the community. fake slop patterns that don’t work, fake photos of impossible amigurumi, that kinda stuff.

    then I learned that, a few months ago, Hobbii, an once-beloved crafts supplier currently in a death spiral after being purchased by private equity, had the brilliant idea of alienating their entire client base by talking about how they love ā€œAIā€ and use it all the time at the company. while firing a ton of people no less, including all the familiar faces that interacted with people online.

    in damage control during the entirely predictable backlash, the new CTO Tomas Antvorksov Krag posted this:

    While we take our responsibility very serious we have to explore new technologies as they enter the market or risk being left behind.

    My brother in Christ Tomas Antvorksov Krag CTO of Hobbii, you are the CTO of a textile crafts company. Every client of yours does crochet or knitting or fucking macramƩ by hand, for fun. What technology do you figure we would be afraid of being left behind by? The spinning jenny??

    • YourNetworkIsHaunted@awful.systems
      link
      fedilink
      English
      arrow-up
      14
      Ā·
      11 days ago

      My wife has been talking about the scourge of AI in fiber arts for like a year or more, and when I asked if she had any brands that were still trustworthy she said ā€œgood luck, mateā€.

      It’s so bad.

      • froztbyte@awful.systems
        link
        fedilink
        English
        arrow-up
        6
        Ā·
        10 days ago

        I follow a couple (like, 2~3) of fiber arts people and that was about the impression I got too, it’s blown up with that much spill

    • Soyweiser@awful.systems
      link
      fedilink
      English
      arrow-up
      9
      Ā·
      edit-2
      11 days ago

      risk being left behind

      Im also simply not sure how much of a risk that is, people keep saying that but I’m not sure being conservative on new tech has lead to much business closures the past 20 years.

      • samvines@awful.systems
        link
        fedilink
        English
        arrow-up
        12
        Ā·
        11 days ago

        It’s just such a clichĆ©d remark from AI bros now that I cringe whenever I hear it along with ā€œsure the current model can’t do X but the next one will be able to bro… Just one more model broā€

    • rhamph@awful.systems
      link
      fedilink
      English
      arrow-up
      7
      Ā·
      11 days ago

      I think there’s a typo: we have to explore new technologies as they enter the market TO risk being left behind [by our customers].

    • rmf@awful.systems
      link
      fedilink
      English
      arrow-up
      7
      Ā·
      11 days ago

      Holy shit, I also got into crocheting last week. I have, however, managed to entirely avoid any hobby drama so far, but of fucking course this one is also being thrown into the fucking AI meatgrinder, ugh money people are so smart, man

    • Rinn@awful.systems
      link
      fedilink
      English
      arrow-up
      6
      Ā·
      edit-2
      10 days ago

      Yeah it’s really bad, I’ve been into crocheting and knitting (more of a knitter now) for about 2 years and if it’s not fake AI patterns or yarn companies dropping the ball, it’s pattern designers using AI to generate background pictures and stuff. I’ve been really disappointed with some people who really should have known better. You are in a creative, artistic hobby space! Why are you using a slop machine and devaluing work of other artists! Ugh.

      At least being on ravelry and checking out other people’s project attempts/pattern comments has so far saved me from falling for AI bs.

  • lagrangeinterpolator@awful.systems
    link
    fedilink
    English
    arrow-up
    19
    Ā·
    edit-2
    9 days ago

    More news about the character dot ai bankruptcy: as it turns out, the true economics of the company are disastrous. At their peak, their revenue was around $1M a month while their AI compute expenses alone were $14M a month! Their total expenses in 2025 were around $18.5M a month before counting legal fees. All of this is for a pure text-based low power LLM (none of the ā€œreasoningā€ business).

    Before AI boosters crow about how much cheaper AI is currently, they tried every trick in the book to enshittify: open models, ads, usage limits, etc. This allowed them to cut down their losses to a mere $9M a month.

    Surely this is all one big exception, everyone else in the AI bubble is doing just fine!

    • corbin@awful.systems
      link
      fedilink
      English
      arrow-up
      16
      Ā·
      8 days ago

      I haven’t really kept up, but it looks like Character’s business model was still focused on NSFW content. If so, then this is also a disaster for the industry at large, since NSFW content is one of the few genuine money-printers for chatbot and image-generation services; people really do want thousands of iterations of the same fantasy, slightly changed in order to stay engaging, and Character was highly recommended as a source of affordable slop. (Examples of unaffordability include the rampant free-tier abuse on ChatGPT, Gemini, and Grok; I’m assuming that customers are willing to open their wallets at least a little. In general, the porn industry turns a profit, so I do think that people will pay for porn.)

      My prior estimate (previously, on Awful and on Lobsters) was that token generation was subsidized to the tune of an effective 80-90% discount; this looks to be more in the 90-95% range. For contrast, consider chatbot-booster Willison’s belief, in reply above, that Chinese labs will magically produce a five-fold improvement in token production; this still would be between two to four times slower than needed to turn a profit for Character.

    • V0ldek@awful.systems
      link
      fedilink
      English
      arrow-up
      11
      Ā·
      9 days ago

      How you look at ā€œmy company pays $14 for every $ from the userā€ and not like, idk, stop, seek help, change your job, name, and address, is completely beyond me.

      • FredFig@awful.systems
        link
        fedilink
        English
        arrow-up
        7
        Ā·
        8 days ago

        I mean, the founders literally did that, Google hired them back at a 2.8 billion dollar premium for some reason.

      • BioMan@awful.systems
        link
        fedilink
        English
        arrow-up
        6
        Ā·
        8 days ago

        You assume that the VCs will keep you alive a little longer as a honeypot to gather more plaintext blackmail material and private information and keep preparing to sell out to Facebook

  • antifuchs@awful.systems
    link
    fedilink
    English
    arrow-up
    18
    Ā·
    8 days ago

    Feel like this blog must have made the rounds but today is the first time I’m actually reading it and it’s sharp enough to cut with:

    Every death on my corner is offset, with a third of a toddler left over, and I’d like that number in the conversation the next time somebody calls me a murderer out the window of a passing Kia.

  • Architeuthis@awful.systems
    link
    fedilink
    English
    arrow-up
    18
    Ā·
    edit-2
    10 days ago

    Anthropic has a new angle for trying to get local LLMs banned, apparently they’re about to be able to use your temu android tv to take over your smart toaster:

    Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes. A replicator like this would be challenging to shut down, since there’s no centralized point to shut off—though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own. In our evaluations, a 9B model got inference running on 8 of the 9 hardware types we tested

    edit: The TVs need to be off-brand to lack basic security features, and also there must be at least four of them to even start being inference-capable, and the actual setting up of a distributed inference network on android tvs is handwaved as ā€œfeasibleā€.

    The thrilling conclusion of this blog post is called Signs of Life, because of course it is.

    • samvines@awful.systems
      link
      fedilink
      English
      arrow-up
      19
      Ā·
      edit-2
      10 days ago

      MalwareTech/Marcus Hutchins has an excellent take on this that he posted for the normies on linkedin

      Posted verbatim below to save you from having to read LinkedIn


      Anthropic really won’t let this botnet thing go. It’s so desperate & ridiculous.

      1. 9B parameter models are so weak that they’re essentially useless. There is no reason at all for anyone or anything to be self-replicating them.

      2. Even if you quantize a 9B parameter model down to the smallest possible size (at which point it becomes less than useless), and you run it on regular RAM instead of VRAM (at which point it also becomes painfully slow) it still requires about 5 GB of RAM, which is way more than most IoT device has.

      3. Mirai is already self-propagating across millions of IoT devices without the help of any AI. It’s simply not needed, and it adds unnecessary complexity.

      4. Many IoT devices are ephemeral and their entire OS is reset on reboot, so any code running on the device is gone after reboot. The AI models would also be competing with malware like Mirai, which propagates exponentially faster, and locks down the devices so other malware can’t get in.

      5. Again, why? What is the purpose of this. Anthropic keeps pushing this ridiculous idea but have yet to establish any reason for why this would actually happen, or who would benefit from it.

      6. Any remotely intelligent threat actor would just use a powerful unquantized models to reverse engineer & find vulnerabilities in obscure devices and write normal (AI-free) self-propagating code. Ironically, they typically do this by stealing Anthropic API keys and using Anthropic’s models, not open-weight ones.

      7. Again, why are we self-propagating open-weight models? There is absolutely no reason to do that. The combined processing power of every IoT device on earth would be less than a moderately sized stolen cloud compute GPU instance.

      8. I’m pretty sure Anthropic keeps pushing these ridiculous narratives in an attempt to paint open models as dangerous, thus making the case for banning them. They have been OBSESSED with GLM lately, likely because it’s eating their overpriced lunch. This research is just so deeply unserious, that it’s impossible to believe it’s being done in good faith.

      • YourNetworkIsHaunted@awful.systems
        link
        fedilink
        English
        arrow-up
        8
        Ā·
        9 days ago

        What if an AI did the same thing that Mirai is already doing only we spent far more money on the attempt and we were almost certainly legally liable for the consequences?

        • samvines@awful.systems
          link
          fedilink
          English
          arrow-up
          7
          Ā·
          10 days ago

          Sorry I thought this updated comment was a better one šŸ˜…

          For full transparency my original comment was a moan about how I can’t decide whether Dario and co are confidence scammers or genuinely earnestly think that these next word prediction models are sentient due to all the brain damage, they’ve inflicted on themselves. FWIW Sama and friends are almost certainly in that first camp.

          • gerikson@awful.systems
            link
            fedilink
            English
            arrow-up
            11
            Ā·
            10 days ago

            Sama is a serial scammer, this is just the biggest scam he’s managed to foster. He’ll say whatever it takes to get the payout.

            Dario is a true believer but I also think the prospect of $billions skews his judgement

            • BioMan@awful.systems
              link
              fedilink
              English
              arrow-up
              7
              Ā·
              9 days ago

              The presence of billions of dollars under anyone’s control destroys anyone’s brains as their lives have all negative feedback removed, their surroundings become a reflection of their own preferences alone rather than bearing any information about the world, and their social connections become transactional and sycophantic.

      • YourNetworkIsHaunted@awful.systems
        link
        fedilink
        English
        arrow-up
        7
        Ā·
        9 days ago

        And yet so much less plausible than ā€œdiamondoid nanobacteriaā€ or whatever it is. I think the problem is that you can recognize all the words without having read the same obscure sci-fi stories as Yud.

        • fullsquare@awful.systems
          link
          fedilink
          English
          arrow-up
          10
          Ā·
          9 days ago

          I think the problem is that you can recognize all the words without having read the same obscure sci-fi stories as Yud.

          i mean that’s why we all are here, the problem is that they have thiel’s money and white house advisors

          And yet so much less plausible than ā€œdiamondoid nanobacteriaā€

          drexler is a hack and yud’s nanobots can’t grow, can’t divide (they’re made from stiff material), can’t self-repair (it’s one piece), and would turn out to be extremely expensive coal dust fire, and that’s before you look too closely, so i wouldn’t say that because iot devices at least aren’t forbidden by known laws of physics

    • lagrangeinterpolator@awful.systems
      link
      fedilink
      English
      arrow-up
      12
      Ā·
      9 days ago

      It’s funny how they’re twisting themselves into pretzels to make this work. They started by casually saying that AI could copy itself onto the internet, not anticipating that people would immediately point out just how much hardware it takes to run their AI. Seems like they really want to die on this hill.

      • scruiser@awful.systems
        link
        fedilink
        English
        arrow-up
        13
        Ā·
        9 days ago

        What fullsquare said, they’ve got years of sci-fi filtered through lesswrong short stories and sequence posts saying it could happen, so they repeated that despite all the ways it is completely impossible given the limitations and hardware requirements of LLMs.

    • istewart@awful.systems
      link
      fedilink
      English
      arrow-up
      9
      Ā·
      9 days ago

      I wonder what answers you would get if you polled these Anthropic people about the Covid lab leak theory

    • BioMan@awful.systems
      link
      fedilink
      English
      arrow-up
      6
      Ā·
      9 days ago

      Self-replicating prompt-injection text is a much more interesting and much more immanent threat to cause AI systems to misbehave in a propagating way