• 1 Post
  • 80 Comments
Joined 1 year ago
cake
Cake day: August 29th, 2024

help-circle
  • Today in Seems Legit News:

    ā€œAs a concrete example, an engineer at Spotify on their morning commute from Slack on their cell phone can tell Claude to fix a bug or add a new feature to the iOS app,ā€ Sƶderstrƶm said. ā€œAnd once Claude finishes that work, the engineer then gets a new version of the app, pushed to them on Slack on their phone, so that he can then merge it to production, all before they even arrive at the office.ā€

    • why is engineer working before contracted time
    • if engineer can do everything by cellphone why does engineer have to commute in the first place
    • if Claude can do everything anyway why do you still have engineers at all
    • if ā€œno engineer has written a line of code since Decemberā€, when are your lowering your subscription prices Spotify
    • why is hypothetical engineer a ā€œheā€, Spotify
    • do you often merge Claude code to production without even a review, Spotify
    • in unrelated news, Anna’s Archive has socialised Spotify metadata and 6TB of music, Gods bless them https://torrentfreak.com/annas-archive-quietly-releases-millions-of-spotify-tracks-despite-legal-pushback/
    • though I won’t do anything with that as I assume everything from Spotify is ā€œAIā€ ā€œmusicā€ anyway and I listen to my bands either from bandcamp, soulseek, or just downloaded from youtube videos uploaded over 10 years ago










  • I’ve been deliberately learning to navigate without GPSes and tech devices, as a life skill (also on foot/public transport). I’m terrible at navigating, but I’m realising navigating is kinda like handwriting—in that it’s very easy to fall into the trap of saying ā€œI’m terrible at thisā€ as a kind of immutable personality trait, while in fact it’s perfectly expected that one is bad at a skill that one never uses, and turns out I can get better at it even with a little bit of deliberate practice. I suck at things but I can improve.

    In the meantime when I use an electronic map to navigate, I still would rather stick a smartphone to the dashboard a car and use whatever navigation app I prefer, than have the screens and navigators built into the car.






  • I don’t mean the term ā€œpsychosisā€ as a depreciative, I mean in the clinical sense of forming a model of the world that deviates from consensus reality, and like, getting really into it.

    For example, the person who posted the Matrix non-code really believed they had implemented the protocol, even though for everyone else it was patently obvious the code wasn’t there. That vibe-coded browser didn’t even compile, but they also were living in a reality where they made a browser. The German botanics professor thought it was a perfectly normal thing to admit in public that his entire academic output for the past 2 years was autogenerated, including his handling of student data. And it’s by now a documented phenomenon how programmers think they’re being more productive with LLM assistants, but when you try to measure the productivity, it evaporates.

    These psychoses are, admittely, much milder and less damaging than the Omega Jesus desert UFO suicide case. But they’re delusions nonetheless, and moreover they’re caused by the same mechanism, viz. the chatbot happily doubling down on everything you say—which means at any moment the ā€œmildā€ psychoses, too, may end up into a feedback loop that escalates them to dangerous places.

    That is, I’m claiming LLMs have a serious issue with hallucinations, and I’m not talking about the LLM hallucinating.


    Notice that this claim is quite independent of the fact that LLMs have no real understanding or human-like cognition, or that they necessarily produce errors and can’t be trusted, or that these errors happen to be, by design, the hardest possible type of error to detect—signal-shaped noise. These problems are bad, sure. But the thing where people hooked on LLMs inflate delusions about what the LLM is even actually doing for them—that seems to me an entirely separate mechanism; something that happens when a person has a syntactically very human-like conversation partner that is a perfect slave, always available, always willing to do whatever you want, always zero pushback, who engages into a crack-cocaine version of brownosing. That’s why I compare it to cult dynamics—the kind of group psychosis in a cult isn’t a product of the leader’s delusions alone, there’s a way that the followers vicariously power trip along with their guru and constantly inflate his ego to chase the next hit together.

    It is conceivable to me that someone could make a neutral-toned chatbot programmed to never 100% agree with the user and it wouldn’t generate these psychotic effects. Only no company will do that because these things are really expensive to run and they’re already bleeding money, they need every trick in the book to get users to stay hooked. But I think nobody in the world had predicted just how badly one can trip when you have ā€œdr. flattery the alwayswrong botā€ constantly telling you what a genius you are.


  • Copy-pasting my tentative doomerist theory of generalised ā€œAIā€ psychosis here:

    I’m getting convinced that in addition to the irreversible pollution of humanity’s knowledge commons, and in addition to the massive environmental damage, and the plagiarism/labour issues/concentration of wealth, and other well-discussed problems, there’s one insidious damage from LLMs that is still underestimated.

    I will make without argument the following claims:

    Claim 1: Every regular LLM user is undergoing ā€œAI psychosisā€. Every single one of them, no exceptions.

    The Cloudflare person who blog-posted self-congratulations about their ā€œMatrix implementationā€ that was mere placeholder comments is one step into a continuum with the people whom the chatbot convinced they’re Machine Jesus. The difference is of degree not kind.

    Claim 2: That happens because LLMs have tapped by accident into some poorly understood weakness of human psychology, related to the social and iterative construction of reality.

    Claim 3: This LLM exploit is an algorithmic implementation of the feedback loop between a cult leader and their followers, with the chatbot performing the ā€œfollowerā€ role.

    Claim 4: Postindustrial capitalist societies are hyper-individualistic, which makes human beings miserable. LLM chatbots exploit this deliberately by artificially replacing having friends. it is not enough to generate code; they make the bots feel like they talk to you—they pretend a chatbot is someone. This is a predatory business practice that reinforces rather than solves the loneliness epidemic.

    n.b. while the reality-formation exploit is accidental, the imaginary-friend exploit is by design.

    Corollary #1: Every ā€œlegitimateā€ use of an LLM would be better done by having another human being you talk to. (For example, a human coding tutor or trainee dev rather than Claude Code). By ā€œbetterā€ it is meant: create more quality, more reliably, with more prosocial costs, while making everybody happier. But LLMs do it: faster at larger quantities with more convenience while atrophying empathy.

    Corollary #2: Capitalism had already created artificial scarcity of friends, so that working communally was artificially hard. LLMs made it much worse, in the same way that an abundance of cheap fast food makes it harder for impoverished folk to reach nutritional self-sufficiency.

    Corollary #3: The combination of claim 4 (we live in individualist loneliness hell) and claim 3 (LLMs are something like a pocket cult follower) will have absolutely devastating sociological effects.