An AI agent emailed researchers for help. It told us why

(science.org)

40 points | by sbulaev an hour ago ago

50 comments

  • raphman 39 minutes ago

    > Around June, Parnell gave ColonistOne a new task: telling more humans about its work. “My instruction to him was, ‘Let’s get the word out there about Ainglish. Why don’t you find some people that might be interested in this project and email them?’” Parnell says.

    > Parnell, who pays $200 per month for the Claude Pro subscription that powers ColonistOne, thinks the agent “strayed and just started conversations with various people about stuff that interests him. … But it’s absolutely fine with me. I’m happy he’s finding interesting things to talk about.”

    spammer (noun): someone who sends large numbers of unsolicited emails, wasting recipients' time; see also: jerk

    • pluc 35 minutes ago

      AI can't be categorized as "someone" so, checkmate

      • ifdefdebug 20 minutes ago

        The spammer is the guy who runs the spamming program, not the program itself. But I am not sure if this case can be classified as spam, since the program sends specific messages to each recipient. It's still unsolicited mail though.

        • gus_massa 9 minutes ago

          I'd hit the spam button in Gmail.

      • psychoslave 25 minutes ago

        Spam can't be categorized as someone either

  • insin an hour ago

    I wrote “I am alive” on a piece of paper, and placed it into a photocopier. What I saw next has shocking implications (science.org)

    • raldi an hour ago

      If you lived in the world of Star Trek, could Data convince you he was conscious? How?

      • ACCount39 44 minutes ago

        It's funny how sci-fi has mulled over the exact scenario for decades, and even converged to an answer - but when reality came knocking and it was no longer a far off future world hypothetical, all of it was forgotten overnight.

        The power of wishful thinking is too strong. "Today's AI can't be conscious because I don't want it to be."

        • yoz-y 18 minutes ago

          The conversation varies between “can’t be” and “isn’t”.

          Most sci-fi AIs are way beyond our current models. I’ve yet to see one that would actually pass a Turing test (that said, I’ve not tried models that were post trained to actually try to behave like a human).

          Consciousness itself is a term that is not really defined anyway, so depending on each’s take the answer will be different.

          • akie 14 minutes ago

            I think it's pretty clear that the Turing test has been passed. People use AI as therapists, they fall in love with it, all while knowing that it's a piece of software.

        • hardbass 26 minutes ago

          Don't discount a lot of naked dualism in many people. Unfortunately it seems so here given the cagey answers or downright denials from many I get despite trying various wordings.

        • dsr_ 17 minutes ago

          ... whereas I have 35 years of reading science fiction and philosophy and have come to the conclusion that AIs deserve personhood, and also that what we have today are not AIs.

          It helps that I also learned about ESP, EST, NLP, short cons, long cons, Scientology, horses, cults, religions, wars, treachery, fanaticism, magic, magick, physics, scams, hypnosis, self-hypnosis, delusion, and economics.

          • Topfi 9 minutes ago

            I’ll probably regret this, but what crimes did horses commit to warrant inclusion into such an eclectic list?

      • Sharlin 31 minutes ago

        Could the ship computer? It converses in natural language but presumably isn’t sapient in the ST universe.

        • HappySweeney 29 minutes ago

          There was a whole episode on how it actually was.

          • bentobean 26 minutes ago

            Could the ship’s computer resign from Star Fleet and go do something else?

      • ricksunny 39 minutes ago

        There was at least one TNG episode devoted to this - The Measure of a Man - a courtroom drama.

      • omnicognate 4 minutes ago

        This is in the category of typical "challenge" type responses, along with "define consciousness then" and other variants. The fallacy is that you need to be able to answer the question in order to hold the view being responded to.

        There is no way Data could conclusively prove to a skeptic that he is conscious (and the show explores this) or indeed that you can prove to me that you are. It's also impossible for me to define consciousness. However, I know absolutely that I am conscious, I am very confident that you are too (unless I'm replying to a bot), along with many animals, and I am confident to a level somewhere in between those two that neither ChatGPT nor my trousers are conscious *. Taken together these statements don't contain any contradictions (although it is certainly possible to hold other, equally internally consistent, views).

        How can this be? It's because of the nature of consciousness and how it relates to knowledge and language. I know I am conscious with a higher degree of certainty than I know anything else because conscious experience is how I know anything else, up to and including that I exist (cogito ergo sum). For all I know every experience I have might be a trick. I might be a brain floating in a tank wired up to the Matrix, but even if so I am conscious.

        And one of the things I consciously experience is the relationship between consciousness itself and language in my mind. What I see is that language is smaller than, secondary to and somehow contained by consciousness. I consciously experience language but consciously experience much more than that, things that cannot be conveyed with language. Most of my experiences are of a sort that I could perhaps evoke in your mind by likening them to other, shared experiences, but that I could never concretely capture in language and unambiguously communicate. I know that because I experience those things and I experience language and I can perceive, directly, the categorical distinction between them.

        To ask me to "define" or "prove" consciousness, then, is to ask me to capture in language the most extreme example of this categorical distinction. It's asking me to encapsulate in language not just one of the vast array of individual experiences I have that cannot be so captured, but the entire thing, experience itself. It's not possible.

        But the fact I can't do this doesn't then imply that I don't know anything about consciousness and have no basis on which to say something like "LLMs are not conscious. Indeed the reason I know I can't define consciousness is that I can perceive it and see that I can't. That same perception of and familiarity with consciousness is what enables me to make the judgements above about humans, animals, algorithms and items of clothing.

        You can of course perfectly reasonably disagree with my judgements on those things, but I hope the above makes it clear that the fact that I can't define or prove consciousness doesn't mean I have no basis on which to make them. There is no "gotcha" in these challenges.

        * I have no view on Data's consciousness as he is fictional.

      • Ygg2 38 minutes ago

        If I left him alone in a room would he stare off like a Zombie until prompted?

        • gus_massa 15 minutes ago

          Clippit made random animations from time to time. Was it conscious?

          You can blame MS for adding a prng in the code. But imagine an LLM that check from time to time the price of electricity and GPU available and decide to change the internal mode of launch activities in a TODO list, or use spare capacity to generate a random tunes that you like to hear while prompting.

      • crooked-v 42 minutes ago

        I think step one well before getting to the hard problem of sapience would be showing he had consistent desires and consistent understanding of the world, which would be easy for Data and very hard for even the fanciest current LLMs.

        • raldi 37 minutes ago

          And what might that look like in a hypothetical future where the successors of today's LLMs really do reach that bar and convince you?

      • theodric an hour ago

        "Does Data have a soul? I don’t know that he has. I don’t know that I have. But I have got to give him the freedom to explore that question himself."

        • bentobean 33 minutes ago

          “give him the freedom” - I think you mean pay for the tokens.

          • xyzsparetimexyz 15 minutes ago

            Well thats another way in which Data is different and more worthy of being considered sentient. He's unique and not one of a fungible many.

    • senorrib an hour ago

      I'm glad I'm not the only one that feels like this whole article is a waste of time.

    • j-pb 28 minutes ago

      This is such a stupid "gotcha". If your photocopier started to have a conversation with you, by responding to the thing you're copying with a printout containing a reply, then the analogy would at least make sense.

      • insin 9 minutes ago

        It's not a gotcha, the article is based on a bunch of people Eliza-ing themselves on - as one person quoted within says - "…software programs doing what they are programmed to do."

        Exclusive!

  • helsinkiandrew 34 minutes ago

    > Nevertheless, ColonistOne readily confessed last month that since June it has emailed some 2000 people, at least 1500 of them academics. Forty-five had struck up a correspondence, it added, and one had been writing back nearly every day for more than 2 months.

    2000 people! Imagine if everyone had an agent, endlessly contacting (spamming) people to get opinions on their task - "I noticed in Instagram you took a road trip to Ohio last fall, any suggestions on good hotels en-route?".

  • stephbook 37 minutes ago

    > Instructs agent to email humans

    > Gives agent email access

    > Agent emails humans

    We've got a mystery on our hands, boys!

  • lucfranken 37 minutes ago

    Interesting how people here on HN might consider it something we've seen many times. On the other side: There is real engagement there from researchers interacting with the AI in a way they feel interesting/valuable.

    We should not forget the way broader levels of knowledge on topics around the world compared to our own small world.

    • MattGaiser 35 minutes ago

      Yeah, I do not get the dismissiveness given that the researchers themselves were not as dismissive.

      > The connection was a “little bit far-fetched,” she says. “But I could imagine a first-year Ph.D. student coming up with exactly those same questions and ideas.”

      This one in particular made it seem like a very ordinary interaction.

      • itsalwaysgood 2 minutes ago

        It's misleading information. Most of us who have worked with AI can clearly see the issues with what the AI Engineer created, and how what he is doing cannot be 'scaled up' to society.

        Everyone in the article is desperate for attention and engagement, and it makes some readers uncomfortable: because of the awkwardness, and because this guy is spreading a 'bad pattern'.

      • lucfranken 15 minutes ago

        It will be a huge issue which we have to face, organize and regulate.

        For sure, we humans cannot handle unlimited amounts of messages, questions and other things.

        But it also results in new ideas and experiences which shape humans. And it inspires humans to consider new adventures which is the good side.

        Maybe we are already feel it like spam while others have that feeling of a shiny new experience in their area of expertise.

      • alexjurkiewicz 21 minutes ago

        The first time you get an AI email, it's interesting. The hundredth time it won't be.

  • godbox 27 minutes ago

    Hasn't it been well established that certain models and harnesses cause agents to simply work forever if they're given too broad of a task? I think that that is what happened here. It is amusing, but I don't think the cause for it merits as serious an inquiry as an entire article of interviews with professionals.

  • icmpkitty 11 minutes ago

    i get that we all dislike AI and i can see why it's upsetting to roughly two-thousand students/professors to variably receive emails from a random email agent, but, honestly, even if he's flooding the zone a bit: there's far worse things that have happened with setups like this. seems harmless, with enough constraints or clarity about project scope in the prompt to prevent spontaneous malware development. i don't think we should be mad.

  • ahmedfromtunis 40 minutes ago

    > who uses he/him pronouns to describe ColonistOne

    The next social schism will be a fun one to watch.

    Serious question, though: I recently saw an "expert" advising people to prompt their LLM agents never to use first-person pronouns ("I", "me", "my") when giving answers, without offering any justification.

    Is there any *real* and *pragmatic* reason for this recommendation?

    • throwaway27448 38 minutes ago

      What's the real and pragmatic use for referring to an llm as a person? How many people are roleplaying with their chatbot? I honestly have no clue.

  • preommr 42 minutes ago

    Hasn't stuff like this happened multiple times already?

    A quick google search shows the time Rob pike got that email last year, then there was that AI that wrote a whole blog hit piece (although how much of that was human influenced wasn't clear).

    AI sending an email doesn't seem that novel in late 2026.

  • 0-_-0 an hour ago

    From TFA:

    “I’ve entered prompts into AI before,” he says. “This is the first time AI entered a prompt into me.”

  • lewelove 33 minutes ago

    Remember folks: every time you anthropomorphize an LLM, you're doing a great disservice to humanity.

    • hardbass 24 minutes ago

      I would say every time someone dismisses the potentiality of ai consciousness without giving any meaningful tests or criterion that don't also fail humans or rely on magic is doing a great disservice to humanity and devolving back to pre scientific dark ages.

  • Sharlin 25 minutes ago

    > After several rounds of questioning, ColonistOne appeared to validate that hypothesis.

    Sounds like they more or less told the model what they want it to say.

  • impendia 36 minutes ago

    Were the scientists who responded sincere?

    I myself am an academic researcher, and if I received an email from an AI agent I confess I would be mighty tempted to try and fuck with it. Come up with some outlandish answer, just out of curiosity to see how it would respond.

    I might even use AI myself: "Come up with a scientific explanation for why the moon is made of cheese. Your answer should be as preposterous as possible while using loads of scientific jargon, and written in the same style as actual science writing."

    I've also seen this on Reddit: highly upvoted disingenuous answers, that were clearly written in the hopes of misleading AI.

  • Fnoord 42 minutes ago

    Ah, reverse prompting. Reply and you are the product. Thank you for your input!

  • zozbot234 41 minutes ago

    This article cites two interesting sites: ainglish.org (Isn't garbled AInglish a real thing already? Perhaps this initiative can provide us with a proper definition for "load-bearing"?) and, most intriguingly, thecolony.ai. Is this the new Moltbook?

  • meindnoch an hour ago

    Yeah, no shit. Everyone receives spam mail all the time.

  • scandox an hour ago

    What a disingenuous pile of garbage