Can a MUD evaluate LLMs? A $99 proof of concept

(cruciblebench.ai)

54 points | by Davisb135 3 hours ago

9 comments

  • krzyk 5 minutes ago
    I was a bit afraid that MUD was an acronym for something else, I clicked in about 10% hope that this will be for multi user dungeons and behold, thank you. Interesting read.

    That brought back memories when we were trying to find free terminal in few departments at our uni to get access to one that has internet connectivity.

  • Varelion 1 hour ago
    A little irrelevant, but I need to vent about something MUD adjacent.

    I really wish MUDs were still a thing. The text-based roleplaying communities were largely swallowed by Discord, where as before they existed scattered over proboards, jcink, and a few others.

    Discord, being the imperfect text platform, limits the text and leaves most interaction limited by awkward formatting. To boot, even when a better ecosystem is invented, players refuse to give it a shot even if directly invited, because they got comfortable with Discord's terribleness.

    It's so frustrating, and I don't know how to voice it, or even what can be done.

    • iepathos 44 minutes ago
      MUDs never disappeared. People stopped finding them engaging enough to use. Blaming Discord reverses the causality. MUDs lost engagement long before disco was created. People congregate on disco because it offers a more compelling social interface, even if it lacks the depth of a persistent game world. The opportunity I'd draw from this isn’t to convince people to return to older and less engaging interfaces. It’s to build persistent roleplaying systems around the interaction modes that have proven to be more engaging already. Potentially combine voice, speech recognition, text, tts, and synthesized characters with a proper world simulation underneath. If roleplaying communities already gather in disco, then deliver the experience where they already are. The interface was always irrelevant to what made MUDs fun anyway. Imagination and connecting with other people was always the thing.
      • Varelion 34 minutes ago
        I specifically outlined non-MUD role playing environments because I know MUDs lost popularity long before Discord. I am outlining that any form of tech, besides Discord, is sidelines -- MUDs and a hopeful resurgence as well.
    • Davisb135 1 hour ago
      I agree with you. You've also got more / easier competition. IIRC, MUDs peaked in the mid-late 90s. MMOs were really the death knell, then the explosion of other types of games, mobile games, then to your point things like Discord.

      One adjacent project we're exploring for the future is if you took all of the current understanding of game design and mechanics with modern AI functionality, knowledge bases, etc, and built a MUD, what would it look like and would it be enough to bring back players from other games?

      • Varelion 1 hour ago
        I honestly worry that the relevance of text-based systems is largely tied with literacy and how it interacts with the player's imagination -- something largely in decline, at least in the US.
        • Davisb135 1 hour ago
          Sadly, you're not wrong... It might not get off the ground, but if we're able to, we'll give it a shot.
          • Varelion 1 hour ago
            I really, really hope you lot manage to!
    • nerevarthelame 52 minutes ago
      Simutronics is has been running and developing Gemstone and DragonRealms for 30+ years and counting. The communities are about 400-1000 players each. They've embraced microtransactions (in the form of quarterly events), but both games are entirely playable without participating in that aspect.

      With modern technology, training in both games is very automatable. But developing and optimizing your training routines is (in my opinion) incredibly fun. Actual roleplaying is less common it was in the 90s, but there are still nightly events where dozens of players will get together to socialize and explore the worlds in-character.

    • ticulatedspline 59 minutes ago
      Technically MUDs still are a thing, there's a good number still running. Though sounds like you don't really just want a MUD where you want to go out mobbing but a more social thing (but not IRC?).

      I suspect MUDs may have a bit of a resurgence soon, I've seen more and more interest in them as of late simply because they seem like an excellent pairing with LLMs which easily consume text. Though I think for your needs someone would need to work on the barrier to entry to getting on a MUD as well as the modern UI/UX expected today.

    • yehoshuapw 19 minutes ago
      discworld mud (any many others) ase still totally worth playing
    • strbean 1 hour ago
      > even what can be done.

      I think this could be solved with bridge bots to make your MUD multiplatform. You could have a dedicated web app, an IRC bridge, a Discord bridge, etc.

      Discord-brained users can still use your MUD and contribute to network effects, and you can provide an off-ramp ("I just want to play VarelionScape, maybe I'll just open the dedicated website on my phone instead of going on Discord and digging through the server list").

      • Davisb135 1 hour ago
        That's a good idea, thanks! We'd only considered the web app + using a client.
    • cmrdporcupine 51 minutes ago
      You would probably like my project https://github.com/timbran-project/moor
    • fragmede 1 hour ago
  • dataviz1000 47 minutes ago
    > the measurements of each LLM

    I think you have the correct idea.

    If you ask an LLM to solve a multiplication problem using reasoning without code tools, depending on the model, it will get 15 digit (15D) * 15 digit multiplication correct (123456789012345 x 998765432109876). It will take between 4000 and 8000 tokens if Sonnet. Eventually with enough digits, it will start only solving the problem 80% ... then 70% of the time until it has so many digits it will never solve. There will be a certain number of digits where it will not converge on a solution nor will it stop working thinking it can solve it. That is very, very expensive. It takes a lot of runs to determine the probability it will solve it.

    What you can do now, this is likely the most important thing, is change the prompt and evaluate how many tokens and at what speed it takes to accomplish the task. Sure the measurements of each LLM are important! Nonetheless, if you can say to a company that you have a technique to tune prompts and evaluate them so instead of spending $1 X 100,000 times a day, they can instead spend $0.90 X 100,000 times a day, you will make a ton of money.

  • thomas536 41 minutes ago
    +1 to using MUD as a learning tool. Learn about agent loops, prompting iteration, etc.

    I vibe coded a few room proof of concept MUD. I added a tools called `oracle` (aka me) that it could ask me questions to help along its way; and the ability to interject in the loop with a hint. Just as I might want an llm to stop to ask me for help instead of plodding along...

    Interesting to limit the number of turns and see where it gets stuck or how quickly it can finish.

  • Towaway69 1 hour ago
    Speaking of MUD, has anyone tried attaching an AI to the Hitchhikers MUD[1]?

    [1]: https://www.bbc.co.uk/programmes/articles/1g84m0sXpnNCv84GpN...

    • imzadi 24 minutes ago
      This is a text-based game, but it is not a MUD. The MU in MUD stands for multi-user.
    • Davisb135 55 minutes ago
      Not Hitchhiker's specifically, we'd considered another existing one but as soon as we started digging into IP law related to MUDs, we decided if we're going to do it, we'll just start our own from scratch.
    • formvoltron 14 minutes ago
      that's funny. the very first thing i had an llm do was to ask it to pretend to be the HHGTTG infocom game. Can't remember how it did.. not amazingly but was interesting still.
  • gbacon 1 hour ago
    That’s a fun design that brings back some old memories.
    • Davisb135 1 hour ago
      Thanks! Yeah, it started with a friend's idea to take a MUD we played in the 90s / early 2000s that had been open sourced and adapt it to mobile games. He's a middle school teacher and wanted to introduce the next generations to MUDs.

      We then decided to pivot and thought, what if this would be a good eval environment for LLMs from the simplistic standpoint of, text is their native environment.

  • artemonster 9 minutes ago
    "We did not choose a MUD because it is charming. We chose it because its constraints make behavior measurable." I so fucking hate these LLMisms