Running a 28.9M parameter LLM on an $8 microcontroller

(github.com)

106 points | by boveyking 9 hours ago

9 comments

  • titzer 2 hours ago
    It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:

    https://milkv.io

    The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!

    • brucehoult 1 hour ago
      Don't forget the 128 bit vector ISA with 32 registers, supporting up to 64 bit int and FP, and with LMUL=8 you can process 1024 bits with a single instruction (at 3 cycles per 128 bits for most operations). Fully supported by GCC and CLANG (xTHeadVector) and compatible with RVV 1.0 with just a command line switch if you use the C intrinsic functions. (a lot of code working on 8 bit elements is binary compatible with RVV 1.0 too e.g. typical memcpy(), memset(), memcmp(), strlen(), strcpy(), strcmp())

      When I bought my 64 MB Duo they were $3!

      Then for a long time they were $5 for the 64 MB, $7 for the 256 MB, and $10 for the 512 MB.

      Sadly, like everything else, they've gone up considerably this year.

      https://arace.tech/products/milk-v-duo

      https://arace.tech/products/milkv-duo-s

    • skippyfish 2 hours ago
      You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task.

      And I hope it stays that way, I don't want MCU shortages...

      • dannyw 11 minutes ago
        Fun and learning is a really good reason. It also reminds me of the damascene: trying to achieve something that doesn't feel possible, and working through all the extreme resource-constrained engineering limits.

        I think most of my embedded projects aren't that useful, but they've taught me a lot.

      • titzer 1 hour ago
        I want to run music models on the Milk-V and have them jam in realtime with me.
      • NooneAtAll3 1 hour ago
        isn't floor price 0.10$ for last couple years?
        • skippyfish 1 hour ago
          Well, that's sub-$0.50. But yeah, CH32V003 is in that ballpark, and some of the cheapest Microchip and Infineon products are around $0.20.

          It's almost never worth it to buy the cheapest chip unless you're making a million of something, but there are very good ones around $1-$2, and $5 is the upscale stuff.

  • rao-v 4 hours ago
    This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
    • Lerc 3 hours ago
      One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions.

      PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton of them running all at the same time.

      The large scale dedicated hardware systems will still have the edge for performance per watt, but the low entry level and slow incline does make these things quite appealing.

      • hgoel 20 minutes ago
        Wouldn't a layer per MCU be heavily bandwidth constrained?
        • monocasa 1 minute ago
          Depending on where you slice the model up, it can be not a whole lot of data. For instance each transformer block outputs a single vector in an embedding space.

          I can see that being cheaper to bitbang with PIO than to actually compute.

          There's certainly some latency stack up, but throughput should be remarkably good.

      • NuclearPM 1 hour ago
        Run the numbers before you waste time here. I doubt this will work.
  • ReactiveJelly 21 minutes ago
    Finally, an LLM that can run on my i5
  • chrishynes 4 hours ago
    Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
    • AussieWog93 2 hours ago
      Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD.

      I think they got something like 10 _seconds per token_ (not tokens per second).

      EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri

      • 3eb7988a1663 1 hour ago
        That's incredible. Sure, not practical for most applications, but if you really want a local top tier model, you can run it on anything as long as you are patient.

        As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, you could probably do something with that given extensive coffee breaks. This speed simulator[0] demos what it looks like.

        [0] https://shir-man.com/tokens-per-second/?speed=1.8

    • Rohansi 2 hours ago
      My guess is because the ESP32's flash is only ~1/4 the bandwidth of the internal SRAM. If you do this on a more powerful system not only is the gap much wider but you also have much more compute you need to keep fed with bandwidth to be efficient.
  • spacedoutman 3 hours ago
    >esp32-s3

    This microcontroller is a beast, currently using it to do dev work on a pi4.

    Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise

    • raphlinus 2 hours ago
      If you want to do this at the $1 price point, you can on RP2350, albeit with some limitations. In particular, it maxes out at full speed (12Mbps). The trick is to use the on-chip USB peripheral for one, and connect the other to GPIO pins backed by PIO.

      This works today with tinyusb and pico-pio-usb, but I'm also playing with a Rust port which I'm hoping will have higher performance.

      • Rohansi 1 hour ago
        $8 ish gets you an ESP32-S3 board with PSRAM, flash, and two USB-C ports. The PSRAM and flash are specifically used for this LLM project. I can't find anything like that with the RP2350 for $1.
        • raphlinus 1 hour ago
          Entirely fair, $1 is just the chip, not the board. No question the ESP32-S3 is incredibly good value.
  • althea_tx 4 hours ago
    This is a really cool project. Thanks for sharing!
  • cr125rider 4 hours ago
    9.7 tokens/sec actually seems like a lot! That’s fun!
  • smy20011 3 hours ago
    Super cool, thank you for sharing!