everybit

Here are some of the projects we've built. We're open about our approach, sharing what we've learned and how we build it.

ShakesGPT

We trained a char-level teeny tiny LM(44k) on a Waveshare ESP32-S3 Touch LCD 1.69 (ESP32-S3R8, 8 MB PSRAM, 16 MB Flash, ST7789V2 240×280 SPI LCD), no external GPU, no pre-trained weights, starting from random initialization. It learns from the full 1,115,394-byte TinyShakespeare corpus (65 unique bytes), embedded directly into firmware .rodata via board_build.embed_txtfiles rather than LittleFS, so a checkpoint upload never wipes the training data. While it trains, the on-board LCD shows live loss, moving average, learning rate, a progress bar, and a loss curve graph. The best checkpoint is saved to flash, and on the next boot the board generates Shakespearesque (is it even a word?) word salad (sadly 5k steps are not enough) from what it learned. There's no autograd anywhere in the pipeline, the entire backward pass is hand-derived, gradient-checked against PyTorch, and validated in QEMU (xtensa) before ever touching hardware. Both LX7 cores split each batch and train in parallel. The build is pinned to espressif32@6.12.0, board esp32-s3-devkitc-1 and framework arduino.

Teeny tiny LM (44k param, 65 vocab) lol

[image or embed]

— aloobun (@aloobun.bsky.social) August 20, 2026 at 8:35 PM

Notes on Challenges & discoveries

• GCC 8.4.0 for Xtensa emits zero SIMD/PIE instructions, the only FMA mechanism is the scalar madd.s instruction, contracted by -ffast-math. Disassembling firmware.elf with xtensa-esp32s3-elf-objdump showed the matvec kernel was a single scalar accumulator per column, no interleaving.
• A reboot loop every ~24s, caused by the task watchdog. Both compute-heavy tasks (one per core) starved the IDLE0 task, triggering task_wdt: Task watchdog got triggered. IDLE0 (CPU 0) did not reset in time. Calling esp_task_wdt_deinit() alone silently failed (ESP_ERR_INVALID_STATE) because IDF 4.4's task_wdt.c requires the subscriber list to be empty first. The fix was esp_task_wdt_delete(xTaskGetIdleTaskHandleForCPU(0)) before esp_task_wdt_deinit(), both now called at the top of setup().
• The malloc() / PSRAM paradox. A first benchmark of matvec(160×160) showed PSRAM and SRAM weights performing identically at 2.39ms, because the "SRAM" test used malloc(), which on this Arduino-ESP32 build is actually PSRAM-backed (CONFIG_SPIRAM_USE_MALLOC=y). Switching to heap_caps_malloc(..., MALLOC_CAP_INTERNAL) revealed the real numbers: 0.61ms (true SRAM) vs 1.79ms (PSRAM), a 3× gap, not the ~20× expected, since the in-order Xtensa LX7 has no hardware prefetcher but its D-cache still handles streaming reads well.
• Dual core cache thrash. A single-core matvec benchmark ran at 210ms/seq; with both cores training simultaneously that rose to 361ms/seq, a ~70% penalty from both cores fighting over the single shared D-cache. The dual-core split still wins overall, but only marginally (1.48s/step vs ~1.68s single-core).
• The linker appends a trailing NUL to embedded blobs. board_build.embed_txtfiles adds one NUL byte after the corpus text, which was initially miscounted as a 66th unique byte and overflowed NV=65. Fixed with CORPUS_LEN = (corpus_end - corpus_start) - 1.
• Loop unrolling helped less than expected. 4 way unrolling the matvec inner loop gave only a 1.3× speedup (2.39ms → 1.79ms), because the real bottleneck was never instruction throughput, it's PSRAM memory latency. Per-FMA cost stays around 39 cycles even with the model weights in SRAM, because gradients and activation caches are still PSRAM-resident.
• The backward pass dominates runtime at roughly 165ms/seq versus 45ms/seq for the forward pass, driven mostly by PSRAM-resident gradient writes (G+= operations) and activation cache array access.

Some more notes on what we built

• Fixed the backlight GPIO mapping (17 → 15) in both platformio.ini and main.cpp.
• Fixed the task watchdog reboot loop by unsubscribing IDLE0 before deinit.
• Shrank the model from 324K to 44K parameters (NC 160 → 56, NF = 224), so it fits entirely in internal SRAM, verified against a gradient check (worst relative error 2.86e-07).
• Moved the model weights to internal SRAM via heap_caps_malloc(sizeof(Model), MALLOC_CAP_INTERNAL), while gradients, velocity, cache, and the best-checkpoint buffer stay in PSRAM via ps_malloc.
• Added -funroll-loops to the build flags and hand-unrolled the matvec inner loop 4 way, with independent accumulator chains and a cleanup loop for the remainder columns.
• Reduced task stack sizes from 32KB to 16KB for trainTask, core0Worker, and genTask.
• Marked the dual-core Core0Job fields volatile to prevent compiler reordering across cores.
• Made the serial header dynamic (%dK params from NPAR/1000) instead of a hardcoded, stale param count.
• Fixed a vocab overflow bug caused by the linker's trailing NUL byte being counted as a 66th unique character.
• Removed the misleading SIMD/auto-vectorization comments and corrected the source documentation to reflect the real memory architecture.
• Stripped out all diagnostic instrumentation (benchmark blocks, per-step core timing) for a clean production firmware build.

Current state

Params:    44K  (NC=56, NF=224, NT=32, NV=65)
RAM:       6.5%  (21,312 / 327,680 B)
Flash:     22.4% (1,466,133 / 6,553,600 B)
Step time: 1,483 ms  →  5,000 steps ≈ 2 hours
Loss:      4.17 (step 0) → 3.11 → 3.27 → 2.83 (step 200)
Gradcheck: PASS, worst relative error 2.86e-07

The board is currently mid-run and will finish autonomously, generating text on the LCD and over serial.

What's next(probably)

• Flip FORCE_RETRAIN from 1 to 0 after the first successful run, so later boots load straight from the saved checkpoint into generation mode.
• Optionally shrink further to NC=40, NF=160 (~23K params) to fit both weights and gradients in SRAM for another ~2× speedup, at some cost to text quality.
• Optionally reduce TOTAL_STEPS for a shorter demo run (1,000 steps ≈ 25 minutes).

Contact via email