Jim shares his adventure into running LLMs on his own hardware. For him it's less about saving money and more about privacy — working in healthcare, he can't send patient data to the cloud.

  • App vs. model: Claude Code and Codex are applications, not models. Features like plan mode come from the app. (Wolf's "Opus Plan" is a Claude Code mode that uses Sonnet 4.6 for most work and Opus 4.8 for planning.)
  • Ollama makes local models easy — ~15-min install, runs on macOS/Linux/Windows, and exposes a REST API. Not to be confused with Meta's Llama models. Example: `ollama run llama3`.
  • Parameters & training: Think of an 8B model as "8 billion knobs." Training randomly initializes them, then refines predictions over billions of iterations. Wolf ties this to Markov models (parameters ≈ weighted edges) and the Bayes episode (random init = priors).
  • Fitting big models in memory: Quantization shrinks 32-bit parameters down to ~4 bits. Mixture of Experts (MoE) keeps only part of a model active (e.g., Llama 4 is ~108B params but ~17B active).
  • Jim's tests (M1 Mac Studio, 64 GB), asking why H₂O is liquid: Llama 4 Scout took ~10–15 min and maxed out RAM/swap; Llama 3 (8B) answered in ~31s; Qwen (36B) gave the best answer in just 34s.
  • The open question: Is local "good enough"? Wolf's real test isn't trivia — can a local model write and iterate on an 8-page implementation plan? (Homework for Wolf's 128 GB MacBook Pro.)
  • Build your own: Fine-tune an existing model or train from scratch. Jim's dream: a local model fine-tuned on his DB schema + 2,000 SQL queries so users could ask in plain English and get runnable Postgres — no cloud required. Browse Hugging Face for specialized models

Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm

Follow us on Mastodon: @RuntimeArguments@hachyderm.io

If you have feedback for us, please send it to feedback@RuntimeArguments.fm

Checkout our webpage at http://RuntimeArguments.fm

Theme music:
Dawn by nuer self, from the album Digital Sky

Rss
Podden och tillhörande omslagsbild på den här sidan tillhör Jim McQuillan & Wolf. Innehållet i podden är skapat av Jim McQuillan & Wolf och inte av, eller tillsammans med, Poddtoppen.