Open Source AI

Microsoft officially supports DeepSeek on Copilot+ PC (7B/14B) via NPU inference. FreeToken open-source tool enables 35B MoE on 8GB GPU at 39 tok/s with dynamic scheduling and agent prefill. FreeToken also runs 753B MoE (GLM-5.2) on single workstation. Muse Glimmer 30B runs on single consumer GPU. Qwen3.8-27B runs on AMD Ryzen AI Max. New technique runs Qwen3.8-27B in 3.33GB VRAM via layer streaming. NobodyWho inference engine released. Practical guides for budget hardware, 8GB VRAM, CPU-only, phones. New tools: HashAgent, Cascadia, TuneMyLLM, Soup CLI, Unsloth Desktop (GUI fine-tuning), Jan AI. Distilled reasoning models deployment specs (8B on 16GB, 32B on 80GB). Qwen 3.8 27B runs on $1K mini PC with 3-bit quantization (8-13 tok/s). Ollama v0.33.0 released with MLX cross-platform fixes and Claude Desktop integration. Code Llama now available on Ollama (7B, 13B, 34B). Shoehorn quantization tool claims to quantize any model for local deployment. OpenJarvis guide provides step-by-step for local AI assistant. Kimi K3 local guide reveals multi-GPU server requirement, challenging democratization for ultra-large models. Critical new finding: inference backends and quantization silently break tool calls—even with identical weights—undermining reproducibility. Recent first-person accounts (Matt's Meandering Mind, personal agent testing) reinforce local deployment benefits and model selection challenges, with one user coding two production services using Devstral1 and local tools. New paper Llama-Mobile introduces 2.7-bit quantization for VLMs, compressing Llama 3.2 11B to 3.7GB for mobile inference (S3D8 format, self-distillation, 3.8 tok/s). A new practical guide for local SLM deployment covers model selection, Ollama setup, and hardware optimization. JANG quantization tool for MLX claims to run 397B Qwen3.5 on 128GB Mac at 2.1-bit. Daedalus-150M, a convolution-attention hybrid designed for CPU inference, beats larger models on CPU. JetBrains released Junie Local, a tuned local coding agent with Qwen3.6-27B, validating local deployment for professional use. TileMix, a new mixed-precision attention kernel, improves long-context inference throughput. Practical optimization case study for Qwen3.6-27B on MacBook M5 shows rolling context reuse, prefix caching, and 2x speedup by disabling reasoning. Security hardening for Ollama+Open WebUI is now a community focus. Perplexity partnered with Nvidia to launch Portable Computer, a fully local AI agent with zero token costs, a major validation of local AI; now shipping with 85.4% on local bench. A critical NemoClaw vulnerability (DNS rebinding via Ollama binding to 0.0.0.0 on Windows/WSL) threatens local agent security. New MLX-DSPark tool offers up to 4× faster decoding on Apple Silicon. An ACL paper on SLM deployment trade-offs finds single-agent with tools best for sub-10B models. University workshops formalize local LLM training with Ollama. Practical guides for running Qwen 3.8 27B uncensored and comparing local tool calling across Gemma 4, Llama 3, Mistral. Apple's new Mac minis (M6/M5 Pro) offer up to 64GB RAM for local AI, though lack of CUDA limits headroom vs AMD/Nvidia. Perplexity's Portable Computer (on Nvidia DGX Spark) is now shipping with full local harness, sandbox, and zero per-token cost for local steps, achieving 85.4% on their local bench—a major validation of local-first agentic AI for enterprises. Liquid AI open-sourced Pipette, a reproducible benchmarking suite for on-device models, quantization, runtime, and hardware, revealing that two 350M models differ 78.4% vs 33.8% in context scaling on the same phone. New SME guides for private knowledge assistants and document processing with open-weight models reinforce democratization. Nvidia's RTX Spark superchip (128GB unified memory, 1 Petaflop FP4) announced for slim laptops, expanding local deployment hardware. A new practical guide for choosing open models emphasizes hardware should follow model evaluation, considering pipeline design, latency, and cloud fallbacks.