Right-Size Your Models, Save Your Power

Oxford’s TIDE programme reminds us that many tasks don’t need giant models. A Kenyan retailer swapped a 13B chatbot for a distilled 3B model on CPUs and saw a 40% drop in energy use with no hit to CSAT. They also pushed inferencing to edge boxes in stores, keeping latency low even during Kenya Power dips and reducing cloud egress charges. Baseline: measure energy per 100 inferences and track first-response quality. Then test a quantised or distilled model and compare. Keep the big model for monthly retrains, but let the slim model serve customers daily. For vision, prune layers and test on low-power accelerators; for speech, evaluate smaller streaming models that can run on-site. The point is not to starve your AI—just to right-size it so watts, shillings, and user experience stay in balance.