Small Language Models and On-Device AI: When You Don't Need the Cloud

The default way to add AI to a product has been to call a large model in the cloud. That is still right for many features, but a growing share of AI work can now happen directly on the user's device, and in some cases that is the better choice.

What changed

Three trends converged. Model makers learned to pack strong performance into small models through better training data, distillation and quantization. Phones and laptops gained dedicated neural processing units. And platform vendors started shipping on-device models and APIs so app developers do not need to bundle their own for common tasks.

When on-device AI wins

• Privacy-sensitive data. Health notes, financial details, personal messages and corporate documents can be processed without leaving the device.
• Offline or unreliable connectivity. Field workers, travel apps, factories, ships and rural areas.
• Instant response. No network round trip, which matters for typing assistance, live camera features and games.
• High volume, low value per request. Autocomplete or classification on every keystroke would be expensive through a paid API.
• Regulated environments where sending data to external processors needs heavy legal review.

When the cloud is still better

Complex reasoning, long documents, broad world knowledge and tasks requiring the strongest available model still favor large cloud models. Small models also need more careful testing because they fail more often outside their focus.

Hybrid patterns

Most real products combine both:

1. Local first, cloud fallback. The device handles the request; if confidence is low or the task is complex, it asks the user before sending to the cloud.
2. Local pre-processing. The device redacts personal data or extracts the relevant part, so only minimal information goes to the cloud.
3. Local for speed, cloud for quality. A quick on-device draft appears instantly and a refined cloud version follows.

Shipping on-device AI in an app

• Choose the runtime. Platform APIs on iOS and Android cover common tasks. Cross-platform runtimes let you ship your own model across devices.
• Mind the download. Even small models are hundreds of megabytes. Download on demand over Wi-Fi rather than bloating the install.
• Test on low-end hardware. Performance on a flagship phone tells you little about a three-year-old mid-range Android device.
• Watch battery and heat. Sustained inference drains batteries. Batch work and use accelerators where available.
• Plan updates. You need a way to ship improved models without a full app release.

Beyond phones

The same models run on edge gateways, kiosks, cars, robots and industrial PCs. In robotics and manufacturing, on-device inference is often a requirement rather than a choice, because decisions must be made in milliseconds. Our articles on edge computer vision and physical AI go deeper.

The takeaway

Ask "does this need to leave the device?" for every AI feature you plan. For a surprising number, the answer is no, and the result is a faster, more private and cheaper product.