Small language models at the edge

What is a small language model?

It is a language model compact enough to run on limited hardware: a phone, a laptop, a single local server or an industrial gateway. There is no agreed size cut-off. The term describes the goal, running close to the data, more than a parameter count.

Vendors use the label for their own compact models. Microsoft describes Phi as "the Microsoft family of small language models (SLMs)."^[1] Meta's Llama 3.2 collection includes 1B and 3B text models that are "expected to be deployed in highly constrained environments, such as mobile devices."^[2]

Why run a model at the edge?

Four reasons come up most often:

• Data stays local. Google's Android documentation says on-device AI works "without needing a network connection or sending data to the cloud."^[5] That matters for sensitive text such as contracts, patient notes or source code.

• It works offline. Field sites, vehicles and secure facilities may not have a reliable connection.

• Response time is steady. There is no network round trip, although Google notes that inference speed then "depends on device hardware."^[5]

• Cost is fixed. You pay for hardware once instead of per call, which can help when the same task runs many times a day.

None of these makes a small model the right choice on its own. They matter when the task is narrow enough for a small model to do well.

Which small models do vendors offer?

Vendors change their lineups often, so check each page before you choose. On the pages we checked:

• Microsoft Phi. The Phi page lists Phi-4, Phi-4-mini and Phi-4-multimodal among its models and says Phi can run "directly on a device without the need for cloud connectivity." Microsoft states the models are open source under the MIT License.^[1]

• Meta Llama 3.2. The 1B and 3B models have 1.23B and 3.21B parameters, with quantized versions for "on-device use-cases with limited compute resources." Use is governed by the Llama 3.2 Community License.^[2]

• Google Gemma 3n. Google describes Gemma 3n as "optimized for use in everyday devices, such as phones, laptops, and tablets."^[3] Read the terms Google links for the specific Gemma model you pick.

• Mistral Ministral 3. Mistral lists Ministral 3 in 3B, 8B and 14B sizes, under Apache 2.0.^[4]

Phone platforms also ship a built-in model. On Android, Gemini Nano runs in the AICore system service, and apps reach it through ML Kit's GenAI APIs for tasks such as summarization, proofreading and rewriting.^[5] Using a platform model means you do not ship or update the model yourself, but you accept its limits.

How do models get small enough?

Mostly through two methods:

• Quantization stores weights at lower precision so the model uses less memory. Meta's quantized Llama 3.2 models use a 4-bit groupwise scheme for the linear layers in every transformer block.^[2]

• Distillation trains a small model to imitate a larger one. Meta used outputs from larger Llama 3.1 models as training targets for the 1B and 3B models, and distillation after pruning "to recover performance."^[2]

Google's Gemma 3n takes a different route: the E2B model loads over 5 billion parameters in standard execution, but with parameter skipping and caching it can run with a memory load of about 1.91 billion.^[3]

Every one of these trades some quality for size. How much depends on the task, which is why testing matters.

What tasks suit a small model?

Bounded tasks with a clear right answer:

• Extracting fields from forms, emails or invoices.

• Routing a request to the right queue or intent.

• Summarizing or rewriting short text.

• Answering questions over a small, local document set.

Tasks that need broad world knowledge, long reasoning chains or many tools usually do better on a larger model. Google itself notes that on-device models have "token limits, device hardware limitations, and model capability boundaries."^[5] If a small model is close but not quite good enough, fine-tuning an open model on your own examples is often the next step.

Should you use a hybrid design?

Often, yes. A common pattern is to let the small model handle routine requests locally and send hard or high-value ones to a larger cloud model. Google's Android documentation points to hybrid inference for cases beyond the on-device model's limits.^[5]

Decide the escalation rule in code, not in the prompt, and log which path each request took. If the cloud path gets document context, our guide to enterprise RAG architecture covers that pipeline.

How do you test a small model?

The same way you would test any model, on your own data:

1. Collect a few hundred real examples with the answers you expect.

2. Run them through the small model and the larger model you would otherwise use.

3. Compare quality on the task, plus speed and memory on the target hardware.

4. Check where it fails, not only how often. Our article on AI hallucination verification covers checks for wrong answers.

5. Re-run the tests whenever you change the model, the quantization or the device.

What else should you plan for?

Updates and governance. A model on a thousand devices needs a way to be updated and rolled back. Licenses differ by vendor and model, so keep a record of which model and version runs where. Data that stays on the device still needs access controls; our article on AI governance for LLM training covers the controls to set.

Our custom AI agents service covers model choice and testing as part of our AI consulting work.