How to Run Edge AI on Small Models

If you want to learn how to run edge AI, the news from this week gives you a pretty dramatic example. Google has just put four TPUs into orbit on its Project Suncatcher satellite and is running Gemma on them, which means a language model is now doing inference hundreds of kilometres above the ground, with no data centre in sight.

Most of us will never launch a satellite. But the lesson transfers straight to the factory floor, the retail shelf and the farm sensor: when bandwidth is limited, power is tight and the round trip to the cloud is too slow, you move the model to the data. This guide walks you through doing exactly that with small models, using tools you can install today. We covered the space story itself in our morning post, and this is the practical follow-up.

What You Need Before You Start

You do not need exotic hardware. A Raspberry Pi 5 with 8 GB of RAM, a Jetson Orin Nano, or even a modern laptop is enough to follow along. On the software side, have Python 3.11, Docker, and a basic comfort with the command line. Pick one real task before you begin, for example classifying defects on a camera feed or summarising sensor logs, because edge AI projects without a concrete job tend to drift.

It also helps to write down three numbers up front: the maximum latency you can tolerate (say 200 ms), the memory you have (say 6 GB usable), and the power budget in watts. Those three figures will drive every decision below. Yeh step skip mat karna, because teams that ignore it usually pick a model that cannot physically fit.

Step 1: Choose a Model That Fits Your Budget

Start small on purpose. For language tasks, look at compact open models such as the Gemma family, where the smaller variants run comfortably in a few gigabytes. For vision, MobileNet and EfficientNet-Lite style networks are still hard to beat on cheap hardware. A good rule of thumb is that a model with 1 to 3 billion parameters, quantised to 4 bits, needs roughly 1 to 2 GB of memory for weights alone.

Benchmark two or three candidates on your actual data, not a public leaderboard. A slightly weaker model that answers in 80 ms beats a brilliant one that takes four seconds, because users and machines both stop waiting.

Step 2: Shrink and Convert the Model

Quantisation is your biggest lever. Converting weights from 16-bit floats to 8-bit or 4-bit integers cuts memory use by half to three quarters and usually speeds up inference, with a small accuracy loss you can measure. Tools like TensorFlow Lite and ONNX Runtime handle the conversion and give you runtimes built for constrained devices.

A sensible order of operations looks like this. First export the trained model to ONNX or TFLite. Then apply post-training quantisation and run your validation set again. If accuracy drops more than two or three points, try quantisation-aware training or a mixed approach where only sensitive layers stay at higher precision. Pruning and distillation are the next tools to reach for, but quantisation alone often gets you most of the way.

Step 3: Package and Deploy to the Device

Wrap the model and its runtime in a small container so every device behaves the same. Keep the image lean: use a slim Python base, install only the inference runtime, and avoid pulling in a full training framework. Push the image to a private registry and have each device pull it on a schedule.

For fleets, add an update channel from day one. You will want to roll out a new model version to five devices first, watch the numbers, then release it to the rest. Pair this with a simple rollback: keep the previous model file on disk so a bad release can be reversed without anyone travelling to the site. If your devices connect over a flaky link, design for offline first, with local queues that sync results when the connection returns.

Step 4: Monitor Accuracy and Resource Use in the Field

A model that worked in the lab will drift in the real world. Lighting changes, sensors age, and new product variants appear. Log the model version, latency, memory, temperature and a confidence score for every inference, and ship summaries (not raw data) back to a central dashboard. Sample a small percentage of low-confidence predictions for human review, and use those to retrain on a regular cadence.

Set alerts on thermal throttling too. Small boards slow themselves down when they overheat, and your latency numbers will quietly double without any code change.

Common Mistakes to Avoid

Optimising for accuracy alone. Teams pick the biggest model that fits on paper, then discover it blows the latency target once the camera pipeline and pre-processing are added. Always measure end to end.

Forgetting security at the edge. Devices sit in places you do not control. Encrypt the model file, sign your updates, and never ship credentials inside the container image.

No plan for updates. A model you cannot update is a liability. Build the rollout and rollback path before the first device ships, not after the first incident.

Key Takeaways

• Start with constraints: Latency, memory and power budgets decide your model, not the other way round.
• Quantise first: 8-bit or 4-bit weights give the largest gain for the least effort.
• Containerise everything: Identical packages make fleets predictable and updates safe.
• Watch the field: Log confidence, latency and temperature, and retrain on real data.
• Plan for rollback: Keep the previous model on the device at all times.

Need Expert Help?

If this feels like a lot to manage alone, TecniForge can handle the heavy lifting. Our team specializes in custom software development and AI integration. Get in touch with our experts.

Also read: Google tests AI in space with a new satellite, our earlier coverage on why this matters today. The original report is on ProPakistani.

Here is your challenge: pick one task in your business this week, set the three budgets, and get a quantised model answering on a single device by Friday. Small and working beats big and planned.