On-Device Inference on a Battery Budget

Running a neural network on a phone without draining the battery in two hours is an engineering problem, not a product aspiration. This post covers the real constraints, the tools that help, and the trade-offs you'll hit before you ship.

Why the Battery Is the Actual Constraint

Mobile SoCs have changed dramatically since 2016. A Snapdragon 8 Gen 3 has a dedicated NPU rated at 98 TOPS. An Apple A17 Pro hits around 35 TOPS on its Neural Engine. The raw throughput is not the bottleneck anymore.

The bottleneck is sustained power draw over a session. A typical 4,000 mAh battery delivers roughly 15–18 Wh of usable energy. An NPU running at full utilisation pulls 1–3 W depending on the chip and the workload. That sounds fine until you account for the rest of the system: display, modem, memory bus, the thermal governor throttling the CPU when the die hits 42°C.

In practice, a model that benchmarks at 12 ms per inference in a laboratory will run at 30–50 ms per inference after 90 seconds of continuous use on a mid-range device, because the thermal headroom is gone.

Your inference budget, therefore, is not just latency. It is latency × frequency × thermal state × battery state of charge. You need to think about all four simultaneously.

What Does "On-Device" Actually Buy You?

Privacy and latency are the two answers engineers reach for first, and both are real.

A model that never sends data off-device has a much simpler data governance story. No API call, no PII leaving the phone, no server-side logging to audit. For health and finance applications this matters enormously. For a photo filter it matters less.

Latency under good network conditions is a wash. A round-trip to a GPU inference endpoint in the same region is typically 80–150 ms including serialisation. A well-quantised MobileNetV3 on an A16 Bionic runs in under 5 ms. The gap is real. But it only matters to the user if the feature is in the critical path of an interaction, not a background process.

The case for on-device inference gets stronger when you factor in reliability. No network means no degraded experience in a tunnel, on a plane, or in rural India where 4G coverage still has gaps. For navigation, real-time translation, and wake-word detection, that offline reliability is not a nice-to-have.

How Do You Actually Fit a Model Into the Power Budget?

This is where the engineering starts.

Quantisation

The single highest-leverage technique. A full FP32 model is four bytes per weight. INT8 is one byte. INT4 is half a byte. The memory bandwidth reduction directly reduces power consumption, because DRAM access is expensive: roughly 3–20 pJ per bit depending on the memory subsystem.

Post-training quantisation (PTQ) with calibration data is the fastest path. For most vision and NLP models quantised to INT8, accuracy degradation is under 1% on standard benchmarks. INT4 is more aggressive: expect 2–5% degradation on language models, more on small models that don't have the redundancy to absorb it.

Quantisation-aware training (QAT) recovers most of that gap but adds a training loop. If you're deploying a model you trained from scratch, build QAT in from the start rather than retrofitting it.

Frameworks: TensorFlow Lite (TFLite) with the quantisation toolkit, PyTorch Mobile with torch.ao.quantization, and Core ML Tools for iOS. ONNX Runtime Mobile is a reasonable cross-platform choice if you need a single model artifact for both Android and iOS.

Pruning and Architecture Choice

Pruning removes weights below a magnitude threshold. Unstructured pruning achieves high sparsity (70–90%) but requires hardware that can exploit sparse computation. Most mobile NPUs don't in 2025. Structured pruning, removing entire channels or heads, gives smaller gains (20–40% parameter reduction) but the resulting dense model runs fast on standard hardware.

Better than pruning an existing model is starting with an architecture designed for mobile. EfficientNet-Lite, MobileNetV3, and MobileViT are all worth considering for vision. For on-device LLM-style inference, Phi-3 Mini (3.8B parameters) and Gemma 2B have both been optimised for low-memory inference. Apple's own on-device models in iOS 18 use a mixture-of-experts approach specifically to keep active parameter count low during inference.

Execution Engine and Delegate Selection

The model format matters less than the execution path. On Android, TFLite with the NNAPI delegate routes to the NPU on Qualcomm and MediaTek devices. Without the delegate, you're on CPU. The difference is 5–10× in latency and power consumption.

On iOS, Core ML is the right choice. It handles delegate selection (Neural Engine, GPU, CPU) automatically and does it well. Do not try to out-schedule Core ML on Apple hardware.

For cross-platform work, ONNX Runtime with the QNN execution provider (Qualcomm AI Engine Direct) is worth the setup cost on Snapdragon devices.

Batching and Scheduling

On-device inference is almost always batch size 1. That's fine. What's not fine is running inference on every camera frame at 30 fps when the feature only needs 5 fps of analysis. Frame-skip logic and event-driven triggering (run inference when the accelerometer detects movement, not on a timer) reduce duty cycle substantially.

Background inference should respect ProcessInfo.isLowPowerModeEnabled on iOS and PowerManager battery saver state on Android. Deferring non-critical inference to when the device is charging is a legitimate strategy.

When On-Device Inference Is the Wrong Choice

There are real cases where you should not do this.

If your model is large and changes frequently, on-device deployment becomes an update distribution problem. A 500 MB model weight file pushed via app update is painful for users and expensive in CDN costs. Server-side inference with a thin client is operationally simpler.

If the task requires more context than fits in device memory, you'll hit limits quickly. A long-context summarisation task that needs a 7B+ parameter model is not a realistic on-device workload in 2025, regardless of what the benchmark sheets say. Running a 7B model on a phone thermal-throttles the device within minutes under continuous load.

If your user base skews to older or low-end devices, the NPU delegate assumptions break down. A device running Android 8 with a Snapdragon 660 will not give you the same execution path. Test on your actual device distribution, not on the flagship you have on your desk.

Conclusion

On-device inference is a sound architecture choice for latency-sensitive, privacy-constrained, or offline-dependent features. The power budget constraint is real but manageable with quantisation, architecture selection, and careful scheduling.

The next concrete step: profile your candidate model on the median device in your target market using Android GPU Inspector or Xcode Instruments' Core ML template. Until you have real power trace data from real hardware, the rest of this is theory.

---

FAQ

Does quantising to INT8 meaningfully hurt model accuracy? For most production models at INT8, accuracy loss is under 1% on standard benchmarks. Smaller models and tasks with narrow output distributions are more sensitive. Always validate on your own held-out dataset, not just the reference benchmark, because distribution shift amplifies quantisation errors.

Is ONNX Runtime or TFLite the better choice for Android? TFLite has a larger ecosystem of pre-converted models and tighter integration with the Android NNAPI delegate. ONNX Runtime is the better choice if you need to share a single model artifact across Android and iOS, or if you're targeting Qualcomm hardware specifically with the QNN execution provider.

How do I measure actual power consumption during inference on device? On Android, use Android GPU Inspector or the Perfetto power rails trace. On iOS, use Xcode Instruments with the Energy Log template. Both give you per-subsystem power draw, not just aggregate battery percentage. Aggregate battery metrics are too coarse to be useful for inference optimisation.

Can I run a large language model on-device in 2025? Yes, with constraints. Models like Phi-3 Mini (3.8B) and Gemma 2B run on flagship devices at acceptable latency for single-turn inference. Sustained multi-turn conversation or long-context tasks will thermal-throttle most phones within minutes. Plan for session length limits and graceful degradation to a server-side fallback.

What is the main reason on-device inference fails in production? Device fragmentation. A model optimised for a Snapdragon 8 Gen 3 with NNAPI delegation may fall back to CPU on 40% of your Android user base. The fallback is slow and power-hungry. Profile your model on your actual device analytics data, not just the top-tier hardware.

Have a project in mind? Contact Sodio Technologies to discuss your requirements and explore the right technology solution for your business.