Edge AI on mobile devices
Edge AI on mobile devices: principles, hardware/software techniques (quantization, pruning, distillation), trade-offs (latency, energy, accuracy), and simple derivations for inference cost and memory.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Edge AI on mobile devices is about running artificial intelligence (AI) models directly on user devices (phones, tablets, wearables) instead of only in the cloud. Engineers use it to reduce latency, save network bandwidth, and improve user privacy by keeping raw data local.
Key ideas
All key terms are defined before use.
- Edge AI — AI computation performed on the "edge" of the network (the user device or nearby gateway) rather than in a remote data center. Assumption: the device has limited CPU, memory, battery, or a specialized accelerator.
- On-device inference — performing model inference (producing outputs from a trained model) on the device. Inference is the use phase of a model; training is usually heavier and often done in the cloud or offline.
- Model compression — any technique that reduces a model's storage size or run-time computation while keeping accuracy acceptable. Main methods: pruning, quantization, knowledge distillation.
- Pruning — removing model parameters (weights) that contribute little to output. Definition: given a trained weight tensor, prune weights whose magnitude is below a threshold or whose importance score is low, then optionally fine-tune.
- Quantization — reducing the number of bits used to represent each parameter or activation (for example from 32-bit float to 8-bit integer). Definition: mapping high-precision values to a lower-precision numeric format, with a quantization function q(x) and dequantization d(q(x)) that approximates x.
- Knowledge distillation — training a small "student" model to mimic a larger "teacher" model. Definition: student minimizes a loss that includes the teacher's soft outputs (probabilities) besides the usual labels.
- Specialized accelerators — hardware units optimized for AI workloads: NPU (Neural Processing Unit), DSP (Digital Signal Processor), GPU (Graphics Processing Unit). Each has different trade-offs in throughput, programmability, and energy.
- Throughput — the number of operations (or inferences) a device can complete per second. Units: FLOPS (floating-point operations per second) or TOPS (tera operations per second) for integer/quantized ops.
- Latency — time taken for a single inference, including preprocessing, memory transfers and compute. Units: seconds (s) or milliseconds (ms).
- Energy — electrical energy consumed per inference. Units: joules (J) or millijoules (mJ). Important because mobile devices have limited battery.
Exact relations and common derivations universities ask for:
Relation between compute, throughput and latency (simple model):
- Let C be compute per inference in operations (e.g., FLOPs), and R be peak device throughput (ops per second). If compute is fully parallelizable and memory is not the bottleneck, then inference compute time T_compute = C / R.
- Total latency T = T_compute + T_mem + T_pre + T_post, where T_mem is memory transfer time, T_pre is preprocessing time, T_post is postprocessing time.
- Assumptions: steady-state execution, no other tasks competing for resources, and model fits in on-chip memory when assuming memory non-dominant.
Relation between energy and operations (simple model):
- Let Eop be average energy per operation (J/op). Then compute energy Ecompute ≈ C · Eop. Total energy per inference E = Ecompute + Emem + Estatic, where Emem is memory access energy and Estatic is baseline system energy for the duration of inference (leakage, display on, etc.).
- Assumptions: Eop constant (approximate), memory accesses counted separately, and no dynamic frequency scaling effects.
Effect of quantization on model size and bandwidth:
- Model storage S_bytes = Nparams · Bbits / 8 (bytes), where Nparams is number of parameters and Bbits is bits per parameter. Quantization reduces Bbits (e.g., 32 -> 8), producing Snew_bytes = Nparams · Bnew / 8 (bytes). Reduction factor = Borig / Bnew. If parameters or activations must be transferred from storage/DRAM or over a link, the bandwidth-limited time is T_mem = Bytes_moved / Bandwidth; reducing Bbits reduces Bytes_moved proportionally.
Exact definitions that are exam-style:
- Definition (Parameter count): The parameter count Nparams of a neural network is the total number of scalar weights and biases stored in the model. This determines storage for parameters: S_bytes = Nparams · (bits per parameter) / 8 (bytes).
- Definition (FLOPs per layer): For a convolution layer with kernel size k×k, Cin input channels, Cout output channels, and output spatial size H×W, the FLOPs per forward pass for that layer = 2 · Cout · Cin · k · k · H · W (2 counts a multiply and an add). Assumption: counting MAC as two FLOPs; some exams count one MAC as one op — be explicit which you use.
Diagrams to draw (describe exactly):
- On-device inference pipeline: Draw boxes left to right: Sensors -> Preprocessing -> Model (inference) -> Postprocessing -> Actuators/UI. Label arrows with data (raw sensor data, preprocessed tensors) and label times T_pre, T_compute, T_post.
- Hardware block diagram: Draw a phone SoC with blocks: CPU, GPU, NPU, SRAM, DRAM, Storage, Sensors. Draw arrows: model parameters moving from storage/DRAM to SRAM/local cache to accelerator. Annotate bandwidth and latency bottlenecks (DRAM access slower/higher energy than SRAM).
Formulas
T_compute = C / R — C = compute per inference (ops), R = peak throughput (ops/s). Units: seconds. Applies when compute dominates and parallelism is ideal.
T_total = T_compute + T_mem + T_pre + T_post — T_mem = memory transfer time, T_pre = preprocessing time, T_post = postprocessing time. Units: seconds.
T_mem = Bytes_moved / Bandwidth — Bytes_moved in bytes, Bandwidth in bytes/s. Roofline bound: T_bound = max(C / R, Bytes_moved / Bandwidth); achieved throughput is min(R, Bandwidth * AI), where arithmetic intensity AI = C / Bytes_moved (ops/byte).
E_compute = C * E_op — E_op = energy per operation (J/op), C = ops. Units: joules. Applies for approximate compute energy.
Model_size = N_params * B_bits / 8 — N_params = number of parameters, B_bits = bits per parameter, divide by 8 for bytes. Units: bytes.
FLOPs_conv = 2 * Cout * Cin * k * k * H * W — For convolutional layers when each MAC counted as two FLOPs. Units: FLOPs.
Size_reduction_factor = B_orig / B_new — Bits per parameter before and after quantization. Dimensionless.
Note: In exams you must state assumptions (e.g., what counts as an operation, whether MAC counted as 1 or 2 FLOPs, whether activations are included).
Worked examples
Example 1 — Model size change by quantization Problem: A trained network has 24,000,000 parameters and uses 32-bit floats. Quantize to 8-bit integers. Compute original and quantized model sizes. Steps:
- Original size S_orig = N_params * 32 bits / 8 = 24,000,000 * 4 bytes = 96,000,000 bytes = 96 MB.
- Quantized size S_q = 24,000,000 * 8 bits / 8 = 24,000,000 * 1 byte = 24,000,000 bytes = 24 MB.
- Reduction factor = 96 MB / 24 MB = 4. Answer: Original 96 MB, quantized 24 MB, 4× smaller. (assuming 1 MB = 10^6 bytes).
Example 2 — Latency estimate from FLOPs and device throughput Problem: A model requires 200 million operations (200 MFLOPs) per inference. A mobile accelerator has peak throughput R = 2 GFLOPS (2 billion ops/s). Estimate ideal compute latency (ignore memory and preprocessing). Steps:
- Convert units: C = 200 MFLOPs = 200 × 10^6 ops. R = 2 GFLOPS = 2 × 10^9 ops/s.
- Compute T_compute = C / R = (200 × 10^6) / (2 × 10^9) = 0.1 s = 100 ms. Answer: Ideal compute latency 100 ms (ignoring memory and other overheads). Note: Real latency is higher due to memory, scheduling and non-ideal parallelism.
Example 3 — Pruning effect on parameter count Problem: A convolutional network has 10 million parameters. A pruning schedule removes 60% of weights and then fine-tunes. What is the new parameter count and storage (assume 8-bit storage after pruning)? Steps:
- New parameter count N_new = 10,000,000 * (1 - 0.60) = 4,000,000 parameters.
- Storage S_new = 4,000,000 * 8 bits / 8 = 4,000,000 bytes = 4 MB. Answer: 4 million parameters, 4 MB when stored as 8-bit integers.
Common mistakes
- Mixing units: GFLOPS vs TOPS vs GIOPS (integer ops). Convert carefully.
- Claiming quantization always preserves accuracy: it usually reduces model size but may reduce accuracy unless carefully calibrated or fine-tuned.
- Forgetting overheads for compression (e.g., sparse formats may need indices which cost memory or extra memory indirection that slows execution).
For your exam
Types of questions you will see:
- Definitions: define Edge AI, on-device inference, pruning, quantization, distillation, NPU. Full-mark answers give concise definition, purpose, and one example implication.
- Short notes: compare pruning vs quantization; explain hardware accelerators; discuss privacy advantages. Full-mark answers give clear definition, algorithm or operation, pros/cons, and a small diagram or formula.
- Derivations: derive latency formula from FLOPs and throughput; compute memory-bandwidth-limited latency. Full marks require stating assumptions and step-by-step algebra.
- Numericals: compute model size change after quantization/pruning; estimate latency from given FLOPs and throughput; calculate energy estimate with given E_op. Full marks use correct unit conversions and show steps.
- Programs/algorithms: pseudocode for iterative magnitude pruning or post-training quantization. Full marks include algorithm steps and mention fine-tuning.
- Diagrams: draw on-device inference pipeline or memory/compute block diagram. Full marks include labeled boxes, arrows and time/energy labels.
- Frameworks/APIs: identify mobile inference stacks (Android NNAPI, TensorFlow Lite, Core ML (Apple Neural Engine), Metal, ONNX Runtime Mobile) and when to use CPU/GPU/NPU delegates. Full marks mention portability, operator coverage, and accelerator access.
Quick check
- What is Edge AI?
- Give the formula relating compute C, throughput R and compute time.
- If you quantize 32-bit model to 8-bit, by what factor does parameter storage reduce (assuming same N)?
Answers: Edge AI = AI computation on the device (edge) not cloud;
T_compute = C / R; Reduction factor = 4×.
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?