Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Edge AI

nounAI Data and Infrastructurealso on-device AI, AI at the edge, embedded AI

In one line

Edge AI is the running of artificial intelligence models directly on local devices such as phones, cameras, vehicles and sensors, near where data is produced, instead of sending the data to a remote cloud server.

Edge AI is the practice of running artificial intelligence models directly on local devices, such as smartphones, cameras, cars, factory machines and sensors, instead of sending data to a distant cloud server. The “edge” is the outer boundary of a network, where data is created. By processing data there, edge AI cuts delay, saves bandwidth, works without a connection and keeps sensitive information on the device [1]. It is the counterpart of cloud AI, and many modern systems combine the two.

How it works

A model is first trained, usually in the cloud or a data center, on large amounts of data. It is then prepared for the device through model compression and packaged for an efficient runtime. On the device, an inference engine loads the model and runs it on available hardware, such as a CPU, a mobile GPU, a digital signal processor or a neural processing unit; see AI chip. Results are used immediately, and only summaries or exceptions may be sent upstream. Updates to the model are delivered over the air.

Why run AI at the edge

  • Low latency. No round trip to a server. This matters for a car braking or a robot arm reacting, where a delay of a fraction of a second is unacceptable.
  • Privacy. Voice, video and health data can stay on the device; see AI privacy.
  • Bandwidth and cost. Sending raw video from thousands of cameras is costly; processing locally sends only events.
  • Reliability. Works with poor or no connectivity, such as in mines, farms, ships or aircraft.
  • Energy and scale. Can be more efficient than moving data across networks, and spreads computing across many devices.

Constraints

Edge devices have limited memory, computing power, battery and cooling. A phone has a few gigabytes of memory shared with other apps, and a microcontroller may have only kilobytes. Models must therefore be small and efficient. Methods include:

  • Quantization. Storing weights as 8-bit or lower numbers instead of 32-bit; see quantization.
  • Pruning. Removing weights or neurons that contribute little [3].
  • Knowledge distillation. Training a small model to imitate a large one; see model distillation.
  • Efficient architectures. Networks designed for devices, such as MobileNet.
  • Hardware-aware design. Tuning models to the chip they will run on.

Han, Mao and Dally showed that combining pruning, quantization and coding could shrink networks by a factor of 35 to 49 with little loss of accuracy, a result that helped launch practical on-device deep learning [3].

TinyML

TinyML is machine learning on microcontrollers, the very small, cheap chips in appliances, wearables and sensors, which may run on milliwatts of power for months or years from a small battery [2]. Typical tasks are keyword spotting (“wake word” detection), gesture recognition, anomaly detection in machine vibration and simple image recognition.

Hardware

Edge hardware ranges from microcontrollers, through smartphone and laptop processors with built-in neural engines, to embedded modules such as NVIDIA Jetson and Google Coral, to industrial gateways and in-vehicle computers. Software stacks include TensorFlow Lite, ONNX Runtime, Core ML, PyTorch Mobile and vendor tools.

Applications

  • Smartphones. Face unlock, photo enhancement, voice assistants, live translation and text prediction.
  • Vehicles. Perception and driver assistance for self-driving cars.
  • Industry. Visual inspection, predictive maintenance and safety monitoring.
  • Healthcare. Wearables that detect irregular heart rhythms, and portable imaging aids.
  • Retail and cities. Smart cameras counting people or vehicles without storing video.
  • Agriculture. Field sensors and drones that spot disease and weeds.
  • Robotics and drones. Onboard perception and motion planning.
  • Smart homes. Speakers and appliances responding without the cloud.

On-device language models

Small language models now run on phones and laptops, supporting offline writing aids, summarization and private assistants. They are far smaller than the largest cloud models and are often combined with them: the device handles routine requests and passes difficult ones to the cloud. See large language models.

Challenges

  • Accuracy trade-offs. Smaller models may be less capable.
  • Fragmented hardware and software. Many chips and toolchains.
  • Updating and managing fleets. Pushing safe updates to millions of devices; see model deployment.
  • Security. Devices can be physically accessed, and models can be stolen or tampered with.
  • Monitoring. Harder to observe performance on devices spread over many places.

Architecture patterns

Edge systems are rarely purely local. Common patterns include device-only inference, where everything runs on the device; edge server inference, where a nearby gateway or on-premises server serves many devices; and split computing, where early layers of a network run on the device and later layers in the cloud, sending only compact intermediate data. Cascades run a tiny model first and wake a larger one only when something interesting is detected, which saves energy: a camera might run a small motion detector constantly and call a stronger recognizer only when a person appears.

Federated learning adds a way to improve models without collecting raw data. Each device trains on its own data and sends only model updates to a server, which combines them; see federated learning. Keyboard suggestions and health models have used this approach to learn from many users while keeping personal data local.

Designing for the edge

Teams building edge AI start from the constraints: the latency target, power budget, memory, cost per device and the cost of a wrong answer. They choose or train a model to fit, measure it on the real hardware instead of in simulation, and test under real conditions such as poor lighting, noise and temperature extremes. Field data is collected, with consent, to find failures, and models are updated in staged rollouts that can be reversed. Safety-critical uses, such as vehicles and medical devices, add certification and fallbacks, covered under AI safety.

Edge AI and the Internet of Things

Billions of connected sensors generate more data than networks and data centers can economically absorb. Edge AI turns this raw stream into useful signals close to the source. A vibration sensor on a pump can learn what normal looks like and send an alert only when the pattern changes. A smart meter can detect faults. A wearable can flag unusual heart rhythms while keeping the raw signal on the wrist. This shift from sending data to sending decisions is one of the main reasons edge AI is spreading, and it connects closely to sensor fusion when several sensors are combined on one device.

FAQs

What is edge AI? It is running AI models directly on local devices instead of in a remote cloud.

What is the difference between edge AI and cloud AI? Edge AI computes on or near the device, giving low delay and privacy, while cloud AI uses remote data centers with more power.

What is TinyML? It is machine learning on very small, low-power microcontrollers.

Why is edge AI good for privacy? Data such as voice and video can be processed locally and need not leave the device.

Can large language models run on the edge? Smaller ones can, on modern phones and laptops, often after quantization.

Key facts

  • Edge AI performs machine learning inference on or near the device that collects the data, which reduces latency and the need to send raw data to the cloud1.
  • TinyML refers to running machine learning on microcontrollers that use milliwatts of power or less2.
  • Techniques such as quantization and pruning can shrink neural networks several times over with small loss of accuracy, making them practical on devices3.

Related concepts

Quick checkWhat is edge AI?Show answer

Edge AI is the running of artificial intelligence models directly on local devices such as smartphones, cameras, vehicles and sensors, near where data is produced, instead of sending the data to a remote cloud server for processing.

Sources

  1. Zhi Zhou et al., Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing, Proceedings of the IEEE, https://doi.org/10.1109/JPROC.2019.2918951
  2. Pete Warden and Daniel Situnayake, TinyML, O'Reilly Media, https://www.oreilly.com/library/view/tinyml/9781492052036/
  3. Song Han, Huizi Mao and William Dally, Deep Compression, International Conference on Learning Representations, https://arxiv.org/abs/1510.00149

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue