Latency
term_id: latency
Category: engineering_practice
Definition
Latency measures the responsiveness of an AI service, typically expressed in milliseconds. It includes inference time, network transmission delays, and processing overhead. Low latency is critical for real-time applications like voice assistants or autonomous driving, where immediate feedback is required. Engineers optimize latency through techniques such as model quantization, pruning, caching, and hardware acceleration, balancing speed against potential trade-offs in accuracy or throughput.
Summary
The time delay between the initiation of a request and the start of the response in an AI system.
Key Concepts
- Inference Time
- Response Time
- Real-time Processing
- Optimization
Use Cases
- Voice recognition systems
- Autonomous vehicle control
- Live translation services