Caching

term_id: caching

Category: engineering_practice

Definition

In AI engineering, caching optimizes performance by keeping recent or frequent query results, model predictions, or intermediate computations in fast memory (like RAM). This reduces the need for expensive recomputation or repeated database queries. Effective cache management strategies, such as Least Recently Used (LRU) eviction policies, ensure that memory usage remains efficient while maximizing throughput for inference engines and data pipelines.

Summary

Caching is a technique of storing frequently accessed data in a temporary, high-speed storage layer to reduce latency and decrease load on primary data sources.

Key Concepts

  • latency reduction
  • memory optimization
  • eviction policies
  • hit rate

Use Cases

  • Storing model inference results
  • Caching database query outputs
  • Pre-computing feature embeddings

Code Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

import redis

# Simple caching example
r = redis.Redis(host='localhost', port=6379, db=0)

def get_prediction(model_id, input_data):
    cache_key = f"pred_{model_id}_{hash(str(input_data))}"
    result = r.get(cache_key)
    if result:
        return result.decode('utf-8')
    # Compute if not cached
    prediction = model.predict(input_data)
    r.setex(cache_key, 3600, str(prediction))
    return prediction

Related