Batch Inference
The process of running model inference on multiple inputs together to improve throughput, resource utilization, and overall serving efficiency.
Explore more about Optimization
Related terms
An optimization technique that combines multiple incoming requests into a single batch to improve throughput and resource utilization.
Inference OptimizationThe process of improving the speed, latency, throughput, memory usage, or cost-efficiency of AI model inference through techniques such as quantization, pruning, batching, caching, hardware acceleration, and optimized model serving.
Latency OptimizationThe practice of reducing response latency in AI systems to improve responsiveness, throughput, and user experience.
Model OptimizationThe process of improving an AI model's performance, efficiency, or resource usage through techniques such as fine-tuning, pruning, quantization, or distillation.
Response CachingA performance optimization technique that stores previously generated responses to immediately serve identical future requests.