Inference Server
Also called: Model Server, Model Inference Server
A software service that hosts one or more AI models and processes inference requests by receiving inputs, executing model predictions, and returning outputs through APIs while managing resources, batching, scaling, and hardware acceleration.
Explore more about Deployment
Related terms
The process of making an AI model available for inference by deploying it behind an interface that accepts requests and returns predictions or generated outputs.
Inference APIAn API that enables applications to send input data to a deployed AI model and receive generated predictions or outputs, providing programmatic access to inference capabilities without managing the underlying model infrastructure.
GPU DeploymentThe practice of deploying AI models or applications on graphics processing units (GPUs) to accelerate inference or training by leveraging massively parallel computation for high-performance workloads.
AutoscalingThe automatic adjustment of computing resources in response to changing workloads to optimize performance, availability, and cost efficiency.