llama.cpp

A tool for running and managing large language models locally, providing a simple interface for downloading, serving, and interacting with models.

An open-source library for high-throughput and low-latency LLM serving and inference optimization.

A software service that hosts one or more AI models and processes inference requests by receiving inputs, executing model predictions, and returning outputs through APIs while managing resources, batching, scaling, and hardware acceleration.

AI capabilities that run directly on local devices, reducing reliance on remote servers and enabling lower-latency, privacy-focused inference.