Run 50+ LLM-related projects locally

Do you run LLMs locally?

Harbor is a containerized LLM toolkit that allows you to run LLMs and additional services using one simple CLI.

Features:

• 50+ LLM-related services
• CLI to run and configure all the services
• A helper desktop app (10Mb, no Electron) to run the CLI via GUI
• Convenience and simplicity are key focus - most of the things are done with a single or very few commands

Examples of what you can do with Harbor:

• Call your LLM with voice
• Access your local LLM setup via a tunnel over the internet (or from phone over WLAN)
• Add Web RAG to your setup
• Build and host LLM-based automation workflows
• Add an optimising proxy between your LLM UI and LLM provider
• Save a complex configuration for your inference engine to reuse it later

Interesting? Let's dive in.

Harbor is built around Docker Compose, but helps overcome the typical scaling pains that make it harder to use for highly dynamic or larger setups with dozens of services.

You'll find it very similar to the Docker Compose and Docker CLIs, but with a much simpler and direct syntax centered around service handles with lots of extra features related to managing supported services.

# Start services
harbor up 

# Manage service configuration
harbor config --help

# Manage service environment
harbor env --help

# Get service URLs for local/LAN/internet
harbor url

# Open service in the browser
harbor open

# Create and manage configuration profiles
# for specific scenarios
harbor profiles --help

# See the history of commands you run
# and repeat them (data is local)
harbor history

# Manage aliases for frequent commands
harbor aliases --help

# Create tunnels to access your
# setup via internet
harbor tunnel

One of the core ideas in Harbor is that you should be able to start with the supported projects in a single (or very few commands). Another one is that services are pre-configured to work together out of the box.

# Running SearXNG automatically enables Web RAG in Open WebUI
harbor up searxng

# Speaches includes OpenAI-compatible SST and TTS
# and connected to Open WebUI out of the box
harbor up speaches

# Run additional/alternative LLM Inference backends
# Open Webui is automatically connected to them.
harbor up llamacpp tgi litellm vllm tabbyapi aphrodite sglang ktransformers

# Run different Frontends
harbor up librechat chatui bionicgpt hollama

# Get a free quality boost with
# built-in optimizing proxy
harbor up boost

# Use FLUX in Open WebUI in one command
harbor up comfyui

# Use custom models for supported backends
harbor llamacpp model https://huggingface.co/user/repo/model.gguf

If you need even more flexibility, Harbor comes with an eject button - that'll give you a Docker Compose setup identical to your current Harbor state.

harbor eject searxng vllm > my-ai-stack.compose.yml

In addition to that, you'll find plenty of QoL features in the CLI itself:

• Automatic capability detection (Nvidia, CDI, ROCm), though not all services support all capabilities
• Argument scrambling: Harbor will handle both harbor logs vllm and harbor vllm logs in the same way
• Quickly launch container shells, inspect images, and many more troubleshooting extras
• Built-in LLM-based help with harbor how
• Get QR codes for your phone to access services in the same network

Even if you prefer to configure and setup your local LLM installation manually - Harbor is still a great guide on self-hosting friendly services and their configuration/setup with the compose stack.

---

Links:

• Installing Harbor Guides to install Harbor CLI and App
• Harbor User Guide High-level overview of working with Harbor
• Harbor App Overview and manual for the Harbor companion application
• Harbor Services Catalog of services available in Harbor
• Harbor CLI Reference Read more about Harbor CLI commands and options. Read about supported services and the ways to configure them.