mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-02 23:56:24 +08:00
169 lines
3.2 KiB
Plaintext
169 lines
3.2 KiB
Plaintext
---
|
|
title: Ollama
|
|
sidebarTitle: Ollama
|
|
description: Local LLM inference with Ollama
|
|
---
|
|
|
|
Run language models locally using Ollama — free, private, and no API keys required.
|
|
|
|
## Overview
|
|
|
|
Ollama lets you run open-source models on your own hardware:
|
|
|
|
- **Completely private** — No data leaves your machine
|
|
- **No API costs** — Just electricity and hardware
|
|
- **Offline capable** — Works without internet
|
|
- **Multiple models** — Llama, Mistral, Qwen, and more
|
|
|
|
## Prerequisites
|
|
|
|
- Ollama installed: https://ollama.com
|
|
- Sufficient RAM (8GB+ recommended)
|
|
- GPU optional but recommended
|
|
|
|
## Installation
|
|
|
|
### macOS
|
|
|
|
```bash
|
|
brew install ollama
|
|
```
|
|
|
|
### Linux
|
|
|
|
```bash
|
|
curl -fsSL https://ollama.com/install.sh | sh
|
|
```
|
|
|
|
### Windows
|
|
|
|
Download from https://ollama.com
|
|
|
|
## Configuration
|
|
|
|
```bash
|
|
# ~/.ironclaw/.env
|
|
|
|
LLM_BACKEND=ollama
|
|
OLLAMA_BASE_URL=http://localhost:11434
|
|
```
|
|
|
|
## Pull a Model
|
|
|
|
Before using a model, pull it:
|
|
|
|
```bash
|
|
# Llama 3.2 (3B parameters, fast)
|
|
ollama pull llama3.2
|
|
|
|
# Llama 3.1 (8B parameters, balanced)
|
|
ollama pull llama3.1
|
|
|
|
# Qwen 2.5 (7B parameters, multilingual)
|
|
ollama pull qwen2.5
|
|
|
|
# Mistral (7B parameters, efficient)
|
|
ollama pull mistral
|
|
```
|
|
|
|
## Setup
|
|
|
|
### Via Wizard
|
|
|
|
```bash
|
|
ironclaw onboard
|
|
```
|
|
|
|
1. Step 3: Select "Ollama"
|
|
2. Step 4: Enter model name (e.g., `llama3.1`)
|
|
|
|
### Manual Configuration
|
|
|
|
```bash
|
|
export LLM_BACKEND=ollama
|
|
export OLLAMA_BASE_URL=http://localhost:11434
|
|
|
|
# Set default model in IronClaw
|
|
ironclaw config set llm.model llama3.1
|
|
```
|
|
|
|
## Popular Models
|
|
|
|
| Model | Size | VRAM | Best For |
|
|
|-------|------|------|----------|
|
|
| `llama3.2` | 3B | 4GB | Fast, simple tasks |
|
|
| `llama3.1` | 8B | 6GB | General purpose |
|
|
| `qwen2.5` | 7B | 6GB | Multilingual |
|
|
| `mistral` | 7B | 6GB | Efficient |
|
|
| `codellama` | 7B | 6GB | Code generation |
|
|
|
|
## Hardware Requirements
|
|
|
|
| Model Size | RAM | GPU VRAM |
|
|
|------------|-----|----------|
|
|
| 3B | 4GB | 4GB |
|
|
| 7B | 8GB | 6GB |
|
|
| 13B | 16GB | 12GB |
|
|
| 70B | 64GB | 48GB |
|
|
|
|
Without GPU, models run slower on CPU.
|
|
|
|
## Custom Base URL
|
|
|
|
For remote Ollama server:
|
|
|
|
```bash
|
|
export OLLAMA_BASE_URL=http://your-server:11434
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="Ollama not running" icon="x-circle">
|
|
```bash
|
|
# Start Ollama
|
|
ollama serve
|
|
|
|
# Or as a service
|
|
brew services start ollama # macOS
|
|
sudo systemctl start ollama # Linux
|
|
```
|
|
</Accordion>
|
|
|
|
<Accordion title="Out of memory" icon="alert">
|
|
- Use a smaller model (3B instead of 7B)
|
|
- Close other applications
|
|
- Add swap space
|
|
- Use a machine with more RAM
|
|
</Accordion>
|
|
|
|
<Accordion title="Slow responses" icon="clock">
|
|
- Use GPU if available
|
|
- Try a smaller model
|
|
- Quantized models run faster
|
|
- Check CPU usage
|
|
</Accordion>
|
|
|
|
<Accordion title="Model not found" icon="search">
|
|
```bash
|
|
# Pull the model first
|
|
ollama pull llama3.1
|
|
|
|
# List available models
|
|
ollama list
|
|
```
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
## Next Steps
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="NEAR AI" icon="star" href="/providers/nearai">
|
|
Cloud option with no hardware requirements
|
|
</Card>
|
|
|
|
<Card title="Tinfoil" icon="shield" href="/providers/tinfoil">
|
|
Private cloud inference with TEE
|
|
</Card>
|
|
</CardGroup>
|