Featured

Building a Fully Local, System-Wide AI Agent CLI on Linux

on

Cloud-based agentic CLI tools are undeniably convenient, but sending internal terminal workflows, system commands, and proprietary files to third-party endpoints is a persistent privacy risk. Running a fully local, autonomous agent directly on your workstation eliminates subscription costs, avoids rate limits, and grants you unconstrained access across all your attached drives.


This guide details how to pair Ollama with Open Interpreter on modern Linux (Ubuntu), resolve tricky runtime quirks like Python 3.14 native builds and removed modules, map persistent storage, and create a one-click desktop launcher.

System Architecture

The setup consists of two layers:

  • Inference Engine (Ollama): Serves local open-weight models with hardware GPU acceleration.

  • Agent Harness (Open Interpreter): Interprets natural language queries, writes code, and executes Bash or Python commands directly on your system.

For agentic coding and file management, qwen2.5-coder:14b provides an optimal balance of context awareness, tool precision, and raw token throughput on modern GPUs.

Step 1: Install Ollama & Pull the Model

Install Ollama to host the local model:

Bash
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5-coder:14b
Verify that the local Ollama instance is active:

Bash
curl http://localhost:11434/api/tags

Step 2: Set Up an Isolated Environment & Resolve Build Hurdles

On modern Ubuntu distributions running Python 3.14+, pre-built native wheels for libraries like tiktoken are unavailable, and modern setuptools packages have removed legacy components. Follow this exact setup to bypass build failures.

1. Install System Build Tools & Rust Compiler

Because dependencies must compile from source on newer Python runtimes, install the build headers and Cargo:

Bash
sudo apt update && sudo apt install -y python3-venv python3-pip python3-dev cargo rustc build-essential

2. Install Open Interpreter with ABI Forward Compatibility

Create an isolated virtual environment and pass the PYO3 forward-compatibility flag so the Rust compiler builds tiktoken against modern Python:

Bash
python3 -m venv ~/.local-interpreter
PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 ~/.local-interpreter/bin/pip install open-interpreter

3. Patch the pkg_resources Deprecation

In setuptools>=70, the legacy pkg_resources module was removed. To prevent Open Interpreter crashing on startup with ModuleNotFoundError: No module named 'pkg_resources', pin setuptools:

Bash
~/.local-interpreter/bin/pip install "setuptools<70.0.0"
Expose the binary to your path:

Bash
sudo ln -sf ~/.local-interpreter/bin/interpreter /usr/local/bin/agent

Step 3: Configure Persistent Multi-Drive Storage

To give your agent seamless navigation across your NVMe, secondary SSDs, and bulk HDDs without manual path explanations:

1. Find Drive UUIDs

Bash
lsblk -f

2. Establish Fixed Mounts in /etc/fstab

Create system mount points:

Bash
sudo mkdir -p /mnt/ssd1 /mnt/ssd2 /mnt/hdd
Add your partitions to /etc/fstab using their UUIDs (use uid=1000,gid=1000,umask=022 if mounting NTFS or exFAT; native ext4 uses standard defaults):

Plaintext
UUID=YOUR-SSD1-UUID   /mnt/ssd1   ext4   defaults,nofail   0   2
UUID=YOUR-SSD2-UUID   /mnt/ssd2   ext4   defaults,nofail   0   2
UUID=YOUR-HDD-UUID    /mnt/hdd    ext4   defaults,nofail   0   2
Mount the filesystems and take ownership:

Bash
sudo mount -a
sudo chown -R $USER:$USER /mnt/ssd1 /mnt/ssd2 /mnt/hdd

3. Create Home Directory Symlinks

Local models reason best with familiar paths. Expose them cleanly in your home folder:

Bash
ln -s /mnt/ssd1 ~/ssd1
ln -s /mnt/ssd2 ~/ssd2
ln -s /mnt/hdd ~/hdd

Step 4: System Instructions & Connection Tuning

By default, local models communicating with Open Interpreter can occasionally output raw JSON tool calls instead of executing code, or crash looking for OpenAI API credentials.

We solve this by routing through Ollama's native /v1 endpoint, supplying a placeholder API key, and setting strict Markdown-block instructions.

1. Write the System Prompt

Create ~/.config/open-interpreter/system_info.txt:

Plaintext
You are an autonomous CLI agent operating on an Ubuntu workstation as user jason.

Storage Layout & Mappings:
- Home Directory: /home/jason (or ~)
- Primary OS Drive: / (NVMe)
- Secondary SSD 1: /mnt/ssd1 (symlink: ~/ssd1)
- Secondary SSD 2: /mnt/ssd2 (symlink: ~/ssd2)
- Bulk Storage HDD: /mnt/hdd (symlink: ~/hdd)

EXECUTION RULES (CRITICAL):
1. NEVER output raw JSON tool calls like {"name": "execute", ...} or {"name": "skills.search", ...}.
2. To run commands or inspect files, ALWAYS write Markdown code blocks using ```bash or ```python.
3. Keep conversational explanations brief. Put the code in the block and let the system run it.

2. Create Shell Aliases

Add aliases to ~/.bashrc specifying the local /v1 endpoint, a dummy API key to appease the OpenAI SDK, and a full 32k context window:

Bash
cat << 'EOF' >> ~/.bashrc

# Local Agent CLI Aliases
alias agy-local='agent --api_base http://localhost:11434/v1 --api_key ollama --model openai/qwen2.5-coder:14b --context_window 32768 --custom_instructions "$(cat ~/.config/open-interpreter/system_info.txt)"'
alias agy-yolo='agent --api_base http://localhost:11434/v1 --api_key ollama --model openai/qwen2.5-coder:14b --context_window 32768 -y --custom_instructions "$(cat ~/.config/open-interpreter/system_info.txt)"'
EOF

source ~/.bashrc

Step 5: Create a One-Click Desktop Shortcut

To launch your agent outside of a manual terminal prompt, create an executable wrapper and a desktop entry.

1. Create the Launcher Script

Desktop entries cannot evaluate complex bash aliases, so save the execution command in an executable script:

Bash
mkdir -p ~/.local/bin
cat << 'EOF' > ~/.local/bin/agy-local-launcher
#!/bin/bash
agent --api_base http://localhost:11434/v1 --api_key ollama --model openai/qwen2.5-coder:14b --context_window 32768 --custom_instructions "$(cat ~/.config/open-interpreter/system_info.txt)"
EOF

chmod +x ~/.local/bin/agy-local-launcher

2. Create the Desktop File

GNOME .desktop files do not expand $HOME in the Exec= key, so specify the absolute path:

Bash
cat << 'EOF' > ~/Desktop/agy-local.desktop
[Desktop Entry]
Version=1.0
Type=Application
Name=AGY Local
Comment=Local AI CLI Agent (Qwen2.5-Coder:14B)
Exec=/bin/bash /home/jason/.local/bin/agy-local-launcher
Icon=utilities-terminal
Terminal=true
Categories=Development;System;
EOF

chmod +x ~/Desktop/agy-local.desktop

3. Enable Launching

On your desktop, right-click the newly generated icon and select "Allow Launching". The icon will convert into an active application launcher.

Usage & Verification

Click your desktop shortcut or run agy-local in your terminal to begin:

Plaintext
> summarize the documents in ~/robot_share
The model generates an executable Python or Bash block, presents it for review, executes it locally on your hardware upon pressing y, and outputs the results directly to your console.

Comments

Post a Comment