Illia Polosukhin 5435b38eca feat(workspace): metadata-driven indexing/hygiene, document versioning, and patch (#1723)
* feat(workspace): metadata-driven indexing/hygiene, document versioning, and patch support

Foundation for the extensible frontend system. Workspace documents now
support metadata flags (skip_indexing, skip_versioning, hygiene config)
via folder-level .config documents and per-file overrides, replacing
hardcoded hygiene targets and indexing behavior.

Key changes:
- DocumentMetadata type with resolution chain (doc → folder .config → defaults)
- Document versioning: auto-saves previous content on write/append/patch
- Workspace patch: search-and-replace editing via memory_write tool
- Hygiene rewrite: discovers cleanup targets from .config metadata
  instead of hardcoded daily/ and conversations/ directories
- memory_read gains version/list_versions params
- memory_write gains metadata/old_string/new_string/replace_all params
- V14 migration adds memory_document_versions table (both PG + libSQL)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review feedback — transaction safety, patch mode, formatting

- Wrap libSQL save_version in a transaction to prevent race condition
  where concurrent writers could allocate the same version number
- Make content optional in memory_write when in patch mode (old_string
  present) — LLM no longer forced to provide unused content param
- Improve metadata update error handling with explicit match arms
- Run cargo fmt across all files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review findings — write-path performance, version pruning, descriptions

1. Resolve metadata once per write: write(), append(), and patch() now
   call resolve_metadata() once and pass the result to both
   maybe_save_version() and reindex_document_with_metadata(), cutting
   redundant DB queries from 3-5 per write down to 1 resolution.

2. Optimize version hash check: replaced get_latest_version_number() +
   get_version() (2 queries) with list_versions(id, 1) (1 query) for
   the duplicate-hash check in maybe_save_version().

3. Wire up version_keep_count: hygiene passes now prune old versions
   for documents in cleaned directories, enforcing the configured
   version_keep_count (default: 50). Removes the TODO comment.

4. Fix misleading tool description: patch mode works with any target
   including 'memory', not just custom paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: wire remaining unwired components — changed_by, layer versioning

1. changed_by now populated: all write paths pass self.user_id as the
   changed_by field in version records instead of None, so version
   history shows who made each change.

2. Layer write/append versioned: write_to_layer() and append_to_layer()
   now auto-version and use metadata-optimized reindexing, matching
   the standard write()/append() paths.

3. append_memory versioned: MEMORY.md appends now auto-version with
   metadata-driven skip and shared metadata resolution.

4. Remove unused reindex_document wrapper: all callers now use
   reindex_document_with_metadata directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: comprehensive coverage for versioning, metadata, patch, and hygiene

26 new tests covering critical and high-priority gaps:

document.rs (7 unit tests):
- is_config_path edge cases (foo.config, empty string, .config/bar)
- content_sha256 with empty string (known SHA-256 constant)
- content_sha256 with unicode (multi-byte UTF-8)
- DocumentMetadata merge: null overlay, nested hygiene replaced wholesale,
  both empty, non-object base

memory.rs (2 schema tests):
- memory_write schema includes patch/metadata params, content not required
- memory_read schema includes version/list_versions params

hygiene.rs (5 integration tests):
- No .config docs → no cleanup happens
- .config with hygiene disabled → directory skipped
- Multiple dirs with different retention (fast=0, slow=9999)
- Documents newer than retention not deleted
- Version pruning during hygiene (keep_count=2, verify pruned)

workspace/mod.rs (14 integration tests):
- write creates version with correct hash and changed_by
- Identical writes deduplicated (hash check)
- Append versions pre-append content
- Patch: single replacement, replace_all, not-found error, creates version
- Patch with unicode characters
- Patch with empty replacement string
- resolve_metadata: no config (defaults), inherits from folder .config,
  document overrides .config, nearest ancestor wins
- skip_versioning via .config prevents version creation

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address zmanian review — PG transaction safety, identity protection, perf

Must-fix:
1. PostgreSQL save_version now uses a transaction with SELECT FOR UPDATE
   to prevent concurrent writers from allocating the same version number,
   matching the libSQL implementation.

2. Restore identity document protection in hygiene cleanup_directory().
   MEMORY.md, SOUL.md, IDENTITY.md, etc. are now protected from deletion
   regardless of which directory they appear in, via is_identity_document()
   case-insensitive check. This restores the safety net that was removed
   when migrating from hardcoded to metadata-driven hygiene.

Should-fix:
3. resolve_metadata() now uses find_config_documents (single query) +
   in-memory nearest-ancestor lookup, instead of O(depth) serial DB
   queries walking up the directory tree.

4. memory_write validates that at least one mode is provided (content
   for write/append, or old_string+new_string for patch) with a clear
   error message upfront, instead of relying on downstream empty checks.

5. Fixed misleading GIN index comment in V15 migration.

9. Added "Fail-open: versioning failures must not block writes" comments
   to all `let _ = self.maybe_save_version(...)` call sites.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: cargo fmt

* fix: address Copilot review — DoS prevention, no-op skip, duplicate hygiene

Security:
- Reject empty old_string in both workspace.patch() and memory_write
  tool to prevent pathological .matches("") behavior (DoS vector)

Correctness:
- Remove duplicate hygiene spawn in multi-user heartbeat — was running
  both via untracked tokio::spawn AND inside the JoinSet, causing
  double work and immediate skip via global AtomicBool guard
- Disallow layer param in patch mode — patch always targets the
  default workspace scope; combining with layer could silently patch
  the wrong document
- Restore trim-based whitespace rejection for non-patch content
  validation (was broken when refactoring required fields)

Performance:
- Short-circuit write() when content is identical to current content,
  skipping versioning, update, and reindex entirely
- Normalize path once at start of resolve_metadata instead of only
  for config lookup (prevents missed document metadata on unnormalized
  paths)

Cleanup:
- Remove duplicate tests/workspace_versioning_integration.rs (same
  tests already exist in workspace/mod.rs versioning_tests module)

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: eliminate flaky hygiene tests caused by global AtomicBool contention

All hygiene tests that used run_if_due() were flaky when running
concurrently because they competed for the global RUNNING AtomicBool
guard. Rewrote them to test the underlying components directly:

- metadata_driven_cleanup_discovers_directories: now uses
  find_config_documents() + cleanup_directory() directly
- multiple_directories_with_different_retention: now uses
  cleanup_directory() per directory directly
- cleanup_respects_cadence: rewritten as a sync unit test that
  validates state file + timestamp logic without touching the
  global guard

Verified stable across 3 consecutive runs (3793 tests, 0 failures).

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address remaining review comments — metadata ordering, PG locking, hygiene safety

1. Metadata applied BEFORE write/patch (#10-11,15): metadata param is
   now set via get_or_create + update_metadata before the write/patch
   call, so skip_indexing/skip_versioning take effect for the same
   operation instead of only subsequent ones.

2. Layer write doc ID (#13-14): metadata no longer re-reads after write
   since it's applied upfront. Removes the stale-scope risk.

3. Version param overflow (#16): validates version is 1..i32::MAX
   before casting, returns InvalidParameters on out-of-range.

4. Hygiene protection list (#18): added HYGIENE_PROTECTED_PATHS that
   includes MEMORY.md, HEARTBEAT.md, README.md (missing from
   IDENTITY_PATHS). cleanup_directory now uses is_protected_document()
   which checks both lists with case-insensitive matching.

5. PG FOR UPDATE on empty table (#22-24): now locks the parent
   memory_documents row (SELECT 1 FROM memory_documents WHERE id=$1
   FOR UPDATE) before computing MAX(version), which works even when
   no version rows exist yet.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address remaining review comments — metadata merge, retention guard, migration ordering

1. **Metadata merge in memory_write tool**: incoming metadata is now
   merged with existing document metadata via `DocumentMetadata::merge()`
   instead of full replacement, so setting `{hygiene: {enabled: true}}`
   no longer silently drops a previously-set `skip_versioning: true`.

2. **Minimum retention_days**: `HygieneMetadata.retention_days` is now
   clamped to a minimum of 1 day during deserialization, preventing an
   LLM from writing `retention_days: 0` and causing mass-deletion on
   the next hygiene pass.

3. **Migration version ordering**: renumbered document_versions migration
   to come after staging's already-deployed migrations (PG: V15→V16,
   libSQL: 15→17). Documented the convention that new migrations must
   always be numbered after the highest version on staging/main.

4. **Duplicate doc comment**: removed duplicated line on
   `reindex_document_with_metadata`.

5. **HygieneSettings**: added `version_keep_count` field to persist
   the setting through the DB-first config resolution chain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: skip metadata pre-apply when layer is specified, clean up stale comment

1. When a layer is specified, skip the metadata pre-apply via
   get_or_create — it operates on the primary scope and would create a
   ghost document there while the actual content write targets the
   layer's scope.

2. Removed stale "See review comments #10-11,15" reference; the
   surrounding comment already explains the rationale.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use BEGIN IMMEDIATE for libSQL save_version to serialize writers

The default DEFERRED transaction only acquires a write lock at the first
write statement (INSERT), not at the SELECT. Two concurrent writers could
both read the same MAX(version) before either inserts, causing a UNIQUE
violation. BEGIN IMMEDIATE acquires the write lock upfront, matching the
existing pattern in conversations.rs.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: document trust boundary on metadata/versioning WorkspaceStore methods

These methods accept bare document UUIDs without user_id checks at the
DB layer. The Workspace struct (the only caller) always obtains UUIDs
through user-scoped queries first. Document this trust boundary
explicitly on the trait so future implementors/callers know not to pass
unverified UUIDs from external input.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address new Copilot review comments — ghost doc, param validation, overflow

1. Skip metadata pre-apply in patch mode to avoid creating a ghost
   empty document via get_or_create when the document doesn't exist,
   which would change a "not found" error into "old_string not found".

2. Validate list_versions and version as mutually exclusive in
   memory_read to avoid ambiguous behavior (list_versions silently won).

3. Clamp version_keep_count to i32::MAX before casting to prevent
   overflow on extreme config values.

4. Mark daily_retention_days and conversation_retention_days as
   deprecated in HygieneSettings — retention is now per-folder via
   .config metadata.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: apply metadata in patch mode via read(), reorder libSQL migrations

1. Metadata is no longer silently ignored in patch mode — uses
   workspace.read() (which won't create ghost docs) instead of skipping
   entirely, so skip_versioning/skip_indexing flags take effect for
   patches on existing documents.

2. Reorder INCREMENTAL_MIGRATIONS to strictly ascending version order
   (16 before 17) to match iteration order in run_incremental().

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove duplicate is_patch_mode, add TODO comments for known limitations

- Remove duplicate `is_patch_mode` binding in memory_write (was
  computed at line 280 and again at line 368).
- Document multi-scope hygiene edge case: workspace.list() includes
  secondary scopes but workspace.delete() is primary-only, causing
  silent no-ops for cross-scope entries.
- Document O(n) reads in version pruning as acceptable for typical
  directory sizes.
- Add TODO on WorkspaceError::SearchFailed catch-all for future
  cleanup into more specific variants.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: reindex on no-op writes for metadata changes, use read_primary in patch

1. write() no longer fully short-circuits when content is unchanged —
   it still resolves metadata and reindexes so that metadata-driven
   flags (e.g. skip_indexing toggled via memory_write's metadata param)
   take effect immediately even without a content change.

2. Patch-mode metadata pre-apply now uses workspace.read_primary()
   instead of workspace.read() to ensure we target the same scope that
   patch() operates on, preventing cross-scope metadata mutation in
   multi-scope mode.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 23:58:27 -07:00
2026-02-22 19:08:43 +00:00
2026-02-22 19:08:43 +00:00
2026-02-21 15:14:57 -07:00
2026-03-18 11:34:19 -07:00
2026-03-18 11:34:19 -07:00

IronClaw

IronClaw

Your secure personal AI assistant, always on your side

License: MIT OR Apache-2.0 Telegram: @ironclawAI Reddit: r/ironclawAI gitcgr

English | 简体中文 | Русский | 日本語

PhilosophyFeaturesInstallationConfigurationSecurityArchitecture


Philosophy

IronClaw is built on a simple principle: your AI assistant should work for you, not against you.

In a world where AI systems are increasingly opaque about data handling and aligned with corporate interests, IronClaw takes a different approach:

  • Your data stays yours - All information is stored locally, encrypted, and never leaves your control
  • Transparency by design - Open source, auditable, no hidden telemetry or data harvesting
  • Self-expanding capabilities - Build new tools on the fly without waiting for vendor updates
  • Defense in depth - Multiple security layers protect against prompt injection and data exfiltration

IronClaw is the AI assistant you can actually trust with your personal and professional life.

Features

Security First

  • WASM Sandbox - Untrusted tools run in isolated WebAssembly containers with capability-based permissions
  • Credential Protection - Secrets are never exposed to tools; injected at the host boundary with leak detection
  • Prompt Injection Defense - Pattern detection, content sanitization, and policy enforcement
  • Endpoint Allowlisting - HTTP requests only to explicitly approved hosts and paths

Always Available

  • Multi-channel - REPL, HTTP webhooks, WASM channels (Telegram, Slack), and web gateway
  • Docker Sandbox - Isolated container execution with per-job tokens and orchestrator/worker pattern
  • Web Gateway - Browser UI with real-time SSE/WebSocket streaming
  • Routines - Cron schedules, event triggers, webhook handlers for background automation
  • Heartbeat System - Proactive background execution for monitoring and maintenance tasks
  • Parallel Jobs - Handle multiple requests concurrently with isolated contexts
  • Self-repair - Automatic detection and recovery of stuck operations

Self-Expanding

  • Dynamic Tool Building - Describe what you need, and IronClaw builds it as a WASM tool
  • MCP Protocol - Connect to Model Context Protocol servers for additional capabilities
  • Plugin Architecture - Drop in new WASM tools and channels without restarting

Persistent Memory

  • Hybrid Search - Full-text + vector search using Reciprocal Rank Fusion
  • Workspace Filesystem - Flexible path-based storage for notes, logs, and context
  • Identity Files - Maintain consistent personality and preferences across sessions

Installation

Prerequisites

  • Rust 1.85+
  • PostgreSQL 15+ with pgvector extension
  • NEAR AI account (authentication handled via setup wizard)

Download or Build

Visit Releases page to see the latest updates.

Install via Windows Installer (Windows)

Download the Windows Installer and run it.

Install via powershell script (Windows)
irm https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.ps1 | iex
Install via shell script (macOS, Linux, Windows/WSL)
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.sh | sh
Install via Homebrew (macOS/Linux)
brew install ironclaw
Compile the source code (Cargo on Windows, Linux, macOS)

Install it with cargo, just make sure you have Rust installed on your computer.

# Clone the repository
git clone https://github.com/nearai/ironclaw.git
cd ironclaw

# Build
cargo build --release

# Run tests
cargo test

For full release (after modifying channel sources), run ./scripts/build-all.sh to rebuild channels first.

Database Setup

# Create database
createdb ironclaw

# Enable pgvector
psql ironclaw -c "CREATE EXTENSION IF NOT EXISTS vector;"

Configuration

Run the setup wizard to configure IronClaw:

ironclaw onboard

The wizard handles database connection, NEAR AI authentication (via browser OAuth), and secrets encryption (using your system keychain). Settings are persisted in the connected database; bootstrap variables (e.g. DATABASE_URL, LLM_BACKEND) are written to ~/.ironclaw/.env so they are available before the database connects.

Alternative LLM Providers

IronClaw defaults to NEAR AI but supports many LLM providers out of the box. Built-in providers include Anthropic, OpenAI, GitHub Copilot, Google Gemini, MiniMax, Mistral, and Ollama (local). OpenAI-compatible services like OpenRouter (300+ models), Together AI, Fireworks AI, and self-hosted servers (vLLM, LiteLLM) are also supported.

Select your provider in the wizard, or set environment variables directly:

# Example: MiniMax (built-in, 204K context)
LLM_BACKEND=minimax
MINIMAX_API_KEY=...

# Example: OpenAI-compatible endpoint
LLM_BACKEND=openai_compatible
LLM_BASE_URL=https://openrouter.ai/api/v1
LLM_API_KEY=sk-or-...
LLM_MODEL=anthropic/claude-sonnet-4

See docs/LLM_PROVIDERS.md for a full provider guide.

Security

IronClaw implements defense in depth to protect your data and prevent misuse.

WASM Sandbox

All untrusted tools run in isolated WebAssembly containers:

  • Capability-based permissions - Explicit opt-in for HTTP, secrets, tool invocation
  • Endpoint allowlisting - HTTP requests only to approved hosts/paths
  • Credential injection - Secrets injected at host boundary, never exposed to WASM code
  • Leak detection - Scans requests and responses for secret exfiltration attempts
  • Rate limiting - Per-tool request limits to prevent abuse
  • Resource limits - Memory, CPU, and execution time constraints
WASM ──► Allowlist ──► Leak Scan ──► Credential ──► Execute ──► Leak Scan ──► WASM
         Validator     (request)     Injector       Request     (response)

Prompt Injection Defense

External content passes through multiple security layers:

  • Pattern-based detection of injection attempts
  • Content sanitization and escaping
  • Policy rules with severity levels (Block/Warn/Review/Sanitize)
  • Tool output wrapping for safe LLM context injection

Data Protection

  • All data stored locally in your PostgreSQL database
  • Secrets encrypted with AES-256-GCM
  • No telemetry, analytics, or data sharing
  • Full audit log of all tool executions

Architecture

┌────────────────────────────────────────────────────────────────┐
│                          Channels                              │
│  ┌──────┐  ┌──────┐   ┌─────────────┐  ┌─────────────┐         │
│  │ REPL │  │ HTTP │   │WASM Channels│  │ Web Gateway │         │
│  └──┬───┘  └──┬───┘   └──────┬──────┘  │ (SSE + WS)  │         │
│     │         │              │         └──────┬──────┘         │
│     └─────────┴──────────────┴────────────────┘                │
│                              │                                 │
│                    ┌─────────▼─────────┐                       │
│                    │    Agent Loop     │  Intent routing       │
│                    └────┬──────────┬───┘                       │
│                         │          │                           │
│              ┌──────────▼────┐  ┌──▼───────────────┐           │
│              │  Scheduler    │  │ Routines Engine  │           │
│              │(parallel jobs)│  │(cron, event, wh) │           │
│              └──────┬────────┘  └────────┬─────────┘           │
│                     │                    │                     │
│       ┌─────────────┼────────────────────┘                     │
│       │             │                                          │
│   ┌───▼─────┐  ┌────▼────────────────┐                         │
│   │ Local   │  │    Orchestrator     │                         │
│   │Workers  │  │  ┌───────────────┐  │                         │
│   │(in-proc)│  │  │ Docker Sandbox│  │                         │
│   └───┬─────┘  │  │   Containers  │  │                         │
│       │        │  │ ┌───────────┐ │  │                         │
│       │        │  │ │Worker / CC│ │  │                         │
│       │        │  │ └───────────┘ │  │                         │
│       │        │  └───────────────┘  │                         │
│       │        └─────────┬───────────┘                         │
│       └──────────────────┤                                     │
│                          │                                     │
│              ┌───────────▼──────────┐                          │
│              │    Tool Registry     │                          │
│              │  Built-in, MCP, WASM │                          │
│              └──────────────────────┘                          │
└────────────────────────────────────────────────────────────────┘

Core Components

Component Purpose
Agent Loop Main message handling and job coordination
Router Classifies user intent (command, query, task)
Scheduler Manages parallel job execution with priorities
Worker Executes jobs with LLM reasoning and tool calls
Orchestrator Container lifecycle, LLM proxying, per-job auth
Web Gateway Browser UI with chat, memory, jobs, logs, extensions, routines
Routines Engine Scheduled (cron) and reactive (event, webhook) background tasks
Workspace Persistent memory with hybrid search
Safety Layer Prompt injection defense and content sanitization

Usage

# First-time setup (configures database, auth, etc.)
ironclaw onboard

# Start interactive REPL
cargo run

# With debug logging
RUST_LOG=ironclaw=debug cargo run

Development

# Format code
cargo fmt

# Lint
cargo clippy --all --benches --tests --examples --all-features

# Run tests
createdb ironclaw_test
cargo test

# Run specific test
cargo test test_name
  • Telegram channel: See docs/TELEGRAM_SETUP.md for setup and DM pairing.
  • Changing channel sources: Run ./channels-src/telegram/build.sh before cargo build so the updated WASM is bundled.

OpenClaw Heritage

IronClaw is a Rust reimplementation inspired by OpenClaw. See FEATURE_PARITY.md for the complete tracking matrix.

Key differences:

  • Rust vs TypeScript - Native performance, memory safety, single binary
  • WASM sandbox vs Docker - Lightweight, capability-based security
  • PostgreSQL vs SQLite - Production-ready persistence
  • Security-first design - Multiple defense layers, credential protection

License

Licensed under either of:

at your option.

Description
IronClaw is OpenClaw inspired implementation in Rust focused on privacy and security IronClaw 基于一个简单的原则:你的 AI 助手应该为你服务,而不是与你为敌。
Readme 1.8 GiB
Languages
Rust 86%
Python 7.5%
JavaScript 3.4%
Shell 1.9%
CSS 0.9%
Other 0.2%