diff --git a/CHANGELOG.md b/CHANGELOG.md index 14a0a6744..7bfa3178e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to NanoClaw will be documented in this file. ## [Unreleased] -- **Opt-in local audit log.** `AUDIT_ENABLED=true` records every `ncl` command (both transports, including scope denials) and every host-routed approval — request, decision, and terminal outcome, covering CLI gates, self-mod, a2a message gates, agent creation (incl. the ungated global-scope door and the channel-registration name flow), the permissions sender/channel cards, and OneCLI credential holds — as one canonical SIEM-shaped event per action, appended to NDJSON day-files under `data/audit/`. Gated chains share a `correlation_id` (the approval id); details pass a recursive secret-key redactor and message-bearing events record shape only (never bodies). `AUDIT_RETENTION_DAYS` (default 90; `0` = keep forever) hard-deletes day-files past the horizon at boot + once daily in the host sweep. Read back with `ncl audit list [--actor --action --resource --outcome --since --until --correlation --limit] [--format ndjson]` — host + global-scope callers only. Off by default: nothing is persisted until enabled, and an enabled box refuses to boot if `data/audit/` isn't writable. See [docs/SECURITY.md](docs/SECURITY.md#6-local-audit-log-opt-in). +- **Opt-in local audit log.** `AUDIT_ENABLED=true` records every `ncl` command (both transports, including scope denials) and every host-routed approval — request, decision, and terminal outcome, covering CLI gates, self-mod, a2a message gates, agent creation (incl. the ungated global-scope door and the channel-registration name flow), the permissions sender/channel cards, and OneCLI credential holds — as one canonical SIEM-shaped event per action, appended to NDJSON day-files under `data/audit/`. Gated chains share a `correlation_id` (the approval id); details pass a recursive secret-key redactor and message-bearing events record shape only (never bodies). `AUDIT_RETENTION_DAYS` (default 90; `0` = keep forever) hard-deletes day-files past the horizon at boot + once daily in the host sweep. Read back with `ncl audit list [--actor --action --resource --outcome --since --until --correlation --limit] [--format ndjson]` — host + global-scope callers only. In-process consumers (future exporters) plug in via `registerAuditHook` — post-write hooks with an init/maintain/shutdown lifecycle that fire only after the local append succeeds, so external systems can never be ahead of the source of truth; no forwarder, credentials, or transport ships in core. Off by default: nothing is persisted until enabled, and an enabled box refuses to boot if `data/audit/` isn't writable. See [docs/SECURITY.md](docs/SECURITY.md#6-local-audit-log-opt-in). - **Optional per-container resource caps.** `CONTAINER_CPU_LIMIT` and `CONTAINER_MEMORY_LIMIT` pass through to `docker run` as `--cpus` / `--memory` (`container-runner.ts`). Both empty by default — no flag added, spawn args byte-identical to today — so existing installs are unaffected. Set them to cap an agent container's CPU/memory so one agent can't monopolize the host (e.g. `CONTAINER_CPU_LIMIT=2`, `CONTAINER_MEMORY_LIMIT=8g`). Swap is intentionally not managed here: `--memory` is a hard cap on a swapless host. - [BREAKING] **Chat SDK pinned to `4.29.0` (was `4.26.0` via `^4.24.0`).** `chat` and the `@chat-adapter/*` channel adapters are version-locked — the adapter's `ChatInstance` must match the bridge's, so a mismatched pair fails to typecheck at `createChatSdkBridge(...)`. `chat` is therefore pinned exactly, and the channel-adapter install pins move with it — the `/add-` SKILL.md steps and `setup/*.sh` scripts on `main`, plus the adapter code on the `channels` branch. Core installs with no channel (only `cli`) are unaffected. **Migration:** if any channel is installed (Slack, Discord, Telegram, Teams, …), re-run its `/add-` skill to pull the matching `4.29.0` adapter. - **Budget/billing-exhausted LLM turns now reach the user instead of being silently dropped.** When a turn ends in a non-retryable provider error (e.g. an Anthropic `403 billing_error`) with no `` wrapping, the agent-runner delivers the provider's notice to the originating channel and stops re-nudging the failing gateway. `providers/claude.ts` now surfaces the SDK's `is_error` flag (and the error subtype's `errors[]` text); `poll-loop.ts` delivers that text and skips the re-wrap retry. Fixes the case where a spend-limit notice produced silence plus a turn-after-turn retry loop. diff --git a/CLAUDE.md b/CLAUDE.md index 72b7b831b..b23f461df 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,7 +71,7 @@ For ad-hoc queries from skills or scripts, use the in-tree wrapper rather than t | `src/command-gate.ts` | Router-side admin command gate — queries `user_roles` directly (no env var, no container-side check) | | `src/modules/approvals/onecli-approvals.ts` | OneCLI credentialed-action approval bridge | | `src/modules/permissions/user-dm.ts` | Cold-DM resolution + `user_dms` cache | -| `src/audit/` | Opt-in local audit log (`AUDIT_ENABLED=true`): single emit seam + wrappers composed at module edges (dispatch middleware, `requestApproval` decorator, approval-resolved observer, permissions/OneCLI/create-agent seams), append-only NDJSON day-files under `data/audit/`, retention prune, `ncl audit` reader. See the "Local Audit Log" section in [docs/SECURITY.md](docs/SECURITY.md) | +| `src/audit/` | Opt-in local audit log (`AUDIT_ENABLED=true`): single emit seam + wrappers composed at module edges (dispatch middleware, `requestApproval` decorator, approval-resolved observer, permissions/OneCLI/create-agent seams), append-only NDJSON day-files under `data/audit/`, retention prune, `ncl audit` reader, and `registerAuditHook` — post-write hooks for in-process exporters (fire only after a successful local append). See the "Local Audit Log" section in [docs/SECURITY.md](docs/SECURITY.md) | | `src/group-init.ts` | Per-agent-group filesystem scaffold (CLAUDE.md, skills) — agent-runner source is a shared read-only mount, not copied per group | | `src/db/container-configs.ts` | CRUD for `container_configs` table (per-group container runtime config) | | `src/backfill-container-configs.ts` | Migrates legacy `container.json` files into the DB on startup | diff --git a/docs/SECURITY.md b/docs/SECURITY.md index c4c9ee46b..02525706d 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -208,6 +208,22 @@ lines for SIEM export. Event fields are chosen to project losslessly onto OCSF and Elastic ECS; forwarding is a mapping exercise, deferred until a forwarder exists. +**Integration surfaces** (no push forwarder ships in core — credentials and +transport for external systems never live here): + +1. **Tail the store** — any external agent (Vector, Filebeat, Fluent Bit, a + custom daemon) tails `data/audit/*.ndjson`; the format is stable and + `schema_version`-stamped. +2. **Pull via the CLI** — poll `ncl audit list --format ndjson --since …` and + dedupe on `event_id`. +3. **In-process post-write hooks** — a module (in-tree or skill-installed) + calls `registerAuditHook({ name, onEvent, init?, maintain?, shutdown? })` + from `src/audit/`. Hooks fire only **after** an event is durably appended + to the local day-file, so anything exported is guaranteed to exist in the + source of truth; a hook that misses events catches up by reading the + day-files (at-least-once). Hook failures are isolated and logged — they + never affect the log, other hooks, or the audited action. + ## Resource Limits Per-container CPU and memory caps are **opt-in and unset by default** — a runaway diff --git a/src/audit/emit.ts b/src/audit/emit.ts index d24f52941..d7366a938 100644 --- a/src/audit/emit.ts +++ b/src/audit/emit.ts @@ -9,6 +9,7 @@ import { randomUUID } from 'crypto'; import { AUDIT_ENABLED } from '../config.js'; import { log } from '../log.js'; +import { notifyAuditHooks } from './hooks.js'; import { redactDetails } from './redact.js'; import { appendAuditLine } from './store.js'; import type { AuditEvent, AuditEventInput } from './types.js'; @@ -32,7 +33,12 @@ export function emitAuditEvent(input: AuditEventInput | (() => AuditEventInput)) correlation_id: input.correlationId ?? null, details: redactDetails(input.details ?? {}), }; - appendAuditLine(JSON.stringify(event)); + const line = JSON.stringify(event); + appendAuditLine(line); + // Post-write hooks: fired only after the append succeeded, so an exporter + // can never know an event the source of truth doesn't. Failures are + // isolated inside notifyAuditHooks. + notifyAuditHooks(event, line); // eslint-disable-next-line no-catch-all/no-catch-all -- fail-open is the contract: auditing must never take down the audited action } catch (err) { // Fail-open + loud: the audited action must proceed even when the log diff --git a/src/audit/hooks.test.ts b/src/audit/hooks.test.ts new file mode 100644 index 000000000..6a5e1693c --- /dev/null +++ b/src/audit/hooks.test.ts @@ -0,0 +1,200 @@ +/** + * Post-write hook contract: hooks observe the LOG (fire only after a + * successful append, exported ⊆ written), failures are isolated everywhere, + * and the lifecycle (init/maintain/shutdown) behaves. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +const state = vi.hoisted(() => ({ enabled: true, appendThrows: false, appended: [] as string[] })); + +vi.mock('../config.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + get AUDIT_ENABLED() { + return state.enabled; + }, + AUDIT_RETENTION_DAYS: 90, + DATA_DIR: '/tmp/nanoclaw-test-hooks-unused', + }; +}); + +vi.mock('./store.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + appendAuditLine: (line: string) => { + if (state.appendThrows) throw new Error('disk full'); + state.appended.push(line); + }, + }; +}); + +vi.mock('../log.js', () => ({ + log: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn(), fatal: vi.fn() }, +})); + +let hooks: typeof import('./hooks.js'); +let emit: typeof import('./emit.js'); +let log: (typeof import('../log.js'))['log']; + +beforeEach(async () => { + state.enabled = true; + state.appendThrows = false; + state.appended.length = 0; + vi.resetModules(); // fresh hook registry per test + hooks = await import('./hooks.js'); + emit = await import('./emit.js'); + log = (await import('../log.js')).log; + vi.clearAllMocks(); +}); + +afterEach(() => { + vi.clearAllMocks(); +}); + +const EVENT_INPUT = { + actor: { type: 'human' as const, id: 'host:test' }, + origin: { transport: 'socket' as const }, + action: 'groups.list', + resources: [{ type: 'agent_group' }], + outcome: 'success' as const, + details: { limit: 5 }, +}; + +describe('post-write notification', () => { + it('calls a registered hook with the parsed event and the exact stored line', () => { + const seen: Array<{ event: import('./types.js').AuditEvent; line: string }> = []; + hooks.registerAuditHook({ name: 'demo', onEvent: (event, line) => seen.push({ event, line }) }); + + emit.emitAuditEvent(EVENT_INPUT); + + expect(state.appended).toHaveLength(1); + expect(seen).toHaveLength(1); + expect(seen[0].line).toBe(state.appended[0]); + expect(JSON.parse(seen[0].line)).toEqual(seen[0].event); + expect(seen[0].event.action).toBe('groups.list'); + }); + + it('does NOT call hooks when the local append fails — exported ⊆ written', () => { + const onEvent = vi.fn(); + hooks.registerAuditHook({ name: 'demo', onEvent }); + state.appendThrows = true; + + expect(() => emit.emitAuditEvent(EVENT_INPUT)).not.toThrow(); // action still proceeds + expect(onEvent).not.toHaveBeenCalled(); + expect(log.error).toHaveBeenCalledWith(expect.stringContaining('Audit append failed'), expect.anything()); + }); + + it('does NOT call hooks when audit is disabled', () => { + const onEvent = vi.fn(); + hooks.registerAuditHook({ name: 'demo', onEvent }); + state.enabled = false; + + emit.emitAuditEvent(EVENT_INPUT); + + expect(state.appended).toHaveLength(0); + expect(onEvent).not.toHaveBeenCalled(); + }); + + it('isolates a throwing hook: the write survives, later hooks still run, the action proceeds', () => { + const second = vi.fn(); + hooks.registerAuditHook({ + name: 'broken', + onEvent: () => { + throw new Error('exporter exploded'); + }, + }); + hooks.registerAuditHook({ name: 'healthy', onEvent: second }); + + expect(() => emit.emitAuditEvent(EVENT_INPUT)).not.toThrow(); + + expect(state.appended).toHaveLength(1); // the log has the event regardless + expect(second).toHaveBeenCalledTimes(1); + expect(log.error).toHaveBeenCalledWith( + expect.stringContaining('Audit hook threw'), + expect.objectContaining({ hook: 'broken', action: 'groups.list' }), + ); + }); +}); + +describe('lifecycle', () => { + it('initAuditHooks surfaces a failing init as a fatal error naming the hook', () => { + hooks.registerAuditHook({ name: 'ok', onEvent: () => {}, init: vi.fn() }); + hooks.registerAuditHook({ + name: 'bad-boot', + onEvent: () => {}, + init: () => { + throw new Error('no route to collector'); + }, + }); + + expect(() => hooks.initAuditHooks()).toThrow(/audit hook "bad-boot" failed to initialize.*no route/); + }); + + it('maintainAuditHooks calls every maintain and isolates throws', () => { + const m1 = vi.fn(() => { + throw new Error('flush failed'); + }); + const m2 = vi.fn(); + hooks.registerAuditHook({ name: 'a', onEvent: () => {}, maintain: m1 }); + hooks.registerAuditHook({ name: 'b', onEvent: () => {}, maintain: m2 }); + hooks.registerAuditHook({ name: 'c', onEvent: () => {} }); // no maintain — fine + + expect(() => hooks.maintainAuditHooks()).not.toThrow(); + expect(m1).toHaveBeenCalledTimes(1); + expect(m2).toHaveBeenCalledTimes(1); + expect(log.error).toHaveBeenCalledWith( + expect.stringContaining('maintenance failed'), + expect.objectContaining({ hook: 'a' }), + ); + }); + + it('shutdownAuditHooks awaits async shutdowns and isolates throws', async () => { + const order: string[] = []; + hooks.registerAuditHook({ + name: 'a', + onEvent: () => {}, + shutdown: async () => { + await Promise.resolve(); + order.push('a'); + }, + }); + hooks.registerAuditHook({ + name: 'b', + onEvent: () => {}, + shutdown: () => { + throw new Error('handle already closed'); + }, + }); + hooks.registerAuditHook({ + name: 'c', + onEvent: () => {}, + shutdown: () => { + order.push('c'); + }, + }); + + await hooks.shutdownAuditHooks(); + + expect(order).toEqual(['a', 'c']); + expect(log.error).toHaveBeenCalledWith( + expect.stringContaining('shutdown failed'), + expect.objectContaining({ hook: 'b' }), + ); + }); + + it('maintainAudit skips hook maintenance when audit is disabled', async () => { + const init = await import('./init.js'); + const maintain = vi.fn(); + hooks.registerAuditHook({ name: 'a', onEvent: () => {}, maintain }); + + state.enabled = false; + init.maintainAudit(); + expect(maintain).not.toHaveBeenCalled(); + + state.enabled = true; + init.maintainAudit(); + expect(maintain).toHaveBeenCalledTimes(1); + }); +}); diff --git a/src/audit/hooks.ts b/src/audit/hooks.ts new file mode 100644 index 000000000..8789af102 --- /dev/null +++ b/src/audit/hooks.ts @@ -0,0 +1,92 @@ +/** + * Post-write audit hooks — the in-process extension seam. + * + * A hook observes the audit LOG, not the event stream: `onEvent` fires only + * after an event has been durably appended to the local day-file, so anything + * a hook exports is guaranteed to exist in the source of truth + * (exported ⊆ written). If the local append fails, hooks are not called — + * and a hook that misses events (crash, restart) catches up by reading the + * day-files, which is the at-least-once story. + * + * Registration follows the tree's observer idiom (registerApprovalResolvedHandler, + * registerResponseHandler, …): an in-tree or skill-installed module calls + * `registerAuditHook(...)` at import time — no core edits, and credentials or + * transport for an external system live in that module, never here. + */ +import { log } from '../log.js'; +import type { AuditEvent } from './types.js'; + +export interface AuditHook { + /** Short identifier used in logs and lifecycle errors. */ + name: string; + /** + * Called after a successful local append. `line` is the exact stored bytes + * (one NDJSON line, no trailing newline); `event` is the parsed record. + * MUST be fast and non-blocking — this runs on the audited action's call + * path. A real exporter buffers here and does its IO from `maintain`/its own + * timers. Throwing is tolerated: isolated and logged, never propagated. + */ + onEvent(event: AuditEvent, line: string): void; + /** Boot hook, called only when audit is enabled. Throw = host refuses to start. */ + init?(): void; + /** Periodic maintenance — called from the 60s host-sweep (enabled boxes only). */ + maintain?(): void; + /** Graceful-shutdown hook (flush buffers, close handles). */ + shutdown?(): void | Promise; +} + +const hooks: AuditHook[] = []; + +export function registerAuditHook(hook: AuditHook): void { + hooks.push(hook); +} + +/** Fan out one written event to every hook, isolating failures per hook. */ +export function notifyAuditHooks(event: AuditEvent, line: string): void { + for (const hook of hooks) { + try { + hook.onEvent(event, line); + // eslint-disable-next-line no-catch-all/no-catch-all -- isolation is the contract: one bad hook must not affect the log, other hooks, or the audited action + } catch (err) { + log.error('Audit hook threw — event is safely in the log', { hook: hook.name, action: event.action, err }); + } + } +} + +/** Boot lifecycle. A hook that can't start is a silent-export-gap risk — fatal. */ +export function initAuditHooks(): void { + for (const hook of hooks) { + try { + hook.init?.(); + } catch (err) { + throw new Error( + `audit hook "${hook.name}" failed to initialize: ${err instanceof Error ? err.message : String(err)}`, + { cause: err }, + ); + } + } +} + +/** Sweep lifecycle — periodic maintenance, isolated per hook. */ +export function maintainAuditHooks(): void { + for (const hook of hooks) { + try { + hook.maintain?.(); + // eslint-disable-next-line no-catch-all/no-catch-all -- one hook's maintenance failure must not stop the others (or the sweep) + } catch (err) { + log.error('Audit hook maintenance failed', { hook: hook.name, err }); + } + } +} + +/** Shutdown lifecycle — awaited by the host's graceful shutdown. */ +export async function shutdownAuditHooks(): Promise { + for (const hook of hooks) { + try { + await hook.shutdown?.(); + // eslint-disable-next-line no-catch-all/no-catch-all -- shutdown must drain every hook even when one throws + } catch (err) { + log.error('Audit hook shutdown failed', { hook: hook.name, err }); + } + } +} diff --git a/src/audit/index.ts b/src/audit/index.ts index dfd1c82a7..06bd2d86c 100644 --- a/src/audit/index.ts +++ b/src/audit/index.ts @@ -7,6 +7,7 @@ */ export * from './types.js'; export { redactDetails } from './redact.js'; -export { AUDIT_DIR, pruneAuditLogIfDue } from './store.js'; -export { initAuditLog } from './init.js'; +export { AUDIT_DIR } from './store.js'; +export { initAuditLog, maintainAudit } from './init.js'; +export { type AuditHook, registerAuditHook } from './hooks.js'; export { runApprovedHandler, withAudit } from './wrappers.js'; diff --git a/src/audit/init.ts b/src/audit/init.ts index 619861a07..e81f56951 100644 --- a/src/audit/init.ts +++ b/src/audit/init.ts @@ -3,12 +3,16 @@ * * The decision observer registers unconditionally (its emits no-op when * disabled). When enabled: assert data/audit/ is writable (refusing to start - * beats running with a silent audit gap) and run the boot prune. + * beats running with a silent audit gap), run the boot prune, and start the + * registered post-write hooks' lifecycle (init here, maintain via the host + * sweep, shutdown via the host's graceful-shutdown registry). */ import { AUDIT_ENABLED, AUDIT_RETENTION_DAYS } from '../config.js'; import { log } from '../log.js'; +import { onShutdown } from '../response-registry.js'; +import { initAuditHooks, maintainAuditHooks, shutdownAuditHooks } from './hooks.js'; import { registerAuditObserver } from './observer.js'; -import { assertAuditWritable, AUDIT_DIR, markPrunedToday, pruneAuditLog } from './store.js'; +import { assertAuditWritable, AUDIT_DIR, markPrunedToday, pruneAuditLog, pruneAuditLogIfDue } from './store.js'; export function initAuditLog(): void { registerAuditObserver(); @@ -23,5 +27,16 @@ export function initAuditLog(): void { } pruneAuditLog(); markPrunedToday(); + initAuditHooks(); // throw → main() exit 1, same posture as the writability assert + onShutdown(() => shutdownAuditHooks()); log.info('Audit log enabled', { dir: AUDIT_DIR, retentionDays: AUDIT_RETENTION_DAYS }); } + +/** + * Host-sweep tick: retention prune (throttled to once per UTC day internally) + * plus every hook's periodic maintenance. No-op when audit is disabled. + */ +export function maintainAudit(): void { + pruneAuditLogIfDue(); + if (AUDIT_ENABLED) maintainAuditHooks(); +} diff --git a/src/host-sweep.ts b/src/host-sweep.ts index 702c7a11b..9832a059d 100644 --- a/src/host-sweep.ts +++ b/src/host-sweep.ts @@ -43,7 +43,7 @@ import { syncProcessingAcks, type ContainerState, } from './db/session-db.js'; -import { pruneAuditLogIfDue } from './audit/index.js'; +import { maintainAudit } from './audit/index.js'; import { log } from './log.js'; import { openInboundDb, openOutboundDb, openOutboundDbRw, inboundDbPath, heartbeatPath } from './session-manager.js'; import { isContainerRunning, killContainer, wakeContainer } from './container-runner.js'; @@ -165,12 +165,13 @@ async function sweep(): Promise { } // MODULE-HOOK:approvals-reason-sweep:end - // Audit retention — unlink day-files past AUDIT_RETENTION_DAYS. Throttled - // to once per UTC day inside the module; no-op when audit is disabled. + // Audit maintenance — retention prune (throttled to once per UTC day inside + // the module) + registered post-write hooks' maintain(). No-op when audit + // is disabled. try { - pruneAuditLogIfDue(); + maintainAudit(); } catch (err) { - log.error('Audit prune failed', { err }); + log.error('Audit maintenance failed', { err }); } setTimeout(sweep, SWEEP_INTERVAL_MS);