Plugin delegation

Use this when: changing plugin delegation registration, revocation, broker identity, or the scope of attempt credentials.

Authority

A delegation is an immutable restriction record owned by an ordinary IAM user. It is neither a permission grant nor a credential. Creating it requires that user's real signed request and admin:CreatePluginDelegation; reading and revoking require the corresponding admin:GetPluginDelegation and admin:RevokePluginDelegation actions. Root, temporary credentials and service accounts cannot own records.

The record binds a deployment, an input bucket, literal input and output prefixes, and an expiry of at most 24 hours. The v1 profile writes to that same bucket; v2 explicitly names the output bucket. Wildcards, control characters and IAM variable expressions are rejected. Its random ID is also its immutable revision. There is no update operation.

The Controller is responsible for validating accepted task identity, current lease, epoch, node session and control generation before requesting credentials. RustFS authenticates the configured broker and enforces the delegation and current parent IAM. RustFS does not independently verify a remote Controller lease.

sequenceDiagram
    participant U as Original IAM user
    participant C as Controller broker
    participant R as RustFS
    participant P as Plugin
    U->>R: Signed delegation registration
    R-->>U: Immutable ID and revision
    C->>C: Validate task and current lease
    C->>R: Signed issue-attempt request
    R->>R: Broker mapping + active record + current IAM
    R->>R: Persist scoped STS and recheck authority
    R-->>C: Short-lived credentials
    C-->>P: Attempt credentials
    P->>R: Exact version GET / attempt-prefix PUT
    R->>R: Current parent IAM intersected with session policy

Persistence and revocation

crates/iam/src/delegation.rs owns record validation and immutable revocation tombstones. Records use the existing IAM store under config/iam/plugin-delegations/v1. Writes require a successful readback. Revocation is idempotent and durable; it does not delete or overwrite the restriction record.

ObjectStore::load_iam_config_strict distinguishes a missing object from an empty object, invalid JSON or failed decryption. Only a genuinely absent tombstone permits an active record. Corrupt or unavailable storage rejects access rather than treating revocation as absent. Existing non-delegation readers keep their compatibility behavior.

Revocation prevents future issuance. It does not promise immediate invalidation of credentials already delivered; those remain subject to their expiration and current parent IAM. Registration, IAM changes and issuance are not one atomic transaction. Issuance rechecks after persistence and deletes the generated STS account if authority changed. Failed persistence or invalidation acknowledgment is reported as unknown issuance, never as safe retry permission.

Manual job submission reads the active delegation before calling the Runner. A submission that passed this check is in flight and may be admitted if revocation races with the outbound request; revocation prevents submissions whose authority check occurs afterward, but does not cancel an admitted job. Callers must treat a successful submit receipt as accepted work even if the delegation is revoked later. After an ambiguous timeout or service error, retry the same idempotency key while the delegation remains active; after revocation or expiry, query status instead of retrying submission.

POST /rustfs/admin/v3/plugin-delegations/{id}/runs/query accepts {version:1,run_id} and requires an ordinary IAM user with admin:GetPluginRun who owns the delegation in {id}. It reads the immutable owner record without checking expiry or revocation, then queries Runner using the record's delegation ID and revision and the authenticated access key. The result can contain only that caller's run status and committed manifest; it does not perform object reads or reauthorize object access. The Runner response must match the requested run ID and the strict status schema before RustFS returns it.

POST /rustfs/admin/v3/plugin-delegations/{id}/runs/cancel accepts the same request and requires admin:CancelPluginRun plus ownership of {id}. Expiry and revocation do not block cancelling an existing run. Runner scopes the run to that owner; missing and other-owner runs share the same not-found response, while successful and failed runs return conflict. Success returns {version:1,run_id,state:"CANCELLED",cleanup_pending} and does not imply that asynchronous cleanup has finished.

POST /rustfs/admin/v3/plugin-delegations/{id}/runs/page accepts {version:1,limit,cursor?} with a limit from 1 to 100 and requires admin:ListPluginRuns plus ownership of {id}. Expiry and revocation do not hide historical runs. RustFS supplies signer and delegation identity from authenticated IAM state; the Runner page is scoped to the exact caller and deployment. The strict response contains run summaries and an opaque-to-clients stable cursor; callers pass the returned cursor unchanged until it is null. RustFS bounds request/response sizes and rejects malformed, unordered, over-limit, or identity-invalid summaries.

POST /rustfs/admin/v3/plugin-delegations/{id}/runs/history accepts {version:1,run_id} and requires admin:ListPluginRuns plus ownership of {id}. RustFS derives signer and delegation identity from the authenticated user and IAM record, then proxies Runner's bounded history response. The response contains at most the latest 32 committed state events in Runner's history-ID order and a has_older flag; timestamps are display data and are not used to reorder events. Missing and foreign runs share the same not-found response. RustFS rejects malformed, empty, over-limit, unexpected-state, or mismatched-run responses and bounds both request and response bodies.

POST /rustfs/admin/v3/plugin-delegations/{id}/runs/replay accepts {version:1,origin_run_id,idempotency_key,deadline_unix_micros} and requires admin:ReplayPluginRun, current IAM authorization, ownership, and an active unexpired delegation. Caller identity fields are not accepted; RustFS derives signer and delegation revision from the authenticated request and IAM record. Runner accepts only an owned failed origin, preserves its fixed plugin/configuration/input version, and requires a new deadline no later than its configured run timeout. RustFS requires a positive Unix-microsecond deadline and verifies the strict receipt echoes it. Missing or foreign origins share not-found behavior; conflicts map to an operation conflict. Requests and receipts are bounded, and responses use Cache-Control: no-store.

Broker profile

rustfs/src/admin/handlers/plugin_attempt.rs implements POST /rustfs/admin/v3/plugin-delegations/{id}/issue-attempt. The signer must be an ordinary IAM user with admin:IssuePluginAttempt, independently listed in RUSTFS_PLUGIN_BROKERS:

{"broker-access-key":"deployment-name"}

This configuration contains public access keys, not secrets. It is immutable per process. Missing or invalid configuration disables broker issuance. The broker is a trusted Controller identity; ordinary users cannot substitute their own request fields for Controller validation.

The request carries tenant, run_id, attempt_id, revision, deployment, input_bucket, input_key, input_version, output_bucket, output_prefix and ttl_seconds. IDs must be non-nil. Both profiles accept one exact input version and an output prefix equal to:

<delegation-output-prefix><lowercase-hex-tenant>/<run-uuid>/<attempt-uuid>/

The session policy permits only s3:GetObjectVersion for that input with an exact s3:VersionId, and s3:PutObject beneath that attempt prefix. Credential lifetime is at most 900 seconds and cannot exceed delegation expiry. The Controller must additionally cap it to its validated lease and hard deadline.

The original user's current IAM remains the parent permission boundary on every S3 request. The server's signing key only seals the token; it is not the plugin's authorization identity. Broker identity and original user identity are separate.

External policy plugins must be disabled: their authorization path does not enforce this built-in session-policy boundary. Initializing, failed and ready plugin states all reject issuance. Attempt tokens carry a signed, persisted marker: after restart or on another node, marked credentials are accepted only by built-in STS authorization. A denied marked request returns a terminal error, so public bucket policies cannot bypass its session scope. The first profile also rejects parent policies containing conditions, because a broker request cannot reconstruct the original user's IP, MFA or other request context. It does not silently replace that context with the broker's.

Response and failures

Successful issuance returns version, delegation_id, delegation_revision, owner_access_key, access_key_id, secret_access_key, session_token and expires_at_unix_seconds, with Cache-Control: no-store. The Controller verifies the returned delegation and owner against the accepted caller before delivery. Credentials must never enter logs, metrics or task result manifests.

Missing records and records owned by a different user have the same read response. Revoked or expired records reject access. Storage failures remain errors. Post-persistence failures carry static error_code, issued and issuance_unknown facts without credentials; callers must reconcile an unknown outcome rather than automatically mint another credential.

The route/action inventory is maintained in rustfs/src/admin/route_policy.rs. This contract does not imply event capture, physical container cleanup, restart reconciliation or upstream release acceptance.

Cross-bucket restriction profile v2

The existing registration endpoint accepts this explicit v2 restriction:

{
  "version": 2,
  "deployment": "runner-production",
  "bucket": "source-images",
  "input_prefix": "inputs/",
  "output_bucket": "plugin-artifacts",
  "output_prefix": "outputs/",
  "expires_at_unix_seconds": 1893456000
}

The example expiry must be replaced by a future value within the existing 24-hour limit. bucket still identifies the input bucket. The existing issue-attempt request and credential response do not change: its input_bucket must equal bucket, and output_bucket must equal the explicit v2 output bucket. Neither a different bucket nor a sibling attempt prefix can be supplied by the broker. The exact input version and deterministic output prefix checks remain in force.

A v1 restriction has neither version nor output_bucket; its JSON fields and same-bucket behavior are unchanged. V2 requires both version: 2 and a nonempty, bounded, literal output_bucket. Missing pairs, explicit nulls, unknown versions and unknown fields are rejected rather than downgraded to v1. The immutable record envelope carries version 1 or 2 to match its restriction profile. Both remain in the existing IAM record namespace; no data migration or mutable record update is introduced. Readers reject mismatched envelope and restriction versions. Older readers reject v2, so deploy compatible readers before creating v2 records.

V2 does not grant the original user additional permissions. Issuance retains its current-parent checks and post-persistence recheck/invalidation, and each S3 request still requires both the original user's current IAM policy and the attempt session policy to allow it. An explicit parent Deny or removal of output permission prevents writes even with an already issued token. No administrator credential substitutes for missing parent permission.

Focused tests are delegation_v1_json_remains_compatible, delegation_v2_requires_explicit_output_bucket, attempt_v2_policy_keeps_input_version_and_output_bucket and delegation_v2_persists_and_vends_with_real_parent_intersection. The last uses the actual issuance handler, persistent IAM and four-disk storage, including a real input version and a third bucket allowed by the parent but forbidden by the delegation. Removing only the output-bucket comparison must fail A03_CROSS_BUCKET_OUTPUT_BOUND; parent intersection is checked independently with issued temporary credentials. These are internal handler/IAM tests, not signed HTTP or complete cross-bucket S3 execution.

This profile adds no bucket-creation API, Runner binding support, credential endpoint or GC permission. Coordinating Runner's pinned binding with its result storage bucket remains a separate integration requirement.

12 KiB Raw Permalink Blame History