Security Audit

06SecuritySecurity audit

Context Graph tribal knowledge

last quarter snapshot · permission baselines · waivers

Sources

AWS

GitHub

Audit log

cron quarterly

Agentcy

Audit agent + Rego

PDF report

Output

PDF

+ Slack

At a glance

The agent's job: every quarter (and on-demand), enumerate human + machine access across AWS and GitHub, summarize policy denies in Agentcy's audit log, and produce a Markdown audit report — diff'ed against last quarter.

Stack

AWS — IAM users, roles, access keys, S3 ACLs, CloudTrail of recent privileged actions.
GitHub — org members, team admins, repo collaborators with elevated scopes.
Agentcy audit log — every policy decision, API key creation, source CRUD.
Artifacts — the report lives there as a Markdown file.
Scheduled Tasks — quarterly cron.

What you'll build

A quarterly task that:

Pulls AWS IAM users, roles, access-key ages, S3 buckets with public/loose ACLs, last-90-days privileged CloudTrail events.
Pulls GitHub org members + their team memberships + their direct repo permissions.
Pulls Agentcy's audit log: top 20 deny decisions, all API-key create/revoke events, all source create/delete events.
Diffs against last quarter's snapshot.
Writes a Markdown report to artifacts. Posts a TL;DR + link to #security-leads.

The same agent answers ad-hoc: "who has admin on the monolith repo?", "are there any IAM keys older than 90 days?", "show me the policy denies in the last week."

Prerequisites

AWS with read-only IAM + S3 + CloudTrail scopes. Recommended: arn:aws:iam::aws:policy/SecurityAudit.
GitHub with org admin scope (PAT) or a GitHub App with members: read, administration: read.
Agentcy policies enabled (AGENTCY_FEATURES_POLICIES=true) — required to have an audit log to summarize.
Slack for the TL;DR post.
Realm — security recommended.

Step-by-step

1. Configure the connectors

Web UIREST API

text

1. Open /connectors → "+ Add Connector".
2. Pick AWS → "Assume role". Paste the role ARN with SecurityAudit
   policy attached. Realm: security. Region: us-east-1 (CloudTrail
   home region).
3. Pick GitHub → "GitHub App" or PAT with admin:org.
4. Slack stays where it is (re-use the existing Slack source).

bash

curl -X POST http://localhost:8080/api/v1/sources \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{
    "name":"aws-audit","connector":"aws","realm":"security",
    "config":{
      "auth":{"kind":"assume_role","role_arn":"arn:aws:iam::1234:role/agentcy-audit"},
      "regions":["us-east-1"],
      "services":["iam","s3","cloudtrail"]
    }
  }'

curl -X POST http://localhost:8080/api/v1/sources \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{
    "name":"github-org-admin","connector":"github","realm":"security",
    "config":{"auth":{"kind":"pat","token":"ghp_..."},"orgs":["acme"]}
  }'

2. Create the audit task

Web UIREST API

text

1. Open /tasks → "+ New Task".
2. Trigger Type: Schedule.
3. Name: security-audit-agent. Realm: security.
4. Cron: 0 6 1 1,4,7,10 *  (1st of Jan/Apr/Jul/Oct at 06:00 UTC).
5. Task Prompt: paste the instruction from the API tab.
6. Connectors: aws-audit, github-org-admin, slack.
7. Cost cap: 20.00 USD/day (audits are heavy reads).
8. Save.

bash

curl -X POST http://localhost:8080/api/v1/tasks \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{
    "name":"security-audit-agent",
    "agent":"default",
    "realm":"security",
    "trigger":{"kind":"schedule","cron":"0 6 1 1,4,7,10 *"},
    "input_template":{
      "instruction":"Produce a quarterly security audit report in Markdown. Sections: (1) AWS IAM — list users with access keys older than 90 days, IAM users with console access without MFA, S3 buckets with permissive ACLs, top 20 most-frequent privileged CloudTrail event names. (2) GitHub — list org admins, repo admins on critical repos (anything tagged production), members without 2FA. (3) Agentcy policy audit — top 10 deny decisions in the last 90 days grouped by rule. (4) Diff vs. last quarter — what's changed, who joined/left, new admins. (5) Top 5 recommended actions ranked by risk. Output as a single Markdown document. Save as an artifact named `security-audit-YYYY-Q?.md`. Then post a 6-line summary plus the artifact link to Slack #security-leads."
    },
    "approval_defaults":{"write":"approve"},
    "cost_cap_usd_per_day": 20.00,
    "max_concurrent_runs": 1
  }'

3. Add an "audit me now" chat path

For ad-hoc audits between cron fires, bind a chat path:

Web UIREST API

text

1. Open /channels → Slack card → Bindings → "+ New".
2. Agent: default · channel: slack · slack_channel: #security-audits ·
   mentioned: yes.
3. Save.

bash

curl -X POST http://localhost:8080/api/v1/bindings \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{
    "agent":"default",
    "match_rule":{"channel":"slack","slack_channel":"#security-audits","mentioned":true}
  }'

4. Test with a partial scope

The full audit takes a few minutes. Test with a narrower instruction first:

Web UIREST API

text

1. Open /chat → new conversation → realm: security.
2. Pin sources: aws-audit, github-org-admin.
3. Ask: "list IAM users without MFA in our prod account, and any
        access keys older than 90 days."
4. Watch the agent call iam.list_users + iam.list_access_keys.
5. If the agent answers correctly, the cron task will work.

bash

CONV=$(curl -s http://localhost:8080/api/v1/chat/conversations \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"title":"audit-smoke","realm":"security","enabled_source_ids":["src_aws_audit","src_github_org_admin"]}' | jq -r .id)

curl -N -X POST "http://localhost:8080/api/v1/chat/conversations/$CONV/messages" \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"content":"list IAM users without MFA and any access keys > 90 days old"}'

Worked example

The full task with retention configured for the artifact:

json

{
  "name": "security-audit-agent",
  "agent": "default",
  "realm": "security",
  "trigger": { "kind": "schedule", "cron": "0 6 1 1,4,7,10 *" },
  "input_template": {
    "instruction": "...as above...",
    "report_filename_template": "security-audit-{{year}}-Q{{quarter}}.md",
    "compare_to_last_quarter": true
  },
  "artifacts": {
    "retention_days": 2555
  },
  "approval_defaults": { "write": "approve" },
  "cost_cap_usd_per_day": 20.00
}

Auditors usually require 7-year retention; 2555 days covers that. Pair with an S3 sink so the artifact also lands in your durable archive bucket.

A read-only policy for this realm:

rego

package agentcy.security_audit

# This task can only read AWS / GitHub. Writes must be human-approved.
deny[msg] {
  input.subject.task == "security-audit-agent"
  input.resource.connector in {"aws", "github"}
  input.resource.tool_effect != "read"
  msg := "audit task is read-only on AWS and GitHub"
}

# But it CAN write the artifact and post to Slack.
allow {
  input.subject.task == "security-audit-agent"
  input.resource.tool == "artifacts.publish"
}
allow {
  input.subject.task == "security-audit-agent"
  input.resource.tool == "slack.post_message"
  input.resource.args.channel == "#security-leads"
}

What good looks like

The Markdown report (~3-5 pages) starts with a TL;DR:

Q2 2026 security audit · 2026-04-01
Summary: 2 IAM users gained admin since last quarter, 4 access keys expired this period, 1 S3 bucket flipped to public (now reverted). Agentcy policies blocked 14 destructive tool calls across 3 incidents. No 2FA gaps in GitHub org. Top action: rotate access key for ci-deploy-bot (107 days old).

Followed by the four sections + the diff. Each section ends with a numbered action list.

The Slack message:

🔒 Q2 audit complete · 2 new admins, 1 bucket-fix, 14 policy denies · top action: rotate ci-deploy-bot key. Full report: security-audit-2026-Q2.md

Tools called: aws.iam_list_users, aws.iam_list_access_keys, aws.s3_list_buckets_with_acls, aws.cloudtrail_summary, github.list_org_members, github.list_repo_collaborators, policies.audit_summary, artifacts.publish, slack.post_message.

Variations

Add SOC 2 control mapping. Inject a system note that maps each finding to the relevant SOC 2 control code. Auditors love this.
Real-time alerting on critical findings. A second task with a policy.deny lifecycle trigger that pages immediately on any new admin grant.
Cross-cloud. Add GCP and Azure connectors when they're available; the recipe shape stays identical.
Vendor inventory. Extend to enumerate which third-party connectors exist in Agentcy itself (/connectors list) — useful for vendor risk reviews.

Troubleshooting

The report is missing the diff. The first run has nothing to diff against. Wait one quarter or pre-seed by manually setting last_audit_artifact_id in the task's input.

AWS source returns "not authorized" on cloudtrail. The role needs cloudtrail:LookupEvents and cloudtrail:GetTrailStatus. The SecurityAudit managed policy includes both.

GitHub org members list is empty. PAT lacks admin:org. Re-issue with at least read:org.

Audit log section is "no data".AGENTCY_FEATURES_POLICIES=false — the audit log only exists when policies are enabled. Turn them on, even if the only policy is allow {true}.

Cost cap hit on the first run. Quarterly audits across a large org can be heavy. Bump the cap to 50 USD for the first run, then drop it back.

Concept: Zero-Trust Policies — required for the audit log section.
How-To: Audit Log — exporting + SIEM integration.
How-To: API Keys — what the audit watches for.

Security Audit ​

Stack ​

What you'll build ​

Prerequisites ​

Step-by-step ​

1. Configure the connectors ​

2. Create the audit task ​

3. Add an "audit me now" chat path ​

4. Test with a partial scope ​

Worked example ​

What good looks like ​

Variations ​

Troubleshooting ​

Next ​

Security Audit

Stack

What you'll build

Prerequisites

Step-by-step

1. Configure the connectors

2. Create the audit task

3. Add an "audit me now" chat path

4. Test with a partial scope

Worked example

What good looks like

Variations

Troubleshooting

Next