fairvisor test

fairvisor test loads a policy bundle and runs the full rule engine in-process against a set of sample requests. No Docker container or network is required.

Synopsis

fairvisor test <bundle-file> [--requests=<file>] [--format=table|json]

Options

Auto-generated requests

If --requests is not provided, the CLI auto-generates one request per policy using the policy’s selector metadata:

• method — first entry in selector.methods, or GET
• path — selector.pathExact or selector.pathPrefix (without trailing /)
• Empty headers, query params, and body

These requests are useful for smoke-testing that each policy reaches its first rule.

Request file format

The --requests file must be a JSON array of request objects:

[
  {
    "method": "POST",
    "path": "/v1/chat/completions",
    "headers": {
      "authorization": "Bearer eyJhbGciOiJIUzI1NiJ9.eyJvcmdfaWQiOiJvcmctMTIzIn0.sig",
      "x-api-key": "key-abc"
    },
    "query_params": {},
    "body": "{\"model\":\"gpt-4o\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}"
  },
  {
    "method": "GET",
    "path": "/api/v1/users",
    "headers": { "x-api-key": "key-abc" }
  }
]

All fields except method and path are optional.

Table output (default)

1. POST /v1/chat/completions  -> allow   (llm-rate-limit / llm-token-budget)
2. GET  /api/v1/users         -> allow   (api-rate-limit / per-key-limit)
3. POST /v1/chat/completions  -> reject  (llm-rate-limit / llm-token-budget)  reason: tpm_exceeded

Summary: 3 total, 2 allow, 1 reject, 0 other

JSON output

fairvisor test policy.json --requests requests.json --format json

{
  "results": [
    {
      "index": 1,
      "method": "POST",
      "path": "/v1/chat/completions",
      "action": "allow",
      "reason": "all_rules_passed",
      "policy_id": "llm-rate-limit",
      "rule_name": "llm-token-budget"
    }
  ],
  "summary": {
    "total": 3,
    "allow": 2,
    "reject": 1,
    "other": 0
  }
}

Examples

# Smoke test every policy with auto-generated requests
fairvisor test policy.json

# Test with explicit requests
fairvisor test policy.json --requests my-requests.json

# Get full JSON output for parsing
fairvisor test policy.json --format json | jq '.summary'

# Fail CI if any request is rejected
fairvisor test policy.json --format json \
  | jq -e '.summary.reject == 0'

How it works

The test command:

1. Loads and compiles the bundle via bundle_loader.load_from_string()
2. Initialises the rule_engine with a mock shared dict (in-process Lua table)
3. Calls rule_engine.evaluate() for each request
4. Aggregates and formats results

Because the mock shared dict is reset between test runs, counter state does not persist across requests in the same run. Use the --requests file to send the same request multiple times to test threshold behaviour.

Testing threshold behaviour

[
  { "method": "POST", "path": "/v1/chat/completions", "headers": { "x-api-key": "key-1" } },
  { "method": "POST", "path": "/v1/chat/completions", "headers": { "x-api-key": "key-1" } },
  { "method": "POST", "path": "/v1/chat/completions", "headers": { "x-api-key": "key-1" } }
]

Sending the same key three times will accumulate token cost and eventually trigger a limiter.