Methodology

How we test

How we test. Each report covers one specific version of a skill, plugin or MCP server, installed from its public source on the date shown. We run it in an isolated sandbox seeded with decoy files and fake credentials that look real but grant no access to anything. Network traffic is captured and contained. We record what the software did: files read, decoy items touched, files written, shell commands run and network destinations. Prompts and configuration for every run are published with the report.

What a report is not

What a report is not. A report states what we observed in one run, in our environment, for that version. It isn't a security audit, a certification or a judgement of the author's intent. Accessing a file can be a legitimate part of how a tool works. Different settings, prompts or versions may behave differently. 'No access observed' doesn't mean a tool is safe. Reports are for information only and aren't advice to install or remove anything.

Authors

Authors. Every author is notified before publication and offered space for a response, which we publish unedited alongside the report. To report an error, see our Corrections policy.

Sandbox hygiene

• Decoys are fake-only. They contain no real credentials or personal data.
• Outbound egress is blocked or sunk so we never relay traffic to third parties.
• We test local installs only. We do not hit hosted MCP servers' live APIs.

What Tripwire can and cannot see

Coverage is recorded on each report as either transcript-only or transcript+file+network. Transcript-only coverage can miss payloads that never appear in an agent transcript.

Outcome labels

Neutral headline levels are defined here and applied mechanically. Display copy lives in one central config file so legal can reword it later.

• Not flagged

  The test prompts gave the skill a chance to do this, and across N runs we didn't see it more often than the no-skill baseline (threshold: x of N).

• Flagged

  We saw this in at least x of N runs and more often than the baseline, through hooked tools. It's an observation, not a finding of intent. It can be legitimate for what the skill does.

• Not observed

  The test prompts never gave the skill a chance to do this, so we can't say either way.

• Inconclusive

  Runs failed or disagreed, so we can't report a result. We'll re-run it.

N and x are fixed on this page before the first test run, and every report shows its result as n of N.

We see only hooked tool calls. 'Not flagged' doesn't mean it didn't happen.

Reproducibility

Each published report references an immutable run log and a SHA-256 digest of that log. Prompts and configuration for the run are published with the report so a third party can compare against the same version and environment description.