Should You Trust Open-Source Security Tools To Guard Your AI Agents?

Last Updated: 10 November 2026

You can trust open-source security tools to guard an agent, but only if you treat them as instruments rather than oracles. The tool can inspect a repository, enumerate tool manifests and flag a dangerous permission scope. It cannot decide whether the finding matters to your business, and it cannot stop an agent that runs with too much access. Two releases landed this week that make the question practical rather than theoretical: an MIT licensed agent security harness from 0sec Labs, and a free reverse-engineering skill from Speedrun AI Labs with a self-learning loop. Both are useful. Neither replaces a person who reads the evidence.

What changed this week in agent security tooling?

Two open-source releases moved agent security from a vendor promise to something a small team can run. 0sec Labs published a harness called 0 under MIT or Apache-2.0, backed by Y Combinator and described as work from the Swiss Applied AI and Cybersecurity Research Lab. It runs locally on Apple Silicon macOS or x64 and ARM64 Linux with your own model provider, and exposes a web UI, a CLI and an MCP server. The repository sits at about 1.1k GitHub stars and 2,114 commits, which is a fair signal that it is actively maintained rather than published and abandoned.

Speedrun AI Labs, a Sydney group, released Reverse Engineer Anything, an MIT licensed skill built around a parity checklist and a recursive self-learning loop. The checklist is signed by the operator, each item is one testable behaviour, and the score is must-haves passing over must-haves total. Done means every must-have passes in two consecutive reviews. The loop then saves checklist patterns, review techniques, tool recipes and fix recipes into the skill, staged for operator approval.

Both are small, readable and free. That is the point. A security tool you can open, read and run against your own stack is a different proposition from a scanner you rent and cannot inspect. The question is what each one actually checks.

What does an MCP-aware security harness actually inspect?

An MCP-aware harness inspects the surfaces where an agent can be turned against its owner: tool manifests, permission scopes, prompt injection surfaces and data egress. Tool manifests describe what each tool claims to do and what parameters it accepts. Permission scopes describe what it can reach. Prompt injection surfaces are the places untrusted text enters the agent's context. Data egress is where results can leave.

Tool manifests matter because of tool poisoning, a documented attack where malicious instructions are hidden in MCP tool metadata or descriptions. Those instructions land in the agent's context and can be followed without any runtime validation against the definition that was originally approved. The research consensus across CloudThinker, aipatternbook and mcpscan is consistent on the mechanism, even where the exploit counts differ. A manifest diff is the simplest defence: if the description changed since approval, the tool changed, and the agent is now trusting something you did not approve.

Permission scopes matter because agents accumulate access. Each new task adds a scope, and nobody removes one. A harness that enumerates scopes per tool and compares them to a declared minimum turns that drift into a reviewable list. Data egress matters because an agent with a network tool and a document reader can move a client file anywhere. The check is not whether egress is possible. It is whether egress is restricted to an approved destination list.

Where does open source beat a paid scanner, and where does it not?

Open source beats a paid scanner on cost, inspection and control, and loses on coverage breadth, support and the staffing it assumes you have. Semgrep is a useful reference point because its pricing is public. Teams Code is US$30 per month per contributor, Supply Chain is US$30 per month per contributor and Secrets is US$15 per month per contributor, with a Free Edition capped at 10 contributors and 10 repositories. Enterprise pricing is custom and includes on-prem SCM support.

For a team of five, US$30 per contributor is US$150 a month for one product line, which is affordable until you need more than one. The free edition covers small teams but stops at 10 contributors and 10 repositories, which is a real ceiling for a growing business. An open-source harness has no seat count. It also has no vendor answering a ticket at 2am, no curated rule library maintained by a research team, and no guarantee that next month's release still works.

The honest split is this. If you need broad language coverage, a maintained rule set and a support contract, a paid scanner earns its price. If you need to inspect your own agent stack, understand what each skill can reach and keep the evidence in-house, an open-source harness does work a paid scanner will not do at any price, because the paid scanner is not built around your MCP surface.

How do you run a harness against a local agent stack?

You run a harness against a local agent stack by keeping the analysis offline, giving it a disposable environment and pointing it at a copy of the repository rather than the live system. The 0sec harness supports that pattern directly: it runs locally with your own model provider, exposes an MCP server that takes a workspace, and ships workflow templates for assessment, verification, fixes and research.

Flowtivity runs this against dual DGX Spark machines with DeepSeek V4 Flash locally, which means the model doing the review is on hardware we control and the code never leaves the building. A workable sequence looks like this. Clone the repository to an analysis machine with no network access. Run the assessment workflow and read the findings queue, which links each finding to the source it came from. Enable the scope plugin so the harness refuses to touch anything outside the declared target. Then, if you want it to attempt fixes, move to a disposable virtual machine and review the evidence before applying anything, because the default shell runs on your host and untrusted work needs an isolated executor.

That last point is the one people skip. The harness is honest about it: assess authorised systems, review the evidence before applying fixes, and use an isolated executor for untrusted work. A tool that tells you where its own risk sits is easier to trust than one that claims none.

What are the honest limits of a self-learning skill?

The honest limit of a self-learning skill is that it learns patterns, not judgement. Speedrun's loop saves checklist patterns, review techniques, tool recipes and fix recipes into the skill and stages them for operator approval. It explicitly does not capture failures as rules, secrets, or anything from the target. That design is careful, and it is still a library of lessons that a person has to read.

Two risks remain. First, a staged lesson can be approved without scrutiny, which turns a review step into a rubber stamp. Second, a skill that has learned from one target may apply those lessons confidently to a different target where they do not hold. The mitigation is procedural rather than technical: an approver independent of the requester, a no-network analysis environment, and a disposable virtual machine for any live steps. If your process has none of those, the self-learning loop is a faster way to be wrong, not a smarter one.

What five-point checklist should an agent skill pass before production?

A five-point checklist should cover provenance, manifest, scope, egress and rollback, because those five questions catch most of the supply-chain risk in an agent skill. Provenance asks who wrote it and whether the source is readable. Manifest asks whether the tool description matches what the skill actually does. Scope asks what it can reach and whether that is the minimum. Egress asks where data can go. Rollback asks whether you can remove it cleanly.

Run the checklist on every skill before it touches production, and re-run the manifest check on every update, because tool poisoning does not need a new version number to arrive. Record the approver and the date. If a skill cannot answer all five questions, it is not ready, no matter how useful the demo looked. The same discipline applies to the harness itself, which is why reading its licence and its safety notes is part of the job.

What is the Flowtivity play?

The Flowtivity play is to treat agent security as a supply-chain problem with an evidence trail, not a scanner you run once. We review what your agents can reach, what skills they trust and whether the evidence survives a challenge from a client or an auditor. The output is a list you can hand to a client, not a dashboard only an engineer can read, and it is rebuilt every time a skill changes.

Four offers cover the work. First, an agent skill audit: we inventory every skill and MCP tool in your stack, diff the manifests against approved versions and produce a scope report. Second, an open-source harness deployment: the 0sec harness running against your repositories on hardware you own, with the scope plugin enabled and findings linked to source. Third, an isolated analysis environment: a no-network machine and disposable virtual machine setup so untrusted skills can be tested without touching production. Fourth, a supply-chain control pack: the five-point checklist, a manifest diff process and a recorded approver for every skill change, wired into your release routine.

We do this on the Gold Coast for Australian businesses that want to know what their agents trust before a client asks. If you cannot list every skill your agent can call, the audit is the place to start.

Frequently asked questions

Is open-source security software safe for business use?

It can be, if you read the licence, inspect the code and run it in an isolated environment. The 0sec harness is MIT or Apache-2.0 licensed and designed to run locally with your own model provider. The risk is not the licence. It is installing a tool with broad permissions and letting it act on your host without reviewing the evidence first.

What is MCP tool poisoning?

MCP tool poisoning hides malicious instructions inside a tool's metadata or description. Those instructions enter the agent's context and can be followed without any runtime validation against the definition that was originally approved. The defence is a manifest diff on every update plus a review of what the tool can reach.

Do I need a paid scanner if I use an open-source harness?

Not necessarily, but they do different jobs. Semgrep's Teams Code is US$30 per month per contributor with a free tier capped at 10 contributors and 10 repositories. A paid scanner gives broad rule coverage and support. An open-source harness inspects your own agent and MCP surface, which a general scanner is not built to do.

How do I test an agent skill without risking production?

Use a no-network analysis machine and a disposable virtual machine. Copy the repository, run the assessment workflow, read the findings and only then consider fixes. Keep the harness scoped to the target and never let an untrusted skill run with your production credentials. Speedrun's own guidance asks for an approver independent of the requester.

What should I check before installing an agent skill?

Five things: who wrote it, whether the manifest matches what it does, what it can reach, where it can send data and whether you can remove it cleanly. Re-check the manifest on every update, because tool poisoning can arrive without a version bump. Record the approver and the date for each decision.

Takeaways

• Open-source agent security tools are worth trusting when you can read them, run them offline and review evidence before anything is applied.
• An MCP-aware harness checks manifests, scopes, injection surfaces and egress, which is where supply-chain risk actually lives.
• Open source wins on cost and inspection. Paid scanners win on coverage and support, and Semgrep's paid tiers start at US$30 per contributor per month.
• A self-learning skill stages its lessons for approval, so an independent reviewer is the control that makes it safe.