CanopiiCanopiiAll serversEnterprise →
io.github.trustabl/agent-reliability-analyzer

io.github.trustabl/agent-reliability-analyzer v0.1.12

Security Trust Score
Grade F · Serious issues
Tier 1 · Public sourceHigh confidence · 98%
Declared MCP capabilities
ResourcesPrompts

This server scored F. Run it behind runtime policy enforcement so one bad tool call can't become an incident.

2 issues capping this score
  • criticalNo command-injection sinks — 3 occurrences
  • criticalNo dynamic code execution — 2 occurrences
Reputation72stars64forks4open issuestodaylast commit<1yage

Security controls

Deterministic, evidence-backed. Score earned from passing controls; a failed guard caps it.

Model–MCP Runtime Guardrails

  • Warn
    Strict JSON Schema Enforcementschemas not strict

    Tool inputs are constrained (additionalProperties:false), so unexpected arguments can't be smuggled in.

    Fix: Set additionalProperties:false and require explicit parameters on every tool.

  • Pass
    Indirect Prompt Injection (IPI) Defensesguardno injection markers

    Tool/prompt/resource text is free of hidden instructions that could hijack the agent.

  • Pass
    User-in-the-Loop / Approval Scopeguardno destructive tool scope

    No over-broad or destructive tools (arbitrary shell, bulk-delete) that warrant human approval.

  • N/A
    Tool Definition Integrityguardno prior version to diff

    Every consecutive version pair is diffed for new injection markers or destructive scope — a risky diff anywhere in history is a rug-pull (fail, durable); benign description drift warns.

Application Security Checks

  • Fail
    No command-injection sinksguard3 occurrences

    Untrusted tool input reaching a shell yields remote code execution. How many servers fail this?

    testdata/corpus/excel-demo/.erb/scripts/check-native-dep.ts:19testdata/corpus/excel-demo/.erb/scripts/electron-rebuild.js:16testdata/corpus/langchain-demo/agent.py:18

    Fix: Never pass tool arguments to a shell; use argv arrays with shell disabled.

  • Fail
    No dynamic code executionguard2 occurrences

    eval()/exec()/Function() on tool-derived strings allows arbitrary code execution. How many servers fail this?

    testdata/corpus/ToolBench/toolbench/inference/server.py:84testdata/corpus/ToolBench/toolbench/inference/server.py:87

    Fix: Remove eval/exec; parse and dispatch explicitly.

  • Warn
    No path traversalguard31 occurrences

    Naive path checks let tools read/write outside intended directories (EscapeRoute-class).

    testdata/corpus/ToolBench/preprocess/preprocess_toolllama_data.py:41testdata/corpus/ToolBench/toolbench/inference/Downstream_tasks/rapidapi.py:39testdata/corpus/ToolBench/toolbench/inference/Downstream_tasks/rapidapi.py:163testdata/corpus/ToolBench/toolbench/inference/server.py:116

    Fix: Resolve to a canonical path and verify containment; reject ../ and symlinks.

  • Warn
    No SSRF sinksguard20 occurrences

    Fetching tool-supplied URLs can pivot into internal networks and metadata services.

    testdata/corpus/ToolBench/data_example/toolenv/tools/Artificial_Intelligence_Machine_Learning/b2b_sales_forecasting/api.py:23testdata/corpus/ToolBench/data_example/toolenv/tools/Business/contacts_api/api.py:23testdata/corpus/ToolBench/data_example/toolenv/tools/Commerce/amazon_data_scapper/api.py:23testdata/corpus/ToolBench/toolbench/inference/Downstream_tasks/rapidapi.py:351

    Fix: Allow-list destinations; reject arbitrary/loopback/link-local URLs.

  • Warn
    No known-vulnerable dependencies113 of 710 runtime deps vulnerable; worst critical

    Runtime dependencies (parsed from the lockfile) are scanned against OSV.dev for published CVEs. Advisory: flagged dependencies lower the score but don't hard-cap it, since transitive reachability is unproven.

    github.com/cloudflare/[email protected] — GHSA-2x5j-vhc8-9cwmgithub.com/cloudflare/[email protected] — GHSA-9763-4f94-gfchgithub.com/cloudflare/[email protected] — GHSA-q9hv-hpm4-hj6xgithub.com/cloudflare/[email protected] — GO-2024-2453

    Fix: Upgrade the flagged dependencies past their fixed versions.

  • Pass
    No unsafe deserializationguardno sinks found

    pickle/yaml.load/etc. on untrusted data can execute code.

  • Pass
    No committed secretsguardno secrets found

    Hardcoded keys/tokens in published source are live credentials an attacker can use.

  • Pass
    Credentials sourced from environmentreads credentials from environment

    Reading secrets from env/secret stores avoids hardcoding them.

  • Pass
    Dependencies pinned (lockfile)lockfile present

    A lockfile makes installs reproducible and resistant to silent dependency swaps.

  • Pass
    Actively maintainedrecent commits

    Unmaintained servers don't receive security fixes.

  • Pass
    Repository not archivedguardactive

    Archived repositories will never be patched.

  • Pass
    Declares a licenseApache-2.0

    A clear license is required for legal enterprise use.

  • Pass
    Has a security policySECURITY.md present

    A SECURITY.md gives a private path to report vulnerabilities.

  • Pass
    Adoption & popularityestablished adoption

    A small, capped nudge from stars/downloads — widely-used servers get more eyes on bugs. It can never offset a real security failure.

  • Not checked
    Signed releasesnot evaluated

    Signed releases let consumers verify artifacts weren't tampered with.

  • N/A
    No install/post-install scriptsguardno published package

    install hooks run arbitrary code on every consumer at install time.

  • N/A
    Package name not typosquattingguardno published package

    Names mimicking popular packages are a common malware delivery vector.

  • N/A
    Published with provenanceno published package

    Build provenance attests the artifact was built from the claimed source by CI.

  • N/A
    Established maintainerno published package

    Brand-new / single anonymous maintainers raise takeover and malware risk.

Transport & Trust Model

  • Warn
    Network Exposurebinds 0.0.0.0

    Binding 0.0.0.0 or exposing debug inspectors widens the attack surface.

    Fix: Bind to 127.0.0.1 by default; never ship open debug endpoints.

  • Pass
    Execution Sandboxingcontainerized

    A container/sandbox image limits blast radius; a server that runs natively has full host access.

  • N/A
    Transport Encryption (TLS)guardno remote endpoints

    Plaintext HTTP exposes traffic and bearer tokens to interception.

  • N/A
    IAM / Authentication Scopingno remote endpoints

    OAuth 2.1 / Protected Resource Metadata gates who can invoke tools.

  • N/A
    Live Endpoint Reachablenot dynamically scanned

    A dynamic scan connected to the declared remote endpoint and it responded — verified live, not a dead URL.

  • N/A
    Authentication Enforced (live)not dynamically scanned

    If the server declares auth is required, it must actually reject anonymous clients. Serving tools to unauthenticated callers is a real exposure.

Tools (41)

sumrespondMy StatesummarizeAdd IncomeTask BoardAdd ExpenseCreate TaskSpending ChartEmail AnalyticsPackage TrackerUpdate My StateMeeting CalendarUpdate Task StatusAI Email ClassifierDaily Email SummaryFinancial DashboardHuman Readable NameTodo Item ExtractorSubscription ManagerFinance Email TrackerArchive Old NewslettersEmail Thread VisualizerExample Action TemplateSmart Urgency DetectionAuto-Archive NewslettersPackage Tracking WatcherBoss Urgent Email WatcherExample AI Email ListenerMeeting Request ExtractorContact Relationship TrackerSummarize CEO Weekly UpdatesAI Finance Email Auto-LabelerSend Payment Reminder to VendorForward Bug Report to EngineeringForward Bug Reports to EngineeringSend Payment Reminder to ACME CorpArchive Old Newsletters from SourcesLabel Urgent Customer Support Emailsarg or the method name, description from theargument (falling back to the method name), description from its
io.github.trustabl/agent-reliability-analyzer v0.1.12 — 14/100 · Canopii Trust Index