A Clean Scan Is a Starting Point: Assessing AI Tools and MCP Servers

A team discovers an MCP server that promises convenient access to business data. Its demonstration looks convincing, the tool descriptions sound reasonable, and a scanner reports no findings. The team approves the connection.

Two weeks later, the server changes.

Which version did the organization approve, and which behavior did it actually evaluate?

The Model Context Protocol provides a way to connect AI applications with capabilities such as tools and data resources. Using that protocol does not, by itself, establish the trustworthiness of the server, its maintainer, or the systems behind it.

Supply-chain assessment should therefore begin with provenance and purpose. Identify who maintains a component, where it runs, how it is updated, which dependencies it includes, and what data leaves the organization. For a hosted service, establish what evidence the provider can supply and which changes customers can observe or control.

My suggested admission process separates description, implementation, and behavior. A tool description says what the tool claims to do. Code and configuration can reveal how it works when available for review. Controlled execution can show what happens under selected conditions. None of these views completely replaces the others.

Cisco's open-source MCP Scanner illustrates how several assessment methods can contribute. Its documented options include YARA rules, LLM-based analysis, the Cisco AI Defense inspection API, and modes for examining exposed MCP content, available source code, and dependencies. Coverage depends on the selected mode, configured analyzers, and accessible material. A metadata-only scan cannot establish everything the server's code might do. Cisco MCP Scanner

Treat findings as evidence for a decision. An absence of findings means the configured checks did not identify a problem in the material examined. It does not establish that every future response or tool execution will be safe.

For a hypothetical customer-record connector, I would first document the permitted use: retrieve a limited set of fields for an assigned account. Then I would test whether requests for another account fail, whether the connector attempts unexpected network connections, and whether an update changes its permissions or returned content.

Run unfamiliar components in an isolated assessment environment with synthetic data and restricted credentials. This applies to the assessment process itself: a scanner mode that launches a local server executes that server, whereas an offline metadata scan has a different exposure.

The approval record should identify the artifact or service version where available, configuration, tool inventory, assessment date, findings, accepted limitations, reviewer, and reassessment triggers. For self-managed software, pin reviewed versions and track updates. For hosted services that cannot be pinned, use contractual change information, monitoring, and repeat evaluations to manage that uncertainty.

Extend the same discipline to models, retrieval sources, agent instructions, and other dependencies. A model replacement may change workflow behavior even when its API stays compatible. A document repository can introduce new content without changing the model at all.

For an initial rollout, create a small approved catalog with named maintainers and explicit reasons for admission. Make removal and rollback part of the catalog process, so that discovering a problem produces an actionable response.

The useful question is broader than whether a scanner passed: what evidence supports using this exact capability, for this task, with this level of access—and what change would invalidate that decision?