When evaluating an API’s readiness for LLM integration via the Model Context Protocol (MCP), we analyze four critical benchmarks:
- Developer Experience
- System Reliability
- Security & Safety
- Semantic Discovery
Experience
The API suffers from poor actionability due to descriptions referencing external APIs (‘use api-config to get all greetings templates’). Documentation quality is inconsistent, with examples sometimes containing hallucinated fields. Sonarqube reports numerous bugs and vulnerabilities, indicating poor code quality. The high number of failed Newman tests also reflects negatively on the developer experience.
Reliability
Response formats are somewhat consistent, using a standard response envelope. However, error responses sometimes lack specific error codes and descriptions, making them less machine-readable. The Newman tests indicate a high failure rate (42 failed tests), negatively impacting reliability. The use of generic ‘Ok’ descriptions for various responses is also problematic.
Security & Safety
The API uses OAuth2 for security, which is machine-friendly. Security types are defined, and the security model seems reasonable. However, the documentation lacks explicit details on scopes and their implications, which could improve agent safety.
Semantic Discovery
The API has several schemas defined, but some examples contain fields not defined in the schema (hallucination trap). The ‘greeting’ schema is well-defined, but the presence of ‘object’ types without properties in error responses and the use of external references for valid values reduce machine interpretability. Composability is hindered by the lack of clear resource identifiers beyond simple IDs.
AI Readiness scoring in APIQuality
Calculating your API’s AI-readiness score is effortless. Simply enable the assessment stage within APIQuality to unlock instant, actionable insights.
Do you want to be sure that you can expose your APIs to MCP safely?