Shipping changes to an MCP server that potentially dozens of agents across the organization depend on needs the same rigor as any critical shared API: automated checks that catch breaking changes before merge, a rollout strategy that limits the blast radius of anything that slips through, and a clear-eyed understanding of what an automated rollback actually undoes, and what it does not.
Schema Contract Checks as a Merge Gate
# scripts/check_schema_contract.py: run in CI on every pull request
import json, sys
def diff_is_breaking(old_schema: dict, new_schema: dict) -> list[str]:
breaks = []
old_required = set(old_schema.get("required", []))
new_required = set(new_schema.get("required", []))
for field in new_required - old_required:
breaks.append(f"New required field added without version bump: '{field}'")
for field, old_prop in old_schema["properties"].items():
if field not in new_schema["properties"]:
breaks.append(f"Field removed: '{field}'")
elif old_prop.get("type") != new_schema["properties"][field].get("type"):
breaks.append(f"Field type changed: '{field}'")
return breaks
if __name__ == "__main__":
published = json.load(open("contracts/published_schemas.json"))
current = json.load(open("contracts/current_schemas.json")) # generated from live tools/list
all_breaks = []
for tool_name, old_schema in published.items():
if tool_name in current and (breaks := diff_is_breaking(old_schema, current[tool_name])):
all_breaks.extend(f"{tool_name}: {b}" for b in breaks)
if all_breaks:
print("BLOCKED: breaking schema changes detected without a new tool version:")
for b in all_breaks:
print(f" - {b}")
sys.exit(1)# .github/workflows/ci.yml
name: MCP Server CI
on: [pull_request]
jobs:
contract-and-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements.txt
- name: Contract test (schema shape, Lesson 26)
run: pytest tests/contract/
- name: Schema diff gate (breaking-change detection)
run: python scripts/check_schema_contract.py
- name: Integration tests (mocked, Lesson 26)
run: pytest tests/integration/Canary Rollout with Automated Rollback
flowchart TD
Deploy["Deploy new version\nalongside stable"] --> Canary10["Route 10% traffic\nto canary"]
Canary10 --> Monitor1{"Error rate & p95 latency\nwithin threshold?"}
Monitor1 -->|no| Rollback["Automatic rollback:\nroute 100% back to stable,\nalert owning team"]
Monitor1 -->|yes| Canary50["Route 50% traffic\nto canary"]
Canary50 --> Monitor2{"Still within threshold?"}
Monitor2 -->|no| Rollback
Monitor2 -->|yes| Full["Route 100% to new version,\nretire old"]
style Deploy fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style Canary10 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style Canary50 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style Monitor1 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style Monitor2 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style Rollback fill:#fef2f2,stroke:#dc2626,color:#0F172A
style Full fill:#f0fdf9,stroke:#0D9488,color:#0F172A# Argo Rollouts: canary strategy with automated analysis and rollback
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: enterprise-mcp-server
spec:
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 10m }
- analysis:
templates: [{ templateName: mcp-error-rate-check }]
- setWeight: 50
- pause: { duration: 10m }
- analysis:
templates: [{ templateName: mcp-error-rate-check }]
- setWeight: 100
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: mcp-error-rate-check
spec:
metrics:
- name: error-rate
interval: 2m
successCondition: result < 0.02 # Roll back automatically above 2% tool-call error rate
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: sum(rate(mcp_tool_call_errors_total{version="canary"}[2m])) / sum(rate(mcp_tool_calls_total{version="canary"}[2m]))What a Rollback Undoes, and What It Does Not
It is important to be precise about what an automated rollback actually accomplishes, since it is easy to treat “rollback complete” as synonymous with “incident resolved.” A version rollback changes which code is currently serving traffic, it has no mechanism to retroactively reverse real-world side effects that a buggy canary version already produced while it was live and serving that 10% (or 50%) of traffic. If the canary’s bug was in a write tool, some real records were genuinely updated incorrectly during the window before the rollback triggered, and rolling back the deployment does nothing to fix those specific records, they remain wrong until someone identifies and corrects them explicitly.
flowchart LR
Bug["Canary version bug\nin a write tool"] --> Writes["Real writes committed\nduring 10% traffic window"]
Rollback["Automated rollback\n(Argo Rollouts)"] -.->|"reverts running CODE"| CodePath["Traffic now on\nstable version"]
Rollback -.->|"does NOT revert"| Writes
Writes --> Audit["Audit log (Lesson 21):\nfilter by version=canary\n+ tool=write_tool_name\n+ time window"]
Audit --> Remediate["Identify and correct\nthe specific affected records\n(separate incident response step)"]
style Bug fill:#fef2f2,stroke:#dc2626,color:#0F172A
style Writes fill:#fef2f2,stroke:#dc2626,color:#0F172A
style Rollback fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style CodePath fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style Audit fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style Remediate fill:#fff7ed,stroke:#f59e0b,color:#0F172AThis is precisely where the audit logging built in Lesson 21 earns its keep beyond security investigation: filtering the audit log by the affected tool name, the canary’s version tag, and the exact time window the canary was live gives an incident responder the specific, bounded list of writes that need review, rather than an undifferentiated search across the entire downstream system with no starting point. A complete incident runbook for a canary rollback affecting a write tool should therefore include this as an explicit follow-up step, not treat “traffic is back on the stable version” as the end of the response.
The final lesson brings every piece from this course together: designing and describing a complete enterprise MCP ecosystem end to end, from a single server’s first tool through federation, security, and production operations.