HomeUse CasesConceptsConfig GuidesDeveloper GuidesAPI Reference
Content

Troubleshooting

IBM Verify MCP Server - Troubleshooting Guide

This guide helps you diagnose and resolve common issues when using IBM Verify MCP Server with MCP clients like Claude Desktop and IBM Bob.

Table of Contents

  1. Connection Issues
  2. Authentication Problems
  3. Server Not Running
  4. Tool Execution Failures
  5. Performance Issues
  6. Configuration Problems
  7. Docker-Specific Issues
  8. Kubernetes-Specific Issues
  9. Collecting Diagnostics
  10. Getting Support
  11. Quick Reference

Connection Issues

MCP Client Cannot Connect to Server

Symptoms:

  • Connection timeout errors
  • "Server not reachable" messages
  • HTTP 502/503/504 errors
  • Client cannot discover tools

Diagnostic Steps:

  1. Verify server is running:

    # Docker
docker ps | grep verify-mcp-server

# Kubernetes
kubectl get pods -n verify-mcp-server -l app=verify-mcp-server

  2. Test health endpoint:

    # Docker (local)
curl http://localhost:8000/health

# Kubernetes (external)
curl https://your-mcp-server-url.com/health

  3. Check MCP endpoint:

    # Docker
curl http://localhost:8000/mcp

# Kubernetes
curl https://your-mcp-server-url.com/mcp

Common Solutions:

  1. Incorrect URL in client configuration:

    • Verify the URL includes the /mcp path
    • Example: https://verify-mcp-server.example.com/mcp
    • For local Docker: http://localhost:8000/mcp

  2. Incorrect client configuration — ensure your config matches the correct format. Example for local development (OAuth, end user):

    {
  "mcpServers": {
    "ibm-verify": {
      "command": "uvx",
      "args": [
        "[email protected]",
        "http://localhost:8000/mcp",
        "--header",
        "persona: end_user"
      ]
    }
  }
}

    For the full set of configuration options, see the Configuring MCP Clients guide.

  3. Firewall or network policy blocking traffic:

    • Check firewall rules
    • Verify network policies in Kubernetes
    • Test from different network

  4. TLS certificate issues:

Client Not Registered Error

Symptoms:

  • Error message: "Client Not Registered"
  • Error message: "The client ID was not found in the server's client registry"
  • MCP client fails to connect after initial setup
  • Connection works initially but fails after some time

Diagnostic Steps:

  1. Check for stale authentication cache:

    # Check if mcp-auth cache exists
ls -la ~/.mcp-auth

  2. Review client logs:

    • Check MCP client logs for "Client Not Registered" errors
    • Look for client ID mismatches

Common Solutions:

  1. Clear MCP authentication cache:

    # On MCP Client system (IBM Bob or Claude Desktop)
rm -rf ~/.fastmcp

  2. Restart MCP client:

    • After clearing the cache, restart your MCP client (IBM Bob or Claude Desktop)
    • The client re-register with the server on next connection

  3. Verify client configuration:

    • Ensure the client configuration hasn't changed
    • Check that the server URL is still correct
    • Verify OAuth client metadata is properly configured

  4. Check server-side client registry:

    # Docker
docker logs verify-mcp-server | grep -i "client.*register"

# Kubernetes
kubectl logs -n verify-mcp-server <pod-name> | grep -i "client.*register"

Note: This issue commonly occurs when:

  • The MCP server has been restarted or redeployed
  • Client authentication tokens have expired
  • There's a mismatch between cached client data and server registry

OAuth Discovery Fails Due to Incorrect MCP Base URL

Symptoms:

  • Connection fails during OAuth discovery
  • 401 Unauthorized with WWW-Authenticate header
  • Error fetching Protected Resource Metadata
  • Error fetching authorization server metadata
  • "The operation was aborted due to timeout"
  • "Connection error: fetch failed"

Example logs:

Received 401 with WWW-Authenticate header

resource_metadata="https://verify-mcp-server.svc.cluster.local:8000/.well-known/oauth-protected-resource/mcp"

Error fetching Protected Resource Metadata
The operation was aborted due to timeout

Error fetching authorization server metadata
The operation was aborted due to timeout

Connection error: fetch failed

Root Cause:

The MCP server is advertising OAuth metadata by using an internal Kubernetes or OpenShift service URL (*.svc.cluster.local) instead of the externally accessible route.

Example of problematic URL:

https://verify-mcp-server.svc.cluster.local:8000

The OAuth discovery process uses the URLs returned by the MCP server. If those URLs are only reachable from inside the cluster, external MCP clients cannot complete OAuth discovery, Dynamic Client Registration (DCR), or token acquisition.

Diagnostic Steps:

  1. Check the configured MCP Base URL:

    # Kubernetes — check base_url in config.yaml mounted via verify-mcp-files ConfigMap
kubectl get configmap -n verify-mcp-server verify-mcp-files -o yaml | grep base_url

# Or if using verify-mcp-config ConfigMap reference:
kubectl get configmap -n verify-mcp-server verify-mcp-config -o yaml 2>/dev/null || true

# Docker (local)
grep IBM_VERIFY_BASE_URL .env 2>/dev/null || grep base_url config/config.yaml

  2. Verify OAuth Protected Resource Metadata endpoint:

    curl -s https://<mcp-base-url>/.well-known/oauth-protected-resource/mcp

  3. Check authorization server metadata endpoint:

    curl -s https://<mcp-base-url>/.well-known/oauth-authorization-server

Common Solutions:

  1. Configure correct MCP Base URL:

    The MCP Base URL should point to the public route exposed by OpenShift or Kubernetes, for example:

    https://verify-mcp-server-dev.5le8.p1.openshiftapps.com

    NOT an internal service URL such as:

    https://verify-mcp-server.svc.cluster.local:8000

  2. Update config.yaml in ConfigMap with correct URL:

    Update base_url under ibm_verify in config.yaml to the public route URL:

    ibm_verify:
  base_url: https://verify-mcp-server.example.com

    Re-apply the verify-mcp-files ConfigMap:

    # Kubernetes / OpenShift
kubectl create configmap verify-mcp-files \
  --namespace=verify-mcp-server \
  --from-file=config.yaml=./config.yaml \
  --dry-run=client -o yaml | kubectl apply -f -

  3. Restart deployment to apply changes:

    # Kubernetes
kubectl rollout restart deployment -n verify-mcp-server verify-mcp-server

# OpenShift
oc rollout restart deployment -n verify-mcp-server verify-mcp-server

Validation Steps:

  1. Verify OAuth Protected Resource Metadata returns public URLs:

    curl -s https://<public-route>/.well-known/oauth-protected-resource/mcp

    Expected response:

    {
  "resource": "https://<public-route>/mcp",
  "authorization_servers": [
    "https://<public-route>"
  ]
}

    NOT expected (internal URLs):

    {
  "resource": "https://<service-name>.<namespace>.svc.cluster.local:8000/mcp",
  "authorization_servers": [
    "https://<service-name>.<namespace>.svc.cluster.local:8000"
  ]
}

  2. Verify authorization server metadata:

    curl -s https://<public-route>/.well-known/oauth-authorization-server

    Ensure all advertised endpoints use the public route and are reachable from the client machine.

  3. Test MCP client connection:

    • Restart your MCP client (IBM Bob or Claude Desktop)
    • Attempt to connect to the MCP server
    • OAuth discovery should now succeed

Important Notes:

  • A 401 Unauthorized response from the MCP endpoint is expected during OAuth discovery and does not indicate a problem by itself.
  • The issue occurs when the OAuth metadata returned by the server contains internal cluster URLs that cannot be reached by the MCP client.
  • Always use the externally accessible route URL for IBM_VERIFY_BASE_URL when deploying in Kubernetes or OpenShift.

Authentication Problems

Login to IBM Verify Fails

Symptoms:

  • Authentication failures
  • 401 Unauthorized or 403 Forbidden errors
  • Login redirect loops
  • "Invalid client" errors

Diagnostic Steps:

  1. Verify tenant URL is accessible:

    curl https://your-tenant.verify.ibm.com/oidc/endpoint/default/.well-known/openid-configuration

  2. Check credentials and configuration:

    # Kubernetes — verify Secret keys exist (does NOT print values)
kubectl get secret -n verify-mcp-server verify-mcp-credentials -o jsonpath='{.data}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(list(d.keys()))"
kubectl get configmap -n verify-mcp-server verify-mcp-files -o yaml
    ⚠️

    Do not use -o yaml on a Secret in production — it prints base64-encoded credentials to the terminal and any attached log collectors.

  3. Review authentication logs:

    # Docker
docker logs verify-mcp-server | grep -i "auth\|token\|oauth"

# Kubernetes
kubectl logs -n verify-mcp-server -l app=verify-mcp-server | grep -i "auth\|token\|oauth"

Common Solutions:

  1. Incorrect OAuth credentials:

    • Verify Subject application client ID and secret match IBM Verify Admin Console.
    • For Kubernetes: check keys in verify-mcp-credentials Secret (client-id, client-secret, sts-client-id, sts-client-secret, actor-client-id, actor-client-secret).

  2. Wrong redirect URI:

    • Ensure base_url matches the actual external server URL.
    • For Docker: http://localhost:8000
    • For Kubernetes: https://your-mcp-server-url.com
    • Verify redirect URI is configured in IBM Verify application.

  3. User not entitled to application:

    • Go to IBM Verify Admin Console
    • Navigate to Applications → MCP Server Subject
    • Go to Entitlement tab
    • Add the user or group

  4. Missing or incorrect scopes:

    • Minimum required scope: openid
    • Verify scope in client configuration matches application configuration

Token Exchange Fails

Symptoms:

  • "Token exchange failed" errors
  • Tools cannot be executed after login
  • invalid_grant errors

Diagnostic Steps:

  1. Verify All Client ID and Secrets configuration: 
    # Kubernetes — list secret key names only (does NOT print values)
kubectl get secret -n verify-mcp-server verify-mcp-credentials -o jsonpath='{.data}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(list(d.keys()))"

# Docker — confirm non-sensitive config only; never pipe full `env` output
docker exec verify-mcp-server env | grep "IBM_VERIFY_TENANT_URL"
    ⚠️

    Do not use kubectl get secret -o yaml or docker exec … env without filtering — both commands print credentials to the terminal.

Common Solutions:

  1. Incorrect STS credentials:

    • Verify STS client ID and secret (sts-client-id, sts-client-secret in K8s Secret or IBM_VERIFY_STS_CLIENT_ID in Docker).
    • Verify STS client is enabled in IBM Verify admin console.

  2. Actor application not configured:

    • Verify Actor client ID and secret (actor-client-id, actor-client-secret in K8s Secret or IBM_VERIFY_ACTOR_CLIENT_ID in Docker).

  3. STS client not enabled:

    • Check STS client is enabled in IBM Verify admin console
    • Verify token exchange grant type is enabled

OAuth Error - Token Exchange with Identity Provider Failed: invalid_client (CSIAQ0155E)

Symptoms:

  • Error: OAuth error - Token exchange with identity provider failed: invalid_client
  • Error code CSIAQ0155E: Client could not be authenticated
  • Login completes but subsequent token exchange fails

Cause:

One or more of the OAuth client credentials configured in the server deployment (Subject, Actor, or STS) do not match the corresponding application definitions in IBM Verify.

Resolution:

Review and correct the credentials in your server configuration (Kubernetes Secret verify-mcp-credentials ):

Kubernetes Secret KeyDocker Env VariableIBM Verify ApplicationField in Admin Console
client-idclient-idMCP Server SubjectClient ID
client-secretclient-secretMCP Server SubjectClient secret
actor-client-idactor-client-idMCP Server ActorClient ID
actor-client-secretactor-client-secretMCP Server ActorClient secret
sts-client-idsts-client-idMCP Server STSClient ID
sts-client-secretsts-client-secretMCP Server STSClient secret

Steps to verify:

  1. In the IBM Verify Admin Console, navigate to Applications and open each of the three applications (Subject, Actor, STS).
  2. Cross-check the Client ID and Client secret values against those stored in verify-mcp-credentials (K8s) or .env (Docker).
  3. Correct any mismatches, then redeploy or restart the server.

Redirect URI Mismatch - CSIAQ0167E

Symptoms:

  • Error: Your request cannot be processed
  • Error code CSIAQ0167E: Redirection URI provided in request is either invalid or doesn't match with any of the OAuth 2.0 client pre-registered redirect URIs
  • User is shown an error page after being redirected from IBM Verify

Cause:

The base URL (base_url in config.yaml or IBM_VERIFY_BASE_URL) set in the server deployment does not match a redirect URI registered in the MCP Server Subject application in IBM Verify.

Resolution:

  1. In the IBM Verify Admin Console, navigate to Applications → MCP Server Subject → Sign-on.
  2. Under Redirect URIs, add the public URL of your MCP server deployment. Examples:
    • https://verify-mcp-server.openshiftapps.com
    • https://<hostname>/mcp
  3. Ensure IBM_VERIFY_BASE_URL in your deployment config is set to the same base URL (without a trailing slash).
  4. Save the application and retry the login flow.

Note: Both the hostname and path must match exactly. A redirect to https://example.com/mcp fails if only https://example.com is registered, and vice versa.

Server Not Running

Pod/Container in CrashLoopBackOff

Symptoms:

  • Container repeatedly restarts
  • Pod status shows CrashLoopBackOff or Error
  • Server endpoint unavailable

Diagnostic Steps:

  1. Check container/pod status:

    # Docker
docker ps -a | grep verify-mcp-server

# Kubernetes
kubectl get pods -n verify-mcp-server

  2. Review logs:

    # Docker
docker logs verify-mcp-server

# Kubernetes
kubectl logs -n verify-mcp-server <pod-name>
kubectl logs -n verify-mcp-server <pod-name> --previous

  3. Check configuration and secret references:

    # Kubernetes — inspect ConfigMap (non-sensitive)
kubectl get configmap -n verify-mcp-server verify-mcp-files -o yaml
# Kubernetes — verify Secret key names exist (does NOT print values)
kubectl get secret -n verify-mcp-server verify-mcp-credentials -o jsonpath='{.data}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(list(d.keys()))"

# Docker — inspect mount points only (avoid printing Env block which contains credentials)
docker inspect verify-mcp-server --format '{{range .Mounts}}{{.Source}} → {{.Destination}}{{println}}{{end}}'
    ⚠️

    Do not use kubectl get secret -o yaml or docker inspect … grep Env — both expose credentials in terminal output.

Common Solutions:

  1. Missing required configuration fields:

    • Ensure all required fields are present in config.yaml (see Installation Guide Step 2 & 3)
    • Required fields in config.yaml:
      • ibm_verify.tenant_url
      • ibm_verify.client_id
      • ibm_verify.client_secret
      • ibm_verify.base_url
      • ibm_verify.sts_client_id
      • ibm_verify.sts_client_secret
      • ibm_verify.actor_client_id
      • ibm_verify.actor_client_secret
    • In Kubernetes, verify the referenced secret keys (client-id, client-secret, sts-client-id, sts-client-secret, actor-client-id, actor-client-secret) exist in verify-mcp-credentials.
    • In Docker, verify corresponding variables in .env are set.

  2. Invalid configuration:

    • Check for typos in config.yaml or .env variable names
    • Verify URLs don't have trailing slashes
    • Ensure secrets don't have extra spaces or newlines

  3. Resource constraints:

    # Kubernetes - check resource usage
kubectl top pod -n verify-mcp-server <pod-name>

# Increase memory/CPU limits if needed
kubectl edit deployment -n verify-mcp-server verify-mcp-server

  4. Image pull errors:

    # Verify image exists
docker pull icr.io/ibm-verify/verify-mcp-server:latest

# Check image pull secrets in Kubernetes
kubectl get secrets -n verify-mcp-server

Tool Execution Failures

Tools Discovered But Cannot Execute

Symptoms:

  • Tools appear in client but fail when invoked
  • Timeout errors during execution
  • Permission denied errors
  • Partial results returned

Diagnostic Steps:

  1. Verify tool appears in discovery:

    • Check MCP client shows the tool
    • Verify tool parameters are correct

  2. Check execution logs:

    # Docker
docker logs verify-mcp-server --tail=100 | grep -i "tool\|error"

# Kubernetes
kubectl logs -n verify-mcp-server <pod-name> --tail=100 | grep -i "tool\|error"

  3. Test IBM Verify API connectivity:

    # Docker
docker exec verify-mcp-server curl -v https://your-tenant.verify.ibm.com/v2.0/Users

# Kubernetes
kubectl exec -n verify-mcp-server <pod-name> -- \
  curl -v https://your-tenant.verify.ibm.com/v2.0/Users

Common Solutions:

  1. Invalid tool parameters:

    • Verify required parameters are provided
    • Check parameter formats match documentation
    • See Using MCP Tools for parameter details

  2. User lacks permissions:

    • Verify user has required entitlements in IBM Verify
    • Check user roles and permissions
    • Ensure STS client has correct entitlements configured

  3. IBM Verify API unavailable:

    • Check tenant is accessible
    • Verify network connectivity
    • Check for IBM Verify service outages

  4. Rate limiting:

    • Reduce request frequency
    • Implement delays between requests
    • Check IBM Verify rate limits

Tools Not Appearing

Symptoms:

  • Expected tools don't appear in client
  • Empty tool catalog
  • Tools appear intermittently

Diagnostic Steps:

  1. Check server startup logs:

    # Docker
docker logs verify-mcp-server | grep -i "tool\|register"

# Kubernetes
kubectl logs -n verify-mcp-server <pod-name> | grep -i "tool\|register"

  2. Verify MCP endpoint:

    curl http://localhost:8000/mcp

Common Solutions:

  1. Server not fully started:

    • Wait for server to complete initialization
    • Check health endpoint returns 200 OK

  2. Client configuration error:

    • Verify client configuration is correct
    • Restart MCP client
    • Check client logs for errors

  3. Authorization issues:

    • Ensure user is authenticated
    • Verify OAuth flow completed successfully

Performance Issues

Slow Response Times

Symptoms:

  • High latency
  • Timeouts
  • Slow tool execution

Diagnostic Steps:

  1. Check resource usage:

    # Docker
docker stats verify-mcp-server

# Kubernetes
kubectl top pod -n verify-mcp-server <pod-name>

  2. Test network latency:

    # Test IBM Verify API response time
time curl -s https://your-tenant.verify.ibm.com/v2.0/Users > /dev/null

  3. Review logs for slow operations:

    docker logs verify-mcp-server | grep -i "slow\|timeout\|latency"

Common Solutions:

  1. Insufficient resources:

    # Docker - increase memory
docker stop verify-mcp-server
docker run -d --name verify-mcp-server \
  --memory="2g" --cpus="2" \
  # ... other options

# Kubernetes - edit deployment
kubectl edit deployment -n verify-mcp-server verify-mcp-server

  2. Network latency:

    • Check network connectivity to IBM Verify
    • Consider deploying closer to IBM Verify region
    • Use CDN or caching if applicable

  3. Increase container resources (Kubernetes):

    Horizontal scaling is not yet supported — the server uses in-process session/cache state that is not shared across pods. Increase CPU/memory limits on the existing single pod instead:

    kubectl edit deployment -n verify-mcp-server verify-mcp-server
# Increase resources.limits.cpu and resources.limits.memory

Memory Leak — OOM Crash (verify-mcp-server Pod Hits Memory Limit)

Symptoms:

  • The verify-mcp-server pod is killed with an OOMKilled status after sustained concurrent usage.
  • Memory usage grows steadily as more users access the MCP Server simultaneously and is never released between sessions.
  • Pod restarts automatically (CrashLoopBackOff or OOMKilled) once it hits its configured memory limit (default: 1 GB).
  • All active user sessions are lost when the pod restarts.

Root Cause:

Memory allocated per user session is not released when sessions are idle or when a user disconnects. Under concurrent load, cumulative in-memory session state accumulates until the pod exhausts its memory limit and is killed by the OS/Kubernetes OOM killer.

Solution / Workaround:

There is currently no automatic session cleanup between user disconnections. When the pod restarts after an OOMKilled event, all users must re-authenticate by logging in to the MCP Server again.

Note: A permanent fix requires implementing server-side session lifecycle management so that memory is released when a user session ends. Until that fix is released, the provided workaround applies.

Configuration Problems

Invalid Configuration Errors

Symptoms:

  • Server fails to start
  • Configuration validation errors
  • Missing required parameters

Diagnostic Steps:

  1. Verify configuration and secrets:

    # Kubernetes — ConfigMap is non-sensitive and safe to print
kubectl get configmap -n verify-mcp-server verify-mcp-files -o yaml
# Kubernetes — list Secret key names only (does NOT print values)
kubectl get secret -n verify-mcp-server verify-mcp-credentials -o jsonpath='{.data}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(list(d.keys()))"

# Docker
cat config/config.yaml
    ⚠️

    Do not use kubectl get secret -o yaml — it prints base64-encoded credentials to the terminal.

  2. Check for typos:

    • Review parameter and secret key names
    • Verify URLs are correct
    • Check for extra spaces or newlines

Common Solutions:

  1. Verify each setting is configured correctly:

    # Kubernetes — ConfigMap is non-sensitive and safe to print
kubectl get configmap -n "${NAMESPACE}" verify-mcp-files -o yaml
# Kubernetes — list Secret key names only (does NOT print values)
kubectl get secret -n "${NAMESPACE}" verify-mcp-credentials -o jsonpath='{.data}' | python3 -c "import sys,json; d=json.load(sys.stdin); print(list(d.keys()))"

# Docker — inspect the mounted config file (does not contain raw secrets when using !enc / !obf / !envvar)
docker exec verify-mcp-server cat /app/config/config.yaml

  2. Update configuration:

    # Docker — edit config.yaml or .env, then restart the container
docker restart verify-mcp-server

# Kubernetes — update ConfigMap/Secret then rollout restart
kubectl create configmap verify-mcp-files \
  --namespace="${NAMESPACE}" \
  --from-file=config.yaml=./config.yaml \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment -n "${NAMESPACE}" verify-mcp-server

Secret or ConfigMap Resolution Fails (!secret / !configmap) in Kubernetes

Symptoms:

  • Pod enters CrashLoopBackOff or fails during startup
  • Server log messages such as:
    • Failed to resolve !secret verify-mcp-credentials/<key>
    • Permission denied: cannot get resource "secrets"
    • 403 Forbidden accessing Kubernetes API from inside the pod

Causes:

  • The ServiceAccount verify-mcp-server-sa is missing or not assigned to the deployment (spec.template.spec.serviceAccountName)
  • The RBAC Role or RoleBinding granting get and list on secrets and configmaps was not created in the target namespace
  • The referenced Secret or ConfigMap name or field key in config.yaml does not exist or has a typo
  • POD_NAMESPACE environment variable is not passed to the container

Solutions:

  1. Check the pod logs for the exact prefix resolution error:

    kubectl logs -n "${NAMESPACE}" -l app=verify-mcp-server --tail=50

  2. Verify the ServiceAccount, Role, and RoleBinding exist:

    kubectl get sa,role,rolebinding -n "${NAMESPACE}"

    If missing, ensure the RBAC resources from the deployment manifest are applied:

    kubectl apply -f verify-mcp-server.yaml

  3. Verify the referenced secret and key exist:

    kubectl get secret -n "${NAMESPACE}" verify-mcp-credentials -o jsonpath='{.data}'

  4. Ensure POD_NAMESPACE is set in the Deployment spec:

    env:
- name: POD_NAMESPACE
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace

Obfuscation Decryption Fails (!obf)

Symptoms:

  • Server fails to start with an error such as bad decrypt or error:06065064:digital envelope routines
  • Log message: Failed to decrypt obfuscated value or invalid padding
  • Server starts but credentials are rejected by IBM Verify (wrong passphrase decrypts to garbage)

Causes:

  • obf_key from secret is missing, empty, or contains a different passphrase than was used to produce the !obf ciphertext
  • The ciphertext was produced with a different OpenSSL version (for example, OpenSSL 1.0.x vs 1.1.1+) — the -pbkdf2 flag requires 1.1.1 or later
  • Whitespace or newline characters were accidentally included in the ciphertext when copying it into config.yaml or a Kubernetes Secret

Solutions:

  1. Confirm OBF_KEY is set and matches the passphrase used during encryption:

    # Docker — confirm the variable is present (prints name only; do not print value in shared terminals)
docker exec verify-mcp-server env | grep -q OBF_KEY && echo "OBF_KEY is set" || echo "OBF_KEY is NOT set"

# Kubernetes — decode the stored passphrase for comparison (run in a secured terminal session)
kubectl get secret -n "${NAMESPACE}" verify-mcp-credentials \
  -o jsonpath='{.data.obf_key}' | base64 -d
    ⚠️

    The Kubernetes command above prints the passphrase to stdout. Run it only in a secured, non-shared terminal session and clear your history afterwards (history -d $(history 1)).

  2. Test decryption offline with the exact passphrase:

    echo "<your-!obf-ciphertext>" | openssl enc -d -aes256 \
    -pbkdf2 -pass pass:"<your-passphrase>" -md sha512 \
    -base64 | tr -d '\n'; echo

    If this fails locally, the passphrase or ciphertext is wrong — re-generate the ciphertext by using the steps in Encryption & Obfuscation.

  3. Check OpenSSL version:

    openssl version
# Must be 1.1.1 or later

  4. Inspect the ciphertext in config.yaml for stray whitespace — the entire Base64 string after !obf must be on a single line with no leading or trailing spaces.

RSA Decryption Fails (!enc)

Symptoms:

  • Server fails to start with error such as Failed to normalize key secrets.obf_key: Failed to decrypt: crypto/rsa: decryption error
  • Container crashes at startup with RSA decryption failed or error:0407006A:rsa routines
  • Log message: Failed to decrypt !enc value or ENC_KEY not set
  • Pod enters CrashLoopBackOff immediately after adding !enc values

Causes:

  • ENC_KEY is not set, or points to an invalid secret or missing file.
  • Kubernetes: the verify-mcp-enc-key Secret was not created, or ENC_KEY is not set to "!secret verify-mcp-enc-key/private.pem".
  • Docker: the private key is not mounted to /app/secrets/private.pem, or ENC_KEY is not set to "!file /app/secrets/private.pem".
  • The private.pem used for decryption does not match the public.pem used to encrypt the !enc values — each ciphertext can only be decrypted by the corresponding private key.
  • The ciphertext was truncated when copying — Base64-encoded RSA ciphertext for a 2048-bit key is exactly 344 characters.

Solutions:

  1. Verify ENC_KEY configuration:

    # Kubernetes — verify ENC_KEY references the secret
kubectl get deployment -n "${NAMESPACE}" verify-mcp-server \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="ENC_KEY")].value}'
# Should output: !secret verify-mcp-enc-key/private.pem

# Docker — confirm the private.pem file exists at the mounted path
docker exec verify-mcp-server ls -la /app/secrets/private.pem

  2. Confirm the Secret was created and contains the private key (Kubernetes):

    kubectl get secret -n "${NAMESPACE}" verify-mcp-enc-key
kubectl describe secret -n "${NAMESPACE}" verify-mcp-enc-key

  3. Test decryption offline using the private key:

    echo "<your-!enc-ciphertext>" | base64 -d | openssl pkeyutl -decrypt \
    -inkey private.pem \
    -pkeyopt rsa_padding_mode:pkcs1

    If this fails, the private key does not match the public key used during encryption — re-generate the ciphertext with the correct public key, following the steps in Encryption & Obfuscation.

  4. Re-create the Secret with the correct key file:

    kubectl delete secret -n "${NAMESPACE}" verify-mcp-enc-key
kubectl create secret generic verify-mcp-enc-key \
  --namespace="${NAMESPACE}" \
  --from-file=private.pem=./private.pem
kubectl rollout restart deployment -n "${NAMESPACE}" verify-mcp-server

Server Starts But Credentials Are Wrong After Adding Encryption

Symptoms:

  • Server is running and healthy (/health returns 200)
  • IBM Verify rejects authentication with invalid_client or CSIAQ0155E
  • Issue appeared immediately after switching from plain-text to !obf or !enc credentials

Causes:

  • The ciphertext was generated from the wrong value (for example, a test value left in place)
  • The Kubernetes Secret still holds a stale !enc/!obf value from a previous attempt
  • The ConfigMap verify-mcp-files still contains the old config.yaml without the !obf/!enc prefixes

Solutions:

  1. Verify the ConfigMap reflects the latest config.yaml:

    kubectl get configmap -n "${NAMESPACE}" verify-mcp-files -o yaml

  2. Re-create the Secret with freshly generated ciphertexts:

    kubectl delete secret -n "${NAMESPACE}" verify-mcp-credentials
kubectl create secret generic verify-mcp-credentials \
  --namespace="${NAMESPACE}" \
  --from-literal=client-secret='!obf <new-ciphertext>' \
  ...
kubectl rollout restart deployment -n "${NAMESPACE}" verify-mcp-server

  3. Temporarily switch back to plain-text to confirm the credentials themselves are correct, then re-apply encryption after confirmed.

Rotating Encryption / Obfuscation Keys (!enc / !obf)

Use this procedure whenever you need to rotate the RSA key pair (for !enc) or the obfuscation key (for !obf) — for example, as part of a scheduled credential rotation, after a key compromise, or after generating new ciphertext values.

⚠️

Order matters: update secrets before restarting the server. The server reads keys at startup; restarting before the new values are in place will cause decryption failures and a CrashLoopBackOff.

Kubernetes

  1. Generate new ciphertext values for all !enc or !obf fields following the steps in Encryption & Obfuscation.

  2. Update the Kubernetes Secrets with the new ciphertext values:

    # Replace the RSA private key used for !enc decryption
kubectl delete secret -n "${NAMESPACE}" verify-mcp-enc-key
kubectl create secret generic verify-mcp-enc-key \
  --namespace="${NAMESPACE}" \
  --from-file=private.pem=./private.pem

# Update credential secrets with re-encrypted / re-obfuscated values
kubectl delete secret -n "${NAMESPACE}" verify-mcp-credentials
kubectl create secret generic verify-mcp-credentials \
  --namespace="${NAMESPACE}" \
  --from-literal=client-secret='!enc <new-ciphertext>' \
  --from-literal=obf-key='!obf <new-obf-value>'
# Add any other secret keys your config.yaml references

  3. Restart the deployment to pick up the new secrets:

    kubectl rollout restart deployment/verify-mcp-server -n "${NAMESPACE}"

  4. Verify the rollout completed successfully:

    kubectl rollout status deployment/verify-mcp-server -n "${NAMESPACE}"
kubectl get pods -n "${NAMESPACE}" -l app=verify-mcp-server

    All pods should reach Running status with READY 1/1. If any pod enters CrashLoopBackOff, check kubectl logs for decryption errors — the most likely cause is a ciphertext generated with a different key than the one now stored in the Secret.

Docker

  1. Generate new ciphertext values for all !enc or !obf fields following the steps in Encryption & Obfuscation.

  2. Update config.yaml with the new !enc or !obf values:

    # Example — replace the old ciphertext with the newly generated one
client_secret: "!enc <new-ciphertext>"
obf_key: "!obf <new-obf-value>"

  3. Restart the container so it re-reads config.yaml at startup:

    docker restart verify-mcp-server

  4. Confirm the container is healthy:

    docker ps --filter name=verify-mcp-server
docker logs verify-mcp-server --tail 30

    Look for the Server started log line. Any Failed to decrypt or RSA decryption failed message means the new ciphertext was not generated with the correct key — regenerate and repeat from step 1.

Docker-Specific Issues

Container Won't Start

Diagnostic Steps:

  1. Check Docker is running:

    docker info

  2. Verify image exists:

    docker images | grep verify-mcp-server

  3. Check port conflicts:

    lsof -i :8000
netstat -an | grep 8000

Common Solutions:

  1. Port already in use:

    # Stop conflicting service or use different port
docker run -d --name verify-mcp-server \
  -p 8001:8000 \
  # ... other options

  2. Pull latest image:

    docker pull icr.io/ibm-verify/verify-mcp-server:latest

  3. Check Docker logs:

    docker logs verify-mcp-server

Platform Mismatch — no matching manifest for linux/arm64/v8

Symptoms:

docker pull icr.io/ibm-verify/verify-mcp-server:latest
latest: Pulling from ibm-verify/verify-mcp-server
no matching manifest for linux/arm64/v8 in the manifest list entries

This error occurs on Apple Silicon (M1/M2/M3) or other ARM64 machines because the image is published for linux/amd64 only.

Solution:

Add --platform linux/amd64 to your docker pull or docker run command:

# Pull with explicit platform
docker pull --platform linux/amd64 icr.io/ibm-verify/verify-mcp-server:latest

# Run with explicit platform
docker run --platform linux/amd64 -d --name verify-mcp-server \
  -p 8000:8000 \
  # ... other options
  icr.io/ibm-verify/verify-mcp-server:latest

Note: Docker Desktop on Apple Silicon automatically enables Rosetta 2 emulation for linux/amd64 images, so performance is generally acceptable for development and testing.

Cannot Access from Host

Symptoms:

  • curl http://localhost:8000/health fails
  • Connection refused errors

Common Solutions:

  1. Verify port mapping:

    docker ps | grep verify-mcp-server
# Should show: 0.0.0.0:8000->8000/tcp

  2. Check container is running:

    docker ps -a | grep verify-mcp-server

  3. Test from inside container:

    docker exec verify-mcp-server curl http://localhost:8000/health

Kubernetes-Specific Issues

Pod Not Scheduling

Symptoms:

  • Pod stuck in Pending state
  • No nodes available

Diagnostic Steps:

  1. Check pod status:

    kubectl describe pod -n verify-mcp-server <pod-name>

  2. Check node resources:

    kubectl top nodes
kubectl describe nodes

Common Solutions:

  1. Insufficient resources:

    • Reduce resource requests
    • Add more nodes to cluster
    • Scale down other workloads

  2. Node selector issues:

    • Remove or update node selectors
    • Add required labels to nodes

Pod Stuck in Pending — Architecture Mismatch (linux/amd64 image on ARM nodes)

Symptoms:

  • Pod stuck in Pending state with an event similar to: 
    0/3 nodes are available: 3 node(s) didn't match node selector.
  • Image pull succeeds but pod never schedules on clusters with mixed or ARM64 node pools.
  • Occurs on clusters running Apple Silicon (M1/M2/M3) worker nodes or AWS Graviton / Azure ARM64 nodes.

Root cause:

The verify-mcp-server image is built for linux/amd64 only. On a cluster that contains ARM64 nodes (or exclusively ARM64 nodes), Kubernetes may schedule the pod onto an incompatible node, causing it to fail to start or become unschedulable.

Solution — pin the deployment to amd64 nodes:

Add a nodeSelector to the spec of your deployment so the scheduler only considers linux/amd64 nodes:

spec:
  # Security context for the pod
  securityContext: {}

  # Pin to amd64 nodes — image was built for linux/amd64 only
  nodeSelector:
    kubernetes.io/arch: amd64

Apply it to your running deployment with:

kubectl patch deployment verify-mcp-server \
  -n verify-mcp-server-test \
  --type=merge \
  -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/arch":"amd64"}}}}}'

Or edit the deployment YAML directly and re-apply:

kubectl edit deployment verify-mcp-server -n verify-mcp-server-test
# Add nodeSelector: kubernetes.io/arch: amd64 under spec.template.spec

Verify the fix:

# Confirm the node selector is in place
kubectl get deployment verify-mcp-server -n verify-mcp-server-test \
  -o jsonpath='{.spec.template.spec.nodeSelector}'

# Confirm the pod is now Running on an amd64 node
kubectl get pods -n verify-mcp-server-test -o wide

Note: If your cluster has no amd64 nodes at all, you must either add an amd64 node pool or rebuild the image for linux/arm64. See the Docker Platform Mismatch section for guidance on building a multi-arch image.

Service Not Accessible

Symptoms:

  • Cannot reach service from outside cluster
  • Route/Ingress not working

Diagnostic Steps:

  1. Check service:

    kubectl get svc -n verify-mcp-server verify-mcp-server
kubectl describe svc -n verify-mcp-server verify-mcp-server

  2. Check endpoints:

    kubectl get endpoints -n verify-mcp-server verify-mcp-server

  3. Check Route/Ingress:

    # OpenShift
oc get route -n verify-mcp-server verify-mcp-server

# Kubernetes
kubectl get ingress -n verify-mcp-server verify-mcp-server

Common Solutions:

  1. Service selector mismatch:

    kubectl edit svc -n verify-mcp-server verify-mcp-server
# Verify selector matches pod labels

  2. Create or fix Route/Ingress:

  3. Test from within cluster:

    kubectl run -it --rm debug --image=curlimages/curl --restart=Never -n verify-mcp-server -- \
  curl http://verify-mcp-server:8000/health

TLS Certificate Problems

Symptoms:

  • Certificate verification errors
  • x509: certificate signed by unknown authority
  • TLS handshake failures

Diagnostic Steps:

  1. Check certificate:

    openssl s_client -connect your-mcp-server-url.com:443 -showcerts

  2. Verify certificate in pod:

    kubectl exec -n verify-mcp-server <pod-name> -- \
  openssl s_client -connect your-tenant.verify.ibm.com:443 -showcerts

Common Solutions:

  1. Install CA certificates:

    # Create ConfigMap with CA cert
kubectl create configmap ca-certificates \
  --from-file=ca-bundle.crt=/path/to/ca-cert.pem \
  -n verify-mcp-server

# Mount in deployment
kubectl edit deployment -n verify-mcp-server verify-mcp-server

  2. Use valid certificate:

    • Obtain certificate from trusted CA
    • Update Secret with new certificate
    • Restart pods

  3. For lab/testing only — install the CA certificate instead:

    # ⚠️ NEVER disable TLS verification (PYTHONHTTPSVERIFY=0) in any environment.
# Disabling TLS verification removes all protection against MITM attacks and
# exposes OAuth tokens and credentials to interception.
#
# The correct fix is to trust the CA certificate. Create a ConfigMap with the CA cert
# and mount it into the container:
kubectl create configmap ca-certificates \
  --from-file=ca-bundle.crt=/path/to/ca-cert.pem \
  -n verify-mcp-server
# Then mount it at /etc/ssl/certs/ca-bundle.crt in the deployment volumeMounts.

Collecting Diagnostics

Information to Collect

When troubleshooting or reporting issues, collect the following:

  1. Server logs:

    # Docker
docker logs verify-mcp-server > mcp-server-logs.txt

# Kubernetes
kubectl logs -n verify-mcp-server <pod-name> > mcp-server-logs.txt
kubectl logs -n verify-mcp-server <pod-name> --previous > mcp-server-logs-previous.txt

  2. Configuration:

    # Docker
docker inspect verify-mcp-server > docker-inspect.json

# Kubernetes
kubectl get deployment -n verify-mcp-server verify-mcp-server -o yaml > deployment.yaml
kubectl get configmap -n verify-mcp-server verify-mcp-files -o yaml > configmap-files.yaml
kubectl get configmap -n verify-mcp-server verify-mcp-config -o yaml > configmap.yaml 2>/dev/null || true
# List Secret key names only — do NOT save raw Secret values to disk
kubectl get secret -n verify-mcp-server verify-mcp-credentials -o jsonpath='{.data}' \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('Secret keys present:', list(d.keys()))" > secret-keys.txt

  3. Support bundle script:

    Docker / local container:

    Save the following as collect-support-info-docker.sh and run it from your host machine. It connects to the running container, collects logs, stats, health-check output, and environment variables (all secrets and UUIDs are redacted automatically), then packages everything into a .tar.gz.

    #!/bin/bash
# collect-support-info-docker.sh
# Support information collection for IBM Verify MCP Server (standalone Docker)
# Collects logs, metrics, and configuration WITHOUT sensitive data

set -euo pipefail

DEFAULT_CONTAINER="verify-mcp-server"
CONTAINER="${1:-$DEFAULT_CONTAINER}"
OUTPUT_DIR="support-bundle-$(date +%Y%m%d-%H%M%S)"

RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
log_info()  { echo -e "${GREEN}[INFO]${NC}  $1"; }
log_warn()  { echo -e "${YELLOW}[WARN]${NC}  $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }

redact() {
    sed -E \
        -e 's/(SECRET|PASSWORD|TOKEN|KEY|client_secret|authorization|bearer)[=:][^ ]*/\1=<REDACTED>/gi' \
        -e 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/00000000-0000-0000-0000-000000000000/gi'
}

cexec() { docker exec "$CONTAINER" sh -c "$1" 2>/dev/null || true; }

# ---- Preflight -----------------------------------------------------------
if ! command -v docker &>/dev/null; then
    log_error "docker not found."; exit 1
fi
if ! docker inspect "$CONTAINER" &>/dev/null; then
    log_error "Container '$CONTAINER' not found."
    echo "Running containers:"; docker ps --format "  {{.Names}}  ({{.Status}})"
    exit 1
fi

# ---- Setup ---------------------------------------------------------------
mkdir -p "$OUTPUT_DIR"
log_info "Output directory : $OUTPUT_DIR"
log_info "Container        : $CONTAINER"

# ---- 1. Docker host metadata ---------------------------------------------
log_info "Collecting Docker host info..."
docker version                        > "$OUTPUT_DIR/docker-version.txt"     2>/dev/null || true
docker info                           > "$OUTPUT_DIR/docker-info.txt"         2>/dev/null || true
docker inspect "$CONTAINER" | redact  > "$OUTPUT_DIR/container-inspect.json" 2>/dev/null || true

# ---- 2. Container status & stats -----------------------------------------
log_info "Collecting container status..."
docker ps -a --filter "name=$CONTAINER"           > "$OUTPUT_DIR/container-status.txt" 2>/dev/null || true
docker stats "$CONTAINER" --no-stream --no-trunc  > "$OUTPUT_DIR/container-stats.txt"  2>/dev/null || \
    log_warn "Stats unavailable (container may be stopped)"

# ---- 3. Application logs -------------------------------------------------
log_info "Collecting stdout/stderr logs (last 5000 lines)..."
docker logs "$CONTAINER" --tail 5000 2>&1 | redact > "$OUTPUT_DIR/logs-stdout.txt" || \
    log_warn "Failed to collect stdout logs"

log_info "Collecting file-based logs (/app/logs/mcp-server.log)..."
cexec "cat /app/logs/mcp-server.log 2>/dev/null || cat /app/logs/mcp.log 2>/dev/null" | redact > "$OUTPUT_DIR/logs-mcp-server.log"

for i in 1 2 3 4 5; do
    content=$(cexec "cat /app/logs/mcp-server.log.$i 2>/dev/null || cat /app/logs/mcp.log.$i 2>/dev/null")
    if [ -n "$content" ]; then
        echo "$content" | redact > "$OUTPUT_DIR/logs-mcp-server.log.$i"
        log_info "Collected rotated log: mcp-server.log.$i"
    fi
done

# ---- 4. Health check -----------------------------------------------------
log_info "Checking health endpoint..."
cexec "curl -sf http://localhost:8000/health" > "$OUTPUT_DIR/health-check.json" || \
    log_warn "Health endpoint did not respond"

# ---- 5. Environment variables (redacted) ---------------------------------
log_info "Collecting env vars (redacted)..."
docker inspect "$CONTAINER" \
    --format '{{range .Config.Env}}{{println .}}{{end}}' 2>/dev/null | \
    redact | sort > "$OUTPUT_DIR/env-vars.txt" || \
    log_warn "Failed to collect env vars"

# ---- 6. System diagnostics inside container ------------------------------
log_info "Collecting system diagnostics from container..."
cexec "free -h 2>/dev/null || cat /proc/meminfo"                                > "$OUTPUT_DIR/sys-memory.txt"
cexec "df -h"                                                                   > "$OUTPUT_DIR/sys-disk.txt"
cexec "uptime"                                                                  > "$OUTPUT_DIR/sys-uptime.txt"
cexec "ps aux 2>/dev/null || ps -ef"                                           > "$OUTPUT_DIR/sys-processes.txt"
cexec "netstat -tuln 2>/dev/null || ss -tuln 2>/dev/null || cat /proc/net/tcp" > "$OUTPUT_DIR/sys-netstat.txt"
cexec "cat /proc/cpuinfo | grep -E 'processor|model name|cpu MHz' | head -20"  > "$OUTPUT_DIR/sys-cpu.txt"

# ---- 7. Summary ----------------------------------------------------------
log_info "Generating summary..."
{
cat <<SUMMARY
IBM Verify MCP Server - Docker Support Bundle
=============================================
Generated    : $(date)
Container    : $CONTAINER
Collected by : $(whoami)@$(hostname)

CONTAINER STATUS
----------------
$(docker ps -a --filter "name=$CONTAINER" 2>/dev/null)

CONTAINER STATS
---------------
$(docker stats "$CONTAINER" --no-stream 2>/dev/null || echo "N/A")

HEALTH CHECK
------------
$(cat "$OUTPUT_DIR/health-check.json" 2>/dev/null || echo "N/A")

LAST 20 LOG LINES
-----------------
$(tail -20 "$OUTPUT_DIR/logs-stdout.txt" 2>/dev/null)

FILES COLLECTED
---------------
$(ls -lh "$OUTPUT_DIR")

NOTES
-----
- All secrets, tokens, and UUIDs have been redacted
- Logs capped at 5000 lines
- Share this bundle with your IBM Support case number
SUMMARY
} > "$OUTPUT_DIR/SUMMARY.txt"

# ---- 8. Compress ---------------------------------------------------------
log_info "Compressing bundle..."
tar -czf "$OUTPUT_DIR.tar.gz" "$OUTPUT_DIR"

BUNDLE_SIZE=$(du -h "$OUTPUT_DIR.tar.gz" | cut -f1)
echo ""
echo "================================================"
echo "  Bundle   : $OUTPUT_DIR.tar.gz"
echo "  Size     : $BUNDLE_SIZE"
echo "  Location : $(pwd)/$OUTPUT_DIR.tar.gz"
echo "================================================"
echo ""

read -rp "Remove uncompressed directory? (y/N) " reply
if [[ "$reply" == "y" || "$reply" == "Y" ]]; then
    rm -rf "$OUTPUT_DIR"
    log_info "Removed."
fi

log_info "Done."

    Run it:

    chmod +x collect-support-info-docker.sh

# default container name (verify-mcp-server-local)
./collect-support-info-docker.sh

# custom container name
./collect-support-info-docker.sh <your-container-name>

    Kubernetes:

    Option A — copy from a running container (preferred):

    docker cp verify-mcp-server:/app/support/scripts/collect-support-info.sh ./collect-support-info.sh
chmod +x ./collect-support-info.sh
./collect-support-info.sh <correct-namespace>

    Option B — stream via kubectl exec when docker cp is not available:

    kubectl exec -n <namespace> <pod-name> -- \
    cat /app/support/scripts/collect-support-info.sh > collect-support-info.sh
chmod +x ./collect-support-info.sh
./collect-support-info.sh <correct-namespace>

    The script generates a support bundle .tar.gz file in your current directory.

    Note: Before sharing the bundle with IBM Support, review its contents and remove or redact any sensitive information such as secrets, tokens, passwords, private keys, or internal hostnames that should not leave your environment.

  4. Environment details:

    # Docker version
docker version

# Kubernetes version
kubectl version

# OS information
uname -a

  5. Network diagnostics:

    # Test connectivity
curl -v http://localhost:8000/health
curl -v https://your-tenant.verify.ibm.com/oidc/endpoint/default/.well-known/openid-configuration

# DNS resolution
nslookup your-mcp-server-url.com

  6. Client configuration:

    • MCP client configuration file (redact secrets)
    • Client version
    • Client logs if available

Enable Debug Logging

  1. Update config.yaml:
    Set logging.level to DEBUG:

    logging:
  level: DEBUG

    • Kubernetes: update the ConfigMap and rollout restart:

      kubectl create configmap verify-mcp-files \
  --namespace=verify-mcp-server \
  --from-file=config.yaml=./config.yaml \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deployment -n verify-mcp-server verify-mcp-server

    • Docker: restart the container to reload config.yaml:

      docker restart verify-mcp-server

  2. Or through environment variable (Docker):

    # Add or update logging level environment variable if referenced in config.yaml.
docker run -d --name verify-mcp-server \
  -e LOG_LEVEL=DEBUG \
  # ... other options

  3. MCP Client (Claude Desktop/Bob): Add --debug to your existing config args to enable verbose client-side logging:

    {
  "mcpServers": {
    "ibm-verify": {
      "command": "uvx",
      "args": [
        "[email protected]",
        "http://localhost:8000/mcp",
        "--header",
        "persona: end_user",
        "--debug"
      ]
    }
  }
}

Getting Support

Before Contacting Support

Complete this checklist:

  • Reviewed this troubleshooting guide
  • Checked server is running and healthy
  • Verified all environment variables are set correctly
  • Tested network connectivity to IBM Verify
  • Collected server logs
  • Documented exact steps to reproduce the issue
  • Noted any error messages with timestamps

What to Include in Support Request

  1. Problem Description:

    • What were you trying to do?
    • What happened instead?
    • When did it start?
    • How often does it occur?

  2. Environment Details:

    • Deployment method (Docker or Kubernetes)
    • MCP Server version/image tag
    • IBM Verify tenant URL (without credentials)
    • MCP client (Claude Desktop, IBM Bob, etc.)
    • Operating system and version

  3. Reproduction Steps:

    • Exact steps to reproduce the issue
    • Expected behavior
    • Actual behavior

  4. Logs and Diagnostics:

    • Server logs (see Collecting Diagnostics)
    • Configuration files (redact secrets)
    • Error messages with timestamps
    • Screenshots if applicable

  5. Impact:

    • Number of users affected
    • Business impact
    • Workarounds attempted

Support Channels

Quick Reference

Common Commands

# Docker - View logs
docker logs verify-mcp-server

# Docker - Restart server
docker restart verify-mcp-server

# Docker - Check status
docker ps | grep verify-mcp-server

# Kubernetes - View logs
kubectl logs -n verify-mcp-server <pod-name>

# Kubernetes - Restart deployment
kubectl rollout restart deployment -n verify-mcp-server verify-mcp-server

# Kubernetes - Check status
kubectl get pods -n verify-mcp-server

# Test health endpoint
curl http://localhost:8000/health

# Test MCP endpoint
curl http://localhost:8000/mcp

Configuration Checklist

Required Settings (config.yaml):

  • tenant_url — IBM Verify tenant URL
  • client_id — Subject application client ID
  • client_secret — Subject application client secret
  • sts_client_id — STS application client ID
  • sts_client_secret — STS application client secret
  • actor_client_id — Actor application client ID
  • actor_client_secret — Actor application client secret
  • base_url (IBM_VERIFY_BASE_URL) — Publicly reachable MCP Server Base URL

Optional / Conditional Settings:

  • secrets.obf_key / OBF_KEY — Passphrase for !obf decryption (required when !obf values are used)
  • ENC_KEY — RSA private key reference for !enc decryption (e.g. "!secret verify-mcp-enc-key/private.pem" in Kubernetes, "!file /app/secrets/private.pem" in Docker)
  • logging.level / LOG_LEVEL — Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
  • transport.type — Transport protocol (streamable-http)

Related Documentation

Updated 4 days ago

Did this page help you?