Upgrade Advisory: Missing External Service Metrics After Istio v1.22 → v1.23 Upgrade

TL;DR

• What changed: Istio 1.23 sets the destination_service_namespace label on telemetry metrics for external services to the namespace of the ServiceEntry (previously "unknown" in 1.22).
• Why it matters: Any Prometheus queries or alerts expecting destination_service_namespace="unknown" for external (off-cluster) traffic will no longer match after the upgrade, leading to missing metrics and silent alerts.
• Quick fix: Update queries and alerts to use the ServiceEntry namespace instead of "unknown".

What Changed & Why It Matters

Istio’s standard request metrics include a label called destination_service_namespace to indicate the namespace of the destination service. In Istio 1.22 and earlier, when the destination was an external service (defined via a ServiceEntry), this label was set to "unknown". Istio 1.23 now labels these metrics with the namespace of the associated ServiceEntry. 

Any existing Prometheus queries or alerts that explicitly filter for "unknown" will no longer detect external traffic, causing silent failures in monitoring dashboards and alerts. Without updating these queries, teams may unknowingly lose visibility into critical external interactions, potentially overlooking service disruptions or performance degradation.

Impact Matrix

Detection Checklist

• Search your Prometheus alert definitions, recording rules, and Grafana panels for any occurrence of destination_service_namespace="unknown".
• Query external service traffic metrics post-upgrade to confirm if it’s showing a real namespace where you previously expected "unknown".
• Identify sudden metric drops for external traffic labeled as "unknown". A sudden drop to zero in 1.23 indicates that those metrics are now being labeled differently.
• Monitor dashboards for unexpected empty or silent external traffic graphs – it usually means your queries are using an outdated label filter.

Deep Dive – Root Cause

In Istio 1.23, the metric label value for external services changed:

• Previously:
  destination_service_namespace="unknown"
• Now:
  destination_service_namespace="<ServiceEntry namespace>"

This labeling change provides clearer, more precise attribution of external traffic by associating metrics directly with the namespace of their defining ServiceEntry. However, this improvement requires teams to proactively update existing monitoring queries to maintain accurate data capture.

Safe Remediation & Upgrade Paths

1. Pre-upgrade preparation: Update Prometheus queries and alerts replacing "unknown" with actual ServiceEntry namespaces.
   
2. Post-upgrade fix: Immediately adjust queries/alerts to match new namespace labeling and reload configurations.‍
3. Verify and backfill: Confirm external traffic metrics appear correctly; adjust queries for historical continuity.
4. Communication and documentation: Notify team, update internal docs, and monitor Istio release notes regularly.
   

How Chkk Upgrade Copilot Helps

Chkk’s Upgrade Copilot not only analyzes release notes and changelogs but also identifies critical Upgrade Advisories based on real-world experiences from users and insights from our pre-verification process. This helps you proactively address potential risks or breakages during your Kubernetes upgrades or open-source project updates.

In addition to Upgrade Advisories, Chkk covers comprehensive tasks including Helm diffs, CRD diffs, dependency mapping, configuration validations, and deprecated API detection, providing thorough coverage of upgrade-related risks. 

Click below to start for free or book a demo to learn more.