In VCF 9.1, components like SDDC Manager’s legacy appliances have been consolidated into the VCF Management Service — a set of components (Log Management/Ops Logs, VCF Operations for Networks, VCF Automation, Identity Broker, Fleet Lifecycle, Salt RaaS, etc.) running as pods inside a Kubernetes cluster called the VCF Services Runtime.

This means troubleshooting has two layers: the UI layer (VCF Operations / Fleet Management) and the Kubernetes layer underneath it. This post walks through both, using real commands, in the order that actually finds problems fastest.

Start in the UI

Before touching a terminal, check:

• Build → Lifecycle → VCF Management → Components — overall component health/inventory. A blank page here (no error, just empty) is a common symptom that something underneath is unhealthy — see Section 3.
• Fleet Management → Passwords — shows stored credentials per component (admin@local, support, consoleuser, etc.) and their expiry. Useful for ruling out expired/rotated credentials.
• Operate → Administration → Integrations — shows data-source adapter instances (e.g., the system-managed integration to VCF Operations for Networks). A “Warning” or “Invalid credentials” status here is a different credential store than Fleet Management Passwords — the two can be out of sync independently.

If the UI shows a specific, named error (SSL thumbprint mismatch, invalid credentials, “already at target version”), that’s usually enough to search Broadcom’s KB directly. If it’s a blank page or a vague “contact support” error, move to the CLI.

SSH in and get cluster-level visibility

ssh vmware-system-user@<control-node-or-VIP>
sudo -i

Root/kubectl access requires elevating with sudo -i first.

If kubectl fails with something like the connection to the server localhost:8080 was refused, the kubeconfig isn’t set for your session — point it at the admin config explicitly:

export KUBECONFIG=/etc/kubernetes/admin.conf

This is usually only needed if root’s ~/.kube/config isn’t already set up on the node you’re on (e.g., a worker node instead of the control-plane node), or if you’re not running as the default configured user.

Check node health first — this rules out or confirms a node-down scenario in seconds:

kubectl get nodes -o wide

All nodes should show Ready. Also note how many are control-plane — a single control-plane node means no HA at that layer, which matters before you consider rebooting anything.

Check overall pod health across all namespaces:

kubectl get pods -A

Filter for anything that isn’t healthy (excludes Completed cronjobs/one-off jobs, which are normal):

kubectl get pods -A --field-selector=status.phase!=Succeeded | grep -vi running

Common namespaces you’ll see:

Namespace	Purpose
ops-logs	Log Management (Ops for Logs) — log-processor, log-store
vcf-fleet-lcm	Fleet Lifecycle services — build/upgrade orchestration
vcf-sddc-lcm	SDDC-level lifecycle management
vmsp-platform	Core VCF Services Runtime platform, Argo workflows, cert trust
vmsp-policies	Platform policies
salt-raas	Salt RaaS automation/orchestration
vcf-fleet-depot	Software depot

Diagnosing a blank/empty UI page

A page that loads with no error is usually a backend or routing problem, not a permissions problem. Check in this order:

1. Browser console/network tab — look for 404s on plugin JS files (stale cached assets after an upgrade). Hard refresh or try incognito first.
2. Backend pod health for the service that renders that page (e.g., vcf-fleet-lcm for the Components/Lifecycle page): bash kubectl get pods -n vcf-fleet-lcm
3. A dependent component may be down, causing the page’s health-aggregation call to fail silently. Walk through each management component’s namespace (see table above) and check for anything not Running.

Diagnosing CrashLoopBackOff

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>

describe shows restart count, last termination reason, and recent events — but the actual error is almost always in the logs of the previous crashed instance, not the current restarting one:

kubectl logs -n <namespace> <pod-name> -c <container-name> --previous --tail=200

Example from this session: log-processor-0 in ops-logs was crash-looping because it couldn’t reach log-store:9200 (Connection refused) — a Spring Boot app failing to initialize because its backend dependency wasn’t available.

When a dependency is the cause, check that dependency’s actual state rather than continuing to chase the crashing pod:

kubectl get statefulsets -n <namespace>

A StatefulSet showing READY 0/0 means it’s scaled to zero — nothing is running, which explains downstream connection failures perfectly. Check whether this is intentional (confirm sizing) and whether data is intact before scaling back up:

kubectl get statefulset <name> -n <namespace> -o yaml | grep -A2 "replicas:"
kubectl get pvc -n <namespace> | grep <name>

If the PVC is still Bound, no data has been lost — scale back up to the confirmed correct replica count:

kubectl scale statefulset <name> -n <namespace> --replicas=<N>
kubectl get pods -n <namespace> -w

Give stateful services (especially anything backed by OpenSearch or similar) a few minutes to fully initialize before expecting dependent pods to recover.

Orphaned or “Unknown” pods

kubectl get pods -n <namespace>

A pod showing STATUS: Unknown means the Kubernetes API server has lost contact with the kubelet that was hosting it — usually a leftover from a workflow (e.g., a system-shutdown job) whose parent resource has already been garbage collected:

kubectl get pod <pod-name> -n <namespace> -o wide   # find the node column directly
kubectl get nodes -o wide                    # confirm all nodes are actually Ready

If all nodes show Ready, the pod itself is just orphaned bookkeeping — safe to clean up once you’ve confirmed nothing depends on it:

kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force

Checking Fleet Lifecycle task/upgrade history

For patch or upgrade failures, the persistent Fleet Lifecycle service pods hold the full task history in their logs — much more useful than chasing ephemeral job pods, which get garbage collected quickly:

kubectl get pods -n vcf-fleet-lcm

Look for the long-running deployment pods (not job-style pods), typically named like vcf-fleet-upgrade-service-fleetupgrade-* and vcf-fleet-build-service-fleetbuild-*. Grep their logs for a specific reference code or step name from the UI error:

kubectl logs -n vcf-fleet-lcm <fleet-upgrade-service-pod> | grep -i -B10 -A30 "<reference-code>"
kubectl logs -n vcf-fleet-lcm <fleet-upgrade-service-pod> | grep -i -B5 -A40 "<step-name>"

This can reveal the entire stage-by-stage history of a task, including every retry attempt with its own reference code and timestamp — critical for telling whether a “new” failure is actually a fresh issue or the tail end of an old stuck task that never recovered.

If the log is too long or has rotated past the relevant window, narrow by time:

kubectl logs -n vcf-fleet-lcm <pod-name> --since=2h | grep -i "<search-term>"

Checking appliance-level (non-Kubernetes) components

Not everything runs as a pod. Components like VCF Operations for Networks Platform run as standalone VMs. If an integration shows “invalid credentials” but the password itself looks fine, check whether the VM is even powered on — via vCenter or PowerCLI:

Get-VM -Name "*ops-networks-platform*" | Select Name, PowerState

A powered-off appliance can surface in the UI as a generic auth/connection failure rather than a clear “unreachable” message. After powering back on, allow time (up to ~15 minutes) for collectors and services to fully reconnect before re-checking status.

General cautions

• Don’t reboot nodes as a first troubleshooting step, especially the control-plane node if you only have one (no HA fallback if something goes wrong on restart). Diagnose to a specific pod/component first; a targeted kubectl delete pod (letting its controller reschedule it) has a much smaller blast radius than a node reboot.
• StatefulSets with persistent data: always confirm PVCs are intact before scaling back up, and check for in-progress Fleet Lifecycle workflows before manually changing replica counts — manual changes can conflict with orchestration expecting to own that state.
• Credentials live in two places: Fleet Management → Passwords (used for Fleet Lifecycle operations) and Operate → Administration → Integrations (used for live data-source adapters). They can go out of sync independently after a password rotation.
• Ephemeral job pods get garbage collected fast. If kubectl logs on a job pod returns nothing, check the persistent service pods that dispatched the job instead — they usually retain the full history.

Quick reference: command cheat sheet

# Cluster health
kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Succeeded | grep -vi running

# Investigate a specific pod
kubectl describe pod <pod> -n <namespace>
kubectl logs -n <namespace> <pod> -c <container> --previous --tail=200

# StatefulSet / dependency issues
kubectl get statefulsets -n <namespace>
kubectl get statefulset <name> -n <namespace> -o yaml | grep -A2 "replicas:"
kubectl get pvc -n <namespace>
kubectl scale statefulset <name> -n <namespace> --replicas=<N>

# Fleet Lifecycle task history
kubectl get pods -n vcf-fleet-lcm
kubectl logs -n vcf-fleet-lcm <fleet-upgrade-service-pod> | grep -i -B10 -A30 "<reference-code>"

# Cleanup orphaned pods
kubectl get pod <pod> -n <namespace> -o wide
kubectl delete pod <pod> -n <namespace> --grace-period=0 --force

Written from real-world troubleshooting of a VCF 9.1 environment, covering Log Management crash loops, stuck StatefulSets, orphaned pods, credential mismatches, and a stuck component upgrade task.