Today we got some complaints from users about slow VMware Cloud Director tasks. It took 40 minutes to create a VM, 10 minutes to power off, and 10 minutes to eject media. Not good! Not good at all!

After logging in to the VAMI of the primary cell we noticed an issue: the cluster health was degraded. The second cell was having some issues but a restart of the cell didn’t solve the problem. A complete restart of the appliance neither.

The first thing I did was to disable the problematic cell in the NSX ALB (that is handling all incoming traffic). The second thing was to redeploy the appliance.

One of the things you need to do is delete the node from the database. In the VCD documentation is a dedicated topic for this: Unregister a Failed Primary or Standby Cell in a Database High Availability Cluster. However, the command that you need to run needs some extra parameters. I ran the following command on the primary cell:

curl -X DELETE "https://<primary cell>:5480/api/1.0.0/nodes/<cell-to-delete>" -k -u "root:<password>"

Now I could redeploy the appliance and run the setup wizard. But the first step failed: Backend validation of NFS mount failed with: /opt/vmware/vcloud-director/data/nfs-test/test-file is owned by an unknown user with UID 4294967294; expected root.

This issue is caused by some miscommunication between the NFS server and the appliance. The solution is very simple:

• Copy the configuration for the NFS mount for transfer file location from the primary node to the node that you are installing. This configuration is located in /etc/fstab.
• Restart the nfs-server service with systemctl restart nfs-server.
• Now you can retry entering the path for the NFS mount in the setup wizard.