Veeam SAN mode Backup failure
I was troubleshooting a strange Veeam backup problem. I could not get Veeam to do SAN mode backup.
The installation was.
-
vCenter 6.5 Update 1c
-
ESXi 6.5 update 1
-
Veeam B&R 9.5 Update 2
-
Backup server Windows 2012 R2
-
Backup Proxy server Windows 2016
-
Subsystem Device Driver Device Specific Module 2.5.0.0
-
Subsystem Device Driver Device Specific Module 2.5.0.0
-
storage IBM Storwize V7000 FC Hyperswap
-
VMFS-6 Datastores
-
VMFS-6 Datastores
-
hard Zoning in SAN switches
The backup was doing failback to NBD, very time running backup jobs on the proxy server.
I looked thru the log files on the Veeam B&R server and the proxy server, but there was no clue to what was wrong, only that it could not do SAN backup, and did a failback to NBD.
I tried:
-
remove Subsystem Device Driver Device Specific Module 2.5.0.0 and Windows MPIO
-
format VMFS-6 datastore to VMFS-5
-
multiple rescan of the vCenter and Windows server i Veeam B&R
-
and other things
Nothing worked.
The Veeam solution is protected by at firewall, and at the end we did look at the firewall, for something we forgot to open, we has opened for TCP port 443 to vCenter, TCP port 902 to the ESXi hostx, DNS, NT and patch management. But what we saw was that the Veeam proxy, was trying to contact the Microsoft Certificate servers.
This is because we have changed the Certificate of the vCenter server, with a signed certificate from the Certificate server, and the Proxy server is not in the Active Directory, so the VDDK could not validate the vCenter certificate. So we opened TCP port 80 to the certificate servers.
I think that we also could have installed the Root CA + intermediate Certificate, but I did not have time to test that.
Update:
Tom Soghter, Principal Solutions Architect at Veeam Software, has written a comment with the explanation for this behavior,.Thanks.
Hi Allan,
just an idea, did you run the vcenter wizard in the Veeam UI again to update as well the certificate? Maybe just run it again, accept the certificate popup if it will be shown and test again with closed CA port ?
Hi Andreas,
I ran the wizard after i change the certificates on vCenter. I tried doing it again, and it works now.
But I am not sure if it is because of the wizard, or that VDDK caches som certificate validation data.
We will keep the firewall rule in place to the certificate server.
While I can’t be 100% sure, the most likely cause of this behavior is that the Microsoft certificate server has been configured with a CRL Distribution Point (CDP) and thus issues certificates with the CDP property set and pointing to its own URL. Generally, a self-signed certificate does not include this propery, and certificate issued from a certificate server where the CDP hasn’t been defined should also not include this property, however, it’s highly recommended to have a CDP defined since certificate revocation is a critical capability for a proper PKI deployment.
When the certificate has a CDP property, the defined client is to contact the CDP via the URL to determine the certificate’s revocation status and, if the CRL is not retrievable, the client should reject the certificate. Normally, the Microsoft certificate server distributes the CRL via AD, but since your proxy was not a domain member, it would instead attempt to use the URL listed in the CDP property, which is almost certainly the certificate server itself.
Great reference article here: https://blogs.technet.microsoft.com/nexthop/2012/12/17/updated-creating-a-certificate-revocation-list-distribution-point-for-your-internal-certification-authority/
Hi Tom,
Thanks for confirm my findings.
Small error in my last comment, in the second paragraph, “defined client” should be “defined client behavior”. Sorry about that.