vCSA /storage/log Running Full

I have been trouble shooting a problem with the /storage/log running full, for a customer.

The problem was that the file /storage/log/vmware/vmware-sps/sps-runtime.log.stderr file was growing very large.

I the log file I found the the vmware-sps (Storage Profile Service) had some problems. The errors looks like this

May 31, 2021 7:18:02 AM org.bouncycastle.jsse.provider.ProvTlsClient notifyAlertRaised
 INFO: Client raised fatal(2) certificate_unknown(46) alert: Failed to read record
 org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
Caused by: java.security.cert.CertPathBuilderException: Certification path could not be validated.   
Caused by: java.security.cert.CertificateException: Unable to construct a valid chain
Caused by: java.security.cert.CertPathValidatorException: Could not validate certificate: certificate expired on 20200917181137GMT+00:00

The last one I did a filter on and could see that the date was different, so that indicated that it was communicating with multiple endpoint, which must be the ESXi server.

So I took a look at the some hosts in the vCenter and found that some of the had expired certificate. I tried to update the certificate from vCenter, but that did not work, and that was because I earlier had used this blog: vCenter 6.7 Update 3, Unable to add Host So I had to set the setting back to the original setting of “vmca” and then I could change the certificates on the hosts.

vpxd.certmgmt.mode = vmca

After all hosts with expired certificates, had them renewed the file stopped growing.

UPDATED June 7, 2021:

There above was only one part of the problem, there was another problem also.

The hosts that was add, when the vCenter was using “thumprint” instead of “vmca” was using a custom certificate that the vmware-sps service could validate, so the also filled up the log file.

To find which hosts certificate was expired or custom, I used the a script from William Lam that can be found thru this blog: Is vCenter Server & ESXi hosts using VMware Certificate Authority (VMCA) or custom CA certificates?

The result of the script looked like this.

To fix the problem I just did a “Renew Certificate” and “Refresh CA Certticates”, on all the hosts with expired and “custom” certificates.