Fixing vCenter 5.5 server’s database usage monitoring issue in service health status

Sometimes an upgrade with swanky new features may be the opposite of what you want. A good example is the inclusion of database usage monitoring within vCenter 5.5 server’s service status.

If your SQL database was or is provisioned as usual and you install vCenter 5.5 or upgrade to it, you will see the following amber warning when you check the vCenter service health status:

“Unable to monitor database storage usage. Refer to VMware KB 2078305 for details.”

But when you fire up that KB, it tells you nothing on how to resolve this warning but rather tells you how to manage your database. This feature, it seems was available since vCenter 5.0 but it never showed any warning if you did not configure your database. But with 5.5 (or maybe even 5.1, sorry we skipped that version), you are force to deal with this and cannot ignore it.

http://www.boche.net/blog/index.php/2011/09/27/enabling-vcenter-server-5-0-database-monitoring/

The link from boche.net tells you that you need to run the following on your SQL database:

grant VIEW SERVER STATE to [vpxuser] --> where vpxuser is the account used to access the vCenter database

However, after making this configuration, the warning never went away and I gave VMware support a call. And it turns out, an additional configuration is required to fix this. So the full configuration is:

 use master
 go
 grant VIEW SERVER STATE to [vpxuser]
 go
 GRANT VIEW ANY DEFINITION TO [vpxuser]
 go

However, once database monitoring is enable, the status turns red with the following message:

“Database storage space is critically low and affects vCenter Server functionality. Refer to VMware KB 2078305 for details.”

If you manage a virtualized estate in a big enterprise like I do, you will know that it is highly unlikely that the databases are that full or else the DBAs will be pounding on my door daily!

The VMware support guy pointed me to this link -> https://communities.vmware.com/thread/493193

It turns out that with database monitoring turned on, vCenter will alert you on ALL drives space issue on your database server, regardless whether they are used by the database or not. Typically, we have the SQL database running off a Microsoft cluster with a quorum of size 1 GB.

The default threshold alerts is found in vCenter’s advanced settings:

config.vpxd.vdb.space.warningMB value=5000
config.vpxd.vdb.space.alertMB value=1000

So I must have at least 5 GB of free space in ALL drives to keep it green. This is obviously impossible in a clustered environment using disk quorum. Further, we have a system volume where the application is installed and which doesn’t grow over time and has about 500 MB free space most of the time.

Here is the vpxd-xxx.log capture identify the drives which are below the default threshold

 
2015-02-05T20:48:30.686-05:00 [09596 info 'utilvpxdVdb'] [VpxdVdb::GetDBSpaceStatus] WarningThreshold(MB): 5000 AlertThreshold(MB): 1000. Updating Health Message Parameter to
 --> Drive: Q, Free Space:479 MB, Used Space: 0 MB
 --> Drive: H, Free Space:454 MB, Used Space: 2320 MB
 --> Drive: I, Free Space:461 MB, Used Space: 0 MB
 --> Drive: C, Free Space:37381 MB, Used Space: 0 MB
 --> Drive: D, Free Space:66157 MB, Used Space: 0 MB
 --> Total Used Space: 2320 MB, Minimum Free Space: 454 MB

My solution is to set the threshold low so that this status is always green. This is not a problem for us because we don’t need to monitor the disk space on the SQL servers, the DBA team does it and will ping us if disk space is low. However, if you are a do-it-all administrator, ensure that you have other disks monitoring tools when doing this.

config.vpxd.vdb.space.warningMB value=200
config.vpxd.vdb.space.alertMB value=100

This is a case of new features adding more noise of some of us. I hope VMware will update this feature to allow us to selective turn on and off monitoring of each disk and change the threshold for each disk.