We are currently using 3 Syslog collector to serve each region (i.e. AMER, EMEA & APAC) running version 5 u2 on Windows 2008 R2 64-bit machines. I had previously posted a question to VMware regarding maximum number of ESXi hosts that can report to it and they categorically said that there are no limits for the service. Technically this is correct, however, practically, the limitation comes from the OS which you run the service one and limits which the OS has per process which limits how many hosts can and should report to it. Recently, we found logs not updated or log folder for the ESXi host empty on our Syslog collectors. Network monitor between the ESXi hosts and Syslog server shows the hosts are continuously sending data via UDP port 514 (this is our setup). So the problem had to be on the Syslog server end.
Upon checking the debug log, we found a lot of these errors: Exceptions.IOError: [Errno 24] Too man open files: u’E:\\syslog\\18.104.22.168\\syslog.log’
So obviously the OS or the process cannot work with too many open files at one time. The 2 Syslog servers which we had had 700 and almost 1,400 hosts pointing to it. Another which was working fine had about 400 hosts. So it looks like the limit is between 400 and below 700 hosts. I restarted the service and waiting for the error to appear on the debug log then counted the number of Syslog folders that had been updated.
Consistently, the number is around 600. Kicking off process explorer (from MS) and looking at the handles for the syslogcollector.exe process, I can see that the handles sort of max out at 700 handles for both of the troubled Syslog servers. On the good one, the handles are about 600. So I will say that there is a limit in the number of hosts your can configure for the Syslog collector, but the limit is due to how the OS works. So to be on the safe side, I will say to limit the number of hosts a Syslog collector works on to about 400 – 500 hosts. This should stress it just right without breaking it apart, however, please do check the debug logs to tune your figures.