VMWare ESXI Gripes: service-control error message not particularly useful

I posted this at the vmware support forum, whose sophisticated text processing interface decided that it knew better than me and reimagineered my pure html post into a rather flat one. I feel like venting and will do so by posting here what it should have looked like there. Note that right now it is just a bunch of WTF-like questions.

We begin this by stating the ESXi cluster in question uses Windows vcenter server; that is not by my choice. I would rather use the appliance

C:\Users\raub> "C:\Program Files\VMware\vCenter Server\bin\service-control" --status --all
Running:
 VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSService VMwareIdentityMgmtService VMwareSTS rhttpproxy vmon vmonapi vmware-cis-config vmware-license vmwareServiceControlAgent
Stopped:
 EsxAgentManager VMWareCAMService VServiceManager content-library mbcs vPostgres vapiEndpoint vimPBSM vmsyslogcollector vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump vmware-perfcharts vpxd vpxd-svcs vsan-health vsphere-ui vspherewebclientsvc

C:\Users\raub>

C:\Users\raub> "C:\Program Files\VMware\vCenter Server\bin\service-control" --start vspherewebclientsvc
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vsphere-client...
Error executing start on service vsphere-client. Details {
    "detail": [
        {  
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'vsphere-client'",
            "args": [
                "vsphere-client"
            ]
        }
    ],
    "resolution": null,
    "problemId": null,
    "componentKey": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'vsphere-client'",
            "args": [
                "vsphere-client"
            ]
        }
    ],
    "resolution": null,
    "problemId": null,
    "componentKey": null
}

C:\Users\raub>

I understand that "An error occurred while starting service 'vsphere-client'", but what is it? Maybe the log file is more helpful. https://kb.vmware.com/s/article/2121043 claims log dir is

C:\ProgamData\VMware\vCenterServer\logs\vsphere-client\logs\

But I can see lots of directories in there but the log one:

C:\Users\raub>dir  "C:\Program Files\VMware\vCenter Server\"
 Volume in drive C has no label.
 Volume Serial Number is F020-F58F

 Directory of C:\Program Files\VMware\vCenter Server

05/21/2018  07:36 PM    <DIR>          .
05/21/2018  07:36 PM    <DIR>          ..
05/21/2018  07:11 PM    <DIR>          apachetomcat
05/21/2018  07:15 PM    <DIR>          autodeploy
05/21/2018  07:17 PM    <DIR>          bin
05/21/2018  07:13 PM    <DIR>          cis-license
05/21/2018  07:10 PM    <DIR>          cis_upgrade_runner
05/21/2018  07:13 PM    <DIR>          cm
05/21/2018  07:10 PM    <DIR>          common-jars
05/21/2018  07:10 PM    <DIR>          common-libs
05/21/2018  07:15 PM    <DIR>          content-library
05/21/2018  07:15 PM    <DIR>          eam
05/21/2018  07:36 PM    <DIR>          eula
05/21/2018  07:12 PM    <DIR>          fips
05/21/2018  07:21 PM    <DIR>          firstboot
05/21/2018  07:15 PM    <DIR>          imagebuilder
05/21/2018  07:10 PM    <DIR>          jmemtool
05/21/2018  07:10 PM    <DIR>          jre
05/21/2018  07:12 PM    <DIR>          jre_ext
05/21/2018  07:15 PM    <DIR>          mbcs
05/21/2018  07:13 PM    <DIR>          netdump
05/21/2018  07:10 PM    <DIR>          openSSL
04/09/2018  01:24 PM         7,398,602 open_source_license.txt
05/21/2018  07:16 PM    <DIR>          perfcharts
05/21/2018  07:11 PM    <DIR>          python
05/21/2018  07:16 PM    <DIR>          python-modules
05/21/2018  07:13 PM    <DIR>          rhttpproxy
05/21/2018  07:12 PM    <DIR>          ruby
05/21/2018  07:15 PM    <DIR>          rvc
05/21/2018  07:13 PM    <DIR>          sca
05/21/2018  07:09 PM    <DIR>          TlsReconfigurator
05/21/2018  07:13 PM    <DIR>          vapi
04/09/2018  01:24 PM            25,214 vcs.ico
05/21/2018  07:14 PM    <DIR>          virgo
05/21/2018  07:12 PM    <DIR>          visl-integration
05/21/2018  07:12 PM    <DIR>          vmafdd
05/21/2018  07:12 PM    <DIR>          vmcad
05/21/2018  07:15 PM    <DIR>          vmcamd
05/21/2018  07:12 PM    <DIR>          vmdird
05/21/2018  07:13 PM    <DIR>          vmdns
05/21/2018  07:13 PM    <DIR>          vmon
05/21/2018  07:15 PM    <DIR>          vmsyslogcollector
05/21/2018  07:13 PM    <DIR>          VMware Identity Services
05/21/2018  07:11 PM    <DIR>          vmware-sasl
04/25/2018  01:20 PM    <DIR>          vmware-sps
05/21/2018  07:13 PM    <DIR>          vmware-sso
05/21/2018  07:14 PM    <DIR>          vPostgres
05/21/2018  07:25 PM    <DIR>          vpxd
05/21/2018  07:14 PM    <DIR>          vpxd-svcs
05/21/2018  07:32 PM    <DIR>          vsan-health
05/21/2018  07:34 PM    <DIR>          vsm
05/21/2018  07:16 PM    <DIR>          vsphere-client
05/21/2018  07:17 PM    <DIR>          vsphere-ui
               2 File(s)      7,423,816 bytes
              51 Dir(s)  69,727,727,616 bytes free

C:\Users\raub>

C:\Users\raub>dir  "C:\Program Files\VMware\vCenter Server\logs"
 Volume in drive C has no label.
 Volume Serial Number is F020-F58F

 Directory of C:\Program Files\VMware\vCenter Server

File Not Found

C:\Users\raub>

Where can I find where vcenter thinks the log files are at?

If you want to see how the ticket looks like at vmware, https://communities.vmware.com/message/2796432#2796432. Try to read the version posted at vmware and you will understand why I am frustrated.

Debugging Thunderbird

While hunting down issues with DKIM, as mentioned in a previous blog entry, we had to figure out how to make Thunderbird provide us with a log file. According to the docs, to debug Thunderbird you should export some environmental variables (I am using Linux here as the OS in this example. Adjust as needed):

# For bash shell (the default shell on most GNU/Linux systems):
export NSPR_LOG_MODULES=imap:5
export NSPR_LOG_FILE=/tmp/imap.log

Make your life easier and do it inside a script:

raub@desktop:/tmp$ cat thunderchicken
#!/bin/sh
# For bash shell (the default shell on most GNU/Linux systems):
export NSPR_LOG_MODULES=smtp:5
export NSPR_LOG_FILE=/tmp/smtp.log
thunderbird
raub@desktop:/tmp$

This way the environment variables you are setting only exist in the context of the script

raub@desktop:/tmp$ sh -x thunderchicken
+ export NSPR_LOG_MODULES=smtp:5
+ NSPR_LOG_MODULES=smtp:5
+ export NSPR_LOG_FILE=/tmp/smtp.log
+ NSPR_LOG_FILE=/tmp/smtp.log
+ thunderbird
[calBackendLoader] Using libical backend at /home/raub/.thunderbird/emfxtc69.default/extensions/
{e2fda1a4-762b-4020-b5ad-a41df1933103}/components/libical.manifest

If you do that, you will not have a beautiful log file you can then examine as you test things in Thunderbird:

raub@desktop:~$ ls -l /tmp/smtp.log
-rw-rw-r--. 1 raub raub 0 Jun 24 16:28 /tmp/smtp.log
raub@desktop:~$ ls -l /tmp/smtp.log
-rw-rw-r--. 1 raub raub 6464 Jun 24 16:29 /tmp/smtp.log
raub@desktop:~$

Getting the SSH_ORIGINAL_COMMAND

Let's say you want to have an account you can ssh into but only run very specific commands in it. A good way to achieve that is to write a wrapper script that is called from your authorized_keys file. So you could have a wrapper that looks like this:

#!/bin/sh
case $SSH_ORIGINAL_COMMAND in
    "/usr/bin/rsync "*)
        $SSH_ORIGINAL_COMMAND
        ;;
    *)
        echo "Permission denied."
        exit 1
        ;;
esac

But, what if you really want to be really precise on the command? Using the above example, not only running rsync but also specifying the path and the arguments? You could cheat and find what the command you are sending is supposed to look like by replacing (temporarily) your wrapper script with this

/bin/sh

DEBUG="logger" # Linux
#DEBUG="syslog -s -l note" # OSX

if [ -n "$SSH_ORIGINAL_COMMAND" ]; then
        $DEBUG "Passed SSH command $SSH_ORIGINAL_COMMAND"
elif [ -n "$SSH2_ORIGINAL_COMMAND" ]; then
        $DEBUG "Passed SSH2 command $SSH2_ORIGINAL_COMMAND"
else
        $DEBUG Not passed a command.
fi

Then you run the ssh command and see what it looks like in the log file. Copy that to your original wrapper script, and you are good to go. So

ssh -t -i /home/raub/.ssh/le_key raub@virtualpork echo "Hey"

Results in

Dec 26 13:34:05 virtualpork syslog[64541]: Passed SSH command echo Hey

While

rsync -avz -e 'ssh -i /home/raub/.ssh/le_key' raub@virtualpork:Public /tmp/backup/

results in

Dec 26 13:28:17 virtualpork syslog[64541]: Passed SSH command rsync --server 
--sender -vlogDtprze.iLs . Public

The latter meaning our little wrapper script would then look like

#!/bin/sh
case $SSH_ORIGINAL_COMMAND in
    "rsync --server --sender -vlogDtprze.iLs . Public")
        $SSH_ORIGINAL_COMMAND
        ;;
    *)
        echo "Permission denied."
        exit 1
        ;;
esac