Setting up zabbix using official instructions and repo: Step 1 we ain't there yet

So I am installing Zabbix. Why, well, you probably know. If not, we can talk about that in a different article. Yes I am testing my ansible playbook in a docker container, but right now that too is not important. The How I did It will be in a different article. This article is the Everything That Went Wrong and How I Got Around That one. Think of it as an insight of how I deal with me being clueless; laughing at my expense is acceptable and maybe even recommended.

I want to install latest version of Zabbix in a CentOS 7 host, as a result I will be using the official zabbix 3.2 install docs, which are the most current when I wrote this article. For now I will be lazy and use the mysql version since it is faster to setup; we can revisit that later.

Dependencies

1. Need the repo. Per the official Zabbix instructions, I am using the official Zabbix repo, which as of the time of this writing can be obtained by

   rpm -ivh http://repo.zabbix.com/zabbix/3.2/rhel/7/x86_64/zabbix-release-3.2-1.el7.noarch.rpm

   I did write a script to get the latest rpm, but it is not important right now. Now, if you are curious, here is the repo config file:

   [root@zabbix ~]# cat /etc/yum.repos.d/zabbix.repo 
   [zabbix]
   name=Zabbix Official Repository - $basearch
   baseurl=http://repo.zabbix.com/zabbix/3.2/rhel/7/$basearch/
   enabled=1
   gpgcheck=1
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
   
   [zabbix-non-supported]
   name=Zabbix Official Repository non-supported - $basearch 
   baseurl=http://repo.zabbix.com/non-supported/rhel/7/$basearch/
   enabled=0
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX
   gpgcheck=1
   [root@zabbix ~]# 

   Before you ask, I am making a point to accidentally post it here for a reason, which will become clearer later.

2. The database server. Thanks to irc user leManu enlightening me, we are not supposed to install mysql (or whatever db) server on the machine that will run zabbix server. With that said, the line

   mysql> grant all privileges on zabbix.* to zabbix@localhost identified by '';

   in the official docs has a very localhost feel to it.

   We better build the db server first, and then go there and create the zabbix user, tying it to the IP for the zabbix server. I used mariadb and then grabbed the required -- FQDN, port, zabbix password -- info and came back to the zabbix server.
   

3. Packages. We have the repo setup, and database server info on standby. We might as well start installing zabbix itself, right?

   [root@zabbix ~]# yum install zabbix-server-mysql zabbix-web-mysql 
   [...]
   --> Finished Dependency Resolution
   Error: Package: zabbix-server-mysql-3.2.4-2.el7.x86_64 (zabbix)
              Requires: fping
   Error: Package: zabbix-server-mysql-3.2.4-2.el7.x86_64 (zabbix)
              Requires: libiksemel.so.3()(64bit)
    You could try using --skip-broken to work around the problem
    You could try running: rpm -Va --nofiles --nodigest
   [root@zabbix ~]# 

   Bummer. Why didn't it grab them from the normal centos repo? I guess maybe it does not have them and we will need to fetch them from another repo. But, before we add another repo, do you remember the file /etc/yum.repos.d/zabbix.repo, whose contents we pasted earlier? It has a zabbix-non-supported; how about if we take a quick look there?

   [root@zabbix ~]# yum whatprovides */fping --enablerepo=zabbix-non-supported
   Loaded plugins: fastestmirror, ovl
   Loading mirror speeds from cached hostfile
    * base: mirrors.gigenet.com
    * extras: mirror.keystealth.org
    * updates: mirror.umd.edu
   fping-3.10-1.el7.x86_64 : Scriptable, parallelized ping-like utility
   Repo        : zabbix-non-supported
   Matched from:
   Filename    : /usr/sbin/fping
   
   
   
   fping-3.10-1.el7.x86_64 : Scriptable, parallelized ping-like utility
   Repo        : @zabbix-non-supported
   Matched from:
   Filename    : /usr/sbin/fping
   
   
   
   [root@zabbix ~]# 

   Short version: grab the two packages we need from it already:

   yum install fping iksemel --enablerepo=zabbix-non-supported

4. Missing setup file (thanks Yum!). Per the docs, we are now supposed to grab a file called /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz and use it to initially populate the zabbix database. Thing is, I can't find that (/usr/share/doc/zabbix-server-mysql-3.2.4) directory, much less the file:

   [root@zabbix ~]# ls  /usr/share/doc/
   coreutils-8.22    pam-1.1.8             python-pycurl-7.19.0
   gnupg2-2.0.22     pygpgme-0.3           unixODBC-2.3.1
   krb5-libs-1.13.2  python-kitchen-1.1.1  zabbix-release-3.2
   [root@zabbix ~]# 

   Maybe /usr/share/doc/zabbix-release-3.2/ is the directory and the docs were off? I will have to expertly crush your hopes:

   [root@zabbix ~]# ls  /usr/share/doc/zabbix-release-3.2/
   GPL
   [root@zabbix ~]# ls -l  /usr/share/doc/zabbix-release-3.2/
   total 20
   -rw-r--r-- 1 root root 18385 Feb 15  2016 GPL
   [root@zabbix ~]# head -10  /usr/share/doc/zabbix-release-3.2/GPL 
   *****************************************************************************
   The following copyright applies to the Red Hat Linux compilation and any 
   portions of Red Hat Linux it does not conflict with. Whenever this
   policy does conflict with the copyright of any individual portion of Red Hat 
   Linux, it does not apply.
   
   *****************************************************************************
   
                       GNU GENERAL PUBLIC LICENSE
                          Version 2, June 1991
   [root@zabbix ~]# 

   Maybe it is somewhere else? Nope.

   [root@zabbix ~]# find / -name create.sql.gz -print
   [root@zabbix ~]# 

   So, where's it? Hey, don't look at me like that. I too have no idea. Let's grab the rpm and then take a look at it:

   root@zabbix:/tmp$ rpm -qlp zabbix-server-mysql-3.2.4-2.el7.x86_64.rpm | grep cre
   ate.sql.gz 
   warning: zabbix-server-mysql-3.2.4-2.el7.x86_64.rpm: Header V4 DSA/SHA1 Signatur
   e, key ID 79ea5ed4: NOKEY
   /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz
   root@zabbix:/tmp$ 

   That is the version we installed, right?

   [root@zabbix ~]# rpm -q zabbix-server-mysql
   zabbix-server-mysql-3.2.4-2.el7.x86_64
   [root@zabbix ~]#

   Looks like it. And yum's log, /var/log/yum.log file think so too:

   Mar 07 14:30:20 Installed: zabbix-web-mysql-3.2.4-2.el7.noarch
   Mar 07 14:30:21 Installed: zabbix-web-3.2.4-2.el7.noarch
   Mar 07 14:30:22 Installed: zabbix-server-mysql-3.2.4-2.el7.x86_64

   This really does not make sense. Let me look again at the contents of the installed package, not at the rpm:

   [root@zabbix ~]# rpm -qlv zabbix-server-mysql
   -rw-r--r--    1 root    root                      132 Mar  2 14:55 /etc/logrotate.d/zabbix-server
   -rw-r-----    1 root    zabbix                  14876 Mar  2 14:55 /etc/zabbix/zabbix_server.conf
   -rw-r--r--    1 root    root                      415 Mar  2 14:29 /usr/lib/systemd/system/zabbix-server.service
   -rw-r--r--    1 root    root                       35 Mar  2 14:29 /usr/lib/tmpfiles.d/zabbix-server.conf
   drwxr-xr-x    2 root    root                        0 Mar  2 14:55 /usr/lib/zabbix/alertscripts
   drwxr-xr-x    2 root    root                        0 Mar  2 14:55 /usr/lib/zabbix/externalscripts
   -rwxr-xr-x    1 root    root                  2220064 Mar  2 14:55 /usr/sbin/zabbix_server_mysql
   drwxr-xr-x    2 root    root                        0 Mar  2 14:55 /usr/share/doc/zabbix-server-mysql-3.2.4
   -rw-r--r--    1 root    root                       98 Feb 27 09:22 /usr/share/doc/zabbix-server-mysql-3.2.4/AUTHORS
   -rw-r--r--    1 root    root                    17990 Feb 27 09:23 /usr/share/doc/zabbix-server-mysql-3.2.4/COPYING
   -rw-r--r--    1 root    root                   742520 Feb 27 09:22 /usr/share/doc/zabbix-server-mysql-3.2.4/ChangeLog
   -rw-r--r--    1 root    root                       52 Feb 27 09:24 /usr/share/doc/zabbix-server-mysql-3.2.4/NEWS
   -rw-r--r--    1 root    root                      188 Feb 27 09:22 /usr/share/doc/zabbix-server-mysql-3.2.4/README
   -rw-r--r--    1 root    root                  1161488 Mar  2 14:49 /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz
   -rw-r--r--    1 root    root                      881 Mar  2 14:55 /usr/share/man/man8/zabbix_server.8.gz
   drwxr-xr-x    2 zabbix  zabbix                      0 Mar  2 14:55 /var/log/zabbix
   drwxr-xr-x    2 zabbix  zabbix                      0 Mar  2 14:55 /var/run/zabbix
   [root@zabbix ~]# ls /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz
   ls: cannot access /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz: No such file or directory
   [root@zabbix ~]# 

   It turns out (kudos to irc user TrevorH for pointing that out) that yum is configured not to install docs

   [root@zabbix ~]# grep -ir tsflags /etc/yum.*
   /etc/yum.conf:tsflags=nodocs
   [root@zabbix ~]# 

   Let's comment it out then and try again

   [root@zabbix ~]# sed -i -e 's/^tsflags=nodocs/#tsflags=nodocs/' /etc/yum.conf
   [root@zabbix ~]# yum reinstall zabbix-server-mysql zabbix-web-mysql --enablerepo=zabbix
   Loaded plugins: fastestmirror, ovl
   Loading mirror speeds from cached hostfile
    * base: dist1.800hosting.com
    * extras: mirror.eboundhost.com
    * updates: mirror.es.its.nyu.edu
   Resolving Dependencies
   --> Running transaction check
   ---> Package zabbix-server-mysql.x86_64 0:3.2.4-2.el7 will be reinstalled
   ---> Package zabbix-web-mysql.noarch 0:3.2.4-2.el7 will be reinstalled
   --> Finished Dependency Resolution
   
   Dependencies Resolved
   
   ================================================================================
    Package                   Arch         Version              Repository    Size
   ================================================================================
   Reinstalling:
    zabbix-server-mysql       x86_64       3.2.4-2.el7          zabbix       1.8 M
    zabbix-web-mysql          noarch       3.2.4-2.el7          zabbix       5.1 k
   
   Transaction Summary
   ================================================================================
   Reinstall  2 Packages
   
   Total download size: 1.8 M
   Installed size: 4.0 M
   Is this ok [y/d/N]: y
   Downloading packages:
   (1/2): zabbix-web-mysql-3.2.4-2.el7.noarch.rpm             | 5.1 kB   00:00     
   (2/2): zabbix-server-mysql-3.2.4-2.el7.x86_64.rpm          | 1.8 MB   00:01     
   --------------------------------------------------------------------------------
   Total                                              1.7 MB/s | 1.8 MB  00:01     
   Running transaction check
   Running transaction test
   Transaction test succeeded
   Running transaction
     Installing : zabbix-web-mysql-3.2.4-2.el7.noarch                          1/2 
     Installing : zabbix-server-mysql-3.2.4-2.el7.x86_64                       2/2 
     Verifying  : zabbix-server-mysql-3.2.4-2.el7.x86_64                       1/2 
     Verifying  : zabbix-web-mysql-3.2.4-2.el7.noarch                          2/2 
   
   Installed:
     zabbix-server-mysql.x86_64 0:3.2.4-2.el7                                      
     zabbix-web-mysql.noarch 0:3.2.4-2.el7                                         
   
   Complete!
   [root@zabbix ~]# ls /usr/share/doc/
   coreutils-8.22    pygpgme-0.3           zabbix-release-3.2
   gnupg2-2.0.22     python-kitchen-1.1.1  zabbix-server-mysql-3.2.4
   krb5-libs-1.13.2  python-pycurl-7.19.0
   pam-1.1.8         unixODBC-2.3.1
   [root@zabbix ~]# ls /usr/share/doc/zabbix-server-mysql-3.2.4/
   AUTHORS  ChangeLog  COPYING  create.sql.gz  NEWS  README
   [root@zabbix ~]# 

   Success at last!

I think that is enough for one article. If you expect this to have any closure or redeeming message, I have news for you sunshine. Just hope that the next zabbix article will talk about actually getting it installed and configured and running. But, I make no guarantees.

Entrypoint scripts and data persistency: mounting a NFS fileshare inside a docker container

This is a quick article; you have been warned.

The traditional way to have persistent data in a docker container is to feed it with a volume from the docker host. You begin by locating a directory, be it in the local docker host drive, network mounted to the docker host, or a volume in another docker container. Then feed it to the container using the VOLUME statement defined in the Dockerfile or to command line. We all know that. But what some (very few we hope) of you might not have been aware of is the volume is only mounted/attached to the container when you tell docker to run the container, which might make running/configuring something that require files in those volumes a bit challenging.

At this point I would expect one of you to shout "Aha! But you can then use an entrypoint script to do all that monkeying that needs to happen after volume is mounted and before whatever service this container provides starts!" And you would be quite right! Here's a quick example in case we lost somebody: if our Dockerfile ended up like this:

# Put the entrypoint script somewhere we can find
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

EXPOSE 22
# Start service
CMD ["/usr/sbin/sshd", "-D"]

we would have created a file called entrypoint.sh in the root path of the container, which is run just after the volume is created and before the, in this example, sshd service is started. Said entrypoint file could do, I don't know:

#!/bin/sh
set -e

# Do something in /home, which is the mounted volume
/do-something-at-home.sh

# And we  are done here
exec "$@"

point is that do-something-at-home.sh is called only after the persistent volume is mounted into the container.

What if you want to mount a network fileshare straight into the container? Before I have time to answer that, someone from the audience will jump and state "Why would you want to do that? You can mount the network fileshare in the docker host and then use VOLUME just like Wicked Witch of the West intended!" What if I don't want to do that? What if I can't for whatever reason mount a network fileshare on the docker host?

The answer is that it is completely doable; you just have to ask the entrypoint script to do the mounting for you. To show what I mean I will use the example of mounting a NFSv4 fileshare that is supposed to be used by some website. So we modify our little entrypoint script a bit

#!/bin/sh
set -e

# /export/www was created in the Dockerfile
mount.nfs4 fileserver.example.com:/export/www /www/some-website 

# And we are done here
exec "$@"

Since we are using NFSv4, chances are you might need to add something like

RUN sed -i -e '/^#Domain/a Domain = example.com' /etc/idmapd.conf

to your dockerfile in addition to telling it to get the nfs client packages, but the general idea should apply for, say, SMB or ZFS or whatever other network filesystem that you fancy: let your entrypoint script do the heavy lifting!

Getting a program/package off the Solaris install CDs

Let's say for whatever reason -- in my case I did a really bare install of Solaris 10 -- you just realized you need a program from the Solaris install CDs/DVDs. How to get it? But, let's make it a bit more challenging: first of all, you can't use Sunfreeware; maybe they just do not have it. Second, the Solaris box you want to put this program in does not have a CD/DVD drive.

Well, we need to start somewhere; how about trying to figure out the name of the package? Yes, you can say some package names reflect the name of the program. But, there are those times the program you want is part of a larger package.

So, we need to cheat.

If you are really luck, you either have or know someone who does have access to another Solaris box which has the program. Then they can use pkgchk ask which package the program in question belongs to. For instance, which package does truss belong to? Let's find out:

# pkgchk -lp `which truss`
Pathname: /usr/bin/truss
Type: linked file
Source of link: ../../usr/lib/isaexec
Referenced by the following packages:
        SUNWtoo        
Current status: installed

#

Ok, we now know truss belong to the package SUNWtoo. Now, we need to get the install CDs and mount them somewhere. I know you can mount them in Solaris (duh!) and in Linux; I have not tried in Windows or OSX. But, let's say you were successful and mounted Disk 1. Now you need to go to the Solaris_10/Product/ directory, where you will find a bunch of directories, each of them has the name of a package. If the package you want is not in this disk, try another one; it better be somewhere, right? In my case, Solaris_10/Product/SUNWtoo is what I want. I tar'ed the package and then copied it to the problematic Solaris box. After untar'ing into the /tmp directory, I then need to install the said package:

# cd /tmp
# tar xf SUNWtoo.tar
# pkgadd -d . SUNWtoo    

Processing package instance  from 

Programming Tools(sparc) 11.10.0,REV=2005.01.21.15.53
Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Using  as the package base directory.
## Processing package information.
## Processing system information.
   10 package pathnames are already properly installed.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

This package contains scripts which will be executed with super-user
permission during the process of installing this package.

Do you want to continue with the installation of  [y,n,?] y

Installing Programming Tools as 

## Installing part 1 of 1.
1982 blocks
## Executing postinstall script.

Installation of  was successful.
# 

So, is it installed? Let's find out!

# pkginfo -x SUNWtoo
SUNWtoo  Programming Tools
         (sparc) 11.10.0,REV=2005.01.21.15.53
# which truss
/usr/bin/truss
#