Friday, July 29, 2016

Checking if RedHat/CentOS has new updates

If you use Debian Linux derivatives, specially Ubuntu, you probably noticed when you login it will tell you (using the MOTD) that there are new updates waiting for you. And, you can use that if, say, you are writing a script to let you know about that; someone I know use that to monitor his Ubunu boxes in Nagios. But, what about in RedHat and derviatives the lazy way?

Let's do some thinking aloud and see if we can come up with something. We know that if we run yum check-update, it should reply with the list of packages needing to be upgraded if any

[root@vmhost ~][raub@duckwitch ~]$ yum check-update
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror.supremebytes.com
 * extras: mirror.hostduplex.com
 * updates: mirror.scalabledns.com

chkconfig.x86_64                        1.3.61-5.el7_2.1                 updates
device-mapper.x86_64                    7:1.02.107-5.el7_2.5             updates
device-mapper-libs.x86_64               7:1.02.107-5.el7_2.5             updates
dracut.x86_64                           033-360.el7_2.1                  updates
glibc.x86_64                            2.17-106.el7_2.6                 updates
glibc-common.x86_64                     2.17-106.el7_2.6                 updates
iproute.x86_64                          3.10.0-54.el7_2.1                updates
kernel.x86_64                           3.10.0-327.22.2.el7              updates
kpartx.x86_64                           0.4.9-85.el7_2.5                 updates
libxml2.x86_64                          2.9.1-6.el7_2.3                  updates
ntpdate.x86_64                          4.2.6p5-22.el7.centos.2          updates
pcre.x86_64                             8.32-15.el7_2.1                  updates
selinux-policy.noarch                   3.13.1-60.el7_2.7                updates
selinux-policy-targeted.noarch          3.13.1-60.el7_2.7                updates
systemd.x86_64                          219-19.el7_2.11                  updates
systemd-libs.x86_64                     219-19.el7_2.11                  updates
systemd-python.x86_64                   219-19.el7_2.11                  updates
systemd-sysv.x86_64                     219-19.el7_2.11                  updates
tzdata.noarch                           2016f-1.el7                      updates
[raub@duckwitch ~]$

As you can see, it does not require you to run that command as root. And you can even check if a repo you use but configured to be normally disabled, like epel in the following example:

raub@duckwitch ~]$ yum check-update --enablerepo=epel
Loaded plugins: fastestmirror
epel/x86_64/metalink                                     |  11 kB     00:00
epel                                                     | 4.3 kB     00:00
(1/3): epel/x86_64/group_gz                                | 170 kB   00:00
(2/3): epel/x86_64/updateinfo                              | 584 kB   00:00
(3/3): epel/x86_64/primary_db                              | 4.2 MB   00:00
Loading mirror speeds from cached hostfile
 * base: mirror.supremebytes.com
 * epel: mirror.chpc.utah.edu
 * extras: mirror.hostduplex.com
 * updates: mirror.scalabledns.com

chkconfig.x86_64                        1.3.61-5.el7_2.1                 updates
device-mapper.x86_64                    7:1.02.107-5.el7_2.5             updates
device-mapper-libs.x86_64               7:1.02.107-5.el7_2.5             updates
dracut.x86_64                           033-360.el7_2.1                  updates
epel-release.noarch                     7-7                              epel
glibc.x86_64                            2.17-106.el7_2.6                 updates
glibc-common.x86_64                     2.17-106.el7_2.6                 updates
iproute.x86_64                          3.10.0-54.el7_2.1                updates
kernel.x86_64                           3.10.0-327.22.2.el7              updates
kpartx.x86_64                           0.4.9-85.el7_2.5                 updates
libxml2.x86_64                          2.9.1-6.el7_2.3                  updates
ntpdate.x86_64                          4.2.6p5-22.el7.centos.2          updates
pcre.x86_64                             8.32-15.el7_2.1                  updates
selinux-policy.noarch                   3.13.1-60.el7_2.7                updates
selinux-policy-targeted.noarch          3.13.1-60.el7_2.7                updates
systemd.x86_64                          219-19.el7_2.11                  updates
systemd-libs.x86_64                     219-19.el7_2.11                  updates
systemd-python.x86_64                   219-19.el7_2.11                  updates
systemd-sysv.x86_64                     219-19.el7_2.11                  updates
tzdata.noarch                           2016f-1.el7                      updates
[raub@duckwitch ~]$

What if there are no upgrades?

[root@server1 ~]# yum check-update
Loaded plugins: fastestmirror
base                                                     | 3.6 kB     00:00
extras                                                   | 3.4 kB     00:00
updates                                                  | 3.4 kB     00:00
Loading mirror speeds from cached hostfile
 * base: mirror.us.leaseweb.net
 * extras: mirror.us.leaseweb.net
 * updates: reflector.westga.edu
[root@server1 ~]#

Sounds like we need to check for the first blank line. We can do that using sed. We can find the blank line by using the /^\s*$/ search pattern in sed. So we could start with something like (the -n is there because we only care when we find said blank line)

yum check-update | `sed -n '/^\s*$/p'`

which if you run does not seem to do much. Reason is that what we really want is to know if the match was successful or not. And, sed actually has the answer: look at these entries I stole from its man pange:

q [exit-code]
              Immediately  quit  the  sed  script  without processing any more
              input, except that if auto-print is  not  disabled  the  current
              pattern  space will be printed.  The exit code argument is a GNU
              extension.

       Q [exit-code]
              Immediately quit the sed  script  without  processing  any  more
              input.  This is a GNU extension.

Let's try it then. First we will find a machine that has a package that needs to be updated. In this case it will be my KVM server, which you have met before when we talked about USB passthrough:

[raub@vmhost ~]$ yum check-update 
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: mirror.vcu.edu
 * extras: centos.mirror.constant.com
 * updates: mirror.vcu.edu

samba4-libs.x86_64                    4.2.10-7.el6_8                     updates
[raub@vmhost ~]$

So as of this writing it has one update. We then try our one-liner, which we want to return 0 if it did not find any pending upgrades and 1 if it did. And that is done by adding q1 past the search string, as in /^\s*$/q1, which means write 1 if search successful. Of course we now need to print the result, which can be done with echo $?. So, let's try it:

[raub@vmhost ~]$ yum check-update | `sed -n '/^\s*$/q1'` ; echo $? 1 [raub@vmhost ~]$

It thinks that it found it. I am not completely sure it works, so we need also to verify it works as it should when we find no updates:

[root@server1 ~]# yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.netdepot.com
 * extras: mirror.solarvps.com
 * updates: ftpmirror.your.org
[root@server1 ~]# yum check-update | `sed -n '/^\s*$/q1'` ; echo $?
0
[root@server1 ~]# 

Sounds like we have a winner. Now all that is left is to wrap something around it to something with that. I will leave that to you. Note that it does not differentiate between normal and security updates though.

Wednesday, July 27, 2016

Starting Tomcat manually in Docker

So I needed to test some settings between CentOS 6 + Apache Tomcat 6 + Java 6 and CentOS 7 + Tomcat7 + Java 8. Best way I found to do these quick tests is to build them in docker: write a quick dockerfile, spool it up, test, and blow it up.

Now, when I built and ran the tomcat 6 + CentOS 6 + Java 6 setup by connecting to the container and manually starting tomcat by typing

service tomcat6 start
It worked fine. So I then built the tomcat 7 + CentOS 7 + Java 8, and tried to start it

[root@tomcat ~]# systemctl start tomcat.service
Failed to get D-Bus connection: Operation not permitted
[root@tomcat ~]# 

Since systemd could not start it, and I could not figure out why (I do not take solace knowing I am not the only one), I tried starting it even more manually:

[root@tomcat /]# /usr/sbin/tomcat start
/usr/sbin/tomcat: line 21: .: /etc/sysconfig/: is a directory
/usr/sbin/tomcat: line 39: /logs/catalina.out: No such file or directory
[root@tomcat /]#

This is the time to do what everyone does at a time like this: look for answers online. All I got was someone asking the very same question. It seems if we want some answers we will need do more exploring on our own.

With that in mind, let's see those two lines we are being barked about:

[root@tomcat /]# sed -n '21p' /usr/sbin/tomcat
    . /etc/sysconfig/${NAME}
[root@tomcat /]# sed -n '39p' /usr/sbin/tomcat
  ${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \
[root@tomcat /]#

Not much help here, but we will revisit that later. First, let's run the bash script again, but this time in a debugging (-x) mode:

[root@tomcat /]# bash -x /usr/sbin/tomcat start
+ '[' -r /usr/share/java-utils/java-functions ']'
+ . /usr/share/java-utils/java-functions
++ _load_java_conf
++ local IFS=:
++ local java_home_save=
++ local java_opts_save=
++ local javaconfdir
++ local conf
++ unset _javadirs
++ unset _jvmdirs
++ set -- /etc/java
++ _log 'Java config directories are:'
++ '[' -n '' ']'
++ for javaconfdir in '"$@"'
++ _log '  * /etc/java'
++ '[' -n '' ']'
++ for javaconfdir in '"$@"'
++ conf=/etc/java/java.conf
++ '[' '!' -f /etc/java/java.conf ']'
++ local IFS
++ local JAVA_LIBDIR
++ local JNI_LIBDIR
++ local JVM_ROOT
++ '[' -f /etc/java/java.conf ']'
++ _log 'Loading config file: /etc/java/java.conf'
++ '[' -n '' ']'
++ . /etc/java/java.conf
+++ JAVA_LIBDIR=/usr/share/java
+++ JNI_LIBDIR=/usr/lib/java
+++ JVM_ROOT=/usr/lib/jvm
++ _javadirs=/usr/share/java:/usr/lib/java
++ _jvmdirs=/usr/lib/jvm
++ _load_java_conf_file /root/.java/java.conf
++ local IFS
++ local JAVA_LIBDIR
++ local JNI_LIBDIR
++ local JVM_ROOT
++ '[' -f /root/.java/java.conf ']'
++ _log 'Skipping config file /root/.java/java.conf: file does not exist'
++ '[' -n '' ']'
++ _javadirs=/usr/share/java:/usr/lib/java
++ _jvmdirs=/usr/lib/jvm
++ '[' -d '' ']'
++ '[' -n '' ']'
++ '[' _ '!=' _off -a -f /usr/lib/abrt-java-connector/libabrt-java-connector.so
-a -f /var/run/abrt/abrtd.pid ']'
++ _log 'ABRT Java connector is disabled'
++ '[' -n '' ']'
+ '[' -z '' ']'
+ TOMCAT_CFG=/etc/tomcat/tomcat.conf
+ '[' -r /etc/tomcat/tomcat.conf ']'
+ . /etc/tomcat/tomcat.conf
++ TOMCAT_CFG_LOADED=1
++ TOMCATS_BASE=/var/lib/tomcats/
++ JAVA_HOME=/usr/lib/jvm/jre
++ CATALINA_HOME=/usr/share/tomcat
++ CATALINA_TMPDIR=/var/cache/tomcat/temp
++ SECURITY_MANAGER=false
+ '[' -r /etc/sysconfig/ ']'
+ . /etc/sysconfig/
/usr/sbin/tomcat: line 21: .: /etc/sysconfig/: is a directory
+ set_javacmd
+ local IFS
+ local cmd
+ '[' -x '' ']'
+ set_jvm
+ local IFS=:
+ local cmd
+ local cmds
+ _set_java_home
+ local IFS=:
+ local jvmdir
+ local subdir
+ local subdirs
+ '[' -n /usr/lib/jvm/jre ']'
+ '[' -z '' ']'
++ readlink -f /usr/lib/jvm/jre/..
+ JVM_ROOT=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el7_2.x86_64
+ return
+ '[' -n /usr/lib/jvm/jre ']'
+ return
+ for cmd in jre/sh/java bin/java
+ JAVACMD=/usr/lib/jvm/jre/jre/sh/java
+ '[' -x /usr/lib/jvm/jre/jre/sh/java ']'
+ for cmd in jre/sh/java bin/java
+ JAVACMD=/usr/lib/jvm/jre/bin/java
+ '[' -x /usr/lib/jvm/jre/bin/java ']'
+ _log 'Using configured JAVACMD: /usr/lib/jvm/jre/bin/java'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ return 0
+ cd /usr/share/tomcat
+ '[' '!' -z '' ']'
+ '[' -n '' ']'
+ CLASSPATH=/usr/share/tomcat/bin/bootstrap.jar
+ CLASSPATH=/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-jul
i.jar
++ build-classpath commons-daemon
+ CLASSPATH=/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-jul
i.jar:/usr/share/java/commons-daemon.jar
+ '[' start = start ']'
+ '[' '!' -z '' ']'
[root@tomcat /]# + /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bo
otstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon
.jar -Dcatalina.base= -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -D
java.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/conf/logg
ing.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
 org.apache.catalina.startup.Bootstrap start
/usr/sbin/tomcat: line 39: /logs/catalina.out: No such file or directory
[root@tomcat /]#

If you never ran a bash script with the -x option, you should since it shows the steps being performed by the script, including tests, as it runs. For instance, you can see it starts by learning a lot about the current Java installation. After that, it loads some file, TOMCAT_CFG=/etc/tomcat/tomcat.conf, and then gives the first error message:

+ TOMCAT_CFG=/etc/tomcat/tomcat.conf
+ '[' -r /etc/tomcat/tomcat.conf ']'
+ . /etc/tomcat/tomcat.conf
++ TOMCAT_CFG_LOADED=1
++ TOMCATS_BASE=/var/lib/tomcats/
++ JAVA_HOME=/usr/lib/jvm/jre
++ CATALINA_HOME=/usr/share/tomcat
++ CATALINA_TMPDIR=/var/cache/tomcat/temp
++ SECURITY_MANAGER=false
+ '[' -r /etc/sysconfig/ ']'
+ . /etc/sysconfig/
/usr/sbin/tomcat: line 21: .: /etc/sysconfig/: is a directory
Now, /etc/tomcat/tomcat.conf (same thing as /usr/share/tomcat/conf/tomcat.conf) defines a few global (to tomcat) to variables. The top of the file also explains it is the file where you should define variables that are custom to your system but global to all tomcat instances being run here. For instance, when I built the tomcat6 container, I had

JAVA_HOME="/usr/lib/jdk1.6.0_41"

because that was the specific java version I wanted to run. Now, if we look not only at line 21 in /usr/sbin/tomcat but also around said line, we can see it wants to load a file in /etc/sysconfig

# Get instance specific config file
if [ -r "/etc/sysconfig/${NAME}" ]; then
    . /etc/sysconfig/${NAME}
fi

If we look at /etc/sysconfig,

[root@tomcat ~]# ls /etc/sysconfig/
network  network-scripts  rdisc  tomcat
[root@tomcat ~]#

It sure makes me think that $NAME = "tomcat" and $NAME is not defined.

For the second error message we should examine the following lines

[root@tomcat /]# + /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bo
otstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon
.jar -Dcatalina.base= -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -D
java.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/conf/logg
ing.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
 org.apache.catalina.startup.Bootstrap start
/usr/sbin/tomcat: line 39: /logs/catalina.out: No such file or directory

That really looks like it wants to write to the log file catalina.out but can't find it. So we take a look at the lines around line 39:

if [ "$1" = "start" ]; then
  ${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \
    -classpath "$CLASSPATH" \
    -Dcatalina.base="$CATALINA_BASE" \
    -Dcatalina.home="$CATALINA_HOME" \
    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
    -Djava.io.tmpdir="$CATALINA_TMPDIR" \
    -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \
    -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \
    org.apache.catalina.startup.Bootstrap start \
    >> ${CATALINA_BASE}/logs/catalina.out 2>&1 &
    if [ ! -z "$CATALINA_PID" ]; then
      echo $! > $CATALINA_PID
    fi

where we find the line

>> ${CATALINA_BASE}/logs/catalina.out 2>&1 &

That makes me think that the $CATALINA_BASE = "/usr/share/tomcat" since

[root@tomcat ~]# ls /usr/share/tomcat/logs/
catalina.out
[root@tomcat ~]#

Now, /etc/sysconfig/tomcat knows about $CATALINA_BASE even though it really does not define it (commented out):

#CATALINA_BASE="/usr/share/tomcat"

Sounds like we need to define $NAME and $CATALINA_BASE somewhere. My vote would be for
/etc/tomcat/tomcat.conf because it claims it is where we put custom stuff.

# For tomcat.service it's /etc/sysconfig/tomcat, for
# tomcat@instance it's /etc/sysconfig/tomcat@instance.

# THE TWO LINES I MENTIONED IN THE ARTICLE
NAME="tomcat"                                   
CATALINA_BASE="/usr/share/tomcat"

# This variable is used to figure out if config is loaded or not.
TOMCAT_CFG_LOADED="1"

# In new-style instances, if CATALINA_BASE isn't specified, it will
# be constructed by joining TOMCATS_BASE and NAME.
TOMCATS_BASE="/var/lib/tomcats/"

After that, I was able to start it and verify it was indeed running

[root@tomcat tomcat]# ps -ef|grep tomcat
root       352     1  8 13:09 ?        00:00:01 /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
root       372     1  0 13:09 ?        00:00:00 grep --color=auto tomcat
[root@tomcat tomcat]#

I wrote a shorter version of this article as a reply to the question I found online and mentioned earlier in this article. I hope it will be useful to the original poster.