Chapter 5: Running `stealth'

Now that stealth has been compiled, the construction of a policy file has been covered, and a service-account on the client has been defined, what must be done to run stealth in practice?

Here's what remains to be done:

In this chapter, these topics are discussed.

5.1: Installing `stealth'

As stealth is mainly a system administrator's tool, it could be installed in /usr/bin. In that case, do (as root) in the directory where stealth was compiled/unpacked: 
    ./build install program
Alternatively, another default location may be specified in the INSTALL.im file or may be provided to the build script. E.g., 
    ./build install program /usr/local/bin/stealth
installing the binary program as /usr/local/bin/stealth.

The provided icmake build script can be started without arguments for an overview of possible commands.

5.2: Stealth command-line and policy file options

Short options are provided between parentheses, immediately following their long option equivalents.

Option descriptions showing (C) can only be used on the command-line, and are ignored when specified in the second section of the policy file.

In the overview of options `<uds>' represents the name of the Unix Domain Socket to use, and `<file-spec>' refers to a (relative or absolute) specification of a file location.

With the first and second synopses relative locations (of the Unix Domain Socket and of other file-specifications) are interpreted relative to the current working directory.

Command-line options overrule options defined in the policy-file.

Only one of the options --daemon, --reload, --resume, --suspend or --terminate can be specified. The options --reload, --rerun, --resume, --suspend, and --terminate ignore any other options.

The following options are still recognized for backward compatibility with stealth pre-3.00 versions and will be removed in a future stealth version. They generate error messages suggesting alternatives:

The following options were discontinued starting since stealth version 3.00.00:

When specifying long options in policy files initial hyphens should be omitted. Here are some examples: 

    %%
    log /tmp/stealth.log
    verbosity 3

5.2.1: Rsyslog filtering

When using rsyslogd(1) property based filters may be used to filter syslog messages and write them to a file of your choice. E.g., to filter messages starting with the syslog message tag (e.g., STEALTH) use 

:syslogtag, isequal, "STEALTH:"   /var/log/stealth.log
:syslogtag, isequal, "STEALTH:"   stop
Note that the colon is part of the tag, but is not specified with the syslog-tag option.

This causes all messages having the STEALTH: tag to be written on /var/log/stealth.log after which they are discarded. More extensive filtering is also supported, see, e.g., http://www.rsyslog.com/doc/rsyslog_conf_filter.html and http://www.rsyslog.com/doc/property_replacer.html

Time stamps written by rsyslogd are not controlled by stealth's --time-stamp option, but, e.g., by a TZ specification in /etc/default/rsyslog. Simply add the line 

    export TZ=UTC
to /etc/default/rsyslog, followed by restarting rsyslogd configures rsyslogd to generate time stamps using UTC.

5.3: Construct one or more policy files

Here we assume that stealth is run by root, and that root wants to store information about the host client under the subdirectory /root/stealth/client.

Furthermore, we assume that reports of stealth integrity-scans should be sent to the user admin@elsewhere, who is only interested in receiving a short summary of changes, as the full report can always be read at the monitor itself. To accomplish this a small support-script was developed, filtering the report generated by stealth down to its essentials.

As the sha1sum program on the client may be compromised, it is a good idea to start the integrity scan by transferring the client's sha1sum program to the monitor first, to verify the integrity of that program locally (i.e., at the monitor), before trusting it to compute sha1sums of the client's files. The same holds true for any libraries and support programs (like find) that are used intensively during integrity scans.

Sha1sum checks should be performed on all setuid and setgid files on the client, and in order to be able reach all files on client, root@monitor is allowed to login to the root@client account using an ssh connection.

Furthermore, sha1sum checks should be performed on all configuration files, living under /etc and on the file /usr/bin/find (which is used intensively when performing the integrity checks).

Next, the construction of the required policy file, implementing the abovementioned requirements, is described in the following subsections.

5.3.1: DEFINE directives

First we write some DEFINE directives simplifying complex command specifications: 

    DEFINE  SSHCMD  /usr/bin/ssh root@client -T -q exec /bin/bash --noprofile
    DEFINE  EXECSHA1 -xdev -perm +u+s,g+s \( -user root -or -group root \) \ 
                    -type f -exec /usr/bin/sha1sum {} \;
The first DEFINE defines the ssh command to use: an ssh-connection will be made to the root account at the client.

The second DEFINE shows the arguments for find(1) when looking for all root setuid or setgid normal files. For all these files the sha1sum(1) program should be run.

5.3.2: USE directives

Next some USE directives, matching our specific, local, situation, are defined: 

    USE BASE        /root/stealth/client
    USE EMAIL       admin@elswhere
    USE MAILER      /root/bin/stealthmail
    USE MAILARGS    "Client STEALTH report"
    USE SSH         ${SSHCMD}

5.3.3: Commands

The following commands are now defined:

In this manual the sha1sum(1) program is frequently used when checking hash values. Stronger hash functions (like sha256sum(1)) might be preferred in practice. sha256sum's hash values are remarkably longer than sha1sum's hash values. When using these longer hash values in the manual it often clobbers the layout of examples. Therefore in this manual sha1sum(1) is continued to be used.

Realize, however, that when updating existing policy files to use sha256sum(1) instead of sha1sum(1), that previously generated log files (that used sha1sum(1)) are incompatible with log files obtained when using sha256sum(1). In practice this means that new log files need to be generated, and any previously geneerated log files must be disregarded.

5.3.3.1: Obtaining the client's sha1sum program

To copy the client's sha1sum program to a local directory we specify: 

    GET /usr/bin/sha1sum /root/tmp
This command must succeed.

5.3.3.2: Checking the integrity of the client's sha1sum program

Next, we check the integrity of the received sha1sum program. For this, we use the monitor's sha1sum program: 

    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
The LABEL command writes the label to the report file just before writing the sha1sum program's output.

The LOCAL command checks the sha1sum of the program copied from the client. The report is written on the file /root/stealth/client/local/sha1. If this fails, stealth terminates, alerting admin@elsewhere that the check failed. This is a serious event, as it indicates that either the monitor's sha1sum is behaving unexpectedly or that the client's sha1sum program has unexpectedly changed.

The sha1sum program may have changed due to a normal upgrade. If so, admin@elsewhere will know this, and can (probably) ignore the warning. The next time stealth is run, the (now updated) SHA1 value is used, and it again compares the obtained SHA1 value to the one obtained for the downloaded sha1sum program.

5.3.3.3: Checking the client's /usr/bin/find program

The client normally uses its find command intensively: find is a great tool for producing reports about almost any conceivable combination of characteristics of sets of files. Of course, the client's find command must itself be OK, as well as the client's sha1sum program. Now that we know that the client's sha1sum program is OK, we can use it to check the client's /usr/bin/find program.

Note that the monitor itself no longer needs to invest any significant processing load: only the client itself is taxed for checking the integrity of its own files: 


    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find

5.3.3.4: Checking the client's setuid/setgid files

Having checked the client's sha1sum and find programs, sha1 checksum checks should be performed on all setuid and setgid files on the client. For this we activate the sha1sum program on the client. In order to check the setuid/setgid files, the following command is added to the policy file: 

    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}

5.3.3.5: Checking the configuration files in the client's /etc/ directory

Finally, the client's configuration files are checked. Some of these files change so frequently that we don't want them to be checked. E.g., /etc/adjtime, /etc/mtab. To check the configuration file, do: 
    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         \ 
          /usr/bin/find /etc -type f -not -perm /6111   \ 
            -not -regex "/etc/\(adjtime\|mtab\)"        \ 
            -exec /usr/bin/sha1sum {} \;

5.3.4: The complete `policy' file

Here is the complete policy file we've constructed so far: 

    DEFINE  SSHCMD  /usr/bin/ssh root@client -T -q exec /bin/bash --noprofile
    DEFINE  EXECSHA1 -xdev -perm +u+s,g+s \( -user root -or -group root \) \ 
                    -type f -exec /usr/bin/sha1sum {} \;

    USE BASE        /root/stealth/client
    USE EMAIL       admin@elswhere
    USE MAILER      /root/bin/stealthmail
    USE MAILARGS    "Client STEALTH report"
    USE SSH         ${SSHCMD}

    USE DD          /bin/dd
    USE DIFF        /usr/bin/diff
    USE REPORT      report
    USE SH          /bin/sh

    GET /usr/bin/sha1sum /root/tmp

    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum

    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
 
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}

    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         \ 
          /usr/bin/find /etc -type f -not -perm /6111   \ 
            -not -regex "/etc/\(adjtime\|mtab\)"        \ 
            -exec /usr/bin/sha1sum {} \;

5.4: Running `stealth' for the first time

When stealth is now run, it creates its initial report files under root/stealth/client.

The first time stealth is run, it is usually run `by hand'. The initial run by hand probably benefits from the --stdout option, as it shows all executed commands on the standard output: 

    stealth --stdout policy
Furthermore, the reports are initialized. Running stealth this way for the policy file constructed in the previous sections produces the following output (lines were wrapped to improve readability): 
    GET /usr/bin/sha1sum /root/tmp
    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / -xdev -perm +u+s,g+s 
            \( -user root -or -group root \) -type f 
            -exec /usr/bin/sha1sum {} \;
    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         /usr/bin/find /etc 
            -type f -not -perm /6111   -not -regex "/etc/\(adjtime\|mtab\)"
            -exec /usr/bin/sha1sum {} \;
    LOCAL /usr/bin/scp -q root@client:/usr/bin/sha1sum /root/tmp
    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / -xdev -perm +u+s,g+s 
            \( -user root -or -group root \) -type f 
            -exec /usr/bin/sha1sum {} \;
    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                        /usr/bin/find /etc 
            -type f -not -perm /6111   -not -regex "/etc/\(adjtime\|mtab\)"
            -exec /usr/bin/sha1sum {} \;

5.4.1: The mailed report

The /root/bin/stealthmail script is called with the following arguments: 

    "Client STEALTH report" admin@elswhere

The mailed report contains information comparable to this: 


    STEALTH (4.00.00) started at Sat, 07 Feb 2015 22:10:56 +0100
        
    Check the client's sha1sum program
    Initialized report on local/sha1
    
    checking the client's /usr/bin/find program
    Initialized report on remote/binfind
    
    suid/sgid/executable files uid or gid root on the / partition     
    Initialized report on remote/setuidgid
    
    configuration files under /etc
    Initialized report on remote/etcfiles

5.4.2: Files under /root/stealth/client

Under /root/stealth/client the following entries are now available:

This completes the information generated by stealth during its first run.

5.5: Subsequent `stealth' runs

5.5.1: All files unaltered

When stealth is subsequently run, it updates its report files under root/stealth/client. If nothing has changed, the log-files remain unaltered. Subsequent runs will, however, add some new info to the file /root/client/report: 

    STEALTH (4.00.00) started at Sat, 07 Feb 2015 22:10:56 +0100
    
    Check the client's sha1sum program
    Initialized report on local/sha1
    
    checking the client's /usr/bin/find program
    Initialized report on remote/binfind
    
    suid/sgid/executable files uid or gid root on the / partition     
    Initialized report on remote/setuidgid
    
    configuration files under /etc
    Initialized report on remote/etcfiles
    
    STEALTH (4.00.00) started at Sat, 07 Feb 2015 22:22:15 +0100
Note that just one extra line was added: a timestamp showing the date/time of the last run. The systems administrator may rotate the report file every once in a while to reclaim some disk space.

5.5.2: Modifications occurred

Basically, three kinds of modifications are possible: additions, modifications, and removals. Here we'll show the effects all these changes have on stealth's output.

For illustrative purposes, the following changes were made to the client's files:

Next, stealth was again run, producing the following output:

5.5.3: Failing LOCAL commands

If the client's sha1sum program itself is altered, a serious situation has developed. In that case, further actions by stealth would be suspect, as their results might easily be currupted. Additional checks will be performed, but a warning is generated on the report file (and in the mail sent to admin@elsewhere): 

    STEALTH (4.00.00) started at Sat, 07 Feb 2015 22:27:15 +0100

    Check the client's sha1sum program
    MODIFIED: /root/tmp/sha1sum
        < fc62fc774999584f1e29e0f94279a652  /root/tmp/sha1sum
        > 45251e259bfaf1951658a7b66c328c52  /root/tmp/sha1sum
    
    *** BE CAREFUL *** REMAINING RESULTS MAY BE FORGED
    
    configuration files under /etc
    REMOVED: /etc/motd.org
        > 945d0b8208e9861b8f9f2de155e619f9  /etc/motd.org
    MODIFIED: /etc/motd
        < 945d0b8208e9861b8f9f2de155e619f9  /etc/motd
        > 7f96195d5f051375fe7b523d29e379c1  /etc/motd
(The report shows the removal of the previously added file motd.org, and the modification of motd. These are real, as the original motd file, modified earlier, was restored at this point).

5.5.4: Skipping (some) integrity checks

Some files or directories may not require integrity checks. Automated processes may modify files which are not threatening the proper functioning of running programs or processes. In those cases a file can be prepared holding the absolute paths of entries to be skipped. Each entry should appear on a line of its own without any additional information.

Stealth can be informed about this file using the --skip-files <file-spec> option. When a relative file specification is used with --skip-files it is interpreted a location relative to the current working directory. The skip-files file itself must contain the paths of the entries to be skipped during file integrity scans. If an entry is already present in a log file then stealth once generates an IGNORING message in the mail sent to the address specified at EMAIL in the policy file. Each entry in a skip-file must be on a line of its own and must be specified using absolute file paths. Entries ending in a slash are assumed to be directories whose full contents must be skipped. Other entries are interpreted as the names of files to skip. Initial and trailing blanks, empty lines and lines having a # as their 1st non blank character are ignored.

Here is an example: 


    stealth -e --skip-files /root/stealth/remote/skipping remote.pol

If an entry /etc/skipme appears in the current logs which is thereafter added to the skippath file then the mail generated by stealth once contains a line like the following: 

    SKIPPING: /etc/skipme
        > a7695bb2d019e60988e757a4b692acfe  /etc/skipme
The reported hash-value is the hash-value at the time of the stealth-run reporting the SKIPPING message.

Entries ending in a slash are assumed to be directories whose contents must (recursively) be skipped.

5.6: Automating repeated `stealth' runs

To automate stealth's integrity scans, a file /etc/cron.d/stealth could be created, containing a line like 

    2,17,32,47 * * * *  root    test -x /usr/bin/stealth && \ 
                           /usr/bin/stealth /root/stealth/client.pol
This starts stealth 2 minutes after every hour. In this example the ssh-key must not require a passphrase, as crontab(1) cannot provide passphrases of ssh-keys. Ssh-keys requiring passphrases can, however, be used if repeated stealth runs are controlled by a program like ssh-cron(1).

In general, randomized events are harder to notice. For this stealth offers the --repeat and --random-interval options. Both options expect an argument in seconds (or in minutes, if an m is appended to the specification). After each integrity scan the next integrity scan starts after the time interval specified by the --repeat option plus a random time value selected from the time interval specified by the --random-interval option. For example, the stealth daemon started by the following command repeatedly performs integrity scans between two and five minutes after the last integrity scan completed: 


    stealth --daemon /root/stealth/client.uds \ 
            --repeat 2m --random-interval 3m /root/stealth/client.pol
Once this program has started its ssh(1) connection to the client host persists, and a possibly required ssh-passphrase is no longer required. Additional (intermediate) integrity scans can always be requested (also without requiring ssh-passphrase specifications) using the command 

    stealth --rerun /root/stealth/client.uds

5.7: Report File Rotation

When stealth performs integrity scans it appends information to the report file. This file therefore eventually grows to a large size, and the systems manager controlling stealth might want to rotate the report file every once in a while (e.g., using a program like logrotate(1), also see the upcoming section 5.7.2). To ensure that no log-rotation takes place while stealth is busy performing integrity scans (thus modifying the report file) the options --suspend and --resume were implemented. Both options require the process-ID file of currently active stealth process as their argument.

For example, if a stealth process was once started using the command 


    stealth --daemon /root/stealth/small.uds --repeat 900 \ 
                    /root/stealth/small.pol
then the --suspend and --resume commands for this process should be called as: 
    stealth --suspend /root/stealth/small.uds
    stealth --resume /root/stealth/small.uds
The stealth process identified in the files provided as arguments to the --suspend and --resume options is called the daemon stealth process below.

The --suspend option has the following effect:

Now that the report file is no longer modified by the daemon stealth process, log-rotation may take place. E.g., a program like logrotate(1) allows its users to specify a command or script just before log-rotation takes place, and `stealth --suspend udsfile' could be specified nicely in such a pre-rotation section.

The --resume option has the following effect:

Note that, once --suspend has been issued, all commands except --resume and --terminate are ignored by the daemon stealth process. While suspended, the --terminate command is acknowledged as a `emergency exit' which may or may not interfere with, e.g., an ongoing log-rotation process. The daemon stealth process should not normally be terminated while it is in its suspended mode. The normal way to terminate a stealth process running in the background is:

5.7.1: Status file cleanup

Whenever stealth is run and it encounters a modified situation the already existing status file summarizing that particular situation is saved and a new status file is created. Eventually, this will result in many status files. While report files can be rotated, it is pointless to rotate old status files, since they are never modified. Instead, status files exceeding a certain age could be removed and more recent files might be zipped to conserve space. In stealth's binary distribution the file /usr/share/doc/stealth/usr/bin/stealthcleanup is provided which can be used to perform such a cleanup. The script expects one argument: a resource file defining the following shell variables: Here is the stealthcleanup script as contained in the binary distribution's /usr/share/doc/stealth/usr/bin directory: 
#!/bin/bash

usage()
{
    echo "
Usage: $0 [-v] rc-file
Where:
    -v: Show the actions that are performed
    rc-file: resource file defining:
            \`directories' - one or more directories containing status files
            \`gzdays'      - number of days status files may exist before they
                             are compressed
            \`rmdays'      - number of days gzipped status files may exist
                             before they are removed. 
"
    exit 1
}


error()
{
    echo "$*" >&2
    exit 1
}

if [ "$1" == "-v" ]
then
    verbose=1
    shift 1
else
    verbose=0
fi

[ $# == 1 ] || usage

# now source the configuration file
. $1           

for x in $directories
do
    cd $x || error "\`$x' must be a directory"
    if [ $verbose -eq 1 ]
    then
        echo "
cd $x"
    fi

    if [ $verbose -eq 1 ]
    then
        echo \
    /usr/bin/find ./ -mtime +$rmdays -type f -regex '.*[0-9]+-[0-9]+\.gz' \
        -exec /bin/rm {} \;
    fi
    /usr/bin/find ./ -mtime +$rmdays -type f -regex '.*[0-9]+-[0-9]+\.gz' \
        -exec /bin/rm {} \;

    if [ $verbose -eq 1 ]
    then
        echo \
    /usr/bin/find ./ -mtime +$gzdays -type f -regex '.*[0-9]+-[0-9]+' \
        -exec /bin/gzip {} \;
    fi 
    /usr/bin/find ./ -mtime +$gzdays -type f -regex '.*[0-9]+-[0-9]+' \
        -exec /bin/gzip {} \;
done

exit 0
Assuming that the status files are written in /var/stealth/target/local and /var/stealth/target/remote; that status file should be compressed when older than 2 days and removed after 30 days, the resource file is: 
directories="
    /var/stealth/target/local
    /var/stealth/target/remote
    "

rmdays=30
gzdays=3
Furthermore assuming that the resourcefile is installed in /etc/stealth/cleanup.rc and the stealthcleanup script itself in /usr/bin/stealthcleanup, the stealthcleanup script could be called as follows: 

    /usr/bin/stealthcleanup /etc/stealth/cleanup.rc
Note that stealthcleanup may be called whether or not there are active stealth processes.

5.7.2: Using `logrotate' to control report- and status files

A program like logrotate(1) allows its users to specify a command or script immediately following log-rotation, and `stealth --resume pidfile' could be specified nicely in such a post-rotation section.

Here is an example of a specification that can be used with logrotate(1). Logrotate (on Debian systems) keeps its configuration files in /etc/logrotate.d, and assuming there is a host target, whose report file is /var/stealth/target/report, the required logrotate(1) specification file (e.g., /etc/logrotate.d/target) could be: 

/root/stealth/report /var/log/stealth/client-small.log {
    weekly
    rotate 12
    compress
    missingok
    copytruncate
    sharedscripts
    prerotate
        /usr/bin/stealth --suspend /root/stealth/small.uds
    endscript
    postrotate 
        /usr/bin/stealth --resume /root/stealth/small.uds
    endscript 
}
Using this specification file, logrotate(1) will Note thet stealth --resume xxx always initiates another file integrity scan.