check_ps v1.1 -- RootKit 'ps' program detector
HISTORY
On September 17, 1997, Duncan Simpson (D.P.Simpson@ECS.SOTON.AC.UK) posted a program called check_ps to the Bugtraq mailing list. The program was in response to having a "rootkit" installed on one of his machines. Rootkit's typically include a hacked version of the 'ps' program that will not list specific processes, thereby hiding them from the SysAdmin and other users of the system (e.g. so the hacker can run a version of Crack without being caught).
This program runs in the background, periodically executing the 'ps' program and checking its contents against the list of processes in a SysV-style /proc file system. Any processes that appear in /proc and DO NOT appear in the information returned by 'ps' are logged and can even be killed. Any processes that appear in the output of 'ps' and not /proc are also reported (this might be done to give you the impression that syslogd is running when it is not, for example). Restriction: non-extant processes with non-fixed pids reported are not detected but easy for humans to detect.
This version with minor additions got called version 1.0, despite a number of serious security problems (part of the reason I posted asking for comments). The only additions were
- Can be run as a one-time check, rather than as a continuous daemon process.
- Can log to an alternative log file so that the cracker will not see the messages from check_ps.
- Can just log that the hidden processes exist, rather than killing them outright and arousing the suspicion of the cracker.
Subsequently I did some fixes and got provoked into creating this version by a message from NASA (informing me that CIAC and various others where interested...obviously this baby fills a need). The code has been improved and various nasty security problems eliminated (it ran ps as root and if ps has been hacked that is obvious a bad idea).
If you use linux you get sophisticated information about what files the processes have open, including the source and destination of the TCP connection to your machine in many cases (even if you use telnet). You also learn the port on your machine, thus which service got cracked. If the cracker is using UDP this does not work of course. It should know about IPX connections too...
N.B. The program is only effective if the cracker fails to notice it until it has told you and done whatever level of zapping you ask it to perform. OTOH A known clean version in single shot mode might be handy in an incident response situation, I guess.
INSTALLATION AND TESTING
Extract the check_ps-1.3.1.tar.gz file into a source directory:
$ cd /usr/local/src
$ tar xzf <download-dir>/check_ps-1.2.tar.gz (GNU tar)
$ cd check-ps-1.4-beta1
$ ./configure
Now edit cfg_smtp.h to more appropiate values. If you fail to do this the compilation process bombs out in log_email.h. This is intended and not a bug.
Where "<download-dir>" is the directory you downloaded the check_ps-1.4beta1.tar.gz file to. Run the configure script to probe your system for the right settings. The items this fixes are marked with (***auto configure***). The script is generated by GNU autoconf from configure.in, aclocal.m4 and acconfig.h, which you can verify to ensure I did not put any evil trojans in that script. Note that configure is smart enough to look up user and group ids by reading /etc/passwd and /etc/group.
System specific but site indepent settings are largely stored in config.h which is an edited copy of config.h.in, a job best left to ./configure. The configure script will fail if run as a user which can not list everyone's process and the same obviously applies to the binary doing good stuff for you (it might find the other processes with kill scanning and complain). There is no other known interaction with the secure linux patch, or patches with the same effect on other systems.
Edit the header lines to suit your system. Items marked CHANGE THIS must be changed or all hell will be let loose. Items marked ***auto confifue*** are set to probably reasonable values by configure. Major items are listed by header file.
cfg_check.h:
PS_PATH path of ps on your system (***auto configure***)
PS_ARGS argument to list all processes (***auto configure***)
PROCPATH path of proc filing system for fetching pids
RECHECKNAP Time to sleep before rechecking PS (once per cycle
maximum) if something suspect appears in secs.
(default 1).
cfg_smtp.h:
RECIPIENTS comma seperated list of recipients in quotes.
Change this or get flamed every time I get an
email you should have got. (CHANGE THIS)
ADDR Address of machine to use (IP numbers)
SENDER Email sender (CHANGE THIS)
SUBJECT Email subject (default "problem report")
MACHINE Machine name to give in HELO command. (CHANGE THIS)
cfg_log.h:
Nothing major here.
cfg_prog.h:
HAVE_FSUID Define this if you have setfsuid(2)
DAEMON_ZAP_MD Deamon mode zap behavour.
0=Do nothing
1=Send SIGSTOP
2=Send SIGKILL (default)
There are also some esoteric items that you probably want to leave alone. If you are an incorrigable #define editor the values are listed below.
cfg_check.h:
MAX_PROCS max processes PS is allowed (default 1)
DEFUID uid to runs ps as (***auto configure***)
DEFGID gid to runs ps as (***auto configure***)
TOLERANCE Max leading white space/trailing junk (default 5120)
NUM_TOLERANCE Max digits in pid (default 10)
SHORTNAP Time to sleep before re-trying getting process id
list (the real one from /proc) in secs. (default 50)
LONGNAP Time to sleep between checks in secs (default 600)
PS_MAXTIME Time to wait before killing PS in secs (default 300)
DISABLE_TIME Time to sleep if ps gets killed by above timeout in
secs (default 1200).
PROCFNAME_MAX Buffer size for making sure a pid is still active,
used by the hidden pid tester to avoid false +ve
cases due to races. Be generous. (Default 1024).
cfg_smtp.h:
USE_GETHOST define this if you are stupid enough to use a hostname
SMTP_MACHINE name of machine to use if the above is defined
SMTP_PORT port to use to send mail (default IPPORT_SMTP aka. 25)
TIMEOUT SMTP response time in secs (default 300 which is <RFC
but too long might give the game away).
cfg_log.h:
INITIAL_BUFFER Initial buffer size (default 1024)
TEXT_INC Increment size when expanding buffer (default 512)
TEXT_MAX Biggest message to send you. (default 10240)
MAXLINE longest line to support (default 4096)
cfg_prog.h:
DAEMON_ARGV argv to exec program with (defualt { "httpd", NULL })
DAEMON_UID real user id to set (***auto configure***)
DAEMON_GID real group id to set (***auto configure***)
linux/os-cfg.h:
FNAME_LEN maximum filename length (default 200)
STAT_BYTES bytes to read from /proc/<pid>/stat (default 2000)
CMD_BYTES bytes to read from /proc/<pid>/cmdline (default 3000)
numbuf.h
MAXX_NUMS Maximum numbers to accept, to stop DoS attacks
(default 10000)
INITIAL_NUMS Initial number buffer size (default 200)
INC Number of numbers to add when expanding (default 50)
Check the Makefile for your operating system and version of the C compiler. As generated it is set for a generic ANSI C compiler and prefers procs fs over using the ps database (which has the problem that support is not implemented yet). Compile check_ps with the 'make' command.
A simple shell script, called 'fake_ps_2', has been provided so that you can test check_ps without actually having to install a rootkit yourself. 'fake_ps_2' just runs the ps command through an 'egrep' filter to block out one or more processes, and adding a fake process, emulating a rootkit ps. Here is the fake_ps script in its entirety:
#!/bin/sh
#
# A simple little fake ps to test check_ps with #
# WARNING: Using this little shell script to test check_ps may generate some # spurious 'hacked ps' messages because the AVOID string will also block out # the call to egrep in the pipeline. The extra process will also generate a # message (unless there really is such a process, which I hope not). #
# Fake pid added by Duncan. Output is "ps ax" format on Linux. REALPS="/bin/ps"
AVOID="sleep"
($REALPS $* | egrep -v $AVOID); echo " 5999 ? R 0:00 not_such_process"
Probably the easiest and safest way to test check_ps is to put a 'sleep' command in the background and then try to find it. Linux offers the best performance, other systems will not tell you all the detail about the "hidden" pid and related processes; to fix this write a processor for /proc/<pid> on your system (linux/process.c is probably a good start).
There is a special version of check_ps build in addition to the production version called check_ps_test that uses ./fake_ps_2 instead of real ps (fake_ps plus the faked process, although you can just "cp -fv fake_ps fake_ps_2" to remove that lie). It also features a few other differences that got added since the demo below, in particular a fgetc(stdin) between a fishing run and its repeat.
$ sleep 1200 &
[1] 512
$ ./check_ps_test
Dec 03 11:54:32 Checking /usr/local/src/check_ps-1.0/fake_ps ax
Dec 03 11:54:33 hacked ps: pid 512 not matched
Dec 03 11:54:43 hacked ps: fake pid 5999 inserted
In this example, process 512 was detected and reported as a hidden process id. The message that saying 'fake pid ... inserted' illustrate another feature of check_ps: it reports non-existent processes that are being listed by the ps command. Rootkit ps programs may do this, for example, to make it look like a syslog daemon is running, when in fact it is not.
There is a demo of the linux output in the file checkps-demo. Note the information shows where the telnet connection was coming from. The second and third runs also demostrate the inode cache noticing details about the ps program (./fake_ps_2 in the test version) changing. I do not have a convient and low impact clock twiddler so there is no demonstration of clock twiddling.
Note that the opporunity to hit ^Z is not featured in the non-test version (the check_ps binary) and the not test version sets fascist resource limits and changes real, effective and saved user id before running ps. It probably should kill off lots of capabilities too. This is a call for patches.
<News from Duncan>
If a non-extant process id is reported by ps in two sucessive checks it is reported. This is designed to stop race conditions generating false hits but detect things like syslogd insertions. These would be obvious to you if the attacker used different pids each time you ran ps! Remember that the program is not a subsitute for you, just a simple trap for those who hack ps and fail to notice it. (If this tool becomes widely avialable someone will probably trojanize a version of this program...)
I was lucky: the attacker failed to deploy the rootkit correctly. After netstat dumped core twice I applied strace and saw untoward system calls. Investigation revealed the rootkit. After determining the damage and re-installing software from known clean sources I added a cron job to remove the rootkit control files every 5 minutes (no trojan version of rm supplied). I think ls and ps were not trojanized but I used "echo " and "echo ." just in case... and re-installed ls from a clean source. I checked ps using echo * in /proc.
Man page deliberately absent so no fool installs it! (Installing it would make "man check_ps" an easy way of detecting the program, which is obviously bad!). The RPM database feature discourages me from rolling a RPM. The code is very resistant to races now.
<End news from Duncan>
COMMAND LINE OPTIONS
check_ps has the following command line options:
--daemon -d Run as a daemon
--kill -k Kill hidden processes (uses KILL)
--halt -h Stop to hidden processes (uses STOP).
--file=<file> -l <file> Log messages to <file> rather than
to the syslog
--email, -e Use email logginf
--version -V version information
--confirm -c Log pid on startup (recommended only with -e)
--killscan -p Use kill scanning in addition to /proc
(just fixing /proc will not work against
kill scanning).
By default, check_ps will run only once, not as a daemon; it will not kill the hidden processes; and it will log its messages to standard output. See the example given above. Note that daemon mode probably should be set to using email because this stops crackers do anything about it. The email is innocous (an email titled "problem report" from MAILER-DAEMON) bar the content. If you have a permanent connection then a hotmail and the like are free and hard to hack while attacking your boxen. The program talks to port 25 directly to avoid all the system logs and local programs..
To run as a daemon, kill processes and log to email execute the following:
$ check_ps -dke
BUGS
Code to kill the parent of the hidden process, originally given by Duncan Simpson, but #ifdef'd out, has not yet been implemented.
AUTHORS
Original check_ps.c, Duncan Simpson (D.P.Simpson@ECS.SOTON.AC.UK).
Code to implement command line options, README, and fake_ps script, Lee E. Brotzman, Allied Technology Group, Inc., NASA Incident Response Capability (NASIRC), leb@nasirc.hq.nasa.gov.
Lots of improvements, code reorganisation and netstat scanning, Duncan Simpson.