Forum OpenACS Q&A: How to bypass rp

Posted by Tom Jackson on 09/19/01 01:28 AM

The AOLserver list has had a thread going today about the nimda worm. Some sites are being hit by hundreds of requests. To avoid the cost of these hits a filter is being used to capture these hits and hold the connection open for a minute or so. This seems to help because the worm tries numerous exploits before moving on to another ip.

The filter I am using is being sourced from the private tcl/init.tcl file, which should guarantee that it is the first to be read at server startup. In that script I register a filter for preauth. This should guarantee that it is the first filter to run for a matching request. For some reason rp_filter is still running. Below is the filter, and some of the server log showing what is being executed per request.


nsv_set nimda conns 0
nsv_set nimda max 20
nsv_set nimda count 0

proc nimda {conn ignore} {

  global breaktrace
  set breaktrace 1
  set req [ns_conn request]
  set reqlist [split $req " "]
  set url [lindex $reqlist 1]
  set host [ns_conn peeraddr]
  
  nsv_incr nimda count
  if {[nsv_get nimda conns] < [nsv_get nimda max]} {
    nsv_incr nimda conns
    ns_sleep 60
    nsv_incr nimda conns -1
  } 
  ns_returnredirect http://$host$url
  return filter_return
  

}

proc nimda_breaktrace {conn ignore} {

  global breaktrace

  if {[info exists breaktrace] && $breaktrace == 1} {
    ns_log Notice "Nimba BreakTrace holding: [nsv_get nimda conns]
total: [nsv_get nimda count] [ns_conn peeraddr]"
    return filter_return
  }
  return filter_ok

}


ns_register_filter preauth GET /scripts/* nimda

ns_register_filter trace  GET /scripts/* nimda_breaktrace


------ Server.log ---
[18/Sep/2001:15:21:13][2986.15375][-conn5-] Notice: Security:
1000851673 sec_generate_session_id_cookie setting 340225, 0.
[18/Sep/2001:15:21:15][2986.25612][-conn11-] Notice: Nimba BreakTrace
holding: 2 total: 241 216.138.84.30

Also, not shown are calls to ad_permission looking for a match on the directory.

Any hints on how rp_filter is getting into the request?

2: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Jerry Asher on 09/19/01 03:56 AM

Tom,

I can't answer your question, but I do want to thank you for the filter. It's completely compatible with an nsvhr master reverse proxy, and it appears to be helpful in keeping nimda in check.

There are some other nimda requests that I'm trapping /_*, /msadc/*, /MSADC/*, /c/* /d/*, and while I understand why you try to direct the attack back against the host, I think channeling it to http://www.microsoft.com is equally reasonable.

3: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Tom Jackson on 09/19/01 04:11 AM

Jerry,

I have reason to believe that our little worm doesn't bother with redirects.

More info later tonight

4: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Jerry Asher on 09/19/01 04:54 AM

Oh I agree, I can't imagine a virus writer caring a whit about 302s.

5: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Tom Jackson on 09/19/01 05:40 AM

Thanks Jerry, you answered my question. I didn't realize the worm was looking in other directories. To my above script, the following ammendments will catch all the worm attempts, eliminating db hits.

# Proc to log all requests to server.log for easier analysis
# probably not for use on heavy traffic sites.

proc all_and_everything {conn ignore} {

  ns_log notice "[ns_conn peeraddr] [ns_conn request]"
  return filter_ok
}

# New ns_register_filters:

ns_register_filter preauth GET /* all_and_everything

ns_register_filter preauth GET /*.exe nimda

ns_register_filter trace  GET /*.exe nimda_breaktrace

# Note order of registration above is important!

6: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Tom Jackson on 09/19/01 05:53 AM

The log below demonstrates that holding the attacker definitely slows
them down. Here three separate attackers slowly make their way through
their program:

[18/Sep/2001:19:42:19][4946.9226][-conn0-] Notice: 216.102.228.12 GET /scripts/root.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:42:21][4946.8201][-sched:11-] Notice: Starting jconn weblog@pathfinderschool.com/ws
[18/Sep/2001:19:42:21][4946.8201][-sched:11-] Notice: Conn is jabber1 and xmlnode is xode1
[18/Sep/2001:19:42:21][4946.3076][jabpool.1] Notice: In thread 1000867341 :: [139961317]
[18/Sep/2001:19:43:19][4946.9226][-conn0-] Notice: Nimba BreakTrace holding: 0 total: 1 released ip: 216.102.228.12
[18/Sep/2001:19:43:21][4946.9226][-conn0-] Notice: 216.43.28.103 GET /scripts/root.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:43:21][4946.10251][-conn1-] Notice: 216.102.228.12 GET /MSADC/root.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:44:21][4946.9226][-conn0-] Notice: Nimba BreakTrace holding: 1 total: 3 released ip: 216.43.28.103
[18/Sep/2001:19:44:21][4946.10251][-conn1-] Notice: Nimba BreakTrace holding: 0 total: 3 released ip: 216.102.228.12
[18/Sep/2001:19:44:24][4946.9226][-conn0-] Notice: 216.43.28.103 GET /MSADC/root.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:44:26][4946.10251][-conn1-] Notice: 216.102.228.12 GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:45:24][4946.9226][-conn0-] Notice: Nimba BreakTrace holding: 1 total: 5 released ip: 216.43.28.103
[18/Sep/2001:19:45:25][4946.9226][-conn0-] Notice: 216.43.28.103 GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:45:26][4946.10251][-conn1-] Notice: Nimba BreakTrace holding: 1 total: 6 released ip: 216.102.228.12
[18/Sep/2001:19:45:28][4946.10251][-conn1-] Notice: 216.102.228.12 GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:46:25][4946.9226][-conn0-] Notice: Nimba BreakTrace holding: 1 total: 7 released ip: 216.43.28.103
[18/Sep/2001:19:46:28][4946.10251][-conn1-] Notice: Nimba BreakTrace holding: 0 total: 7 released ip: 216.102.228.12
[18/Sep/2001:19:46:34][4946.9226][-conn0-] Notice: 216.43.28.103 GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:47:14][4946.10251][-conn1-] Notice: 216.70.28.5 GET /scripts/root.exe?/c+dir HTTP/1.0
[18/Sep/2001:19:47:34][4946.9226][-conn0-] Notice: Nimba BreakTrace holding: 1 total: 9 released ip: 216.43.28.103
[18/Sep/2001:19:48:14][4946.10251][-conn1-] Notice: Nimba BreakTrace holding: 0 total: 9 released ip: 216.70.28.5
[18/Sep/2001:19:48:17][4946.9226][-conn0-] Notice: 216.70.28.5 GET /MSADC/root.exe?/c+dir HTTP/1.0

7: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Gilbert Wong on 09/19/01 05:55 AM

Thanks Tom! I'm also getting requests for /xmlstatus.asp. Though, I'm not sure if that's related to the nimba worm.

8: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by David Eison on 09/19/01 04:13 PM

Tom, thanks for the filter.

I can't replicate the behavior you described w/ the requestprocessor also filtering something that was handled by your filter. If you turn on DebugP and LogDebugP in your site map/ACS Kernel parameters you should get RP logging notices when the request processor is invoked; are you sure you're getting these notices, and that your registrations are done in /home/aol3whatever/modules/tcl/init.tcl? (If you put your code into /web/sitename/tcl/init.tcl, it will be run after rp_filter is registered, so then the described behavior would make sense)

9: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Tom Jackson on 09/19/01 06:33 PM

The problem was that I wasn't catching all the worms, only those matching /scripts/*. Replacing the filter glob to /*.exe, allowed me to catch all worm attempts. If you are using my filter, or the original by Jim, I would recommend the change. Also, if you are using ACS, you must use a trace filter as well, otherwise you will run the risk of numerous db errors showing up in your log file. The global var I used probably isn't necessary.

10: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by David Walker on 09/20/01 01:54 PM

my solution (using linux) is to exec a script using sudo that calls
a script that calls ipchains and adds their IP to an ipchains block
list and then does an ns_conn close.

It appears to me that this makes their client wait, thinking the
connection is still open while on the server side the connection is
closed and no extra server resources are expended. I don't know how
big the block list can get before it affects linux performance but I
have a cron job clear it periodically.

I can post the scripts and instructions if anyone is interested.

11: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Jerry Asher on 09/20/01 04:16 PM

Yes, please post the scripts. Thanks.

12: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by David Walker on 09/20/01 05:14 PM

Prerequisites:
ipchains support (enabled by default in most cases)
sudo (you might have to install this)

First create a chain to hold the blocked IPs and place it at the head of the input and output chains. (These will be erased if you reboot. Make sure they appear above any other rules you might have)

        /sbin/ipchains -N blockip
        /sbin/ipchains -A input -p tcp -j blockip
        /sbin/ipchains -A output -p tcp -j blockip

The blockip script is simply intended to keep the web server from having the right to do anything other than the intended action with ipchains. I place it in /root/bin/blockip and execute it using sudo

#!/usr/bin/tclsh
# blockip takes a single IP address and adds an ipchains rule to
# block it in both directions

set ip [lindex $argv 0]
if {$argc != 1 || ![string is digit [string map { . ""} $ip]]} {
        puts "give me the IP and just the IP"
}
 
puts [exec /sbin/ipchains -A blockip -p tcp -s $ip -j DENY ]
puts [exec /sbin/ipchains -A blockip -p tcp -d $ip -j DENY ]

use visudo to add this line to /etc/sudoers

nsadmin ALL=NOPASSWD: /root/bin/blockip

Create a .tcl file containing the following and place it in /web/<servername>/tcl/ (or wherever your server's tcl directory is)

ad_register_filter -critical t -priority 1 preauth * /*.exe 
vt_blockip
 
proc vt_blockip {conn ignore} {
        exec sudo /root/bin/blockip [ns_conn peeraddr]
        ns_conn close
        return "filter_break"
}

Edit tcl/ad-security.tcl and change
ad_register_filter -critical t -priority 1 preauth * /* sec_read_security_info
to
ad_register_filter -critical t -priority 2 preauth * /* sec_read_security_info
so that the blocking filter will be higher priority than the security filter.

and, finally, here is the cron script to clear the blocked IPs

#!/bin/sh
 
/sbin/ipchains -F blockip

13: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by Tom Jackson on 09/20/01 07:12 PM

David

ns_conn close does not work to slow down the worm. The ipchains idea is doing all the work. Here is some output, with the use of ns_conn close instead of ns_sleep 60.

[20/Sep/2001:09:05:45][16725.9226][-conn0-] Notice: 216.254.126.216 GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0
[20/Sep/2001:09:05:45][16725.9226][-conn0-] Notice: Nimba BreakTrace holding: 0 total: 13 released ip: 216.254.126.216
[20/Sep/2001:09:05:45][16725.9226][-conn0-] Notice: 216.254.126.216 GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0
[20/Sep/2001:09:05:45][16725.9226][-conn0-] Notice: Nimba BreakTrace holding: 0 total: 14 released ip: 216.254.126.216
[20/Sep/2001:09:05:46][16725.9226][-conn0-] Notice: 216.254.126.216 GET /scripts/..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0
[20/Sep/2001:09:05:46][16725.9226][-conn0-] Notice: Nimba BreakTrace holding: 0 total: 15 released ip: 216.254.126.216
[20/Sep/2001:09:05:46][16725.9226][-conn0-] Notice: 216.254.126.216 GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0

Another thing I'd like to figure out is why some posts are nicely formatted when I use Plain Text and others are not. Is it just the q-and-a-post-reply-form is working correctly?

14: Response to How to bypass rp_filter: Nimba Worm (response to 1)

Posted by David Walker on 09/20/01 07:49 PM

the ns_conn close is intended to make sure that aolserver knows that the connection is closed. It probably isn't even needed.

Forum OpenACS Q&A: How to bypass rp_filter: Nimba Worm