~ overflow ~

Problem:
So i noticed a real slow down on my server recently, nothing new was installed, no new sites launched. Investigating it a little further i saw like +100 sendmail processes running in background, WTF!

Solution:

Looking at huge maillogs, i found that there’s a botnet hammering my mail server, trying to authenticate with random strings and send unauthenticated emails, as if my server were configured as a promiscuous email server.

So, i had to put in practice my never used before iptables config skills.

First i added a rule to avoid global hammering:

-N SMTP-BLOCK
-A SMTP-BLOCK -s 127.0.0.1 -j ACCEPT
-A SMTP-BLOCK -s ##YOUR_SERVER_IPS## -j ACCEPT
-A SMTP-BLOCK -m limit --limit 1/m --limit-burst 3 -j LOG --log-level notice --log-prefix "iptables SMTP-BLOCK "
-A SMTP-BLOCK -m recent --name SMTPBLOCK --set -j DROP
-A INPUT -p tcp --dport 25 -m state --state NEW -m recent --name SMTPBLOCK --rcheck --seconds 360 -j SMTP-BLOCK
-A INPUT -p tcp --dport 25 -m state --state NEW -m recent --name SMTP --set
-A INPUT -p tcp --dport 25 -m state --state NEW -m recent --name SMTP --rcheck --seconds 60 --hitcount 10 -j SMTP-BLOCK
-A INPUT -p tcp --dport 25 -m state --state NEW -j ACCEPT

This will avoid hammering on port 25, SMTP, make sure you add your server’s ip here, otherwise it might slowdown your own server from sending emails out. This will ban users for 60 seconds if they send more than 3 packets a minute on port 25, pretty much enough what you need to do.

Now the bot net will still hammering, since they have MANY different ips , 800 so far, they can still hammering and eating up your server resources and bandwidth, since they will not stop i wrote a script to analyze maillogs and ban by ip:

// (c) z3n - R1V1@100605 - www.overflow.biz - rodrigo.orph@gmail.com

if (!isset($argv[2])) die("Usage: ".$_SERVER['PHP_SELF']." <input file> <ip tables>");

function _gii($s,$s_1,$s_2,$st=1) { // what, start string, end string, strip tags bool
	if ((stripos($s,$s_1) !== false) && (stripos($s,$s_2) !== false)) {
		$p=strlen($s_1)+stripos($s,$s_1);
		return ($st == 0) ? substr($s,$p,stripos($s,$s_2,$p)-$p) : strip_tags(substr($s,$p,stripos($s,$s_2,$p)-$p));
	} else {
		return "";
	}
}
function _r($x) { echo $x."\n"; }

$ips=array();
$handle=fopen($argv[1],"r");
while (!feof($handle)) {
	$buf=fgets($handle,4096);
	$x=explode("\n",$buf);
	foreach ($x as $v) {
		if (strpos($v,"Relaying denied") !== false) {
			$ip=_gii($v," [","]",0);
			if (!in_array($ip,$ips))
				$ips[]=$ip;
		}
	}
}

for ($r="",$i=0,$j=count($ips),_r("Found: ".$j." entries, building ban list..."),$ip_tables=file_get_contents($argv[2]);$i < $j;$i++)
	if (strpos($ips[$i],$ip_tables) === false)
		$r.="-A INPUT -s ".$ips[$i]." -j REJECT\n";

if (!empty($r)) {
	file_put_contents("maillog.ban",$r);
	_r("Wrote maillog.ban");
} else {
	_r("No new entries found");
}

..not so good on preg, but it works :P
once you generated the maillog.ban file just paste it on the iptables.

Besides that you may want to tune up your MTA child limit in order to avoid server overloading.

You may also want to keep this:

-A INPUT -p tcp -m tcp --syn -m limit -m state --limit 1/second --limit-burst 5 --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --syn -m state --state NEW -j REJECT

on for a while, it will ban ips based on their sync packet usage, let’s say that our unhappy script kiddie who owns the botnet starts to ping the server instead, this will avoid server from having resources wasted, but may also cause issues with legit clients.

I also found this article that shows how to disable ping echoing straight on kernel just by adding net.ipv4.icmp_echo_ignore_all = 1 at /etc/sysctl.conf.

Problem:

Convert VirtualBox Disk Image (VDI) to VMWare Harddisk Format (VDMK)

Solution:

Solely using qemu will generate a broken image, the right solution is:

 VBoxManage --convertSettings internalcommands converttoraw INPUT.vdi OUTPUT.raw && qemu-img convert -O vmdk OUTPUT.raw FINAL.vmdk && rm -f OUTPUT.raw

Source:

commandlinefu

Problem:

While installing virtualbox on my server i found out that it had a hell of deps, which i can’t seem to find.

Solution:

Search around i found out that yum is able to find and install deps off a rpm, like this:

yum localinstall --nogpgcheck VirtualBox-2.1.2_41885_fedora9-1.i386.rpm

I had to umount a drive but it wasn’t working, not even with -f (force) option.

So i looked man pages and found out about lazy option (-l) which solved the issue;

umount -l /path/to/mount/point

Problem:
My fedora 10 repos stopped working.

Solution:

My host had the dedicated configurated with a single static mirror. I figured out that by removing the baseurl from repos and enabling the mirrorlist it would work just fine.

Download:

Fedora 10 Mirror Repos

Problem:

Zend requires registration in order to dowload the free optimizer, it’s annoying and you keep receiving spam.

Solution:

http://downloads.zend.com/optimizer/3.3.3/ZendOptimizer-3.3.3-linux-glibc23-x86_64.tar.gz (linux 64-bit)
http://downloads.zend.com/optimizer/3.3.3/ZendOptimizer-3.3.3-linux-glibc23-i386.tar.gz (linux 32-bit)

neat.

Problem:

When using regular linux ssh it keeps timeouting or simply freezes when no inputs/outputs happen.

Solution:

Usually ssh has a TCP keep alive switch which you can enable to avoid this, although depending on your router and other network odds this might not help, also you may need to send keep alives in a shorter time than what’s the default of ssh. So, in order to have this working you need to edit /etc/ssh/ssh_config adding this to Host *, assuming you want to have this settings globally for every host.

ServerAliveInterval 60
ServerAliveCountMax 99999999999999

Why so many 9’s? There’s no way to do a endless keep alive, if you put 0 it will just drop the connection after the first keep alive, -1 is invalid, so we go high…….

Problem:

When starting mysqld i got the following error:

Timeout error occurred trying to start MySQL Daemon.

Solution:

After checking mysqld.log i found out that the tunning i did on my.cnf had some options that aren’t compatible with the mysqld version.

Problem:

Need to search for a string in many files and return the filenames.

Solution:

find -depth -name *.file_extension | xargs grep -sl “string“

Problem:

You can receive emails with your own mail server, but can’t send to “non-local” domains.

Solution:

You need to setup auth method before sending emails outside the local realm. You can enable ecrypted passwords and such.

Change at sendmail config:

define(`confAUTH_OPTIONS’, `A’)dnl
TRUST_AUTH_MECH(`LOGIN PLAIN’)dnl
define(`confAUTH_MECHANISMS’, `LOGIN PLAIN’)dnl

at ssh:

set auth_options to `A’
service saslauthd restart (don’t forget it’s importaint)
service sendmail restart