Note: This is an article published in Linkedin published on 2018.

Background:

In an ISP that hosts email accounts of more than 500.000 active users with an imaps and pop3s’s services to allow the users to check they email. Many of these users have a desktop computer and have the same IP all the time, but a big percentage of users check the emails from different places and devices and we can’t know or control where they will go !!!

The ISP have a typical security configuration on mail servers:

• Firewall to block all traffic, except traffic to 25,587,993,995 / tcp.
• Antivirus/Antispam filters.
• White and black list of IPs.
• NIDS (Network intrusion detection system) to check the traffic.
• HIDS (Host-based intrusion detection system) to check the logs files.
• Mail server with a built-in brute-force detection system.

These securities points help the ISP to:

• Check inside of the network traffic to detect exploits or dangerous traffic.
• Check the logs files to detect abnormal activity, for example, when a user change the country many times in the last 30 minutes.
• Detect and stop attacks of brute-force.

The brute-force attack for the mail system was worked fine in the last 5 years. Basically, it checks if a client tries to connect with a username and an invalid password more than 5 times, if yes, the system will block this “user account” for 5 minutes.

Additional to mail system to mitigate the brute-force attack, exist another system (OSSEC) that work very similar, but it tries to find in the logs of mail servers dangerous patterns, like:

• Invalid accounts users
• Invalid authentications
• Invalid SMTP/POP3/IMAPs commands

If OSSEC detect that a same IP tried to connect 3 times with an incorrect password in the last minutes, the OSSEC will block this IP in the firewall for 5 minutes. After 5 minutes, the IP will be unblocked and if the same IP insist with invalids password, the OSSEC will block the IP by 15 minutes now and the block time will be increasing until the maximum time of 1 day.

That is very nice, because is the IP from the attacker that is blocked in the first 3 tries and the firewall won’t allow the traffic to the mail servers, so, in this case the “user account” isn’t usually blocked by an attacker and the “real user” can continue working without problem.

The problem:

How I explained before, to stop a brute-force attack by IP we need that an attacker try the combination of a user/invalid password less 3 times from the same IP. But, the problem was a bit complicate to the ISP:

The brute-force attack come from a botnet with miles of different IPs and the attacker allows use a combination of a user/password only once by IP.

And after five connections to the mail server with an incorrect user/password the “users accounts” were locked by 5 minutes, when the account was active, the mail system detect the brute-force and lock the account again.

The mail servers didn’t have performance problem or go down in any moment, the DDoS attack put down other very important system on this ISP, the “Support Center” by a hundred of clients calling or writing to “unlock” the account one and other time.

Suggested solutions:

After much research on Internet, calling the people that “have passed” to a similar problem, we recompiled a list of recommended solutions:

Deactivate the brute-force system at mail server. (WTF ??!!) Use captcha. (Captcha in the IMAP/POP3 protocol ??!!) Use VPN at clients side. (There are thousands of a normal users.) Create a blacklist. (Very difficult because the IPs were from many countries/ranges.)

The final solution to us:

1. After two days of hard working to stop this attack, we create a solution that worked very well to us:

• Changing OSSEC to NUDeS (Network Users Detection Systems):
• Created a configuration to detect “real users”, checking in the logs files for patterns that confirm that the user/IP have been connected correctly to the mail server.
• Put the IP from this successfully authentication in a firewall greylist(*).

If the account doesn’t connect from the same IP again in the next 7 days, the IP will be deleted from the greylist. Note: We use the greylist, because the idea is that after the authentication, the system still use the others securities controls.

2. Rejecting all connections by default on firewall:

By default, all the connections to mail servers will be blocked by 60 seconds, except if the IP is on the greylist. After 60 seconds, the firewall allowed only “one connection” to be forward to the mail servers to try to authenticate in it. If the authentication is correct, the OSSEC will get the IP and put it in a greylist. Note: After detecting a performance problem with the iptables and thousands of rules, we changed OSSEC to use “ipset” and manage all the rules better.

We were worried if the default lock changes in 60 seconds would affect the real users or not. But users have the mail client software configured to verify the email every 1-5 minutes, on the first attempt the user receives a network error and the client thought it was temporary error, in the second attempt to access the mail server, they have connected without problems.

Maybe that is not the “correct” solution, but this graphic show the results that has worked with us until now.