How to Block 100,000+ Individual IP addresses
PhpApacheSecurity.HtaccessPhp Problem Overview
Introduction
How do you Block large number of IP address from your web application/server. Obviously that can easily be done in PHP or any programming language
$ipList = []; // array list or from database
if (in_array(getIP(), $ipList)) {
// Log IP & Access information
header("https://www.google.com.ng/search?q=fool"); // redirect
exit(); // exit
}
Or Using htaccess
order allow,deny
deny from 123.45.6.7
deny from 012.34.5.
# .... the list continues
allow from all
The issues
- Am trying to block a whole
100k plus individual IPsnotsubnets - Am trying to avoid user getting to PHP before blocking such IP
- 100000+ is over 1.5MB and that is a lot if information to be loading in
htaccessall the time - Database of IP still growing ... and they would be nee to dynamically add more values
- To set bans in
iptablesfor 100000+ is just ridiculous (Might Be wrong)
Stupid Idea
order allow,deny
deny from database <-------- Not sure if this is possible
allow from all
Question
- Is it possible for
htaccessto get the list from database (Redis,Crunchbase,Mongo, MySQL or even Sqlite) ... any - Is there a visible solution to manage such kind of issue in production
- I know the best solution is
Block the IPs at the firewall levelis there any way to pragmatically add/remove IP to the firewall
Finally
My approach might be totally wrong ... all I want is a visible solution since spammers and botnets are on the rise ...
Please this has nothing to do with DOS attack its a simple ... get lost response
Update
- Firewall : Cisco PIX 515UR
Php Solutions
Solution 1 - Php
Something that you can try is keeping a list of the IP addresses you want to block in a text file or convert it to a dbm hash file, then use mod_rewrite's RewriteMap. You'd have to set this up in your server/vhost config. You cannot initialize a map in an htaccess file.
RewriteEngine On
RewriteMap deny_ips txt:/path/to/deny_ips.txt
RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0
RewriteRule ^ - [L,F]
The /path/to/deny_ips.txt file would look something like this:
12.34.56.78 1
11.22.33.44 1
etc.
Essentially, an IP that you want to deny and a space then a "1". Any IP in this text file will cause the server to return a 403 Forbidden. To speed things up a bit you can use the httxt2dbm to generate a dbm hash and then you'd define the mapping as so:
RewriteMap deny_ips dbm:/path/to/deny_ips.dbm
I'm not sure what the performance hit is for using mod_rewrite like this with a lot of IPs, but a quick benchmark test on apache 2.2 running on a 3Ghz i686 under linux, the difference between 5 IPs in the list versus 102418 is negligible. According to ab's output, they're nearly identical.
Addressing specific questions:
> Is it possible for htaccess to get the list from database (Redis,Crunchbase,Mongo, MySQL or even Sqlite) ... any
Using a rewrite map, you can use the "prg" map type to run an external program for a mapping type. You can then write a perl, php, etc. script to talk to a database in order to look up an IP address. Also note that caveats listed under "Caution". You'd then use this map like you would any other map (RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0). This would essentially create a bottleneck for all requests. Not the best solution for talking to a database.
In apache 2.4 though, there is a dbd/fastdbd map type, which allows you to create queries through mod_dbd. This is a much better option and the mod_dbd module manages connections to the database, pools connections, etc. So the map definition would look something like:
RewriteMap deny_ips "fastdbd:SELECT active FROM deny_ips WHERE source = %s"
Assuming you have a table "deny_ips" with 2 columns "source" (the IP address) and "active" (1 for active, 0 for inactive).
> Is there a visible solution to manage such kind of issue in production
If you are storing all of the blocked IPs in the database, it's a matter of managing the contents of your database table. If you are using the dbm map type, I know at least perl has a DBI for managing dbm files, so you can use that to add/remove IP entries from the deny list. I've never used it before so I can't really say much about it. Managing a flat text file is going to be a lot trickier, especially if you plan on removing entries, and not just append to it. Outside of using a database and apache 2.4's mod_dbd, I don't think any of these solutions are out of the box or production ready. It's going to require custom work.
> I know the best solution is Block the IPs at the firewall level is there any way to pragmatically add/remove IP to the firewall
For IPtables, there is a perl interface that's marked as Beta, but I've never used it before. There's libiptc but according to netfilter's faq:
>Is there an C/C++ API for adding/removing rules? > >The answer unfortunately is: No. > >Now you might think 'but what about libiptc?'. As has been pointed out numerous times on the mailinglist(s), libiptc was NEVER meant to be used as a public interface. We don't guarantee a stable interface, and it is planned to remove it in the next incarnation of linux packet filtering. libiptc is way too low-layer to be used reasonably anyway. > >We are well aware that there is a fundamental lack for such an API, and we are working on improving that situation. Until then, it is recommended to either use system() or open a pipe into stdin of iptables-restore. The latter will give you a way better performance.
So I don't know how viable a libiptc solution is if there's no API stability.
Solution 2 - Php
ANOTHER PERSPECTIVE
Hello. You can check if an address is blocked or not, via accessing two bytes in two data chunks each 8KB long. Yes, I am serious... Please be patient because it takes a little bit long to explain it.
THE THEORY
An IP address is an address, actually a 4 byte number.
The question is, what if we make it to address bit positions?.
The answer: Well ok, we will have
2^32 = 4 Giga Bits
of addressing space and that will take
4Gb/8 = 512 Mega Bytes
of allocation. Ouch! But do not worry, we are not going to block everything in the ipverse and 512MB is an exaggeration.
This can open us a path to the solution.
The Lilliputian Case
Think of a Lilliputian world which there exists only ip addresses from 0 to 65535. So addresses are like 0.1 or 42.42 up to 255.255.
Now King of this world wants to block several L-IP (lilliput ip) addresses.
First he builds a virtual 2D bit map which is 256 * 256 bits long that takes up :
64 K Bits = 8 K Bytes.
He decides to block that nasty "revolution" site which he hates because he is the king, the address is 56.28 for instance.
Address = (56 * 256) + 28 = 14364.(bit position in whole map)
Byte in map = floor(14364 / 8) = 1795.
Bit position= 14364 % 8 = 4.(modulus)
He opens the map file, accesses 1795th byte and sets the bit 4 (by an | 16), then writes it back to mark the site as blocked.
When his script sees the 56.28, it does the same calculation and looks at the bit, and if it is set, blocks the address.
Now what is the moral of the story? Well we can use this lilliputian structure.
THE PRACTICE
The Real World Case
We can apply the Lilliputian case to real world with a "use it when you need" approach since allocating a 512MB file is not a good choice.
Think of a database table named BLOCKS with entries like that:
IpHead(key): unsigned 16 bit integer,
Map : 8KB BLOB(fixed size),
EntryCount : unsigned 16 bit integer.
And another table with just one entry with the structure below named BASE
Map : 8KB BLOB(fixed size).
Now lets say you have an incoming address 56.28.10.2
Script accesses BASE table and gets the Map.
It looks up the higher order IP numbers 56.28:
Address = (56 * 256) + 28 = 14364.(bit position in whole map)
Byte in map = floor(14364 / 8) = 1795.
Bit position= 14364 % 8 = 4.(modulus)
Looks at byte 1795 bit 4 in the Map.
If bit is not set no further operation is needed meaning there is no blocked ip address in range 56.28.0.0 - 56.28.255.255 .
If bit is set then the script accesses the BLOCKS table.
The higher order IP numbers were 56.28 which gives 14364 so the script queries the BLOCKS table with index IpHead = 14364. Fetches the record. The record should exist since it is marked at BASE.
Script does the calculation for lower order IP address
Address = (10 * 256) + 2 = 2562.(bit position in whole map)
Byte in map = floor(2562 / 8) = 320.
Bit position= 2562 % 8 = 2.(modulus)
Then it checks if address is blocked by looking at bit 2 of byte 320 of field Map.
Job done!
Q1: Why do we use BASE at all, we could directly query BLOCKS with 14364.
A1: Yes we could but BASE map lookup will be faster then BTREE search of any database server.
Q2: What is the EntryCount field in BLOCKS table for?
A2: It is the count of ip addresses blocked in the map field at the same record. So if we unblock ip's and EntryCount reaches 0 that BLOCKS record becomes unnecessary. It can be erased and the corresponding bit on BASE map will be unset.
IMHO this approach will be lightning fast. Also for the blob allocation is 8K per record. Since db servers keep blobs in seperate files, file systems with 4K, 8K or multiples of 4K paging will react fast.
In case blocked addresses are too dispersed
Well that is a concern, which will make the database BLOCKS table to grow unnecessarily.
But for such cases the alternative is to use a 256256256 bit cube which is 16777216 bits long, equaling to 2097152 bytes = 2MB.
For our previous example Higher Ip resolving is :
(56 * 65536)+(28 * 256)+10
So BASE will become a 2MB file instead of a db table record, which will be opened (fopen etc.) and bit will be addressed via seeking (like fseek, never read whole file contents, unnecessary) then access the BLOCKS table with structure below:
IpHead(key): unsigned 32 bit integer, (only 24 bit is used)
Map : 32 unsigned 8 bit integers(char maybe),(256 bit fixed)
EntryCount : unsigned 8 bit integer.
Here is the php example code for block checking of bitplane-bitplane (8K 8K) version:
Side Note: This script can be optimized further via elimination of several calls etc.. But written like this for keeping it easy to understand.
<?
define('BLOCK_ON_ERROR', true); // WARNING if true errors block everyone
$shost = 'hosturl';
$suser = 'username';
$spass = 'password';
$sdbip = 'database';
$slink = null;
$slink = mysqli_connect($shost, $suser, $spass, $sdbip);
if (! $slink) {
$blocked = BLOCK_ON_ERROR;
} else {
$blocked = isBlocked();
mysqli_close($slink); // clean, tidy...
}
if ($blocked) {
// do what ever you want when blocked
} else {
// do what ever you want when not blocked
}
exit(0);
function getUserIp() {
$st = array(
'HTTP_CLIENT_IP',
'REMOTE_ADDR',
'HTTP_X_FORWARDED_FOR'
);
foreach ( $st as $v )
if (! empty($_SERVER[$v]))
return ($_SERVER[$v]);
return ("");
}
function ipToArray($ip) {
$ip = explode('.', $ip);
foreach ( $ip as $k => $v )
$ip[$k] = intval($v);
return ($ip);
}
function calculateBitPos($IpH, $IpL) {
$BitAdr = ($IpH * 256) + $IpL;
$BytAdr = floor($BitAdr / 8);
$BitOfs = $BitAdr % 8;
$BitMask = 1;
$BitMask = $BitMask << $BitOfs;
return (array(
'bytePos' => $BytAdr,
'bitMask' => $BitMask
));
}
function getBaseMap($link) {
$q = 'SELECT * FROM BASE WHERE id = 0';
$r = mysqli_query($link, $q);
if (! $r)
return (null);
$m = mysqli_fetch_assoc($r);
mysqli_free_result($r);
return ($m['map']);
}
function getBlocksMap($link, $IpHead) {
$q = "SELECT * FROM BLOCKS WHERE IpHead = $IpHead";
$r = mysqli_query($link, $q);
if (! $r)
return (null);
$m = mysqli_fetch_assoc($r);
mysqli_free_result($r);
return ($m['map']);
}
function isBlocked() {
global $slink;
$ip = getUserIp();
if($ip == "")
return (BLOCK_ON_ERROR);
$ip = ipToArray($ip);
// here you can embed preliminary checks like ip[0] = 10 exit(0)
// for unblocking or blocking address range 10 or 192 or 127 etc....
// Look at base table base record.
// map is a php string, which in fact is a good byte array
$map = getBaseMap($slink);
if (! $map)
return (BLOCK_ON_ERROR);
$p = calculateBitPos($ip[0], $ip[1]);
$c = ord($map[$p['bytePos']]);
if (($c & $p['bitMask']) == 0)
return (false); // No address blocked
// Look at blocks table related record
$map = getBlocksMap($slink, $p[0]);
if (! $map)
return (BLOCK_ON_ERROR);
$p = calculateBitPos($ip[2], $ip[3]);
$c = ord($map[$p['bytePos']]);
return (($c & $p['bitMask']) != 0);
}
?>
I hope this helps.
If you have questions on the details, I will be happy to answer.
Solution 3 - Php
Block the traffic before it reaches the www server using iptables and ipset.
Catch the blacklisted IP traffic in the filter table of the INPUT chain assuming your web server is on the same machine. If you are blocking IPs on a router you will want the FORWARD chain.
First create the ipset:
ipset create ip_blacklist hash:ip
IPs can be added via:
ipset add ip_blacklist xxx.xxx.xxx.xxx
Add the ipset match rule to your iptables (DROP all packets match to ipset):
iptables --table filter --insert INPUT --match set --match-set ip_blacklist src -j DROP
This will stop the blacklisted traffic before the www server.
Edit: I had a chance to look up the default maximum size and it is 65536 so you will need to adjust this to support 100000+ entries:
ipset create ip_blacklist hash:ip maxelem 120000
You can also tweak the hash size:
ipset create ip_blacklist hash:ip maxelem 120000 hashsize 16384 (Must be a power of 2)
My experience is ipset lookup has negligible effect on my system (~45000 entries). There are a number of test cases on the net. Memory for the set is a limiting factor.
Solution 4 - Php
You need to do this with an external firewall, not in PHP. I recommend pfSense or PF. I have used it before and it is very easy to use, very intuitive, and extremely powerful. It is the choice of the best sys-admins. I run it on FreeBSD, but it works great on OpenBSD as well. I am a Linux guy so it pains me to say this, but don't try to run it on Linux. BSD is easy, and you can figure it out quickly.
An awesome feature for pfSense is the ability to configure using scripts and restricting the configuration access to a single network interface (so that only things on the LAN can configure it). It also has a couple of ID10T level features to keep you from cutting off your own access accidentally.
You should also be aware that many spammers can switch IPs quickly using things like Tor. To fix this you should include in your block list the addresses that are known tor exit nodes (this list is available from various places).
Solution 5 - Php
If you want a way to add / remove via code take a look at denyhosts. You could either maintain the IP list via code or patch the source to read from whatever location you want.
Solution 6 - Php
There is a project with netfilter for that called ipset so you can add or remove ip to a list and you just have to create a rule against this list
Solution 7 - Php
If you're blocking IPs, you really should be doing this at the firewall level (you don't want users from unwelcome IP addresses getting very far into your system). Thus, I suggest writing a bash script that queries the database and modifies your firewall configuration file accordingly (this assumes that you want a solution which utilizes IP addresses stored in your web database -- there very well might be a better place to store such information).
EDIT: If you wanted to add IP addresses to the blacklist at the PHP level, as @Populus suggested, here is the manual on how to use system calls in PHP: http://php.net/manual/en/function.system.php
And here are the commands you would need to use to add an ip address to your blacklist if you're using iptables: http://www.cyberciti.biz/faq/linux-iptables-drop/
Solution 8 - Php
I know a way
its by php.
as you mentioned in the very beginig of this question.
$ipList = []; // array list or from database
if (in_array(getIP(), $ipList)) {
// Log IP & Access information
header("https://www.google.com.ng/search?q=fool"); // redirect
exit(); // exit
}
I read thing you have wrote. but just wait a minute.
why you want to do this.is not important.but why with this situation.
I mean why would you want to avoid accessing php because of its speed or just because of preventing and because its so hard to call the function in all pages?
if the only reason of avoiding some ip from accessing php is.avoiding theme to see contents.
so I got an idea and I.suggest you this way.
using one entry point.
I have worked with this solution.
first with a simple htaccess you send all requests to one page called entry point.(like index.php)
with a simple rewriting rule i'll give it to you.
so when the user requests
mysite.com/some/path/page.php or anything
htaccess will execute something like the following without changing the url. so the user would not feel anything.
mysite.com/index.php?r=some/path/page.php
so every request became one request with different $_GET['r'] params. so for evey request we will have index.php executed. and now we can do somthing like this in index.php
$ipList = []; // array list or from database
if (in_array(getIP(), $ipList)) {
// Log IP & Access information
header("https://www.google.com.ng/search?q=fool"); // redirect
exit(); // exit
}
//if after this execute means the IP is not banned
//now we can include file that $_GET['r'] points to
include $_GET['r'];
its so simple.and its real one is so complicated.but the main idea is the same. what do you think?
Solution 9 - Php
It seems that most of us agree to block at the firewall level.
You could have a program that listens to your website for ips to block and generates a script:
ip = getNextIpToBlock()
an = increment_unique_alphanum_generator()
script = generate_script(ip, an)
script would look something like this (where [an] is an alphanumeric value and [ip] is the ip you block):
en [enter]
*password* [enter]
conf t [enter]
access-list [an] deny ip [ip] 0.0.0.0 any [enter]
access-group [an] in interface outside [enter]
Then you load this script to another program that executes remote telnet or ssh calls to your FW CLI.
Don't forget to logout and maybe every 100 ips you copy the running config to start config.
I don't know but you may want to know now what are the limitations for your firewall.
Best,
Solution 10 - Php
Do a geo-lookup on the IPs in your list. My own experience has shown most malicious (i.e. spam) connections have originated from China. If you find the same to be the case for you, and you have no specific need to serve China, see if you can efficiently block the entire country at the firewall level.
Solution 11 - Php
IMHO, there are several angles from which this question can be considered
- You have a very large set of unique IP addresses to block. The earlier you block them on your server, the less processing power you will waste on them. You can add/remove IPs in your firewall through appropriate system calls from PHP.
Considering the above option, the only relevant questions are:
- Do they visit often? => If so, then solution (1) is your best bet.
- Can your firewall handle it efficiently? => If not, you may want to consider an other solution.
The .htaccess would be my second choice.
Solution 12 - Php
Why don't you block by ANS instead? i.e.:
function ip2asn($ip)
{
return trim(shell_exec("whois -h whois.cymru.com $ip|sed 1,1d|cut -d '|' -f 1"));
}
$asn = ip2asn($_SERVER['REMOTE_ADDR']);
# 16247 https://ipinfo.io/AS16247
# 9009 https://ipinfo.io/AS9009
$bad_asn = array("16247", "9009");
if(in_array($asn, $bad_asn)){
http_response_code(404);
exit();
}