How to rate limit a web server?

[Brian Miller]

I've noticed whenever I hit too many links on LinkedIn, it gives me an HTTP 429 error.  Apparently this is their version of a rate limit notice.

It's rather interesting and I would like to do similar things on my own server. I have a basic Apache Web server at home and I'm wondering how I could configure it to perform a rate limit. I guess the operating system or Apache would have to identify the individual and not can convoluted with other peoples traffic and then once they've access too many pages in a given amount of time, a generic HTML page is presented rather than the dynamic PHP content.

 

Any advice would be appreciated. I'm not really sure where to start looking for this stuff. I don't think this is done at the PHP level. I suspect it's done at the Apache of operating system level. Even at the firewall level.

[+Nik Louch Subscriber²]

You’re looking for an apache mod. There are numerous available, a quick Google search brought up plenty of information - but I’m assuming you want more than that, you’d prefer a user here to give their own recommendations

You are spot on about this occurring WAY before you get to languages such as PHP, you want this to be implemented when they request is first received.

[Brian Miller]

I think so Barrett but I don't really understand how it works. Based on what I'm experiencing, I think either Apache or the operating system is intercepting my connection. Either that or there's some intelligent firewall going on. I'm really fascinated by this technology. I guess LinkedIn gets hit hard every day and they need to rate limit all the users. Kind of lame considering they make so much money every day.

[+InsaneNutter MVC]

You could configure Apache to limit connections with a module for example.

I have an Apache server which only purpose is to serve downloads for a website I run, however to stop bots crawling the site and scraping every file (putting immense load on the server) a module is used (limitipconn_module) that limits the max number of connections per IP address to two.

If someone attempts to download more than two files at once they are simply shown an error message. When one download has finished that person is free to start another download.

To do this you would edit your configuration file to load the module and configure the desired options:

# Limit to two Connections
LoadModule limitipconn_module modules/mod_limitipconn.so
<IfModule limitipconn_module>
MaxConnPerIP 2
</IfModule>

This could potentially be an issue if two users we're behind a CGNAT on the same ISP and therefor shared the same WAN IP address, however my site is no LinkedIn so that's very unlikely to happen. However I thought it was worth brining up a potential limitation of doing that.

You can also increase / decrease the request per second on Apache: https://ubiq.co/tech-blog/increase-apache-requests-per-second/

LinkedIn will be load balancing the traffic, using a Web Application Firewall (WAF) to analyse traffic and then either allow or deny it, probably a whole lot more also.

[+Nik Louch Subscriber²]

On 31/10/2023 at 08:27, Brian Miller said:

Based on what I'm experiencing, I think either Apache or the operating system is intercepting my connection

As I said - it’s Apache

[Brian Miller]

Thank you @InsaneNutter