• 0

How to rate limit a web server?


Question

I've noticed whenever I hit too many links on LinkedIn, it gives me an HTTP 429 error.  Apparently this is their version of a rate limit notice.

It's rather interesting and I would like to do similar things on my own server. I have a basic Apache Web server at home and I'm wondering how I could configure it to perform a rate limit. I guess the operating system or Apache would have to identify the individual and not can convoluted with other peoples traffic and then once they've access too many pages in a given amount of time, a generic HTML page is presented rather than the dynamic PHP content.

 

Any advice would be appreciated. I'm not really sure where to start looking for this stuff. I don't think this is done at the PHP level. I suspect it's done at the Apache of operating system level. Even at the firewall level.

Link to comment
https://www.neowin.net/forum/topic/1435050-how-to-rate-limit-a-web-server/
Share on other sites

5 answers to this question

Recommended Posts

  • 0

You’re looking for an apache mod. There are numerous available, a quick Google search brought up plenty of information - but I’m assuming you want more than that, you’d prefer a user here to give their own recommendations :)

You are spot on about this occurring WAY before you get to languages such as PHP, you want this to be implemented when they request is first received.

  • 0

I think so Barrett but I don't really understand how it works. Based on what I'm experiencing, I think either Apache or the operating system is intercepting my connection. Either that or there's some intelligent firewall going on. I'm really fascinated by this technology. I guess LinkedIn gets hit hard every day and they need to rate limit all the users. Kind of lame considering they make so much money every day.

  • 0

You could configure Apache to limit connections with a module for example.

I have an Apache server which only purpose is to serve downloads for a website I run, however to stop bots crawling the site and scraping every file (putting immense load on the server) a module is used (limitipconn_module) that limits the max number of connections per IP address to two.

If someone attempts to download more than two files at once they are simply shown an error message. When one download has finished that person is free to start another download.

To do this you would edit your configuration file to load the module and configure the desired options:

# Limit to two Connections
LoadModule limitipconn_module modules/mod_limitipconn.so
<IfModule limitipconn_module>
MaxConnPerIP 2
</IfModule>

This could potentially be an issue if two users we're behind a CGNAT on the same ISP and therefor shared the same WAN IP address, however my site is no LinkedIn so that's very unlikely to happen. However I thought it was worth brining up a potential limitation of doing that.

You can also increase / decrease the request per second on Apache: https://ubiq.co/tech-blog/increase-apache-requests-per-second/

LinkedIn will be load balancing the traffic, using a Web Application Firewall (WAF) to analyse traffic and then either allow or deny it, probably a whole lot more also.

  • Like 1

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • If you look at all RAID implementations that exist, you're going to find exceptions. However, all the modern consumer varieties tend to have some things in common (by default). I'll stick to describing those. When you add a disk to a RAID array, metadata is stored at the end of the disk. It records the array the disk is part of, which other disks are in the array, etc. This is called the RAID superblock. If you create a RAID 1 array, your operating system will see them all as a single disk that is very slightly smaller than a single disk (due to the superblock). Everything you write to the RAID disk gets written identically to each of its member disks by the storage controller. Technically, disks are read/written in blocks (each block is multiple sectors in size), but this is all transparent to the user. Every file you create or change or delete is created/changed/deleted on every member disk simultaneously. This is true whether you have 2 disks in the array or more than 2.  If one disk completely fails, you can still operate just fine off the remaining disk(s) (but see the caution below). If you remove one disk and attach it to another PC, it should work fine. The partition information and everything is all at the front of the disk, just as expected. The superblock will just appear as some extra junk at the very end of the disk, outside any partition. In some scenarios, where it is recognized as a RAID member disk from another PC, there might be an extra step before it will let you use it, but it's all very doable. Caution:  Blocks are read from the disks in a staggered fashion. For example, with 2 disks, all the odd blocks are read from one disk and all the even blocks are read from the other. By working together like this, read speeds can be practically doubled. But this comes with a huge drawback. If a disk doesn't fail completely, but instead develops bad sectors, you may not realize it. The bad sectors may happen to be in blocks that are never read on that disk. In some cases, people have had bad sectors develop on one disk, then had the other disk fail, and only then realized that the remaining disk has bad sectors and corrupt data. Every backup method has its pros and cons. Never trust just RAID, or just an external HDD, or just the cloud. Use multiple methods to backup important data.
    • Evidence that it affects "most" people negatively? Based on what? The fact that their are millions of users in fact show me the opposite, that "most" are quite happy.
  • Recent Achievements

    • Conversation Starter
      Kavin25 earned a badge
      Conversation Starter
    • One Month Later
      Leonard grant earned a badge
      One Month Later
    • Week One Done
      pcdoctorsnet earned a badge
      Week One Done
    • Rising Star
      Phillip0web went up a rank
      Rising Star
    • One Month Later
      Epaminombas earned a badge
      One Month Later
  • Popular Contributors

    1. 1
      +primortal
      537
    2. 2
      ATLien_0
      205
    3. 3
      +FloatingFatMan
      167
    4. 4
      Michael Scrip
      151
    5. 5
      Som
      127
  • Tell a friend

    Love Neowin? Tell a friend!