Threats to your instance’s stability are everywhere. Maybe it’s a poorly written DevOps metrics script that’s pulling data on recent commits and pull requests too frequently, or an internally built app that’s checking users’ permissions and sending lots of unnecessary requests. These scenarios endanger the stability and performance of your instance; when a single (or a few) bad actors or automated scripts bombard your instance with a massive number of requests and consume the majority of its resources, you’ll experience degraded performance, and possibly even a Denial of Service (DoS). The larger the organization, the more catastrophic this kind of misuse can be.
These threats come at a significant cost to businesses. End users are stuck staring at a throbber (a loading icon) or waiting for pages to load and actions to complete, while admins scramble (often after hours) to analyze logs looking for the culprit. And to prevent future abuse, admins then have to contact the bad actor so they can modify a script or stop an action. All the while, the organization is bleeding money due to lost productivity and wasted resources.
Solutions like CPU or more robust hardware are really just Band-aids – you need a better way to protect your instance against these threats. This is where rate limiting, our most recent tool for administrators, comes in handy.
What is rate limiting?
Rate limiting is a commonly used tool that allows you to control the rate of incoming or outgoing traffic. A popular algorithm for rate limiting is called a token bucket, also known as a “leaky bucket” – this is what we used for Bitbucket Data Center. Each user is given a full token bucket (according to their token bucket size), and every time they make an HTTP request, one token is removed from their bucket. Over a period of time, new tokens will accumulate at a constant rate (the token bucket refill rate) until the bucket is full again. If a user’s bucket is empty, they cannot make requests.
Additional layers of control and security
Admins have a couple more options that give them the control and flexibility they need to meet their organization’s unique needs.
One is the ability to identify and view historical bad actors. Admins can easily see users that have been rate limited in the past 24 hours in the rate limiting settings page, or from the access log using the event label “rate-limited”. With access to this list, admins can provide recurring offenders with suggestions to help them avoid being rate limited in the future, or set exemptions for edge case scenarios. Global settings default the token bucket size to 60 and the refill rate to 5, but there are two types of exemptions:
- Allow custom settings – gives a user a different token bucket size and refill rate.
- Allow unlimited requests – removes users from rate limiting altogether.
We are also exploring ways to expand on this capability and provide you with more control to meet your organization’s specific needs.
A real-life example
So, what does this look like in practice?
One team had a bot that was making over 10,000 requests per hour for their wall board, unintentionally placing significant load on Bitbucket Data Center. This is indicated in the graph below by the spike around 30:00. Shortly thereafter, rate limiting kicked in, and they started seeing errors (HTTP 429 response code) in their scripts. The team quickly modified their scripts to make less frequent requests, indicated by the sudden decrease in the number of requests (almost 40 percent fewer) at 20 hours.
Throughout this cycle, Bitbucket Data Center’s resource usage reduced without impacting the wall board’s functionality. Even better, this change didn’t require the admin to chase down the bad actor to pinpoint the problem and fix it.
Rate limiting drives a natural change in behavior among your users because when they abuse the system, they are negatively impacted. They have an incentive to avoid these types of misuse.
Learn more about other recent investments in our Data Center product line in our webinar, or evaluate Bitbucket Data Center to see for yourself.