paritybit.ca

Unnamed repository; edit this file 'description' to name the repository.
Log | Files | Refs | README | LICENSE

commit eebea1630df1a52205497c81a780b7acf0423ab9
parent 41407e5905092abaa659c295d245d51653085fec
Author: Jake Bauer <jbauer@paritybit.ca>
Date:   Wed, 29 Jul 2020 23:24:15 -0400

Publish new blog post

Diffstat:
Mpages/blog.md | 1+
Apages/blog/my-git-server-was-ddosed.md | 151++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mpages/home.md | 4++--
Mpublic/feeds/sitewide-feed.xml | 54++++++++++++++++++++++++++++++++++++++++++++++++++++++
Apublic/img/banned-ips-now.png | 0
Apublic/img/banned-ips.png | 0
Apublic/img/more-banned-ips.png | 0
Mpublic/sitemap.xml | 1+
8 files changed, 209 insertions(+), 2 deletions(-)

diff --git a/pages/blog.md b/pages/blog.md @@ -27,6 +27,7 @@ href="https://pleroma.paritybit.ca/jbauer">Pleroma</a>. ### 2020 <ul> + <li>2020-07-29 <a href="blog/my-git-server-was-ddosed">My Git Server Was DDoSed</a></li> <li>2020-07-25 <a href="blog/mastodon-to-pleroma-3-shutting-down-mastodon">Mastodon to Pleroma — 3 — Shutting Down Mastodon</a></li> <li>2020-07-25 <a href="blog/i-was-wrong">I Was Wrong</a></li> <li>2020-07-24 <a href="blog/switching-to-alacritty">"Switching From St to Alacritty" and "The Separation of Creators From Their Work"</a></li> diff --git a/pages/blog/my-git-server-was-ddosed.md b/pages/blog/my-git-server-was-ddosed.md @@ -0,0 +1,151 @@ +## My Git Server Was DDoSed + +[//]: # "I noticed some strange behaviour on my git server and it turns out someone was DDoSing it. Here's what I had to do to get it back up and running again." + +[//]: # "main.min.css" + +[//]: # + +<div class="byline"> +<b>Written By:</b> Jake Bauer | + <b>Posted:</b> 2020-07-29 | + <b>Last Updated:</b> 2020-07-29 +</div> + +On the evening of the 27th of July, 2020 (two days ago), I noticed some odd +behaviour with my git server. UptimeRobot, the service which provides +[status.paritybit.ca](https://status.paritybit.ca), kept sending me emails every +dozen or so minutes telling me that my git server was down and then up again. +This happened a number of times but I just chalked it up to a misbehaving server +or heavier-than-normal traffic. + +It wasn't until the notifications became so numerous and noticing that turning +the machine off and on again that I figured something wasn't right. I shut down +the server for the night to prevent it from being continuously overwhelmed (and +thereby me receiving a hundred emails overnight) and went to sleep with the idea +of fixing it in the morning. + +When I woke up, the first thing I tried was turning the server back on and +waiting to see if it went down again. Maybe whatever was going on fixed it self +in the ~13 hours that the server was offline. Unfortunately, within 10 minutes +the server was down again. + +My next step in diagnosing the problem was to check the server logs. Normally, I +have both the `access.log` and `error.log` turned off on my servers because I +don't need the content of those logs under normal circumstances. I proceeded to +turn on logging again just for [git.paritybit.ca](https://git.paritybit.ca) and +watched them with `tail -f`. + +I noticed a flood of various sketchy looking user agents constantly hitting +various URLs in a pattern that was clearly indicative of bots. These bots were +looking at what seemed like every single diff of every single file of every +single commit and this was bringing my poor PineA64+ down to its knees trying to +keep up with syntax highlighting all the code and generating all the `.tar.gz`, +`.tar.bz2` and `.zip` files for every single snapshot. Luckily, it was just the +webserver that kept going down; git access through SSH was unaffected. + +These are the kinds of user agents I noticed: + +``` +"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)" + +"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)" + +"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0" + +"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11" + +"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)" +``` + +My first thought upon seeing this traffic was that they were just some +overzealous bots. I changed my `robots.txt` file to deny scraping all but the +index page of my git server and I changed the robots setting in `cgitrc` to +`robots=none` to prevent the site from being indexed. + +Unfortunately, that didn't help one bit. When a bot ignores robots directives, +it's more than likely a malicious bot (or at the very least, one made by an +incompetent programmer). My next action was to set up Fail2Ban to catch these +malicious actors and ban them for a number of hours from accessing my server. So +I set up an `nginx-badbots` filter (by copying the `apache-badbots` filter and +adding the malicious user agents to the `badbots-custom` variable), I set the +ban time to 2 days, the retry count to 2 times, and the action to add a drop +rule to my `nftables` firewall. + +In the first few minutes, there were about 10 or so IPs added to the list. I let +this keep going for an hour all the while monitoring it and watching my git +server continue to come back online only to be knocked down again. After about +15 minutes, I got the idea to run a GeoIP lookup on the addresses which were +banned and this is what I saw: + +<figure> + <a href="/img/banned-ips.png"><img src="/img/banned-ips.png" alt="A + screenshot of the GeoIP lookup output showing 235 IPs from China and 1 IP + from Germany."/></a> +</figure> + +Clearly, either there was someone using a botnet with a lot of infected Chinese +computers or someone in China was DDoSing my server using whatever IPs they +controlled. I let it get this bad before deciding I had to take much more +drastic measures: + +<figure> + <a href="/img/more-banned-ips.png"><img src="/img/more-banned-ips.png" + alt="A screenshot of another GeoIP lookup showing 1 IP from Canada, 876 IPs + from China, and 1 IP from Germany."/></a> +</figure> + +At this rate, Fail2Ban was not working and, after an hour of adding hundreds of +IPs to the blocklist, my server was still going down. It was suggested to me by +someone on the Fediverse (I was ranting about this whole thing over there) that +I might be able to stop the attack by pre-emptively banning all Chinese IPs. I +left that as a last resort because it's an extreme measure that has the very +real implication of locking out good people from viewing my sites. It's not like +everyone in China is a malicious actor and there may even be people over there +who find my stuff useful. + +Though, unfortunately, the attack was showing no signs of slowing down and I had +to take this action just to get my server back up and running again. I found a +list of IPv4 address ranges for China and added all of those IPs to my firewall +(it was an extra ~2200 rules). The server pretty much instantly came back +online and the list of IPs banned by Fail2Ban stopped growing. Since then, my +git server has stayed up without any further issues. + +It doesn't seem to be over though. It would appear the script kiddies caught on +to my blanket IP ban and have been hitting the server again from various +countries. I just checked as of writing this blog post, and this is now the +output of Fail2Ban + GeoIP lookup: + +<figure> + <a href="/img/banned-ips-now.png"><img src="/img/banned-ips-now.png" alt="A + screenshot of another GeoIP lookup showing 876 Chinese IPs, 464 IP addresses + not found, 170 Russian IPs, 155 American IPs, 85 Australian IPs, 76 British + IPs, 64 Turkish IPs, 62 Canadian IPs, 59 Indian IPs, 59 German IPs, 31 + Brazilian IPs, and so on for nearly every developed country in the + world."/></a> +</figure> + +Clearly the attack is still going on but, thanks to Fail2Ban, it's no longer +significant enough to keep bringing down my Git server. I also don't notice any +lag out of the ordinary when using it myself which is a good sign. + +What perplexes me about this DDoS is that this person or group, whoever they +are, is only attacking [git.paritybit.ca](https://git.paritybit.ca) and none +of my other services. My website, Pleroma, and Matrix all seem to be behaving +just fine and there's no extraneous bandwidth usage from any of them. My git +server is also just a web server for people to look at and clone my +repositories; there are no accounts to take over or databases to hack. Is this +some automated attack designed to scrape all my code in an attempt to find +credentials? Who knows... + +I'll give it a few more days and see if I can unblock China from accessing my +web services because I'd rather not block an entire country from my stuff just +because of one malicious actor. Unfortunately, if the attack continues to the +point of bringing down my git server again, I'll have to leave the block in +place (though I may try rate-limiting IPs that Fail2Ban catches instead of +outright blocking them). + +_This is my eighty-second post for the +[#100DaysToOffload](https://social.paritybit.ca/tags/100DaysToOffload) +challenge. You can learn more about this challenge over at +[https://100daystooffload.com](https://100daystooffload.com)._ diff --git a/pages/home.md b/pages/home.md @@ -18,6 +18,8 @@ This site will soon™️ be available over Gopher and Gemini. <a class="rss-icon" href="/feeds/sitewide-feed.xml"> <img src="/img/feed-icon.png" width="15" height="15" alt="Click for RSS Feed"/> </a> </div> +2020-07-29 <a class="feed-item" href="blog/my-git-server-was-ddosed">My Git Server Was DDoSed</a> + 2020-07-25 <a class="feed-item" href="blog/mastodon-to-pleroma-3-shutting-down-mastodon">Mastodon to Pleroma — 3 — Shutting Down Mastodon</a> 2020-07-25 <a class="feed-item" href="blog/i-was-wrong">I Was Wrong</a> @@ -36,8 +38,6 @@ This site will soon™️ be available over Gopher and Gemini. 2020-07-16 <a class="feed-item" href="blog/why-i-hate-the-term-modern">Why I Hate The Term "Modern"</a> -2020-07-15 <a class="feed-item" href="blog/be-a-good-netizen">Be A Good Netizen</a> - ### What is a Parity Bit? It is a bit (in the 1's and 0's sense) used in checking for errors in digital diff --git a/public/feeds/sitewide-feed.xml b/public/feeds/sitewide-feed.xml @@ -7,6 +7,60 @@ <description>The feed that covers all notable additions, updates, announcements, and other changes for the entire paritybit.ca website.</description> <item> + <title>My Git Server Was DDoSed</title> + <link>https://www.paritybit.ca/blog/my-git-server-was-ddosed</link> + <guid>https://www.paritybit.ca/blog/my-git-server-was-ddosed</guid> + <pubDate>Wed, 29 Jul 2020 23:23:00 -0400</pubDate> + <description><![CDATA[<h2 id="my-git-server-was-ddosed">My Git Server Was DDoSed</h2> +<div class="byline"> +<p><b>Written By:</b> Jake Bauer | <b>Posted:</b> 2020-07-29 | <b>Last Updated:</b> 2020-07-29</p> +</div> +<p>On the evening of the 27th of July, 2020 (two days ago), I noticed some odd behaviour with my git server. UptimeRobot, the service which provides <a href="https://status.paritybit.ca">status.paritybit.ca</a>, kept sending me emails every dozen or so minutes telling me that my git server was down and then up again. This happened a number of times but I just chalked it up to a misbehaving server or heavier-than-normal traffic.</p> +<p>It wasn’t until the notifications became so numerous and noticing that turning the machine off and on again that I figured something wasn’t right. I shut down the server for the night to prevent it from being continuously overwhelmed (and thereby me receiving a hundred emails overnight) and went to sleep with the idea of fixing it in the morning.</p> +<p>When I woke up, the first thing I tried was turning the server back on and waiting to see if it went down again. Maybe whatever was going on fixed it self in the ~13 hours that the server was offline. Unfortunately, within 10 minutes the server was down again.</p> +<p>My next step in diagnosing the problem was to check the server logs. Normally, I have both the <code>access.log</code> and <code>error.log</code> turned off on my servers because I don’t need the content of those logs under normal circumstances. I proceeded to turn on logging again just for <a href="https://git.paritybit.ca">git.paritybit.ca</a> and watched them with <code>tail -f</code>.</p> +<p>I noticed a flood of various sketchy looking user agents constantly hitting various URLs in a pattern that was clearly indicative of bots. These bots were looking at what seemed like every single diff of every single file of every single commit and this was bringing my poor PineA64+ down to its knees trying to keep up with syntax highlighting all the code and generating all the <code>.tar.gz</code>, <code>.tar.bz2</code> and <code>.zip</code> files for every single snapshot. Luckily, it was just the webserver that kept going down; git access through SSH was unaffected.</p> +<p>These are the kinds of user agents I noticed:</p> +<pre><code>&quot;Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)&quot; + +&quot;Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)&quot; + +&quot;Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0&quot; + +&quot;Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11&quot; + +&quot;Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)&quot;</code></pre> +<p>My first thought upon seeing this traffic was that they were just some overzealous bots. I changed my <code>robots.txt</code> file to deny scraping all but the index page of my git server and I changed the robots setting in <code>cgitrc</code> to <code>robots=none</code> to prevent the site from being indexed.</p> +<p>Unfortunately, that didn’t help one bit. When a bot ignores robots directives, it’s more than likely a malicious bot (or at the very least, one made by an incompetent programmer). My next action was to set up Fail2Ban to catch these malicious actors and ban them for a number of hours from accessing my server. So I set up an <code>nginx-badbots</code> filter (by copying the <code>apache-badbots</code> filter and adding the malicious user agents to the <code>badbots-custom</code> variable), I set the ban time to 2 days, the retry count to 2 times, and the action to add a drop rule to my <code>nftables</code> firewall.</p> +<p>In the first few minutes, there were about 10 or so IPs added to the list. I let this keep going for an hour all the while monitoring it and watching my git server continue to come back online only to be knocked down again. After about 15 minutes, I got the idea to run a GeoIP lookup on the addresses which were banned and this is what I saw:</p> +<figure> +<a href="/img/banned-ips.png"><img src="/img/banned-ips.png" alt="A + screenshot of the GeoIP lookup output showing 235 IPs from China and 1 IP + from Germany."/></a> +</figure> +<p>Clearly, either there was someone using a botnet with a lot of infected Chinese computers or someone in China was DDoSing my server using whatever IPs they controlled. I let it get this bad before deciding I had to take much more drastic measures:</p> +<figure> +<a href="/img/more-banned-ips.png"><img src="/img/more-banned-ips.png" + alt="A screenshot of another GeoIP lookup showing 1 IP from Canada, 876 IPs + from China, and 1 IP from Germany."/></a> +</figure> +<p>At this rate, Fail2Ban was not working and, after an hour of adding hundreds of IPs to the blocklist, my server was still going down. It was suggested to me by someone on the Fediverse (I was ranting about this whole thing over there) that I might be able to stop the attack by pre-emptively banning all Chinese IPs. I left that as a last resort because it’s an extreme measure that has the very real implication of locking out good people from viewing my sites. It’s not like everyone in China is a malicious actor and there may even be people over there who find my stuff useful.</p> +<p>Though, unfortunately, the attack was showing no signs of slowing down and I had to take this action just to get my server back up and running again. I found a list of IPv4 address ranges for China and added all of those IPs to my firewall (it was an extra ~2200 rules). The server pretty much instantly came back online and the list of IPs banned by Fail2Ban stopped growing. Since then, my git server has stayed up without any further issues.</p> +<p>It doesn’t seem to be over though. It would appear the script kiddies caught on to my blanket IP ban and have been hitting the server again from various countries. I just checked as of writing this blog post, and this is now the output of Fail2Ban + GeoIP lookup:</p> +<figure> +<a href="/img/banned-ips-now.png"><img src="/img/banned-ips-now.png" alt="A + screenshot of another GeoIP lookup showing 876 Chinese IPs, 464 IP addresses + not found, 170 Russian IPs, 155 American IPs, 85 Australian IPs, 76 British + IPs, 64 Turkish IPs, 62 Canadian IPs, 59 Indian IPs, 59 German IPs, 31 + Brazilian IPs, and so on for nearly every developed country in the + world."/></a> +</figure> +<p>Clearly the attack is still going on but, thanks to Fail2Ban, it’s no longer significant enough to keep bringing down my Git server. I also don’t notice any lag out of the ordinary when using it myself which is a good sign.</p> +<p>What perplexes me about this DDoS is that this person or group, whoever they are, is only attacking <a href="https://git.paritybit.ca">git.paritybit.ca</a> and none of my other services. My website, Pleroma, and Matrix all seem to be behaving just fine and there’s no extraneous bandwidth usage from any of them. My git server is also just a web server for people to look at and clone my repositories; there are no accounts to take over or databases to hack. Is this some automated attack designed to scrape all my code in an attempt to find credentials? Who knows…</p> +<p>I’ll give it a few more days and see if I can unblock China from accessing my web services because I’d rather not block an entire country from my stuff just because of one malicious actor. Unfortunately, if the attack continues to the point of bringing down my git server again, I’ll have to leave the block in place (though I may try rate-limiting IPs that Fail2Ban catches instead of outright blocking them).</p> +<p><em>This is my eighty-second post for the <a href="https://social.paritybit.ca/tags/100DaysToOffload">#100DaysToOffload</a> challenge. You can learn more about this challenge over at <a href="https://100daystooffload.com">https://100daystooffload.com</a>.</em></p>]]></description> + </item> +<item> <title>Mastodon to Pleroma — 3 — Shutting Down Mastodon</title> <link>https://www.paritybit.ca/blog/mastodon-to-pleroma-3-shutting-down-mastodon</link> <guid>https://www.paritybit.ca/blog/mastodon-to-pleroma-3-shutting-down-mastodon</guid> diff --git a/public/img/banned-ips-now.png b/public/img/banned-ips-now.png Binary files differ. diff --git a/public/img/banned-ips.png b/public/img/banned-ips.png Binary files differ. diff --git a/public/img/more-banned-ips.png b/public/img/more-banned-ips.png Binary files differ. diff --git a/public/sitemap.xml b/public/sitemap.xml @@ -3,6 +3,7 @@ <url><loc>https://www.paritybit.ca</loc></url> <url><loc>https://www.paritybit.ca/home</loc></url> <url><loc>https://www.paritybit.ca/blog</loc></url> + <url><loc>https://www.paritybit.ca/blog/my-git-server-was-ddosed</loc></url> <url><loc>https://www.paritybit.ca/blog/mastodon-to-pleroma-3-shutting-down-mastodon</loc></url> <url><loc>https://www.paritybit.ca/blog/i-was-wrong</loc></url> <url><loc>https://www.paritybit.ca/blog/switching-to-alacritty</loc></url>