all repos — cgit @ 23debef62104c70600be2b745ec3957538eeac6e

a hyperfast web frontend for git written in c 

robots.txt: disallow access to snapshots

My dmesg is filled with the oom killer bringing down processes while the
Bingbot downloads every snapshot for every commit of the Linux kernel in
tar.xz format. Sure, I should be running with memory limits, and now I'm
using cgroups, but a more general solution is to prevent crawlers from
wasting resources like that in the first place.

Suggested-by: Natanael Copa <ncopa@alpinelinux.org>
Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Jason A. Donenfeld Jason@zx2c4.com
Tue, 28 May 2013 14:17:00 +0200
commit

23debef62104c70600be2b745ec3957538eeac6e

parent

830eb6f6ff9e6dded2bfae99d7cb46f049790470

2 files changed, 4 insertions(+), 0 deletions(-)

jump to
M Makefile → Makefile 
@@ -78,6 +78,7 @@ $(INSTALL) -m 0755 -d $(DESTDIR)$(CGIT_DATA_PATH)
 	$(INSTALL) -m 0644 cgit.css $(DESTDIR)$(CGIT_DATA_PATH)/cgit.css
 	$(INSTALL) -m 0644 cgit.png $(DESTDIR)$(CGIT_DATA_PATH)/cgit.png
 	$(INSTALL) -m 0644 favicon.ico $(DESTDIR)$(CGIT_DATA_PATH)/favicon.ico
+	$(INSTALL) -m 0644 robots.txt $(DESTDIR)$(CGIT_DATA_PATH)/robots.txt
 	$(INSTALL) -m 0755 -d $(DESTDIR)$(filterdir)
 	$(COPYTREE)  filters/* $(DESTDIR)$(filterdir)
A robots.txt 
@@ -0,0 +1,3 @@
+User-agent: *
+Disallow: /*/snapshot/*
+Allow: /