From: Nicolas Lœuillet <nicolas@loeuillet.org>
Date: Sat, 16 Dec 2017 08:02:40 +0000 (+0100)
Subject: Merge pull request #3510 from wallabag/update-robotstxt
X-Git-Tag: 2.3.1~12
X-Git-Url: https://git.immae.eu/?a=commitdiff_plain;h=9b606725fe7cb060433fd08a52cf49799734eda9;hp=a68a80f654be86ae25d49420b86099302ce55bb6;p=github%2Fwallabag%2Fwallabag.git

Merge pull request #3510 from wallabag/update-robotstxt

Updated robots.txt to prevent crawling
---

diff --git a/web/robots.txt b/web/robots.txt
index 214e4119..1f53798b 100644
--- a/web/robots.txt
+++ b/web/robots.txt
@@ -1,4 +1,2 @@
-# www.robotstxt.org/
-# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
-
 User-agent: *
+Disallow: /