diff options
| author | Nicolas Lœuillet <nicolas@loeuillet.org> | 2017-12-16 09:02:40 +0100 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2017-12-16 09:02:40 +0100 |
| commit | 9b606725fe7cb060433fd08a52cf49799734eda9 (patch) | |
| tree | 0ea708172e9455ae63791879f1bb611f7c2af699 | |
| parent | a68a80f654be86ae25d49420b86099302ce55bb6 (diff) | |
| parent | e4ae6b976fbccb5fff7f03e97b6cf33704328e16 (diff) | |
| download | wallabag-9b606725fe7cb060433fd08a52cf49799734eda9.tar.gz wallabag-9b606725fe7cb060433fd08a52cf49799734eda9.tar.zst wallabag-9b606725fe7cb060433fd08a52cf49799734eda9.zip | |
Merge pull request #3510 from wallabag/update-robotstxt
Updated robots.txt to prevent crawling
| -rw-r--r-- | web/robots.txt | 4 |
1 files changed, 1 insertions, 3 deletions
diff --git a/web/robots.txt b/web/robots.txt index 214e4119..1f53798b 100644 --- a/web/robots.txt +++ b/web/robots.txt | |||
| @@ -1,4 +1,2 @@ | |||
| 1 | # www.robotstxt.org/ | ||
| 2 | # www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 | ||
| 3 | |||
| 4 | User-agent: * | 1 | User-agent: * |
| 2 | Disallow: / | ||