-<h2 id="restricting-search-engines-and-web-crawler-traffic">Restricting search engines and web crawler traffic</h2>
-<p>Creating a <code>robots.txt</code> witht he following contents at the root of your Shaarli installation will prevent "honest" web crawlers from indexing each and every link and Daily page from a Shaarli instance, thus getting rid of a certain amount of unsollicited network traffic.</p>
-<pre><code>User-agent: *
-Disallow: /</code></pre>
-<p>See: <a href="http://www.robotstxt.org/" class="uri">http://www.robotstxt.org/</a>, <a href="http://www.robotstxt.org/robotstxt.html" class="uri">http://www.robotstxt.org/robotstxt.html</a>, <a href="http://www.robotstxt.org/meta.html" class="uri">http://www.robotstxt.org/meta.html</a></p>