From 714abba58dfb33e6776adcc4caa2a47c1789c0c1 Mon Sep 17 00:00:00 2001 From: Sylvain Pace Date: Tue, 21 Aug 2018 12:07:10 +0200 Subject: [PATCH] fix and closes algolia/docsearch-configs-private#82 --- docs/src/config-file.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/src/config-file.md b/docs/src/config-file.md index fe7f1c01..60fe4c13 100644 --- a/docs/src/config-file.md +++ b/docs/src/config-file.md @@ -398,13 +398,13 @@ If used, `min_indexed_level` is ignored. ### `scrape_start_urls` _Optional_ -By default, the crawler will not extract content from the pages defined in -`starts_urls`. If you have valuable content on your homepage, you should set -this to `true`. +By default, the crawler will extract content from the pages defined in +`starts_urls`. If you do have valuable content on your `starts_urls` or these pages are duplicates from another one, you should set +this to `false`. ```json { - "scrape_start_urls": true + "scrape_start_urls": false } ```