From 42af8459d8f46caa59f1d94263e06550b07456ce Mon Sep 17 00:00:00 2001 From: Sylvain Pace Date: Wed, 8 Aug 2018 17:31:28 +0200 Subject: [PATCH 1/3] Doc/update content (#396) * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: chore(deps): update dependency onchange to v4.1.0 integrate previous work enhance tyle/content reformat part 1 wait for review adance start_urls enhance attributes description fix typo proofread documentation/docsearch add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review # This is the commit message #2: update README #269 # This is the commit message #3: fix json # This is the commit message #2: enhance as algolia/docsearch-configs#387 # This is the commit message #3: updating flavicon # This is the commit message #2: Update 1-customize-configuration-file.html.md.erb * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: chore(deps): update dependency onchange to v4.1.0 integrate previous work enhance tyle/content reformat part 1 wait for review adance start_urls enhance attributes description fix typo proofread documentation/docsearch add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review # This is the commit message #2: update README #269 # This is the commit message #3: fix json # This is the commit message #2: enhance as algolia/docsearch-configs#387 # This is the commit message #3: updating flavicon # This is the commit message #2: Update 1-customize-configuration-file.html.md.erb * documenting algolia/docsearch-scraper#387 * documenting algolia/docsearch-scraper#387 * fixing changelog * fixing typo * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: chore(deps): update dependency onchange to v4.1.0 integrate previous work enhance tyle/content reformat part 1 wait for review adance start_urls enhance attributes description fix typo proofread documentation/docsearch add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review # This is the commit message #2: update README #269 # This is the commit message #3: fix json # This is the commit message #2: enhance as algolia/docsearch-configs#387 # This is the commit message #3: updating flavicon # This is the commit message #2: Update 1-customize-configuration-file.html.md.erb * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: chore(deps): update dependency onchange to v4.1.0 integrate previous work enhance tyle/content reformat part 1 wait for review adance start_urls enhance attributes description fix typo proofread documentation/docsearch add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review # This is the commit message #2: update README #269 # This is the commit message #3: fix json # This is the commit message #2: enhance as algolia/docsearch-configs#387 # This is the commit message #3: updating flavicon # This is the commit message #2: Update 1-customize-configuration-file.html.md.erb * documenting algolia/docsearch-scraper#387 * documenting algolia/docsearch-scraper#387 * fixing typo * fixing style issue --- .../2-docsearch-scraper/2-config-options.html.md.erb | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb b/docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb index 44e8537c..966f0cf3 100644 --- a/docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb +++ b/docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb @@ -67,11 +67,9 @@ We mostly attribute it on our own regarding plenty of underlying factors. The `a ### `start_urls` _Mandatory_ You can pass either a string or an array of urls. The crawler will go to each page in order, following every link it finds on the page. It will only stop if -the domain is outside of the `allowed_domains` or if the link is blacklisted in +the domain is outside of the `allowed_domains` or if the link is blacklisted from the `stop_urls`. -`stop_urls`. - -Note that it currently does not follow *301* redirects. +Note that we currently do not follow *301* redirects. This parameter also behaves as a [regular expression](https://en.wikipedia.org/wiki/Regular_expression). If you don't use a sitemap, you must define at least one reachable URL (HTTP 20x). Otherwise the scraping will fail. @@ -149,7 +147,7 @@ Finally, If your website contains different parts and layouts, you can define sp } ``` -#### `start_urls.tags` +### `start_urls.tags` Tags will be apllied to every record from the matchin matched pages, i.e., its URL is matchin the `url`. These `tags` will [be processed as `attributesForFaceting`](https://www.algolia.com/doc/api-reference/api-parameters/attributesForFaceting/) . If you want further detail, [check the original documentation](https://www.algolia.com/doc/api-reference/api-parameters/facetFilters/). We do recommend to use `tags` if you want [to refine the scope of your search](https://www.algolia.com/doc/guides/searching/faceting/#faceting-overview). From 49bacccbce5284a82a78bcc1da858334150368ea Mon Sep 17 00:00:00 2001 From: Danny Guo Date: Thu, 9 Aug 2018 04:33:32 -0400 Subject: [PATCH 2/3] docs: Fix a typo (#397) --- .../4-docsearch-FAQ/3-does-my-project-qualify.html.md.erb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/documentation/4-docsearch-FAQ/3-does-my-project-qualify.html.md.erb b/docs/source/documentation/4-docsearch-FAQ/3-does-my-project-qualify.html.md.erb index a711eb1a..6c28ee32 100644 --- a/docs/source/documentation/4-docsearch-FAQ/3-does-my-project-qualify.html.md.erb +++ b/docs/source/documentation/4-docsearch-FAQ/3-does-my-project-qualify.html.md.erb @@ -14,7 +14,7 @@ Please note that you can [always run DocSearch on your own infra](https://commun If you want us to host everything for you, there are some few requirements you still need to comply with: - Be **an owner** of the snippet. You will need to inject some JS snippets. -- Your website must be **publicly available**. If your website is not, you will need to run our scrapper on a whitelisted server or implemnet your own authentification strategy (not supported so far). +- Your website must be **publicly available**. If your website is not, you will need to run our scrapper on a whitelisted server or implement your own authentification strategy (not supported so far). - Your website is a **documentation**. We will reduce the scope of our scrap to helpful pages only. - Your website is **production ready**. We will not accept lorem ipsum website. From 989f26b4907585a3e12d6c5c5d4a40941aea3a44 Mon Sep 17 00:00:00 2001 From: Sylvain Pace Date: Thu, 9 Aug 2018 13:47:00 +0200 Subject: [PATCH 3/3] Doc/update content (#398) * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: chore(deps): update dependency onchange to v4.1.0 integrate previous work enhance tyle/content reformat part 1 wait for review adance start_urls enhance attributes description fix typo proofread documentation/docsearch add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review # This is the commit message #2: update README #269 # This is the commit message #3: fix json # This is the commit message #2: enhance as algolia/docsearch-configs#387 # This is the commit message #3: updating flavicon # This is the commit message #2: Update 1-customize-configuration-file.html.md.erb * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: chore(deps): update dependency onchange to v4.1.0 integrate previous work enhance tyle/content reformat part 1 wait for review adance start_urls enhance attributes description fix typo proofread documentation/docsearch add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review # This is the commit message #2: update README #269 # This is the commit message #3: fix json # This is the commit message #2: enhance as algolia/docsearch-configs#387 # This is the commit message #3: updating flavicon # This is the commit message #2: Update 1-customize-configuration-file.html.md.erb * documenting algolia/docsearch-scraper#387 * documenting algolia/docsearch-scraper#387 * fixing changelog * fixing typo * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: chore(deps): update dependency onchange to v4.1.0 integrate previous work enhance tyle/content reformat part 1 wait for review adance start_urls enhance attributes description fix typo proofread documentation/docsearch add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review # This is the commit message #2: update README #269 # This is the commit message #3: fix json # This is the commit message #2: enhance as algolia/docsearch-configs#387 # This is the commit message #3: updating flavicon # This is the commit message #2: Update 1-customize-configuration-file.html.md.erb * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: # This is a combination of 3 commits. # This is the 1st commit message: chore(deps): update dependency onchange to v4.1.0 integrate previous work enhance tyle/content reformat part 1 wait for review adance start_urls enhance attributes description fix typo proofread documentation/docsearch add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review # This is the commit message #2: update README #269 # This is the commit message #3: fix json # This is the commit message #2: enhance as algolia/docsearch-configs#387 # This is the commit message #3: updating flavicon # This is the commit message #2: Update 1-customize-configuration-file.html.md.erb * documenting algolia/docsearch-scraper#387 * documenting algolia/docsearch-scraper#387 * fixing typo * fixing style issue * fixing recurrnet issue --- .../2-docsearch-scraper/2-config-options.html.md.erb | 2 +- .../1-customize-configuration-file.html.md.erb | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb b/docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb index 966f0cf3..3bdeb022 100644 --- a/docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb +++ b/docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb @@ -383,7 +383,7 @@ Default is `0`. ### `custom_settings` _Optional_ This object is [any custom Algolia settings](https://www.algolia.com/doc/api-client/settings/#the-scope-of-settings-and-parameters) you would like to pass to the index -settings. You will [look under the hood of algolia](https://www.algolia.com/doc/). +settings. Please use the python syntax. You will [look under the hood of algolia](https://www.algolia.com/doc/). ### `nb_hits_max` _Optional_ diff --git a/docs/source/documentation/4-docsearch-FAQ/1-customize-configuration-file.html.md.erb b/docs/source/documentation/4-docsearch-FAQ/1-customize-configuration-file.html.md.erb index 30355613..a20f8a1b 100644 --- a/docs/source/documentation/4-docsearch-FAQ/1-customize-configuration-file.html.md.erb +++ b/docs/source/documentation/4-docsearch-FAQ/1-customize-configuration-file.html.md.erb @@ -76,4 +76,11 @@ You can override the way these elements are impacting the search thanks to `cust "desc(weight.level)" ] } - ``` \ No newline at end of file + ``` + +## Editing settings + +### Why the settings made from the algolia dashboard are not persistent? + +This situation only happens when you are running our DocSearch scraper on your own. +Every time you are running the crawler, every algolia settings are overridden. If you wan to use custome algolia settings, [please use `custom_settings`](https://community.algolia.com/docsearch/documentation/docsearch-scraper/config-options/#custom_settings-optional) instead. \ No newline at end of file