From b5a1d1c7a743da845df9099a69c994378955644a Mon Sep 17 00:00:00 2001 From: Pixelastic Date: Fri, 17 Aug 2018 14:15:23 +0200 Subject: [PATCH] Finishing the config doc rewrite --- docs/src/config-file.md | 245 +++++++++++++++++++--------------------- docs/src/faq.md | 20 ++++ 2 files changed, 135 insertions(+), 130 deletions(-) diff --git a/docs/src/config-file.md b/docs/src/config-file.md index 9313cb9d..61dad093 100644 --- a/docs/src/config-file.md +++ b/docs/src/config-file.md @@ -46,8 +46,7 @@ name you'd like. This array contains the list of urls that will be used to start crawling your website. The crawler will recursively follow any links on those pages. It will -not follow links that are on another domain (unless they are allowed in -`allowed_domains`) and never follow links defined in `stop_urls`. +not follow links that are on another domain and never follow links defined in `stop_urls`. ```json { @@ -294,6 +293,35 @@ a `default_value` to fallback to. } ``` +### Removing unnecessary characters + +Some documentations add special characters to headings, like `#` or `›`. Those +characters have a stylistic value but no meaning and shouldn't be indexed in the +search results. + +You can define a list of characters you want to exclude from the final indexed +value by setting the `strip_chars` key. + +```json +{ + "selectors": { + "lvl0": { + "selector": "#content header h1", + "strip_chars": "#›" + } + } +} +``` + +Note that you can also define `strip_chars` directly at the root of the config +and it will be applied to all selectors. + +```json +{ + "strip_chars": "#›" +} +``` + ### Targeting elements using XPath instead of CSS CSS selectors are a clear and concise way to target elements of a page, but they @@ -324,11 +352,25 @@ browser first, making sure they match what you're expecting. ## Other options -### `allowed_domains` _Optional_ +### `custom_settings` _Optional_ + +This key can be used to overwrite your Algolia index settings. We don't +recommend changing it as the default settings are meant to work for all +websites. + +One use case would be to configure the `separatorsToIndex` setting. By default +Algolia will consider all special character as a word separator. In some contexts, like for method names, you might want `_`, `/` or `#` to keep their meaning. + +```json +{ + "custom_settings": { + "separatorsToIndex": "_/" + } +} +``` + +Check the [Algolia documentation][2] for more information on the settings. -You can pass an array of strings. This is the whitelist of -domains the crawler will browse. If a link targets a page that is not in the -whitelist, the crawler will not follow it. ### `min_indexed_level` _Optional_ @@ -342,6 +384,35 @@ This is useful when your documentation has pages that share the same `lvl0` and records, but want to keep the one matching content that is different across pages. +```json +{ + "min_indexed_level": 2 +} +``` + +### `nb_hits` _Special_ + +`nb_hits` automatically updated by DocSearch every time it runs your config. It +is set to the number of records that were extracted and indexed. We monitor this +key internally to keep track of any unintended spike or drop that could reveal +a misconfiguration. + +You don't have to set nor modify this key, we're only documenting it here in +case you were wondering what this was about. + +### `only_content_level` _Optional_ + +When `only_content_level` is set to `true`, then the crawler won't create +records for each `lvlX` selectors, but only for the `text` selectors. + +If used, `min_indexed_level` is ignored. + +```json +{ + "only_content_level": true +} +``` + ### `scrape_start_urls` _Optional_ By default, the crawler will not extract content from the pages defined in @@ -393,6 +464,9 @@ Note that this is often used to avoid duplicate content, by adding ## Using a sitemap +If your website has a `sitemap.xml` file, you can let DocSearch know and it will +use it to define which pages to crawl. + ### `sitemap_urls` _Optional_ You can pass an array of urls pointing to your sitemap(s) files. If this value @@ -435,149 +509,60 @@ crawled. ``` +## Using JavaScript rendering +By default DocSearch expect websites to have server-side rendering, meaning that +HTML source is returned directly by the server. If your content is generated by the +front-end, you have to tell DocSearch to emulate a browser through Selenium. - - - - - - - - - - - - - - - - - - - - - - -### `only_content_level` _Optional_ - -When `only_content_level` is set to `true`, we only index builded records which match the `text` selectors. Every other record will be skipped. This parameter is more flexible than `min_indexed_level`. Once `only_content_level` is used, `min_indexed_level` becomes pointless. - -Default is `false` +_As client-side crawling is orders of magnitude slower than server-side +crawling, we highly encourage you to update your website to enable server-side +rendering._ ### `js_render` _Optional_ -The HTML code that we crawl is sometimes generated using Javascript. In those -cases, the `js_render` option must be set to `true`. It will enable our -internal proxy (Selenium) to render pages before crawling them. - -We highly recommend avoiding client-side rendering. It mainly decreases [the performance of your website][3]. - -Default is `false` - -### `js_wait` _Optional_ - -When `js_render` is set to `true`, the `js_wait` parameter lets you change the default waiting time (in seconds) to render the -webpage with the Selenium emulator. - -Default is `0`s - -### `use_anchors` _Optional_ - -The `use_anchors` needs to be set to True for a javascript doc when the hash is -used to route the query. Internally, this will disable the canonicalize feature that -is removing the hash from the url. - -This parameter is optional and is set to `false` by default. - -### `strip_chars` _Optional_ - -A list of characters to remove from the indexed text. - -You can also override the default `strip_chars` per level +Set this value to true if your website requires client-side rendering. This will +make DocSearch spawn a Selenium proxy to fetch all your webpages. ```json -"selectors": { - "lvl0": { - "selector": "#content article h1", - "strip_chars": " .,;:" - } +{ + "js_render": true } ``` -### `nb_hits` _Mandatory_ +### `js_wait` _Optional_ -Each time the configuration is locally run, this attribute is set to the number of records indexed. +If your website is slow to load, you can use `js_wait` to tell DocSearch to wait +a specific amount of time (in seconds) for the page to load before extracting +its content. -This attribute is used for purposed monitoring. We keep a track of its evolution in order to detect main changes. -Default is `0`. +Note that this option might have a very large impact on the time required +to crawl your website and we would encourage you to enable server-side rendering +on your website instead. -### `custom_settings` _Optional_ - -This object is [any custom Algolia settings][4] you would like to pass to the index -settings. You will [look under the hood of algolia][5]. - -### `nb_hits_max` _Optional_ - -The number of maximum records allowed for the whole indexing. If the scrapping is bigger, it will fail. - -This value is not meant to be set from anyone except DocSearch maintainer. - -Default is `600 000` (arbitrary, might change) - -## Possible issues - -### Duplicated content - -It could happen that the crawled website returned duplicated data. Most of the time, this is because the crawled pages got the same urls with two different schemes. - -If we have URLs like `http://website.com/page` and `http://website.com/page/` (notice the second one ends with `/`), the scraper will consider them as different. This can be fixed by adding a regex to the `stop_urls` in the `config.json`: +This option has no impact if `js_render` is set to `false`. ```json -"stop_urls": [ - "/$" -] +{ + "js_render": true, + "js_wait": 2 +} ``` -In this attribute, you can also list the pages you want to skip: +### `use_anchors` _Optional_ + +Websites using client-side rendering often don't use full urls, but instead take +advantage of the url hash (the part after the `#`). + +If your website is using such urls, you should set `use_anchors` to `true` for +DocSearch to index all your content. ```json -"stop_urls": [ - "http://website.com/page/" -] +{ + "js_render": true, + "use_anchors": true +} ``` -### Anchors - -The scraper will also consider pages with anchors as different pages. Make sure you remove any hash sign from the urls that you put in the stop & start URLs: - -*Bad:* - -```json -"stop_urls": [ - "http://website.com/page/#foo" -] -``` - -*Good:* - -```json -"stop_urls": [ - "/$" -] -``` - -Or : - -```json -"stop_urls": [ - "http://website.com/page/" -] -``` - - [1]: https://github.com/algolia/docsearch-configs/tree/master/configs -[2]: https://github.com/algolia/docsearch-configs/issues/83) -[3]: https://medium.com/walmartlabs/the-benefits-of-server-side-rendering-over-client-side-rendering-5d07ff2cefe8 -[4]: https://www.algolia.com/doc/api-client/settings/#the-scope-of-settings-and-parameters -[5]: https://www.algolia.com/doc/ +[2]: https://www.algolia.com/doc/api-reference/settings-api-parameters/ diff --git a/docs/src/faq.md b/docs/src/faq.md index ab751bdf..80554009 100644 --- a/docs/src/faq.md +++ b/docs/src/faq.md @@ -75,6 +75,26 @@ What we recommend instead is to exclude the code blocks from the indexing (by using the `selectors_exclude` option in your config), and instead structure your content so the method names are actual headers. +## Why do I have duplicate content in my results? + +This can happen when you have several urls pointing to the same content, for +example with `./docs`, `./docs/` and `./docs/index.html` or even both `http` and +`https` in place. + +This can be fixed by `stop_urls` to all the patterns you want to exclude. The +following example will exclude all urls ending with `/` or `index.html` as well +as those starting with `http://`. + +```json +{ + "stop_urls": [ + "/$", + "/index.html$", + "^http://" + ] +} +``` + ## My favorite project documentation does not use DocSearch. What can I do? We'd love to help!