nikon/docsearch
1
0
Fork 0
Code Activity Actions

Finishing the config doc rewrite

Browse source
This commit is contained in:
Pixelastic 2018-08-17 14:15:23 +02:00
parent 51e4de0d77
commit b5a1d1c7a7
2 changed files with 135 additions and 130 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

245
docs/src/config-file.md
View file

@ -46,8 +46,7 @@ name you'd like.
This array contains the list of urls that will be used to start crawling your
website. The crawler will recursively follow any links on those pages. It will
not follow links that are on another domain (unless they are allowed in
`allowed_domains`) and never follow links defined in `stop_urls`.
not follow links that are on another domain and never follow links defined in `stop_urls`.
```json
{
@ -294,6 +293,35 @@ a `default_value` to fallback to.
}
```
### Removing unnecessary characters
Some documentations add special characters to headings, like `#` or `›`. Those
characters have a stylistic value but no meaning and shouldn't be indexed in the
search results.
You can define a list of characters you want to exclude from the final indexed
value by setting the `strip_chars` key.
```json
{
"selectors": {
"lvl0": {
"selector": "#content header h1",
"strip_chars": "#›"
}
}
}
```
Note that you can also define `strip_chars` directly at the root of the config
and it will be applied to all selectors.
```json
{
"strip_chars": "#›"
}
```
### Targeting elements using XPath instead of CSS
CSS selectors are a clear and concise way to target elements of a page, but they
@ -324,11 +352,25 @@ browser first, making sure they match what you're expecting.
## Other options
### `allowed_domains` _Optional_
### `custom_settings` _Optional_
This key can be used to overwrite your Algolia index settings. We don't
recommend changing it as the default settings are meant to work for all
websites.
One use case would be to configure the `separatorsToIndex` setting. By default
Algolia will consider all special character as a word separator. In some contexts, like for method names, you might want `_`, `/` or `#` to keep their meaning.
```json
{
"custom_settings": {
"separatorsToIndex": "_/"
}
}
```
Check the [Algolia documentation][2] for more information on the settings.
You can pass an array of strings. This is the whitelist of
domains the crawler will browse. If a link targets a page that is not in the
whitelist, the crawler will not follow it.
### `min_indexed_level` _Optional_
@ -342,6 +384,35 @@ This is useful when your documentation has pages that share the same `lvl0` and
records, but want to keep the one matching content that is different across
pages.
```json
{
"min_indexed_level": 2
}
```
### `nb_hits` _Special_
`nb_hits` automatically updated by DocSearch every time it runs your config. It
is set to the number of records that were extracted and indexed. We monitor this
key internally to keep track of any unintended spike or drop that could reveal
a misconfiguration.
You don't have to set nor modify this key, we're only documenting it here in
case you were wondering what this was about.
### `only_content_level` _Optional_
When `only_content_level` is set to `true`, then the crawler won't create
records for each `lvlX` selectors, but only for the `text` selectors.
If used, `min_indexed_level` is ignored.
```json
{
"only_content_level": true
}
```
### `scrape_start_urls` _Optional_
By default, the crawler will not extract content from the pages defined in
@ -393,6 +464,9 @@ Note that this is often used to avoid duplicate content, by adding
## Using a sitemap
If your website has a `sitemap.xml` file, you can let DocSearch know and it will
use it to define which pages to crawl.
### `sitemap_urls` _Optional_
You can pass an array of urls pointing to your sitemap(s) files. If this value
@ -435,149 +509,60 @@ crawled.
</url>
```
## Using JavaScript rendering
By default DocSearch expect websites to have server-side rendering, meaning that
HTML source is returned directly by the server. If your content is generated by the
front-end, you have to tell DocSearch to emulate a browser through Selenium.
### `only_content_level` _Optional_
When `only_content_level` is set to `true`, we only index builded records which match the `text` selectors. Every other record will be skipped. This parameter is more flexible than `min_indexed_level`. Once `only_content_level` is used, `min_indexed_level` becomes pointless.
Default is `false`
_As client-side crawling is orders of magnitude slower than server-side
crawling, we highly encourage you to update your website to enable server-side
rendering._
### `js_render` _Optional_
The HTML code that we crawl is sometimes generated using Javascript. In those
cases, the `js_render` option must be set to `true`. It will enable our
internal proxy (Selenium) to render pages before crawling them.
We highly recommend avoiding client-side rendering. It mainly decreases [the performance of your website][3].
Default is `false`
### `js_wait` _Optional_
When `js_render` is set to `true`, the `js_wait` parameter lets you change the default waiting time (in seconds) to render the
webpage with the Selenium emulator.
Default is `0`s
### `use_anchors` _Optional_
The `use_anchors` needs to be set to True for a javascript doc when the hash is
used to route the query. Internally, this will disable the canonicalize feature that
is removing the hash from the url.
This parameter is optional and is set to `false` by default.
### `strip_chars` _Optional_
A list of characters to remove from the indexed text.
You can also override the default `strip_chars` per level
Set this value to true if your website requires client-side rendering. This will
make DocSearch spawn a Selenium proxy to fetch all your webpages.
```json
"selectors": {
"lvl0": {
"selector": "#content article h1",
"strip_chars": " .,;:"
}
{
"js_render": true
}
```
### `nb_hits` _Mandatory_
### `js_wait` _Optional_
Each time the configuration is locally run, this attribute is set to the number of records indexed.
If your website is slow to load, you can use `js_wait` to tell DocSearch to wait
a specific amount of time (in seconds) for the page to load before extracting
its content.
This attribute is used for purposed monitoring. We keep a track of its evolution in order to detect main changes.
Default is `0`.
Note that this option might have a very large impact on the time required
to crawl your website and we would encourage you to enable server-side rendering
on your website instead.
### `custom_settings` _Optional_
This object is [any custom Algolia settings][4] you would like to pass to the index
settings. You will [look under the hood of algolia][5].
### `nb_hits_max` _Optional_
The number of maximum records allowed for the whole indexing. If the scrapping is bigger, it will fail.
This value is not meant to be set from anyone except DocSearch maintainer.
Default is `600 000` (arbitrary, might change)
## Possible issues
### Duplicated content
It could happen that the crawled website returned duplicated data. Most of the time, this is because the crawled pages got the same urls with two different schemes.
If we have URLs like `http://website.com/page` and `http://website.com/page/` (notice the second one ends with `/`), the scraper will consider them as different. This can be fixed by adding a regex to the `stop_urls` in the `config.json`:
This option has no impact if `js_render` is set to `false`.
```json
"stop_urls": [
"/$"
]
{
"js_render": true,
"js_wait": 2
}
```
In this attribute, you can also list the pages you want to skip:
### `use_anchors` _Optional_
Websites using client-side rendering often don't use full urls, but instead take
advantage of the url hash (the part after the `#`).
If your website is using such urls, you should set `use_anchors` to `true` for
DocSearch to index all your content.
```json
"stop_urls": [
"http://website.com/page/"
]
{
"js_render": true,
"use_anchors": true
}
```
### Anchors
The scraper will also consider pages with anchors as different pages. Make sure you remove any hash sign from the urls that you put in the stop & start URLs:
*Bad:*
```json
"stop_urls": [
"http://website.com/page/#foo"
]
```
*Good:*
```json
"stop_urls": [
"/$"
]
```
Or :
```json
"stop_urls": [
"http://website.com/page/"
]
```
[1]: https://github.com/algolia/docsearch-configs/tree/master/configs
[2]: https://github.com/algolia/docsearch-configs/issues/83)
[3]: https://medium.com/walmartlabs/the-benefits-of-server-side-rendering-over-client-side-rendering-5d07ff2cefe8
[4]: https://www.algolia.com/doc/api-client/settings/#the-scope-of-settings-and-parameters
[5]: https://www.algolia.com/doc/
[2]: https://www.algolia.com/doc/api-reference/settings-api-parameters/

20
docs/src/faq.md
View file

@ -75,6 +75,26 @@ What we recommend instead is to exclude the code blocks from the indexing (by
using the `selectors_exclude` option in your config), and instead structure your
content so the method names are actual headers.
## Why do I have duplicate content in my results?
This can happen when you have several urls pointing to the same content, for
example with `./docs`, `./docs/` and `./docs/index.html` or even both `http` and
`https` in place.
This can be fixed by `stop_urls` to all the patterns you want to exclude. The
following example will exclude all urls ending with `/` or `index.html` as well
as those starting with `http://`.
```json
{
"stop_urls": [
"/$",
"/index.html$",
"^http://"
]
}
```
## My favorite project documentation does not use DocSearch. What can I do?
We'd love to help!