nikon/docsearch
1
0
Fork 0
Code Activity Actions

Updating config doc

Browse source
This commit is contained in:
Pixelastic 2018-08-16 19:26:09 +02:00
parent b5733d2bc9
commit 1eb2e148bd
5 changed files with 471 additions and 325 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
docs/src/_layouts/two-columns.pug
View file

@ -18,9 +18,9 @@ block nav
block content
.bgtb-moon-white-pure.relative
.md_flex.max-w-lg.m-auto
.navigation-sidebar.absolute.hidden.md_static.bg-white.md_bg-transparent.w-100.md_w-25.md_block.md_border-0.text-left.md_pl-2
.navigation-sidebar.sticky.pin-t.top-3.h-100vh-3.overflow-y-auto.hidden.md_block.bg-white
// Adding subsections from markdown
.sticky.pin-t.h-100vh-3.overflow-y-auto.pt-2.w-5.m-auto.sm_flex.sm_w-90.md_block.md_w-100
.pt-2.w-5.m-auto.sm_flex.sm_w-90.md_block.md_w-100.md_pl-2
each category in sidebar
.flcnw.flspa.sm_flex-auto.sm_w-33.sm_p-1.md_w-100.md_p-0
.category.text-nebula.poppins.uppercase.mb-1.text-2=category.title
@ -37,7 +37,7 @@ block content
li
a.block.py-0x.text-telluric(href=`${page.url}#${heading.anchor}`)=heading.title
.documentation-content.text-1.pt-2.pb-1.px-2.w_100.md_w-75
.documentation-content.text-2.pt-2.pb-1.px-2.w_100.md_w-75
.block.md_hidden.mb-1
a(href=site.url) DocSearch
span  > 

21
docs/src/_styles/docs.css
View file

@ -30,6 +30,27 @@
@apply .inline-block;
}
}
h3 {
@apply .text-nebula .text-3 .normal;
@apply .mb-1 .mt-2;
/* Trick to have heading scroll into view and not being hidden by header */
&:before {
content: "";
@apply hidden .h-3;
}
&:target:before {
@apply block;
}
.anchor {
@apply .hidden;
&:after {
content: "#";
}
}
&:hover .anchor {
@apply .inline-block;
}
}
}
.navigation-sidebar{

648
docs/src/config-file.md
View file

@ -1,77 +1,86 @@
---
layout: two-columns
title: Make the most of your configuration
title: Config Files
---
// TODO
For each DocSearch request we receive, we'll create a custom JSON config file
that will define how the crawler should behave. You can find all the configs in
[this repository][1].
## Introduction
The DocSearch scraper will use a configuration file specifying:
- the Algolia index name that will store the records resulting from the crawling
- the URLs it needs to crawl
- the URLs it shouldn't crawl
- the (hierarchical) CSS selectors to use to extract the relevant content from your webpages
- the CSS selectors to skip
- An optional sitemap URL that will be crawled and then scraped
- additional options you might provide to fine-tune the scraping
## How it works
Once you run the DocSearch scraper on a specific configuration, it will:
- crawl all the URLs you specified (from the `start_urls` or the `sitemap`)
- follow all the hyperlinks mentioned in the page, and continue the crawling there
- stop the crawling as soon as you've reached a URL that is not specified in your configuration or affiliated to a start url
- extract the content of every single crawled page following the logic you defined using the CSS selectors
- push the resulting records to the Algolia index you configured
### Does the crawl encompass several domains/sub-domains?
The `start_urls` define the allowed domain for our crawler. Basically, we take the main domain of every URLs. We will not go outside this whitelisted domain list. If you want to encompass a wider domain, please include it as a new start_url.
## Configuration format
A configuration file looks like:
A DocSearch looks like this:
```json
{
"index_name": "stripe",
"start_urls": [
"https://stripe.com/docs"
],
"stop_urls": [
"https://stripe.com/docs/api"
],
"selectors": {
"lvl0": "#content header h1",
"lvl1": "#content article h1",
"lvl2": "#content section h3",
"lvl3": "#content section h4",
"lvl4": "#content section h5",
"lvl5": "#content section h6",
"text": "#content header p,#content section p,#content section ol"
},
"selectors_exclude": [
".method-list",
"aside.note"
],
// additional options
[...]
"index_name": "example",
"start_urls": [
"https://www.example.com/docs"
],
"selectors": {
"lvl0": "#content header h1",
"lvl1": "#content article h1",
"lvl2": "#content section h3",
"lvl3": "#content section h4",
"lvl4": "#content section h5",
"lvl5": "#content section h6",
"text": "#content header p,#content section p,#content section ol"
},
}
```
It must be **a valid JSON file**
## ``index_name``
## DocSearch options
This is the name of the Algolia index where your records will be pushed. The
`apiKey` we will share with you will be restricted to work on this index.
### `index_name` _Mandatory_
When using the free DocSearch crawler, the `indexName` will always be the name
of the config. If you're running DocSearch yourself, you can of course use any
name you'd like.
Name of the Algolia index where all the data will be pushed.
```json
{
"index_name": "example"
}
```
**On our own infrastructure, this name must be equal to the configuration file name**
## `start_urls`
This array contains the list of urls that will be used to start crawling your
website. The crawler will recursively follow any links on those pages. It will
not follow links that are on another domain (unless they are allowed in
`allowed_domains`) and never follow links defined in `stop_urls`.
```json
{
"start_urls": [
"https://www.example.com/docs"
]
}
```
### Using regular expressions
The `start_urls` option also allows for passing an object in place of a string,
to express more complex patterns. This object must contain a `url` key
containing the regular expression matching the url, as well as a `variables` key
that will be used to replace the named matches.
The following example will make it clearer:
```json
{
"start_urls": [
{
"url": "http://www.example.com/docs/(?P<lang>.*?)/(?P<version>.*?)/",
"variables": {
"lang": ["en", "fr"],
"version": ["latest", "3.3", "3.2"]
}
}
]
}
```
<<<<<<< HEAD
We mostly attribute it on our own regarding plenty of underlying factors. The `apiKey` that we provide is generated with a restriction on the `index_name`. Changing the `index_name` would require to ask for a new key. Thus if you want to **change the name**, please **submit a new configuration**, we will generate a new key accordingly.
### `start_urls` _Mandatory_
You can pass either a string or an array of urls. The crawler will go to each
@ -79,240 +88,241 @@ page in order, following every link it finds on the page. It will only stop if
the domain is outside of the `allowed_domains` or if the link is blacklisted from the `stop_urls`.
Note that we currently do not follow *301* redirects.
=======
The beneficial side effect of using this syntax is that all records that will
then be extracted from crawling `http://www.example.com/docs/en/latest` will have
`lang: en` and `version: latest` added to it, allowing you to then filter based on
those values.
This parameter also behaves as a [regular expression](https://en.wikipedia.org/wiki/Regular_expression). If you don't use a sitemap, you must define at least one reachable URL (HTTP 20x). Otherwise the scraping will fail.
The following example shows how you can filter results matching specifics
language and version from the front-end
You can build a more advanced URL. You will need to use a JSON object with a `variables` attribute. This attribute is an array of variables that will be injected into the URLs:
```js
docsearch({
[…],
algoliaOptions: {
'facetFilters': ["lang:en", "version:latest"]
},
});
```
>>>>>>> Updating config doc
### Using custom tags
You can also apply custom tags to some pages without the need to use regular
expressions. In that case, add the list of tags to the `tags` key. Note
that those tags will be automatically added as facets in Algolia, allowing you
to filter based on their values as well.
**Example:**
```json
{
"url": "http://example.com/docs/(?P<lang>.*?)/(?P<version>.*?)/",
"variables": {
"version": [
"latest",
"3.3",
[...]
],
"lang": [
"en",
"fr",
[...]
]
"start_urls": [
{
"url": "http://www.example.com/docs/concepts/",
"tags": ['concepts', 'terminology']
}
]
}
```
The whole pattern `(?P<version>.*?)` will be replaced by the value assigned in the related key `version`.
The variable name is not fixed. Please note that those variables will behave as
[`attributesForFaceting`](https://www.algolia.com/doc/api-reference/api-parameters/attributesForFaceting/) which may help you [restrain the scope of the search from the snippet](https://www.algolia.com/doc/guides/searching/faceting/).
Thus you can limit the scope of the search to the records from the pages encompassed by `http://example.com/docs/en/latest/*` thanks to the following snippet:
```js
<script type="text/javascript">
docsearch({
apiKey: ${apiKey},
indexName: ${indexName},
inputSelector: '#search'
algoliaOptions: { 'facetFilters': ["lang:en", "version:latest"] },
docsearch({
[…],
algoliaOptions: {
'facetFilters': ["tags:concepts"]
},
});
</script>
```
In order to promote some pages, you can set the `page_rank` attribute (default: `0`, can be a positive or negative integer).
### Using Page Rank
Finally, If your website contains different parts and layouts, you can define specific `selectors` for each part and apply them using `selectors_key`:
If you want to give more weight to some pages to boost their ranking in the
results, you can attribute a custom `page_rank` to specific urls. Pages with
highest `page_rank` will be returned before pages with a lower `page_rank`. Note
that you can pass any numeric value, including negative values.
**Example:**
```json
{
"index_name": "example",
"start_urls": [
"http://example.com/docs/latest/",
"start_urls": [
{
"url": "http://example.com/docs/concepts/",
"page_rank": 1,
"selectors_key": "concepts"
"url": "http://www.example.com/docs/concepts/",
"page_rank": 5,
},
{
"url": "http://www.example.com/docs/contributors/",
"page_rank": 1,
}
],
]
}
```
In this example, results extracted from the _Concepts_ page will be ranked
higher than results extracted from the _Contributors_ page.
### Using custom selectors per page
If the markup of your website is so different from one page to another that you
can't have generic selectors, you can namespace your selectors and specify which
set of selectors should be applied to specific pages.
```json
{
"start_urls": [
"http://www.example.com/docs/",
{
"url": "http://www.example.com/docs/concepts/",
"selectors_key": "concepts"
},
{
"url": "http://www.example.com/docs/contributors/",
"selectors_key": "contributors"
}
]
"selectors": {
"default": {
"lvl0": ".docSearch-content h1",
"lvl1": ".docSearch-content h2",
"lvl2": ".docSearch-content h3",
"lvl3": ".docSearch-content h4",
"lvl4": ".docSearch-content h5",
"text": ".docSearch-content p, .docSearch-content li"
"lvl0": ".main h1",
"lvl1": ".main h2",
"lvl2": ".main h3",
"lvl3": ".main h4",
"lvl4": ".main h5",
"text": ".main p"
},
"concepts": {
"lvl0": ".docSearch-header h2",
"lvl1": ".docSearch-content h1",
"lvl2": ".docSearch-content h2",
"lvl3": ".docSearch-content h3",
"lvl4": ".docSearch-content h5",
"text": ".docSearch-content p"
"lvl0": ".header h2",
"lvl1": ".main h1.title",
"lvl2": ".main h2.title",
"lvl3": ".main h3.title",
"lvl4": ".main h5.title",
"text": ".main p"
},
"contributors": {
"lvl0": ".main h1",
"lvl1": ".contributors .name",
"lvl2": ".contributors .title",
"text": ".contributors .description"
}
}
[...]
}
```
<<<<<<< HEAD
### `start_urls.tags`
Tags will be apllied to every record from the matchin matched pages, i.e., its URL is matchin the `url`. These `tags` will [be processed as `attributesForFaceting`](https://www.algolia.com/doc/api-reference/api-parameters/attributesForFaceting/) . If you want further detail, [check the original documentation](https://www.algolia.com/doc/api-reference/api-parameters/facetFilters/).
We do recommend to use `tags` if you want [to refine the scope of your search](https://www.algolia.com/doc/guides/searching/faceting/#faceting-overview).
=======
Here, all documentation pages will use the selectors defined in
`selectors.default` while the page under `./concepts` will use
`selectors.concepts` and those under `./contributors` will use
`selectors.contributors`.
>>>>>>> Updating config doc
From your search UI, you will need to use the following input:
```js
algoliaOptions: { 'facetFilters': ["tags:$VALUE"] },
```
Default not used. Must be an array.
**Example:**
From the config:
```json
"start_urls": [
{
"url": "http://example.com/docs/concepts/",
"selectors_key": "concepts"
}
```
From the search UI:
```js
algoliaOptions: { 'facetFilters': ["tags:concepts"] },
```
### `scrape_start_urls`
This boolean let you decide if you want to extract the content of the starting pages.
Default is `false`
### `stop_urls` _Optional_
This array can be used to blacklist URLs. The crawler will stop on these and
will not consider their content. Likewise, if a link within a crawled webpage
targets such pages, the crawler will not follow the link.
You can use a regular expression as well as plain urls.
Note: It is sometimes needed to add `http://www.example.com/index.html` pages to
the `stop_urls` list if you set `http://www.example.com` as a `start_urls`, to
avoid duplicated content.
### `selectors` _Mandatory_
## `selectors`
This object contains all the CSS selectors that will be used to create the
record hierarchy. It can contains up to 6 levels (`lvl0`, `lvl1`, `lvl2`, `lvl3`, `lvl4`,
`lvl5`) and `text`.
record hierarchy. It can contains up to 6 levels (`lvl0`, `lvl1`, `lvl2`,
`lvl3`, `lvl4`, `lvl5`) and `text`.
A default config would be to target the page `title` or `h1` as `lvl0`, the `h2`
as `lvl1` and `h3` as `lvl2`. `text` is usually any `p` of text.
as `lvl1` and `h3` as `lvl2` and `p` as `text`, but this is highly dependent on
the markup.
We recommend making use of at least the three first levels for better relevancy.
The `text` key is mandatory, but we highly recommend setting also `lvl0`, `lvl1`
and `lvl2` to have a decent level of relevance.
### `global` selectors _Optional_
It's possible to make a selector global which means that all records from the page will have
this value. This is useful when you have a title that is in the right sidebar and
the sidebar is placed after the content in the DOM.
`global` attributes should be seen as a way to extract the matching elements from the HTML flow. These global elements will not be considered as breaking ones when we encounter them along the flow. If this token is enabled :
- we will not create a new record from the current builded stack.
- we will apply its contextual value (or `default_value` if not matched) to every records
We mostly use this parameter for page that miss context. It enables us to pick up the context from another common part without having duplicates.
```json
"selectors": {
"lvl0": {
"selector": "#content header h1",
"global": true
}
}
```
### Xpath selector _Optional_
By default, `selectors` are considered to be [css selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors) but you can specify that a selector is an [XPath one](https://developer.mozilla.org/en-US/docs/Web/XPath).
This is useful when you want to do more complex selection like selecting the parent of a target.
```json
"selectors": {
"lvl0": {
"selector": "//li[@class=\"chapter active done\"]/../../a",
"type": "xpath"
}
}
```
### Default value _Optional_
You have the possibility to add a default value which will be used if the selector doesn't match anything.
```json
"selectors": {
"lvl0": {
"selector": "#content article h1",
"default_value": "Documentation"
}
}
```
### `selectors_exclude` _Optional_
By default, the `selectors` search is applied page-wide. If there are some parts
of the page that you do not want to include (e.g. a table of content, a sidebar or a footer),
you can add them to the `selectors_exclude` key.
### Sitemap crawling _Optional_
Our crawler offers you to crawl a site by discovering the URLs using Sitemaps. Thus, you can define the direct URL(s) to your sitemap XML file, `sitemap_urls`. In order to parse it, you should establish regex(s), `sitemap_urls_regex`, which will match the URLs to crawl. Otherwise it will use the `start_urls` pattern in order to match the expected URLs.
For sites that use Sitemap index files that point to other sitemap files, all those sitemaps will be followed.
### `sitemap_urls` _Optional_
A list of urls pointing to the sitemaps (or sitemap index) you want to crawl. Must be provided if you want docsearch to discover your via sitemaps.
### `sitemap_urls_regexs` _Optional_
A list of regular expressions that will be applied to each URL from the sitemap. If one of the patterns match a URL, this link will be scraped. If no regular expression is defined, the start_urls will be taken as a pattern.
### `force_sitemap_urls_crawling` _Optional_
Specifies if the matched URLs should not respect the same rules as the crawled hyperlink. If set to true, each URL will be scraped even if it does comply with the `start_urls` or `stop_urls`. Default is `force_sitemap_urls_crawling` set to `false`
**Example:**
```json
{
[...]
"sitemap_urls": [
"https://www.mySite.com/sitemap.xml"
],
"sitemap_urls_regexs": [
"/doc/"
],
"force_sitemap_urls_crawling": true,
[...]
"selectors": {
"lvl0": "#content header h1",
"lvl1": "#content article h1",
"lvl2": "#content section h3",
"lvl3": "#content section h4",
"lvl4": "#content section h5",
"lvl5": "#content section h6",
"text": "#content header p,#content section p,#content section ol"
},
}
```
Given this configuration, every webpage of the sitemap whose URL contains '/doc/' will be scraped even if they don't comply with `start_urls` or `stop_urls`.
### `sitemap_alternate_links` _Optional_
Selectors can be passed as string, or as objects containing a `selector` key.
Other special keys can be set, as documented below.
This parameter is only useful when you are using a sitemap to crawl your website.
It specifies if alternate links should be followed. Your sitemap should inlcude localized versions of your page in such format:
```
<url>
<loc>http://example.com/</loc>
<xhtml:link rel="alternate" hreflang="de" href="http://example.com/de"/>
</url>
```json
{
"selectors": {
"lvl0": {
"selector": "#content header h1",
}
}
}
```
If `sitemap_alternate_links` is not set, the link "http://example.com/de" will not be parsed from the sitemap.
### Using global selectors
Default is `false`
The default way of extracting content through selectors is to read the HTML
markup from top to bottom. This works well with semi-structured content, like
a hierarchy of headers. This breaks when relevant information is not part of the
same node flow. For example when the title is in a header or a sidebar.
For that reason, you can set a selector as global, meaning that it will match on
the whole page, and will be the same for all records extracted on this page.
```json
{
"selectors": {
"lvl0": {
"selector": "#content header h1",
"global": true
}
}
}
```
### Setting a default value
If your selector might not match a valid element on the page, you can define
a `default_value` to fallback to.
```json
{
"selectors": {
"lvl0": {
"selector": "#content header h1",
"default_value": "Documentation"
}
}
}
```
### Targeting elements using XPath instead of CSS
CSS selectors are a clear and concise way to target elements of a page, but they
have a limitations. For example, you cannot go _up_ the cascade with CSS.
If you need a more powerful selector mecanism, you can write your selectors
using XPath by setting `type: xpath`. You should also set `global: true` on the
same selector.
The following example will look for a `li.chapter.active.done` and then go up
two levels in the DOM until it found a `a`. The content of this `a` will then be
used as the value of the `lvl0` selector.
```json
{
"selectors": {
"lvl0": {
"selector": "//li[@class=\"chapter active done\"]/../../a",
"type": "xpath",
"global": true
}
}
}
```
XPath selector can be hard to read. We highly encourage you to test them in your
browser first, making sure they match what you're expecting.
## Other options
### `allowed_domains` _Optional_
@ -320,21 +330,134 @@ You can pass an array of strings. This is the whitelist of
domains the crawler will browse. If a link targets a page that is not in the
whitelist, the crawler will not follow it.
### Sitemap crawling _Optional_
Default is the domain of the first elements in the `start_urls`.
### `min_indexed_level` _Optional_
Lets you define the minimum level at which you want a record to be indexed. For
example, with a `min_indexed_level: 1`, you will only index records that have at
least a `lvl0` and a `lvl1` field.
The default value is `0`. By increasing it, you can chose to not index some
records if they don't have enough `lvlX` matching. For example, with
a `min_indexed_level: 2`, records that have at least `lvl0`, `lvl1` and
`lvl2` matching something will be indexed.
This is useful when your documentation has pages that share the same `lvl0` and
`lvl1` for example. In that case, you don't want to index all the shared
records, but want to keep the one matching content that is different across
pages.
### `scrape_start_urls` _Optional_
By default, the crawler will not extract content from the pages defined in
`starts_urls`. If you have valuable content on your homepage, you should set
this to `true`.
```json
{
"scrape_start_urls": true
}
```
### `selectors_exclude` _Optional_
This expects an array of CSS selectors. Any element matching one of those
selectors will be removed from the page before any data is extracted from it.
This can be used to remove a table of content, a sidebar or a footer, to
make other selectors easier to write.
```json
{
"selectors_exclude": [
".footer",
"ul.deprecated"
],
}
```
### `stop_urls` _Optional_
This an array of strings or regular expressions. Whenever the crawler is about
to visit a link, it will first check if the link matches something in the array.
If it does, it will not follow the link. This should be used to restrict pages
the crawler should visit.
Note that this is often used to avoid duplicate content, by adding
`http://www.example.com/docs/index.html` if you already have
`http://www.example.com/docs/` as a `start_urls`.
```json
{
"stop_urls": [
"https://www.example.com/docs/index.html",
"license.html"
],
}
```
## Using a sitemap
### `sitemap_urls` _Optional_
You can pass an array of urls pointing to your sitemap(s) files. If this value
is set, DocSearch will try to read urls from your sitemap(s) instead of
following every link of your `starts_urls`.
```json
{
"sitemap_urls": [
"http://www.example.com/docs/sitemap.xml"
],
}
```
### `sitemap_alternate_links` _Optional_
Sitemaps can contain _alternative links_ for urls. Those are other versions of
the same page, in a different language, or with a different url. By default
DocSearch will ignore those urls.
Set this to `true` if you want those other version to be crawled as well.
```json
{
"sitemap_urls": [
"http://www.example.com/docs/sitemap.xml"
],
"sitemap_alternate_links": true
}
```
With the above config and the `sitemap.xml` below, both
`http://www.example.com/docs/` and `http://www.example.com/docs/de/` will be
crawled.
```
<url>
<loc>http://www.example.com/docs/</loc>
<xhtml:link rel="alternate" hreflang="de" href="http://www.example.com/de/"/>
</url>
```
This is especially useful when the documentation is split into several pages and
all pages duplicate the main title or introduction (see [this issue][https://github.com/algolia/docsearch-configs/issues/83]).
With `min_indexed_level`, you can ignore the duplicated title.
Default is `0`
### `only_content_level` _Optional_
@ -348,7 +471,7 @@ The HTML code that we crawl is sometimes generated using Javascript. In those
cases, the `js_render` option must be set to `true`. It will enable our
internal proxy (Selenium) to render pages before crawling them.
We highly recommend avoiding client-side rendering. It mainly decreases [the performance of your website](https://medium.com/walmartlabs/the-benefits-of-server-side-rendering-over-client-side-rendering-5d07ff2cefe8).
We highly recommend avoiding client-side rendering. It mainly decreases [the performance of your website][3].
Default is `false`
@ -391,8 +514,8 @@ Default is `0`.
### `custom_settings` _Optional_
This object is [any custom Algolia settings](https://www.algolia.com/doc/api-client/settings/#the-scope-of-settings-and-parameters) you would like to pass to the index
settings. Please use the python syntax. You will [look under the hood of algolia](https://www.algolia.com/doc/).
This object is [any custom Algolia settings][4] you would like to pass to the index
settings. You will [look under the hood of algolia][5].
### `nb_hits_max` _Optional_
@ -451,3 +574,10 @@ Or :
"http://website.com/page/"
]
```
[1]: https://github.com/algolia/docsearch-configs/tree/master/configs
[2]: https://github.com/algolia/docsearch-configs/issues/83)
[3]: https://medium.com/walmartlabs/the-benefits-of-server-side-rendering-over-client-side-rendering-5d07ff2cefe8
[4]: https://www.algolia.com/doc/api-client/settings/#the-scope-of-settings-and-parameters
[5]: https://www.algolia.com/doc/

7
docs/src/crawler-overview.md
View file

@ -3,10 +3,9 @@ layout: two-columns
title: Crawler Overview
---
The DocSearch crawler is written in python and heavily based on the
[Scrapy][1] framework. It will crawl all pages of your
website, and extract content from the HTML structure to populate an Algolia
index.
The DocSearch crawler is written in python and heavily based on the [Scrapy][1]
framework. It will crawl all pages of your website and extract content from the
HTML structure to populate an Algolia index.
It will automatically follow every internal link to make sure we are not missing
any content, and will use the semantics of your HTML structure to construct its

114
docs/src/inside-the-engine.md
View file

@ -3,67 +3,63 @@ layout: two-columns
title: Inside the engine
---
// TODO
This page will explain in more details how the crawler extracts content from your
page, and how it ranks it in the results.
## Building your index
## Crawling all your pages
Building records using the scraper is pretty intuitive. According to your settings,
we extract the payload of your webpage and index it, preserving your data's structure.
This is achieved in a simple way:
* We **read top down** your web page following your HTML flow and pick out your
matching elements according their **levels** (`selectors_level`)
* We create a record for each paragraph along with its hierarchical path.
This construction is based on their **time of appearance** along the flow.
* We **index** these records with the appropriate global settings (e.g. metadata, tags, etc.)
Each crawl will begin its journey by the value of the `start_urls` you have in
your config. It will read those pages and recursively follow every link in those
pages until it has crawled all your website.
_**Note:** The above process performs sanity tests as it scrapes, in order to detect errors.
If indeed there are any serious warnings, it will abort and therefore not overwrite your current index.
These checks ensure that your dedicated index isn't flushed._
If it detects a `sitemap.xml`, it will use this link list instead of crawling
all pages.
## Extracting content from each page
Then, for each page, it will read the HTML markup from top to bottom. It will
look for HTML elements matching your CSS `selectors`. It will specifically look
for elements matching your `text` selector (`<p>` by default). Each of those
matches will be later transformed into an Algolia record.
For each matching `text` element, the crawler will also keep in memory the
current hierarchy of headers (identified by the `lvl0` to `lvl5` selectors) that
it had to traverse to get to this text. This hierarchical information, as well
as some generic page metadata (such has the page url) are then pushed to
Algolia.
Note that the crawler performs sanity checks before pushing data to Algolia. For
example if you changed the markup of your website, the selectors might not match
anything. If we detect that something is wrong with your current crawl, we don't
overwrite your previous index.
## Ranking records
Algolia always returns the most relevant results first, using a [tie-breaking
approach][1]. DocSearch will first search for exact matches in your keywords then
fallback to partial matches. Those results will then be ordered based, once
again, on the page hierarchy, as extracted from the `selectors`.
The default strategy is to first look at the closest header of the matching
text. If a matching paragraph of text is under `Advanced Settings / API Options
/ verySpecificMethod()`, it will be ranked higher than if it is only found under
`Gettings Started > Installation`. The idea here is that if you have a match
under a very deep hierarchy, chances are that this match is very specific and
might be more interesting that something found in a very broad topic.
But this does not work in all cases as some documentations don't have deep
hierarchy. In that case, we use the paragraph position. The first paragraph of
the page will be ranked higher than the last one.
You also have a way to boost some pages directly in your config by using the
`page_rank` option. This accepts a numeric value, and all pages with
a `page_rank` of 5 will be returned before pages with a `page_rank` of 1.
If you want to get fancy, you could even overwrite the default
`customRanking` used by the index by using the `custom_settings` option of
your config.
## How are my DocSearch records ranked?
DocSearch empowers the Algolia ranking strategy. The formula is completely
based on [the tie-breaking approach](https://www.algolia.com/doc/guides/ranking/ranking-formula/#tie-breaking-approach).
The special feature of DocSearch's ranking resides in [the custom ranking](https://www.algolia.com/doc/guides/ranking/custom-ranking/):
We have defined 3 main weight indicators for every record. These values are
ordered by importance following the tie-breaking approach:
1. `page_rank`: this value, equal to `0` by default, can be set from the
`start_urls` object. It can be customized in order to boost or restrain some
records depending on **their webpage's URL**. It will need to match a **specific
regular expression pattern**.
**Example:**
```json
{
"index_name": "example",
"start_urls": [
{
"url": "http://example.com/docs/api/v1\\.[0-9]",
"page_rank": 1
}
]
}
```
2. `level`: this value depends on the `level` of the record. A record's level is
its **deepest level attribute not null**. `text` records have a weight of 0.
3. `position`: This value is the position of the matching element within every
picked up element along the original HTML flow. The **sooner** the record appears,
the **higher** it will be ranked.
You can override the way these elements are impacting the search thanks to `custom_settings`.
**Example:**
```json
"custom_settings": {
"customRanking": [
"asc(weight.position)",
"desc(weight.page_rank)",
"desc(weight.level)"
]
}
```
[1]: https://www.algolia.com/doc/guides/ranking/ranking-formula/#tie-breaking-approach
[2]: https://www.algolia.com/doc/guides/ranking/ranking-formula/#tie-breaking-approach
[3]: https://www.algolia.com/doc/guides/ranking/custom-ranking/