diff --git a/docs/src/_layouts/two-columns.pug b/docs/src/_layouts/two-columns.pug index 67a3e18a..63bb7468 100644 --- a/docs/src/_layouts/two-columns.pug +++ b/docs/src/_layouts/two-columns.pug @@ -18,9 +18,9 @@ block nav block content .bgtb-moon-white-pure.relative .md_flex.max-w-lg.m-auto - .navigation-sidebar.absolute.hidden.md_static.bg-white.md_bg-transparent.w-100.md_w-25.md_block.md_border-0.text-left.md_pl-2 + .navigation-sidebar.sticky.pin-t.top-3.h-100vh-3.overflow-y-auto.hidden.md_block.bg-white // Adding subsections from markdown - .sticky.pin-t.h-100vh-3.overflow-y-auto.pt-2.w-5.m-auto.sm_flex.sm_w-90.md_block.md_w-100 + .pt-2.w-5.m-auto.sm_flex.sm_w-90.md_block.md_w-100.md_pl-2 each category in sidebar .flcnw.flspa.sm_flex-auto.sm_w-33.sm_p-1.md_w-100.md_p-0 .category.text-nebula.poppins.uppercase.mb-1.text-2=category.title @@ -37,7 +37,7 @@ block content li a.block.py-0x.text-telluric(href=`${page.url}#${heading.anchor}`)=heading.title - .documentation-content.text-1.pt-2.pb-1.px-2.w_100.md_w-75 + .documentation-content.text-2.pt-2.pb-1.px-2.w_100.md_w-75 .block.md_hidden.mb-1 a(href=site.url) DocSearch span  >  diff --git a/docs/src/_styles/docs.css b/docs/src/_styles/docs.css index a3b71400..919004ba 100644 --- a/docs/src/_styles/docs.css +++ b/docs/src/_styles/docs.css @@ -30,6 +30,27 @@ @apply .inline-block; } } + h3 { + @apply .text-nebula .text-3 .normal; + @apply .mb-1 .mt-2; + /* Trick to have heading scroll into view and not being hidden by header */ + &:before { + content: ""; + @apply hidden .h-3; + } + &:target:before { + @apply block; + } + .anchor { + @apply .hidden; + &:after { + content: "#"; + } + } + &:hover .anchor { + @apply .inline-block; + } + } } .navigation-sidebar{ diff --git a/docs/src/config-file.md b/docs/src/config-file.md index ca0bdc96..9313cb9d 100644 --- a/docs/src/config-file.md +++ b/docs/src/config-file.md @@ -1,77 +1,86 @@ --- layout: two-columns -title: Make the most of your configuration +title: Config Files --- -// TODO +For each DocSearch request we receive, we'll create a custom JSON config file +that will define how the crawler should behave. You can find all the configs in +[this repository][1]. -## Introduction - -The DocSearch scraper will use a configuration file specifying: - - the Algolia index name that will store the records resulting from the crawling - - the URLs it needs to crawl - - the URLs it shouldn't crawl - - the (hierarchical) CSS selectors to use to extract the relevant content from your webpages - - the CSS selectors to skip - - An optional sitemap URL that will be crawled and then scraped - - additional options you might provide to fine-tune the scraping - -## How it works - -Once you run the DocSearch scraper on a specific configuration, it will: - - crawl all the URLs you specified (from the `start_urls` or the `sitemap`) - - follow all the hyperlinks mentioned in the page, and continue the crawling there - - stop the crawling as soon as you've reached a URL that is not specified in your configuration or affiliated to a start url - - extract the content of every single crawled page following the logic you defined using the CSS selectors - - push the resulting records to the Algolia index you configured - - - -### Does the crawl encompass several domains/sub-domains? - -The `start_urls` define the allowed domain for our crawler. Basically, we take the main domain of every URLs. We will not go outside this whitelisted domain list. If you want to encompass a wider domain, please include it as a new start_url. - -## Configuration format - -A configuration file looks like: +A DocSearch looks like this: ```json { - "index_name": "stripe", - "start_urls": [ - "https://stripe.com/docs" - ], - "stop_urls": [ - "https://stripe.com/docs/api" - ], - "selectors": { - "lvl0": "#content header h1", - "lvl1": "#content article h1", - "lvl2": "#content section h3", - "lvl3": "#content section h4", - "lvl4": "#content section h5", - "lvl5": "#content section h6", - "text": "#content header p,#content section p,#content section ol" - }, - "selectors_exclude": [ - ".method-list", - "aside.note" - ], - // additional options - [...] + "index_name": "example", + "start_urls": [ + "https://www.example.com/docs" + ], + "selectors": { + "lvl0": "#content header h1", + "lvl1": "#content article h1", + "lvl2": "#content section h3", + "lvl3": "#content section h4", + "lvl4": "#content section h5", + "lvl5": "#content section h6", + "text": "#content header p,#content section p,#content section ol" + }, } ``` -It must be **a valid JSON file** +## ``index_name`` -## DocSearch options +This is the name of the Algolia index where your records will be pushed. The +`apiKey` we will share with you will be restricted to work on this index. -### `index_name` _Mandatory_ +When using the free DocSearch crawler, the `indexName` will always be the name +of the config. If you're running DocSearch yourself, you can of course use any +name you'd like. -Name of the Algolia index where all the data will be pushed. +```json +{ + "index_name": "example" +} +``` -**On our own infrastructure, this name must be equal to the configuration file name** +## `start_urls` +This array contains the list of urls that will be used to start crawling your +website. The crawler will recursively follow any links on those pages. It will +not follow links that are on another domain (unless they are allowed in +`allowed_domains`) and never follow links defined in `stop_urls`. + +```json +{ + "start_urls": [ + "https://www.example.com/docs" + ] +} +``` + +### Using regular expressions + +The `start_urls` option also allows for passing an object in place of a string, +to express more complex patterns. This object must contain a `url` key +containing the regular expression matching the url, as well as a `variables` key +that will be used to replace the named matches. + +The following example will make it clearer: + +```json +{ + "start_urls": [ + { + "url": "http://www.example.com/docs/(?P.*?)/(?P.*?)/", + "variables": { + "lang": ["en", "fr"], + "version": ["latest", "3.3", "3.2"] + } + } + ] +} +``` + +<<<<<<< HEAD We mostly attribute it on our own regarding plenty of underlying factors. The `apiKey` that we provide is generated with a restriction on the `index_name`. Changing the `index_name` would require to ask for a new key. Thus if you want to **change the name**, please **submit a new configuration**, we will generate a new key accordingly. ### `start_urls` _Mandatory_ You can pass either a string or an array of urls. The crawler will go to each @@ -79,240 +88,241 @@ page in order, following every link it finds on the page. It will only stop if the domain is outside of the `allowed_domains` or if the link is blacklisted from the `stop_urls`. Note that we currently do not follow *301* redirects. +======= +The beneficial side effect of using this syntax is that all records that will +then be extracted from crawling `http://www.example.com/docs/en/latest` will have +`lang: en` and `version: latest` added to it, allowing you to then filter based on +those values. -This parameter also behaves as a [regular expression](https://en.wikipedia.org/wiki/Regular_expression). If you don't use a sitemap, you must define at least one reachable URL (HTTP 20x). Otherwise the scraping will fail. +The following example shows how you can filter results matching specifics +language and version from the front-end -You can build a more advanced URL. You will need to use a JSON object with a `variables` attribute. This attribute is an array of variables that will be injected into the URLs: +```js +docsearch({ + […], + algoliaOptions: { + 'facetFilters': ["lang:en", "version:latest"] + }, +}); +``` +>>>>>>> Updating config doc + +### Using custom tags + +You can also apply custom tags to some pages without the need to use regular +expressions. In that case, add the list of tags to the `tags` key. Note +that those tags will be automatically added as facets in Algolia, allowing you +to filter based on their values as well. -**Example:** ```json { - "url": "http://example.com/docs/(?P.*?)/(?P.*?)/", - "variables": { - "version": [ - "latest", - "3.3", - [...] - ], - "lang": [ - "en", - "fr", - [...] - ] + "start_urls": [ + { + "url": "http://www.example.com/docs/concepts/", + "tags": ['concepts', 'terminology'] + } + ] } ``` -The whole pattern `(?P.*?)` will be replaced by the value assigned in the related key `version`. -The variable name is not fixed. Please note that those variables will behave as -[`attributesForFaceting`](https://www.algolia.com/doc/api-reference/api-parameters/attributesForFaceting/) which may help you [restrain the scope of the search from the snippet](https://www.algolia.com/doc/guides/searching/faceting/). - -Thus you can limit the scope of the search to the records from the pages encompassed by `http://example.com/docs/en/latest/*` thanks to the following snippet: ```js - ``` -In order to promote some pages, you can set the `page_rank` attribute (default: `0`, can be a positive or negative integer). +### Using Page Rank -Finally, If your website contains different parts and layouts, you can define specific `selectors` for each part and apply them using `selectors_key`: +If you want to give more weight to some pages to boost their ranking in the +results, you can attribute a custom `page_rank` to specific urls. Pages with +highest `page_rank` will be returned before pages with a lower `page_rank`. Note +that you can pass any numeric value, including negative values. -**Example:** ```json { - "index_name": "example", - "start_urls": [ - "http://example.com/docs/latest/", + "start_urls": [ { - "url": "http://example.com/docs/concepts/", - "page_rank": 1, - "selectors_key": "concepts" + "url": "http://www.example.com/docs/concepts/", + "page_rank": 5, + }, + { + "url": "http://www.example.com/docs/contributors/", + "page_rank": 1, } - ], + ] +} +``` + +In this example, results extracted from the _Concepts_ page will be ranked +higher than results extracted from the _Contributors_ page. + +### Using custom selectors per page + +If the markup of your website is so different from one page to another that you +can't have generic selectors, you can namespace your selectors and specify which +set of selectors should be applied to specific pages. + +```json +{ + "start_urls": [ + "http://www.example.com/docs/", + { + "url": "http://www.example.com/docs/concepts/", + "selectors_key": "concepts" + }, + { + "url": "http://www.example.com/docs/contributors/", + "selectors_key": "contributors" + } + ] "selectors": { "default": { - "lvl0": ".docSearch-content h1", - "lvl1": ".docSearch-content h2", - "lvl2": ".docSearch-content h3", - "lvl3": ".docSearch-content h4", - "lvl4": ".docSearch-content h5", - "text": ".docSearch-content p, .docSearch-content li" + "lvl0": ".main h1", + "lvl1": ".main h2", + "lvl2": ".main h3", + "lvl3": ".main h4", + "lvl4": ".main h5", + "text": ".main p" }, "concepts": { - "lvl0": ".docSearch-header h2", - "lvl1": ".docSearch-content h1", - "lvl2": ".docSearch-content h2", - "lvl3": ".docSearch-content h3", - "lvl4": ".docSearch-content h5", - "text": ".docSearch-content p" + "lvl0": ".header h2", + "lvl1": ".main h1.title", + "lvl2": ".main h2.title", + "lvl3": ".main h3.title", + "lvl4": ".main h5.title", + "text": ".main p" + }, + "contributors": { + "lvl0": ".main h1", + "lvl1": ".contributors .name", + "lvl2": ".contributors .title", + "text": ".contributors .description" } } - [...] } ``` +<<<<<<< HEAD ### `start_urls.tags` Tags will be apllied to every record from the matchin matched pages, i.e., its URL is matchin the `url`. These `tags` will [be processed as `attributesForFaceting`](https://www.algolia.com/doc/api-reference/api-parameters/attributesForFaceting/) . If you want further detail, [check the original documentation](https://www.algolia.com/doc/api-reference/api-parameters/facetFilters/). We do recommend to use `tags` if you want [to refine the scope of your search](https://www.algolia.com/doc/guides/searching/faceting/#faceting-overview). +======= +Here, all documentation pages will use the selectors defined in +`selectors.default` while the page under `./concepts` will use +`selectors.concepts` and those under `./contributors` will use +`selectors.contributors`. +>>>>>>> Updating config doc -From your search UI, you will need to use the following input: -```js -algoliaOptions: { 'facetFilters': ["tags:$VALUE"] }, -``` - -Default not used. Must be an array. - - -**Example:** - - From the config: -```json -"start_urls": [ - { - "url": "http://example.com/docs/concepts/", - "selectors_key": "concepts" - } - ``` - - From the search UI: - ```js - algoliaOptions: { 'facetFilters': ["tags:concepts"] }, - ``` -### `scrape_start_urls` - -This boolean let you decide if you want to extract the content of the starting pages. - -Default is `false` - -### `stop_urls` _Optional_ - -This array can be used to blacklist URLs. The crawler will stop on these and -will not consider their content. Likewise, if a link within a crawled webpage -targets such pages, the crawler will not follow the link. -You can use a regular expression as well as plain urls. - -Note: It is sometimes needed to add `http://www.example.com/index.html` pages to -the `stop_urls` list if you set `http://www.example.com` as a `start_urls`, to -avoid duplicated content. - -### `selectors` _Mandatory_ +## `selectors` This object contains all the CSS selectors that will be used to create the -record hierarchy. It can contains up to 6 levels (`lvl0`, `lvl1`, `lvl2`, `lvl3`, `lvl4`, -`lvl5`) and `text`. +record hierarchy. It can contains up to 6 levels (`lvl0`, `lvl1`, `lvl2`, +`lvl3`, `lvl4`, `lvl5`) and `text`. A default config would be to target the page `title` or `h1` as `lvl0`, the `h2` -as `lvl1` and `h3` as `lvl2`. `text` is usually any `p` of text. +as `lvl1` and `h3` as `lvl2` and `p` as `text`, but this is highly dependent on +the markup. -We recommend making use of at least the three first levels for better relevancy. +The `text` key is mandatory, but we highly recommend setting also `lvl0`, `lvl1` +and `lvl2` to have a decent level of relevance. -### `global` selectors _Optional_ - -It's possible to make a selector global which means that all records from the page will have -this value. This is useful when you have a title that is in the right sidebar and -the sidebar is placed after the content in the DOM. - -`global` attributes should be seen as a way to extract the matching elements from the HTML flow. These global elements will not be considered as breaking ones when we encounter them along the flow. If this token is enabled : -- we will not create a new record from the current builded stack. -- we will apply its contextual value (or `default_value` if not matched) to every records - -We mostly use this parameter for page that miss context. It enables us to pick up the context from another common part without having duplicates. - -```json -"selectors": { - "lvl0": { - "selector": "#content header h1", - "global": true - } -} -``` - -### Xpath selector _Optional_ - -By default, `selectors` are considered to be [css selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors) but you can specify that a selector is an [XPath one](https://developer.mozilla.org/en-US/docs/Web/XPath). -This is useful when you want to do more complex selection like selecting the parent of a target. - -```json -"selectors": { - "lvl0": { - "selector": "//li[@class=\"chapter active done\"]/../../a", - "type": "xpath" - } -} -``` - -### Default value _Optional_ - -You have the possibility to add a default value which will be used if the selector doesn't match anything. - -```json -"selectors": { - "lvl0": { - "selector": "#content article h1", - "default_value": "Documentation" - } -} -``` - -### `selectors_exclude` _Optional_ - -By default, the `selectors` search is applied page-wide. If there are some parts -of the page that you do not want to include (e.g. a table of content, a sidebar or a footer), -you can add them to the `selectors_exclude` key. - -### Sitemap crawling _Optional_ - -Our crawler offers you to crawl a site by discovering the URLs using Sitemaps. Thus, you can define the direct URL(s) to your sitemap XML file, `sitemap_urls`. In order to parse it, you should establish regex(s), `sitemap_urls_regex`, which will match the URLs to crawl. Otherwise it will use the `start_urls` pattern in order to match the expected URLs. - -For sites that use Sitemap index files that point to other sitemap files, all those sitemaps will be followed. - -### `sitemap_urls` _Optional_ -A list of urls pointing to the sitemaps (or sitemap index) you want to crawl. Must be provided if you want docsearch to discover your via sitemaps. - -### `sitemap_urls_regexs` _Optional_ -A list of regular expressions that will be applied to each URL from the sitemap. If one of the patterns match a URL, this link will be scraped. If no regular expression is defined, the start_urls will be taken as a pattern. - -### `force_sitemap_urls_crawling` _Optional_ -Specifies if the matched URLs should not respect the same rules as the crawled hyperlink. If set to true, each URL will be scraped even if it does comply with the `start_urls` or `stop_urls`. Default is `force_sitemap_urls_crawling` set to `false` - -**Example:** ```json { - [...] - "sitemap_urls": [ - "https://www.mySite.com/sitemap.xml" - ], - "sitemap_urls_regexs": [ - "/doc/" - ], - "force_sitemap_urls_crawling": true, - [...] + "selectors": { + "lvl0": "#content header h1", + "lvl1": "#content article h1", + "lvl2": "#content section h3", + "lvl3": "#content section h4", + "lvl4": "#content section h5", + "lvl5": "#content section h6", + "text": "#content header p,#content section p,#content section ol" + }, } ``` -Given this configuration, every webpage of the sitemap whose URL contains '/doc/' will be scraped even if they don't comply with `start_urls` or `stop_urls`. -### `sitemap_alternate_links` _Optional_ +Selectors can be passed as string, or as objects containing a `selector` key. +Other special keys can be set, as documented below. -This parameter is only useful when you are using a sitemap to crawl your website. - -It specifies if alternate links should be followed. Your sitemap should inlcude localized versions of your page in such format: - -``` - - http://example.com/ - - +```json +{ + "selectors": { + "lvl0": { + "selector": "#content header h1", + } + } +} ``` -If `sitemap_alternate_links` is not set, the link "http://example.com/de" will not be parsed from the sitemap. +### Using global selectors -Default is `false` +The default way of extracting content through selectors is to read the HTML +markup from top to bottom. This works well with semi-structured content, like +a hierarchy of headers. This breaks when relevant information is not part of the +same node flow. For example when the title is in a header or a sidebar. + +For that reason, you can set a selector as global, meaning that it will match on +the whole page, and will be the same for all records extracted on this page. + +```json +{ + "selectors": { + "lvl0": { + "selector": "#content header h1", + "global": true + } + } +} +``` + +### Setting a default value + +If your selector might not match a valid element on the page, you can define +a `default_value` to fallback to. + +```json +{ + "selectors": { + "lvl0": { + "selector": "#content header h1", + "default_value": "Documentation" + } + } +} +``` + +### Targeting elements using XPath instead of CSS + +CSS selectors are a clear and concise way to target elements of a page, but they +have a limitations. For example, you cannot go _up_ the cascade with CSS. + +If you need a more powerful selector mecanism, you can write your selectors +using XPath by setting `type: xpath`. You should also set `global: true` on the +same selector. + +The following example will look for a `li.chapter.active.done` and then go up +two levels in the DOM until it found a `a`. The content of this `a` will then be +used as the value of the `lvl0` selector. + +```json +{ + "selectors": { + "lvl0": { + "selector": "//li[@class=\"chapter active done\"]/../../a", + "type": "xpath", + "global": true + } + } +} +``` + +XPath selector can be hard to read. We highly encourage you to test them in your +browser first, making sure they match what you're expecting. + +## Other options ### `allowed_domains` _Optional_ @@ -320,21 +330,134 @@ You can pass an array of strings. This is the whitelist of domains the crawler will browse. If a link targets a page that is not in the whitelist, the crawler will not follow it. -### Sitemap crawling _Optional_ - -Default is the domain of the first elements in the `start_urls`. - ### `min_indexed_level` _Optional_ -Lets you define the minimum level at which you want a record to be indexed. For -example, with a `min_indexed_level: 1`, you will only index records that have at -least a `lvl0` and a `lvl1` field. +The default value is `0`. By increasing it, you can chose to not index some +records if they don't have enough `lvlX` matching. For example, with +a `min_indexed_level: 2`, records that have at least `lvl0`, `lvl1` and +`lvl2` matching something will be indexed. + +This is useful when your documentation has pages that share the same `lvl0` and +`lvl1` for example. In that case, you don't want to index all the shared +records, but want to keep the one matching content that is different across +pages. + +### `scrape_start_urls` _Optional_ + +By default, the crawler will not extract content from the pages defined in +`starts_urls`. If you have valuable content on your homepage, you should set +this to `true`. + +```json +{ + "scrape_start_urls": true +} +``` + +### `selectors_exclude` _Optional_ + +This expects an array of CSS selectors. Any element matching one of those +selectors will be removed from the page before any data is extracted from it. + +This can be used to remove a table of content, a sidebar or a footer, to +make other selectors easier to write. + +```json +{ + "selectors_exclude": [ + ".footer", + "ul.deprecated" + ], +} +``` + +### `stop_urls` _Optional_ + +This an array of strings or regular expressions. Whenever the crawler is about +to visit a link, it will first check if the link matches something in the array. +If it does, it will not follow the link. This should be used to restrict pages +the crawler should visit. + +Note that this is often used to avoid duplicate content, by adding +`http://www.example.com/docs/index.html` if you already have +`http://www.example.com/docs/` as a `start_urls`. + +```json +{ + "stop_urls": [ + "https://www.example.com/docs/index.html", + "license.html" + ], +} +``` + +## Using a sitemap + +### `sitemap_urls` _Optional_ + +You can pass an array of urls pointing to your sitemap(s) files. If this value +is set, DocSearch will try to read urls from your sitemap(s) instead of +following every link of your `starts_urls`. + +```json +{ + "sitemap_urls": [ + "http://www.example.com/docs/sitemap.xml" + ], +} +``` + +### `sitemap_alternate_links` _Optional_ + +Sitemaps can contain _alternative links_ for urls. Those are other versions of +the same page, in a different language, or with a different url. By default +DocSearch will ignore those urls. + +Set this to `true` if you want those other version to be crawled as well. + +```json +{ + "sitemap_urls": [ + "http://www.example.com/docs/sitemap.xml" + ], + "sitemap_alternate_links": true +} +``` + +With the above config and the `sitemap.xml` below, both +`http://www.example.com/docs/` and `http://www.example.com/docs/de/` will be +crawled. + +``` + + http://www.example.com/docs/ + + +``` + + + + + + + + + + + + + + + + + + + + + + -This is especially useful when the documentation is split into several pages and -all pages duplicate the main title or introduction (see [this issue][https://github.com/algolia/docsearch-configs/issues/83]). -With `min_indexed_level`, you can ignore the duplicated title. -Default is `0` ### `only_content_level` _Optional_ @@ -348,7 +471,7 @@ The HTML code that we crawl is sometimes generated using Javascript. In those cases, the `js_render` option must be set to `true`. It will enable our internal proxy (Selenium) to render pages before crawling them. -We highly recommend avoiding client-side rendering. It mainly decreases [the performance of your website](https://medium.com/walmartlabs/the-benefits-of-server-side-rendering-over-client-side-rendering-5d07ff2cefe8). +We highly recommend avoiding client-side rendering. It mainly decreases [the performance of your website][3]. Default is `false` @@ -391,8 +514,8 @@ Default is `0`. ### `custom_settings` _Optional_ -This object is [any custom Algolia settings](https://www.algolia.com/doc/api-client/settings/#the-scope-of-settings-and-parameters) you would like to pass to the index -settings. Please use the python syntax. You will [look under the hood of algolia](https://www.algolia.com/doc/). +This object is [any custom Algolia settings][4] you would like to pass to the index +settings. You will [look under the hood of algolia][5]. ### `nb_hits_max` _Optional_ @@ -451,3 +574,10 @@ Or : "http://website.com/page/" ] ``` + + +[1]: https://github.com/algolia/docsearch-configs/tree/master/configs +[2]: https://github.com/algolia/docsearch-configs/issues/83) +[3]: https://medium.com/walmartlabs/the-benefits-of-server-side-rendering-over-client-side-rendering-5d07ff2cefe8 +[4]: https://www.algolia.com/doc/api-client/settings/#the-scope-of-settings-and-parameters +[5]: https://www.algolia.com/doc/ diff --git a/docs/src/crawler-overview.md b/docs/src/crawler-overview.md index b06e708f..c73c41ee 100644 --- a/docs/src/crawler-overview.md +++ b/docs/src/crawler-overview.md @@ -3,10 +3,9 @@ layout: two-columns title: Crawler Overview --- -The DocSearch crawler is written in python and heavily based on the -[Scrapy][1] framework. It will crawl all pages of your -website, and extract content from the HTML structure to populate an Algolia -index. +The DocSearch crawler is written in python and heavily based on the [Scrapy][1] +framework. It will crawl all pages of your website and extract content from the +HTML structure to populate an Algolia index. It will automatically follow every internal link to make sure we are not missing any content, and will use the semantics of your HTML structure to construct its diff --git a/docs/src/inside-the-engine.md b/docs/src/inside-the-engine.md index ee803406..52a436dc 100644 --- a/docs/src/inside-the-engine.md +++ b/docs/src/inside-the-engine.md @@ -3,67 +3,63 @@ layout: two-columns title: Inside the engine --- -// TODO +This page will explain in more details how the crawler extracts content from your +page, and how it ranks it in the results. -## Building your index +## Crawling all your pages -Building records using the scraper is pretty intuitive. According to your settings, -we extract the payload of your webpage and index it, preserving your data's structure. -This is achieved in a simple way: -* We **read top down** your web page following your HTML flow and pick out your -matching elements according their **levels** (`selectors_level`) -* We create a record for each paragraph along with its hierarchical path. -This construction is based on their **time of appearance** along the flow. -* We **index** these records with the appropriate global settings (e.g. metadata, tags, etc.) +Each crawl will begin its journey by the value of the `start_urls` you have in +your config. It will read those pages and recursively follow every link in those +pages until it has crawled all your website. -_**Note:** The above process performs sanity tests as it scrapes, in order to detect errors. -If indeed there are any serious warnings, it will abort and therefore not overwrite your current index. -These checks ensure that your dedicated index isn't flushed._ +If it detects a `sitemap.xml`, it will use this link list instead of crawling +all pages. + +## Extracting content from each page + +Then, for each page, it will read the HTML markup from top to bottom. It will +look for HTML elements matching your CSS `selectors`. It will specifically look +for elements matching your `text` selector (`

` by default). Each of those +matches will be later transformed into an Algolia record. + +For each matching `text` element, the crawler will also keep in memory the +current hierarchy of headers (identified by the `lvl0` to `lvl5` selectors) that +it had to traverse to get to this text. This hierarchical information, as well +as some generic page metadata (such has the page url) are then pushed to +Algolia. + +Note that the crawler performs sanity checks before pushing data to Algolia. For +example if you changed the markup of your website, the selectors might not match +anything. If we detect that something is wrong with your current crawl, we don't +overwrite your previous index. + +## Ranking records + +Algolia always returns the most relevant results first, using a [tie-breaking +approach][1]. DocSearch will first search for exact matches in your keywords then +fallback to partial matches. Those results will then be ordered based, once +again, on the page hierarchy, as extracted from the `selectors`. + +The default strategy is to first look at the closest header of the matching +text. If a matching paragraph of text is under `Advanced Settings / API Options +/ verySpecificMethod()`, it will be ranked higher than if it is only found under +`Gettings Started > Installation`. The idea here is that if you have a match +under a very deep hierarchy, chances are that this match is very specific and +might be more interesting that something found in a very broad topic. + +But this does not work in all cases as some documentations don't have deep +hierarchy. In that case, we use the paragraph position. The first paragraph of +the page will be ranked higher than the last one. + +You also have a way to boost some pages directly in your config by using the +`page_rank` option. This accepts a numeric value, and all pages with +a `page_rank` of 5 will be returned before pages with a `page_rank` of 1. + +If you want to get fancy, you could even overwrite the default +`customRanking` used by the index by using the `custom_settings` option of +your config. - -## How are my DocSearch records ranked? - -DocSearch empowers the Algolia ranking strategy. The formula is completely -based on [the tie-breaking approach](https://www.algolia.com/doc/guides/ranking/ranking-formula/#tie-breaking-approach). - -The special feature of DocSearch's ranking resides in [the custom ranking](https://www.algolia.com/doc/guides/ranking/custom-ranking/): - -We have defined 3 main weight indicators for every record. These values are -ordered by importance following the tie-breaking approach: -1. `page_rank`: this value, equal to `0` by default, can be set from the -`start_urls` object. It can be customized in order to boost or restrain some -records depending on **their webpage's URL**. It will need to match a **specific -regular expression pattern**. - -**Example:** - ```json -{ - "index_name": "example", - "start_urls": [ - { - "url": "http://example.com/docs/api/v1\\.[0-9]", - "page_rank": 1 - } - ] -} -``` -2. `level`: this value depends on the `level` of the record. A record's level is -its **deepest level attribute not null**. `text` records have a weight of 0. -3. `position`: This value is the position of the matching element within every -picked up element along the original HTML flow. The **sooner** the record appears, -the **higher** it will be ranked. - -You can override the way these elements are impacting the search thanks to `custom_settings`. - - **Example:** - - ```json -"custom_settings": { - "customRanking": [ - "asc(weight.position)", - "desc(weight.page_rank)", - "desc(weight.level)" - ] -} - ``` +[1]: https://www.algolia.com/doc/guides/ranking/ranking-formula/#tie-breaking-approach +[2]: https://www.algolia.com/doc/guides/ranking/ranking-formula/#tie-breaking-approach +[3]: https://www.algolia.com/doc/guides/ranking/custom-ranking/