@@ -21,7 +21,7 @@ applications, so please give us a couple of days to get back to you :)
### We create a configuration
-Once we receive your application, we'll have a look at your website and create a
+Once we receive [your application][2], we'll have a look at your website and create a
custom configuration file for it. This file defines which URLs we should crawl
or ignore, as well as the specific CSS selectors to be used for selecting
headers, subheaders, etc. All configs are publicly available in our [config
@@ -48,3 +48,4 @@ All we ask is that you keep the "search by Algolia" logo next to your search
results.
[1]: https://github.com/algolia/docsearch-configs/tree/master/configs
+[2]: apply.html
diff --git a/docs/src/inside-the-engine.md b/docs/src/inside-the-engine.md
index fab46a84..3c9787f5 100644
--- a/docs/src/inside-the-engine.md
+++ b/docs/src/inside-the-engine.md
@@ -4,58 +4,57 @@ title: Inside the engine
---
This page will explain in more details how the crawler extracts content from
-your page, and how it ranks it in the results.
+your page every 24h, and how it ranks it in the results.
## Crawling
Each crawl will begin its journey by the value of the `start_urls` you have in
-your config. It will read those pages and recursively follow every link in those
-pages until it has crawled all your website.
+your config. It will read those pages and recursively extract and follow every
+links in those pages until it has browsed every compliant pages.
-If it detects a `sitemap.xml`, it will use this link list instead of crawling
-all pages.
+If you have explictly defined a `sitemap.xml`, our crawler will scrap every
+provided and compliant pages. We do recommend to use [a sitemap][1] since it
+clearly exposes URLs to crawl and avoid missing page that aren't linked from
+another one.
## Extracting content
-Then, for each page, it will read the HTML markup from top to bottom. It will
-look for HTML elements matching your CSS `selectors`. It will look for elements
-matching your `text` selector (`` by default). Each of those matches will be -later transformed into an Algolia record. +Building records using the scraper is pretty intuitive. According to your settings, +we extract the payload of your webpage and index it, preserving your data's structure. +This is achieved in a simple way: +* We **read top down** your web page following your HTML flow and pick out your +matching elements according their **levels** based on the `selectors_level` defined. +* We create a record for each paragraph along with its hierarchical path. +This construction is based on their **time of appearance** along the flow. +* We **index** these records with the appropriate global settings (e.g. metadata, tags, etc.) -For each matching `text` element, the crawler will also keep in memory the -current hierarchy of headers (identified by the `lvl0` to `lvl5` selectors) that -it had to traverse to get to this text. This hierarchical information, as well -as some generic page metadata (such has the page url) are then pushed to -Algolia. - -Note that the crawler performs sanity checks before pushing data to Algolia. For -example if you changed the markup of your website, the selectors might not match -anything. If we detect that something is wrong with your current crawl, we don't -overwrite your previous index. +_**Note:** The above process performs sanity tests as it scrapes, in order to detect errors. +If indeed there are any serious warnings, it will abort and therefore not overwrite your current index. +These checks ensure that your dedicated index isn't flushed._ ## Ranking records Algolia always returns the most relevant results first, using a [tie-breaking -approach][1]. DocSearch will first search for exact matches in your keywords +approach][2]. DocSearch will first search for exact matches in your keywords then fallback to partial matches. Those results will then be ordered based, once again, on the page hierarchy, as extracted from the `selectors`. -The default strategy is to first look at the closest header of the matching -text. For example, a paragraph under `Settings / API / verySpecificMethod()`, -will be ranked higher than one under `Settings / API`. The idea here is that if -you have a match under a deep hierarchy, chances are that this match is specific -and might be more interesting that something found in a broad topic. +The default strategy is to promote records having matching words in the highetst level fist. +Thus if two results have the same matching words, the one having them in the highest level +(lvl0) will be reanked higher. We also use the position of the matching words. The sooner +they appear within the HTML flow, the higher the record will be ranked. -But this does not work in all cases as some documentations don't have deep -hierarchy. In that case, we use the paragraph position. The first paragraph of -the page will be ranked higher than the last one. +The relevancy is based on several factors and can be customised according to the +algolia tie-breaking method. -You also have a way to boost some pages directly in your config by using the -`page_rank` option. This accepts a numeric value, and all pages with a -`page_rank` of 5 will be returned before pages with a `page_rank` of 1. +You can boost pages depending ont their URLs. This is done from the `start_urls` and its +`page_rank` attributes. It is a numeric value, default to 0. The bigger it is, the higher results from the matching pages will be ranked. +For example all pages with a `page_rank` of 5 will be returned before pages with a `page_rank` of 1. -You could even overwrite the default `customRanking` used by the index by using +You could even change the relevancy strategy by [overwriting the default `customRanking`][3] used by the index by using the `custom_settings` option of your config. -[1]: +[1]:https://www.sitemaps.org/ +[2]: https://www.algolia.com/doc/guides/ranking/ranking-formula/#tie-breaking-approach +[3]:[https://www.algolia.com/doc/guides/ranking/custom-ranking/] diff --git a/docs/src/tips.md b/docs/src/tips.md index 50a9e682..aa1a8f89 100644 --- a/docs/src/tips.md +++ b/docs/src/tips.md @@ -9,8 +9,8 @@ share some tips on how you can make the most out of DocSearch. ### Use a `sitemap.xml` -If your website has a sitemap, DocSearch will use it to get the list of pages to -index. If it can't find one, it will follow every link of every page instead. +If you provide a sitemap from your configuration, DocSearch will use it to diretly browse the pages to +index. Pages are still crawled which means we extract every compliant link. We highly recommend you add a `sitemap.xml` to your website if you don't have one already. This will make the indexing faster, but will also give you more @@ -24,20 +24,23 @@ Sitemaps are also considered good practice for other aspects, including SEO DocSearch works better on structured documentation. Relevance of results is based on the structural hierarchy of content. In simpler terms it means that we read the `