diff --git a/docs/src/inside-the-engine.md b/docs/src/inside-the-engine.md index 2cd8f455..0075b905 100644 --- a/docs/src/inside-the-engine.md +++ b/docs/src/inside-the-engine.md @@ -3,18 +3,18 @@ layout: two-columns title: Inside the engine --- -This page will explain in more details how the crawler extracts content from -your page every 24h, and how it ranks it in the results. +This page will explain in more detail how the crawler extracts content from +your page every 24h, and how it ranks the results. ## Crawling Each crawl will begin its journey by the value of the `start_urls` you have in your config. It will read those pages and recursively extract and follow every -links in those pages until it has browsed every compliant pages. +link in those pages until it has browsed every compliant page. If you have explicitly defined a `sitemap.xml`, our crawler will scrap every -provided and compliant pages. We do recommend to use [a sitemap][1] since it -clearly exposes URLs to crawl and avoid missing page that aren't linked from +provided and compliant page. We do recommend using [a sitemap][1] since it +clearly exposes URLs to crawl and avoid missing pages that aren't linked from another one. ## Extracting content @@ -23,8 +23,8 @@ Building records using the scraper is pretty intuitive. According to your settings, we extract the payload of your web page and index it, preserving your data's structure. This is achieved in a simple way: -- We **read top down** your web page following your HTML flow and pick your - matching elements out according their **levels** based on the +- We **read top down** your web page following your HTML flow and pick out your + matching elements according to their **levels** based on the `selectors_level` defined. - We create a record for each paragraph along with its hierarchical path. This construction is based on their **time of appearance** along the flow. @@ -43,13 +43,13 @@ approach][2]. DocSearch will first search for exact matches in your keywords then fallback to partial matches. Those results will then be ordered based, once again, on the page hierarchy, as extracted from the `selectors`. -The default strategy is to promote records having matching words in the highetst +The default strategy is to promote records having matching words in the highest level fist. Thus if two results have the same matching words, the one having them in the highest level (lvl0) will be reanked higher. We also use the position of the matching words. The sooner they appear within the HTML flow, the higher the record will be ranked. -The relevancy is based on several factors and can be customised according to the +The relevancy is based on several factors and can be customized according to the Algolia tie-breaking method. You can boost pages depending ont their URLs. This is done from the `start_urls`