nikon/docsearch
1
0
Fork 0
Code Activity Actions

Update inside-the-engine.md

Browse source
This commit is contained in:
Peter Villani 2018-09-03 14:47:09 +02:00 committed by GitHub
parent f01fa04306
commit 6a50015816
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 9 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
docs/src/inside-the-engine.md
View file

@ -3,18 +3,18 @@ layout: two-columns
title: Inside the engine
---
This page will explain in more details how the crawler extracts content from
your page every 24h, and how it ranks it in the results.
This page will explain in more detail how the crawler extracts content from
your page every 24h, and how it ranks the results.
## Crawling
Each crawl will begin its journey by the value of the `start_urls` you have in
your config. It will read those pages and recursively extract and follow every
links in those pages until it has browsed every compliant pages.
link in those pages until it has browsed every compliant page.
If you have explicitly defined a `sitemap.xml`, our crawler will scrap every
provided and compliant pages. We do recommend to use [a sitemap][1] since it
clearly exposes URLs to crawl and avoid missing page that aren't linked from
provided and compliant page. We do recommend using [a sitemap][1] since it
clearly exposes URLs to crawl and avoid missing pages that aren't linked from
another one.
## Extracting content
@ -23,8 +23,8 @@ Building records using the scraper is pretty intuitive. According to your
settings, we extract the payload of your web page and index it, preserving your
data's structure. This is achieved in a simple way:
- We **read top down** your web page following your HTML flow and pick your
matching elements out according their **levels** based on the
- We **read top down** your web page following your HTML flow and pick out your
matching elements according to their **levels** based on the
`selectors_level` defined.
- We create a record for each paragraph along with its hierarchical path. This
construction is based on their **time of appearance** along the flow.
@ -43,13 +43,13 @@ approach][2]. DocSearch will first search for exact matches in your keywords
then fallback to partial matches. Those results will then be ordered based, once
again, on the page hierarchy, as extracted from the `selectors`.
The default strategy is to promote records having matching words in the highetst
The default strategy is to promote records having matching words in the highest
level fist. Thus if two results have the same matching words, the one having
them in the highest level (lvl0) will be reanked higher. We also use the
position of the matching words. The sooner they appear within the HTML flow, the
higher the record will be ranked.
The relevancy is based on several factors and can be customised according to the
The relevancy is based on several factors and can be customized according to the
Algolia tie-breaking method.
You can boost pages depending ont their URLs. This is done from the `start_urls`