nikon/docsearch
1
0
Fork 0
Code Activity Actions

Update crawler-overview.md

Browse source
This commit is contained in:
mblandineau 2019-08-01 14:49:46 +02:00 committed by Sylvain Pace
parent 5a984d1200
commit ac0049b643
1 changed files with 2 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
docs/src/crawler-overview.md
View file

@ -6,7 +6,7 @@ title: Scraper Overview
## How?
The DocSearch scraper is written in python and heavily based on the [Scrapy][1]
framework. It will scrap all pages of your website and extract content from the
framework. It will go through all pages of your website and extract content from the
HTML structure to populate an Algolia index.
It will automatically follow every internal links to make sure we are not
@ -16,7 +16,7 @@ will be used for the hierarchy, and each `p` of text will be used as a potential
result.
Those CSS selectors can be overwritten, and each website actually has its own
JSON configuration file that describes in more detail how the scrap should
JSON configuration file that describes in more detail how the scraper should
behave. You can find the complete list of options in [the related section][2].
## When?