nikon/docsearch
1
0
Fork 0
Code Activity Actions

Doc/update content (#379)

Browse source 
* # This is a combination of 2 commits.
# This is the 1st commit message:

# This is a combination of 3 commits.
# This is the 1st commit message:

# This is a combination of 3 commits.
# This is the 1st commit message:

chore(deps): update dependency onchange to v4.1.0

integrate previous work

enhance tyle/content

reformat part 1

wait for review

adance start_urls

enhance attributes description

fix typo

proofread documentation/docsearch

add apiKey mention

intefrate review and small fixes

finished proofreading

update sclient-rendering

use unseen review

# This is the commit message #2:

update README #269

# This is the commit message #3:

fix json

# This is the commit message #2:

enhance  as algolia/docsearch-configs#387

# This is the commit message #3:

updating flavicon

# This is the commit message #2:

Update 1-customize-configuration-file.html.md.erb

* documenting algolia/docsearch-scraper#387
This commit is contained in:
Sylvain Pace 2018-07-31 13:19:40 +02:00 committed by GitHub
parent b510acba1b
commit 52acd0a25a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 21 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/source/documentation/1-docsearch/3-recommendations.html.md.erb
View file

@ -91,4 +91,4 @@ For those reasons we highly recommend that you use a [**Sitemap**](https://www.s
This lists every page of your web site and will be used as the **main source of truth**
and it will define the roadmap of our scraping.
Beside this exhaustivity, using a sitemap introduces a significant performance improvement for our scraper.
Beside this exhaustivity, using a sitemap introduces a significant performance improvement for our scraper.

21
docs/source/documentation/2-docsearch-scraper/2-config-options.html.md.erb
View file

@ -64,11 +64,11 @@ Name of the Algolia index where all the data will be pushed.
**On our own infrastructure, this name must be equal to the configuration file name**
We mostly attribute it on our own regarding plenty of underlying factors. The `apiKey` that we provide is generated with a restriction on the `index_name`. Changing the `index_name` would require to ask for a new key. Thus if you want to **change the name**, please **submit a new configuration**, we will generate a new key accordingly.
### `start_urls` _Mandatory_
You can pass either a string or an array of urls. The crawler will go to each
page in order, following every link it finds on the page. It will only stop if
the domain is outside of the `allowed_domains` or if the link is blacklisted in
`stop_urls`.
Note that it currently does not follow *301* redirects.
@ -290,12 +290,31 @@ Specifies if the matched URLs should not respect the same rules as the crawled h
```
Given this configuration, every webpage of the sitemap whose URL contains '/doc/' will be scraped even if they don't comply with `start_urls` or `stop_urls`.
### `sitemap_alternate_links` _Optional_
This parameter is only useful when you are using a sitemap to crawl your website.
It specifies if alternate links should be followed. Your sitemap should inlcude localized versions of your page in such format:
```
<url>
<loc>http://example.com/</loc>
<xhtml:link rel="alternate" hreflang="de" href="http://example.com/de"/>
</url>
```
If `sitemap_alternate_links` is not set, the link "http://example.com/de" will not be parsed from the sitemap.
Default is `false`
### `allowed_domains` _Optional_
You can pass an array of strings. This is the whitelist of
domains the crawler will browse. If a link targets a page that is not in the
whitelist, the crawler will not follow it.
### Sitemap crawling _Optional_
Default is the domain of the first elements in the `start_urls`.
### `min_indexed_level` _Optional_