integrate previous work (#269)
* integrate previous work enhance tyle/content reformat part 1 wait for review * adance start_urls * enhance attributes description * fix typo * proofread documentation/docsearch * add apiKey mention intefrate review and small fixes finished proofreading update sclient-rendering use unseen review
This commit is contained in:
Sylvain Pace
GitHub
parent
c8a319e4af
commit
c85ff1fc6f
9 changed files with 423 additions and 234 deletions
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
title: Overview
|
||||
---
|
||||
|
||||
We're scratching our own itch here. As developers, we spend a lot of time
|
||||
reading documentation, and it isn't always easy to find the information we need.
|
||||
|
||||
Not blaming anyone here. Building a good search for a documentation is a complex
|
||||
challenge. We happen to have a lot of experience doing that, and we want to
|
||||
share it with the world.
|
||||
|
||||
|
||||
DocSearch will provide, for free and out of the box, the Algolia search
|
||||
experience for your documentation website. You only need to [ask us to configure
|
||||
your site](https://community.algolia.com/docsearch/) or
|
||||
[submit you own configuration](https://github.com/algolia/docsearch-configs). We'll
|
||||
get back to you with what you need to integrate your new search into your website.
|
||||
|
||||
1. We'll **configure** your search experience,
|
||||
2. We'll **crawl** and **scrap** your documentation pages every 24h accordingly,
|
||||
3. You'll only need to **add few lines to your website**: a Javascript snippet along with 2 libraries (JS & CSS).
|
||||
|
||||
**NB:** If you prefer to DIY, you can [run the scraper](https://github.com/algolia/docsearch-scraper)
|
||||
in your own infra. Feel free to fork!
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
---
|
||||
title: How does it work?
|
||||
---
|
||||
|
||||
## How to start DocSearch?
|
||||
|
||||
It all starts with a simple config file. The config file contains your
|
||||
doc site's structure, for example your specific CSS selectors. It also
|
||||
contains a number of attributes and directives that tell
|
||||
[our DocSearch dedicated](https://github.com/algolia/docsearch-scraper)
|
||||
tool how to find the searchable information from your website.
|
||||
|
||||
## The Scraper, to extract content
|
||||
|
||||
At a **regular time interval** (24h), our scraper will index the content of your
|
||||
whole website. This **brand new** index will collect the public data of your
|
||||
documentation website and push it onto our dedicated server in order to enable
|
||||
your DocSearch instance. The process follows the directives contained in your
|
||||
**customized configuration** file to tailor the generic behavior of our scraper,
|
||||
with the aim of taking the most important information and organizing it for search optimization.
|
||||
|
||||
This scraping process runs on our infrastructure, not yours. You don't have to bother
|
||||
with it. However, our [scraper is also totally open source](https://github.com/algolia/docsearch-scraper)
|
||||
and so you are free to run it on your own servers.
|
||||
|
||||
## Browsing through your pages
|
||||
|
||||
In order to discover your pages, we highly recommend the use of a [sitemap](https://www.sitemaps.org/)
|
||||
which will be our source of truth and is a good practice for SEO purposes.
|
||||
|
||||
In any case, our tool is crawling your website. It follows every hyperlink
|
||||
embedded within the scraped pages as long as the URL belongs to the scope
|
||||
defined by the `start_urls` (and/or `sitemap_urls_regexs` applied to your sitemap).
|
||||
|
||||
## Building your index
|
||||
|
||||
Building records using the scraper is pretty intuitive. According to your settings,
|
||||
we extract the payload of your webpage and index it, preserving your data's structure.
|
||||
This is achieved in a simple way:
|
||||
* We **read top down** your web page following your HTML flow and pick out your
|
||||
matching elements according their **levels** (`selectors_level`)
|
||||
* We create a record for each paragraph along with its hierarchical path.
|
||||
This construction is based on their **time of appearance** along the flow.
|
||||
* We **index** these records with the appropriate global settings (e.g. metadata, tags, etc.)
|
||||
|
||||
_**Note:** The above process performs sanity tests as it scrapes, in order to detect errors.
|
||||
If indeed there are any serious warnings, it will abort and therefore not overwrite your current index.
|
||||
These checks ensure that your dedicated index isn't flushed._
|
||||
|
|
@ -0,0 +1,92 @@
|
|||
---
|
||||
title: Recommendations
|
||||
---
|
||||
|
||||
Here we look at several ingredients that help guarantee best results.
|
||||
You can also have a look at our [blog that discusses the pitfalls and recommendations](https://blog.algolia.com/how-to-build-a-helpful-search-for-technical-documentation-the-laravel-example/) that create the DocSearch adventure!
|
||||
|
||||
## Page Structure
|
||||
|
||||
For DocSearch to work, your **documentation** must be structured in a certain way. Structure reflects **clarity**, **exactness**, and **steadiness**. A helpful document will always have an understandable structure in one glance.
|
||||
It allows a user to get the ins and outs of the information and to quickly assimilate the **context**. It will help her/him to establish the **potential relevancy** of the document regarding her/his **intent**.
|
||||
|
||||
Last but not least the structure introduces a lot of meta information that are game changers at indexing time, such as **document hierarchy** or the use of [facets](https://www.algolia.com/doc/guides/searching/faceting/#faceting-overview).
|
||||
|
||||
## Document hierarchy
|
||||
|
||||
As previously mentioned, the **meta information** is almost as important as the payload itself.
|
||||
The hierarchical path of a document is one of the most important pieces of metadata information.
|
||||
It brings out the **context** of the document and thus empowers its content with an additional relationship.
|
||||
(Here we are referring to Custom Ranking).
|
||||
|
||||
Finding the **right depth** of your tree and how to split-up your content is one of the **most complex tasks**.
|
||||
For large documents, we usually recommend having 4 levels (from lvl0 until lvl3).
|
||||
Three different levels should be the minimum.
|
||||
|
||||
The **depth** of a record is also really enlightening.
|
||||
It will be considered within our [ranking formula](https://www.algolia.com/doc/guides/ranking/ranking-formula/) thanks to the [attributeToIndex](https://www.algolia.com/doc/api-reference/api-parameters/searchableAttributes/) settings.
|
||||
Definitely the higher up the content is, the more impact it will have.
|
||||
|
||||
In order to deliver the best user experience, it is key to open the page at the exact position of the match.
|
||||
The location of the match is easier to find in a **fragmented** document thanks to anchors.
|
||||
This hierarchy must be total and complete in order to be meaningful,
|
||||
this is why you should try to avoid heterogeneity.
|
||||
|
||||
## Consistency
|
||||
|
||||
Consistency is a pillar of a meaningful documentation. In addition to increasing
|
||||
the **intelligibility** of a document it also shortens the time required for a user
|
||||
to find the coveted information. The document's **topic** should be easily **identifiable**
|
||||
and its **outline** sharply demarcated.
|
||||
|
||||
The hierarchy should always have the same size. Try to **avoid orphan records**
|
||||
such like the implicit introduction/conclusion or asides. The selectors must be
|
||||
efficient for **every document** and highlight the proper hierarchy.
|
||||
They need to match only the coveted elements depending on their level.
|
||||
Be careful to avoid the **edge effect** by matching unexpected **superfluous elements**.
|
||||
|
||||
We encourage you to build selectors that are **flexible from a DOM structure** point of view
|
||||
but really **discriminating regarding the documentation's information**.
|
||||
They should be as much as possible GUI-agnostic and focus on the **characterization of the payload**.
|
||||
|
||||
Selectors should only match information from **real document webpage**
|
||||
and remain ineffective for others ones (e.g., landing page, table of content, etc.).
|
||||
We urge the maintainer to define a **dedicated class** for the **main DOM container**
|
||||
that includes the actual document content such as `.docSearch-content`
|
||||
|
||||
Since documentation should be **interactive**, it is a key point to
|
||||
**verbalize concepts with standardized words**.
|
||||
This **redundancy**, empowered with the **search experience** (dropdown),
|
||||
will even enable the user to **learn at searching time**.
|
||||
The **way to find the information** plays a key role in **leading** the user to the
|
||||
**retrieved knowledge itself**. You can also use the **synonym feature**.
|
||||
|
||||
## Unicity
|
||||
|
||||
The more time-consuming reading documentation is, the more painful and reluctant its use will be.
|
||||
You must avoid hazy points or catch-all. In addition to it being unhelpful,
|
||||
the catch-all document may be **confusing** and **counterproductive**.
|
||||
|
||||
Last but not least duplicates introduce noise and mislead users. This is why
|
||||
you should always focus on the relevant content and avoid duplicating content
|
||||
within your site (e.g. landing page which contains all of the information, summing up, etc.).
|
||||
In cases where the duplicates's existence is expected since it belongs to another
|
||||
dataset (e.g. a different version), you should use [facets](https://www.algolia.com/doc/guides/searching/faceting/).
|
||||
|
||||
## Conciseness
|
||||
|
||||
What is clearly thought out is clearly and concisely expressed.
|
||||
|
||||
## The Crawler, The Discovery Process, Exhaustivity
|
||||
|
||||
Since every documentation should be complete, the search experience must **cover the whole of it**.
|
||||
By default our scraper is crawling your website: it **follows hyperlinks referenced** from the scraped pages.
|
||||
If these links point to a page which belongs to the allowed domain,
|
||||
this precise page will be scraped, crawled and so forth.
|
||||
This **de facto discovery** is really practical but not exhaustive enough.
|
||||
If for any reason, a webpage could not be referenced from another covered one,
|
||||
the scope of the search will be imperfect.
|
||||
For those reasons we highly recommend that you use a **Sitemap** (#).
|
||||
This lists every page of your web site and will be used as the **main source of truth**
|
||||
and it will define the roadmap of our scraping.
|
||||
Beside this exhaustivity, using a sitemap introduces a significant performance improvement for our scraper.
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
---
|
||||
title: Docsearch overview
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
We're scratching our own itch here. As developers, we spend a lot of time
|
||||
reading documentation, and it isn't always easy to find the information we need.
|
||||
|
||||
Not blaming anyone here. Building a good search for a documentation is a complex
|
||||
challenge. We happen to have a lot of experience doing that, and we want to
|
||||
share it with the world. For free.
|
||||
|
||||
Just submit the form on the [website](https://community.algolia.com/docsearch/) and we'll get back to you with what
|
||||
you need to integrate your new search into your website.
|
||||
|
||||
1. We'll crawl your documentation pages,
|
||||
2. We'll configure your search experience,
|
||||
3. You'll need to add a bit of JavaScript and CSS code to your website.
|
||||
|
||||
If you prefer to DIY, you can run the [scraper](https://github.com/algolia/docsearch-scraper) in your own infra.
|
||||
|
|
@ -1,29 +1,32 @@
|
|||
---
|
||||
title: Install
|
||||
title: DocSearch on your own infrastructure
|
||||
---
|
||||
|
||||
This is the repository for the scraper for the [DocSearch project](https://community.algolia.com/docsearch/).
|
||||
You can run it on your own, or [ask us](https://community.algolia.com/docsearch/) to crawl your documentation.
|
||||
In the regular use of DocSearch, you don't have to bother, we **handle the whole stack**.
|
||||
|
||||
DocSearch is composed by 3 different projects:
|
||||
* The front-end of DocSearch: https://github.com/algolia/docsearch
|
||||
* The scraper which browses & indexes web pages: https://github.com/algolia/docsearch-scraper
|
||||
* The configurations for the scraper: https://github.com/algolia/docsearch-configs
|
||||
In some specific use cases, you may want to look under the hood and DIY.
|
||||
This is welcome with DocSearch since every tool is open source.
|
||||
DocSearch is composed of 3 different projects:
|
||||
* [The front-end of DocSearch](https://github.com/algolia/docsearch)
|
||||
* [The scraper](https://github.com/algolia/docsearch-scraper), which browses & indexes web pages
|
||||
* [The configuration repo](https://github.com/algolia/docsearch-configs) for the scraper.
|
||||
|
||||
## The DocSearch backend: [our scraper](https://github.com/algolia/docsearch-scraper):
|
||||
|
||||
This project is a collection of submodules, each one in its own directory:
|
||||
* cli: A command line tool to manage DocSearch. Run `./docsearch` and follow the steps
|
||||
* deployer: Tool used by Algolia to deploy the configuration in our mesos infrastructure
|
||||
* deployer: Tool used by Algolia to deploy the configuration in our Apache Mesos infrastructure
|
||||
* doctor: A monitoring/repair tool to check if the indices built by the scraper are in good shape
|
||||
* playground: An HTML page to easily test DocSearch indices
|
||||
* scraper: The core of the scraper. It reads the configuration file, fetches the web pages and indexes them in Algolia.
|
||||
|
||||
## Install Docsearch
|
||||
## Install
|
||||
|
||||
The DocSearch scraper is based on [Scrapy](https://scrapy.org), a famous python-based web scraper. Because it might need some JavaScript to render the pages it crawls, the scraper is also depending on [selenium](http://www.seleniumhq.org).
|
||||
|
||||
To ease the setup process, a Docker container is provided to help you run the scraper.
|
||||
|
||||
### Environment
|
||||
### Environment:
|
||||
|
||||
- Install `python` & `pip`
|
||||
- `brew install python # will install pip`
|
||||
|
|
@ -33,7 +36,7 @@ To ease the setup process, a Docker container is provided to help you run the sc
|
|||
- `cd docsearch-scraper`
|
||||
- `pip install --user -r requirements.txt`
|
||||
|
||||
### With docker
|
||||
### With docker:
|
||||
|
||||
- Build the underlying Docker image: `./docsearch docker:build`
|
||||
|
||||
|
|
@ -61,7 +64,7 @@ Options:
|
|||
--help Display help message
|
||||
|
||||
Available commands:
|
||||
bootstrap Boostrap a docsearch config
|
||||
bootstrap Bootstrap a docsearch config
|
||||
run Run a config
|
||||
playground Launch the playground
|
||||
docker
|
||||
|
|
@ -74,17 +77,17 @@ Available commands:
|
|||
|
||||
### Create a config
|
||||
|
||||
To use DocSearch, the first thing you need is to create a crawler config. For more details about configs, check out [https://github.com/algolia/docsearch-configs](https://github.com/algolia/docsearch-configs), you'll have a list of options you can use and a lot of live and working examples.
|
||||
To use DocSearch, the first thing you need is to create a crawler config. For more details about configs, check out [our configuration repo](https://github.com/algolia/docsearch-configs), you'll have a list of options you can use and a lot of live and working examples.
|
||||
|
||||
### Crawl the website
|
||||
|
||||
**With docker:**
|
||||
#### With docker:
|
||||
|
||||
```sh
|
||||
$ ./docsearch docker:run /path/to/your/config
|
||||
```
|
||||
|
||||
**Without docker:**
|
||||
#### Without docker
|
||||
|
||||
```sh
|
||||
$ ./docsearch run /path/to/your/config
|
||||
|
|
@ -119,7 +122,7 @@ To add the DocSearch dropdown menu to your website, add the following snippet to
|
|||
|
||||
And you are good to go!
|
||||
|
||||
### Specify appId
|
||||
### Specify your own appId
|
||||
|
||||
If you are running the scraper on your own, you will need to tell the widget about your Algolia application ID via the `appId` parameter.
|
||||
|
||||
|
|
@ -134,7 +137,7 @@ If Algolia is handling the crawling of your site, you do not need to specify `ap
|
|||
|
||||
## Admin task
|
||||
|
||||
If you are Algolia employee and want to manage a DocSearch account,
|
||||
If you are an Algolia employee and want to manage a DocSearch account,
|
||||
you'll need to add the following variables in your `.env` file:
|
||||
|
||||
```
|
||||
|
|
@ -148,24 +151,24 @@ DEPLOY_KEY=
|
|||
|
||||
The cli will then have more commands for you to run.
|
||||
|
||||
For some actions like deploying you might need to use different credentials than the ones in the .env file.
|
||||
For some actions like deploying you might need to use different credentials than the ones in the _.env_ file.
|
||||
To do this you need to override them when running the cli tool:
|
||||
|
||||
```
|
||||
APPLICATION_ID= API_KEY= ./docsearch deploy:configs
|
||||
```sh
|
||||
APPLICATION_ID=<your APPLICATION_ID> API_KEY=<your API_KEY> ./docsearch deploy:configs
|
||||
```
|
||||
|
||||
## Run the tests
|
||||
|
||||
### With docker
|
||||
### With docker:
|
||||
|
||||
```sh
|
||||
$ ./docsearch test
|
||||
```
|
||||
|
||||
### Without docker
|
||||
### Without docker:
|
||||
|
||||
```sh
|
||||
$ pip install pytest
|
||||
$ API_KEY='test' APPLICATION_ID='test' python -m pytest
|
||||
```
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
title: Config options
|
||||
title: Make the most of your configuration
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
|
@ -16,7 +16,7 @@ The DocSearch scraper will use a configuration file specifying:
|
|||
## How it works
|
||||
|
||||
Once you run the DocSearch scraper on a specific configuration, it will:
|
||||
- crawl all the URLs you specified (from the *start_urls* or the *sitemap*)
|
||||
- crawl all the URLs you specified (from the `start_urls` or the `sitemap`)
|
||||
- follow all the hyperlinks mentioned in the page, and continue the crawling there
|
||||
- stop the crawling as soon as you've reached a URL that is not specified in your configuration or affiliated to a start url
|
||||
- extract the content of every single crawled page following the logic you defined using the CSS selectors
|
||||
|
|
@ -53,67 +53,139 @@ A configuration file looks like:
|
|||
}
|
||||
```
|
||||
|
||||
### `index_name` ***Mandatory***
|
||||
It must be **a valid JSON file**
|
||||
|
||||
Name of the Algolia index where all the data will be pushed. If the `PREFIX` environment variable is defined, it will be prefixed
|
||||
with it.
|
||||
## DocSearch options
|
||||
|
||||
*This name must be equal to the configuration file name*
|
||||
### `index_name` _Mandatory_
|
||||
|
||||
### `start_urls` ***Mandatory***
|
||||
Name of the Algolia index where all the data will be pushed.
|
||||
|
||||
**On our own infrastructure, this name must be equal to the configuration file name**
|
||||
|
||||
We mostly attribute it on our own regarding plenty of underlying factors. The `apiKey` that we provide is generated with a restriction on the `index_name`. Changing the `index_name` would require to ask for a new key. Thus if you want to **change the name**, please **submit a new configration**, we will generate a new key accordingly.
|
||||
|
||||
### `start_urls` _Mandatory_
|
||||
You can pass either a string or an array of urls. The crawler will go to each
|
||||
page in order, following every link it finds on the page. It will only stop if
|
||||
the domain is outside of the `allowed_domains` or if the link is blacklisted in
|
||||
`stop_urls`.
|
||||
Strings will be considered as regex.
|
||||
|
||||
Note that it currently does not follow 301 redirects.
|
||||
|
||||
### `selectors` ***Mandatory***
|
||||
This parameter also behaves as a [regular expression](https://en.wikipedia.org/wiki/Regular_expression). If you don't use a sitemap, you must define at least one reachable URL (HTTP 20x). Otherwise the scraping will fail.
|
||||
|
||||
You can build a more advanced URL. You will need to use a JSON object with a `variables` attribute. This attribute is an array of variables that will be injected into the URLs:
|
||||
|
||||
Example:
|
||||
```json
|
||||
{
|
||||
"url": "http://example.com/docs/(?P<lang>.*?)/(?P<version>.*?)/",
|
||||
"variables": {
|
||||
"version": [
|
||||
"latest",
|
||||
"3.3",
|
||||
[...]
|
||||
],
|
||||
"lang": [
|
||||
"en",
|
||||
"fr",
|
||||
[...]
|
||||
]
|
||||
}
|
||||
```
|
||||
The whole pattern `(?P<version>.*?)` will be replaced by the value assigned in the related key `version`.
|
||||
|
||||
The variable name is not fixed. Please note that those variables will behave as
|
||||
[`attributesForFaceting`](https://www.algolia.com/doc/api-reference/api-parameters/attributesForFaceting/) which may help you [restrain the scope of the search from the snippet](https://www.algolia.com/doc/guides/searching/faceting/).
|
||||
|
||||
Thus you can limit the scope of the search to the records from the pages encompassed by `http://example.com/docs/en/latest/*` thanks to the following snippet:
|
||||
```js
|
||||
<script type="text/javascript">
|
||||
docsearch({
|
||||
apiKey: ${apiKey},
|
||||
indexName: ${indexName},
|
||||
inputSelector: '#search'
|
||||
algoliaOptions: { 'facetFilters': ["lang:en", "version:latest"] },
|
||||
});
|
||||
</script>
|
||||
```
|
||||
|
||||
You can also define some `tags` that will also behave as [`attributesForFaceting`](https://www.algolia.com/doc/api-reference/api-parameters/attributesForFaceting/).
|
||||
|
||||
In order to promote some pages, you can set the `page_rank` attribute (default: `0`, can be a positive or negative integer).
|
||||
|
||||
Finally, If your website contains differents parts and layouts, you can define specific `selectors` for each part and apply them using `selectors_key`:
|
||||
|
||||
Example:
|
||||
```json
|
||||
{
|
||||
"index_name": "example",
|
||||
"start_urls": [
|
||||
"http://example.com/docs/latest/",
|
||||
{
|
||||
"url": "http://example.com/docs/concepts/",
|
||||
"tags": [
|
||||
"concepts"
|
||||
],
|
||||
"page_rank": 1,
|
||||
"selectors_key": "concepts"
|
||||
},
|
||||
"selectors": {
|
||||
"default": {
|
||||
"lvl0": ".docSearch-content h1",
|
||||
"lvl1": ".docSearch-content h2",
|
||||
"lvl2": ".docSearch-content h3",
|
||||
"lvl3": ".docSearch-content h4",
|
||||
"lvl4": ".docSearch-content h5",
|
||||
"text": ".docSearch-content p, .docSearch-content li"
|
||||
},
|
||||
"concepts" {
|
||||
"lvl0": ".docSearch-header h2",
|
||||
"lvl1": ".docSearch-content h1",
|
||||
"lvl2": ".docSearch-content h2",
|
||||
"lvl3": ".docSearch-content h3",
|
||||
"lvl4": ".docSearch-content h5",
|
||||
"text": ".docSearch-content p"
|
||||
}
|
||||
}
|
||||
[...]
|
||||
}
|
||||
```
|
||||
|
||||
### `scrape_start_urls`
|
||||
|
||||
This boolean let you decide if you want to extract the content of the starting pages.
|
||||
|
||||
Default is `false`
|
||||
|
||||
### `stop_urls` _Optional_
|
||||
|
||||
This array can be used to blacklist URLs. The crawler will stop on these and
|
||||
will not consider their content. Likewise, if a link within a crawled webpage
|
||||
targets such pages, the crawler will not follow the link.
|
||||
You can use a regular expression as well as plain urls.
|
||||
|
||||
Note: It is sometimes needed to add `http://www.example.com/index.html` pages to
|
||||
the `stop_urls` list if you set `http://www.example.com` as a `start_urls`, to
|
||||
avoid duplicated content.
|
||||
|
||||
### `selectors` _Mandatory_
|
||||
|
||||
This object contains all the CSS selectors that will be used to create the
|
||||
record hierarchy. It contains 6 levels (`lvl0`, `lvl1`, `lvl2`, `lvl3`, `lvl4`,
|
||||
`lvl5`) and `text`. You should fill at least the three first levels for better
|
||||
relevance.
|
||||
record hierarchy. It can contains up to 6 levels (`lvl0`, `lvl1`, `lvl2`, `lvl3`, `lvl4`,
|
||||
`lvl5`) and `text`.
|
||||
|
||||
A default config would be to target the page `title` or `h1` as `lvl0`, the `h2`
|
||||
as `lvl1` and `h3` as `lvl2`. `text` is usually any `p` of text.
|
||||
|
||||
## Sitemap crawling ***Optional***
|
||||
We recommend making use of at least the three first levels for better relevancy.
|
||||
|
||||
Our crawler offers you to crawl a site by discovering the URLs using Sitemaps. Thus, you will need to define the direct url(s) to your sitemap XML file, `sitemap_urls` , and then establish regex(s), `sitemap_urls_regexs` , which will match the URLs to crawl. Otherwise it will use the `start_urls` pattern in order to match the URLs available within the site map.
|
||||
### Global selectors _Optional_
|
||||
|
||||
For sites that use Sitemap index files that point to other sitemap files, all those sitemaps will be followed.
|
||||
|
||||
### `sitemap_urls`
|
||||
A list of urls pointing to the sitemaps (or sitemap index) you want to crawl. Must be provided if you want to discover though sitemap.
|
||||
|
||||
### `sitemap_urls_regexs`
|
||||
A list of regular expression that will be applied to each URL from the sitemap. If the pattern match an URL, this link will be scrapped. If none regular expressions is defined, the start_urls will be taken as pattern.
|
||||
|
||||
### `force_sitemap_urls_crawling`
|
||||
Specifies if matched URL should not respect the same rules as the hyperlink crawled. If set to true, each URL will be scraped no matter if it suited the `start_urls` or `stop_urls`. Default is `force_sitemap_urls_crawling` disabled
|
||||
|
||||
#### Example
|
||||
```json
|
||||
[...]
|
||||
"sitemap_urls": [
|
||||
"https://www.mySite.com/sitemap.xml"
|
||||
],
|
||||
"": [
|
||||
"/doc/"
|
||||
],
|
||||
"force_sitemap_urls_crawling": true,
|
||||
[...]
|
||||
```
|
||||
Given this configuration, each webpage whose the URL contains '/doc/' will be scrapped even if they don't complied the `start_urls` or `stop_urls`
|
||||
|
||||
|
||||
### Global selectors
|
||||
|
||||
It's possible to make a selector global which means that all records for the page will have
|
||||
It's possible to make a selector global which means that all records from the page will have
|
||||
this value. This is useful when you have a title that is in the right sidebar and
|
||||
the sidebar is placed after the content in the dom.
|
||||
the sidebar is placed after the content in the DOM.
|
||||
|
||||
```json
|
||||
"selectors": {
|
||||
|
|
@ -124,10 +196,10 @@ the sidebar is placed after the content in the dom.
|
|||
}
|
||||
```
|
||||
|
||||
### Xpath selector
|
||||
### Xpath selector _Optional_
|
||||
|
||||
By default `selectors` is considered to be a [css selector](https://www.w3schools.com/cssref/css_selectors.asp) but you can specify that a selector is a [xpath query](https://www.w3schools.com/xml/xpath_intro.asp).
|
||||
This is useful when you want to do more complex selection like selecting the parent of a node.
|
||||
By default, `selectors` are considered to be [css selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors) but you can specify that a selector is an [XPath one](https://developer.mozilla.org/en-US/docs/Web/XPath).
|
||||
This is useful when you want to do more complex selection like selecting the parent of a target.
|
||||
|
||||
```json
|
||||
"selectors": {
|
||||
|
|
@ -138,10 +210,9 @@ This is useful when you want to do more complex selection like selecting the par
|
|||
}
|
||||
```
|
||||
|
||||
### Default value
|
||||
### Default value _Optional_
|
||||
|
||||
You have the possibility to add a default value. If the given selector doesn't match anything in a page
|
||||
then for each record the default value will be set
|
||||
You have the possibility to add a default value which will be used if the selector doesn't match anything.
|
||||
|
||||
```json
|
||||
"selectors": {
|
||||
|
|
@ -152,9 +223,99 @@ then for each record the default value will be set
|
|||
}
|
||||
```
|
||||
|
||||
### Strip Chars
|
||||
### `selectors_exclude` _Optional_
|
||||
|
||||
You can override the default `strip_chars` per level
|
||||
By default, the `selectors` search is applied page-wide. If there are some parts
|
||||
of the page that you do not want to include (e.g. a table of content, a sidebar or a footer),
|
||||
you can add them to the `selectors_exclude` key.
|
||||
|
||||
### Sitemap crawling _Optional_
|
||||
|
||||
Our crawler offers you to crawl a site by discovering the URLs using Sitemaps. Thus, you can define the direct URL(s) to your sitemap XML file, `sitemap_urls`. In order to parse it, you should establish regex(s), `sitemap_urls_regex`, which will match the URLs to crawl. Otherwise it will use the `start_urls` pattern in order to match the expected URLs.
|
||||
|
||||
For sites that use Sitemap index files that point to other sitemap files, all those sitemaps will be followed.
|
||||
|
||||
### `sitemap_urls` _Optional_
|
||||
A list of urls pointing to the sitemaps (or sitemap index) you want to crawl. Must be provided if you want docsearch to discover your via sitemaps.
|
||||
|
||||
### `sitemap_urls_regexs` _Optional_
|
||||
A list of regular expressions that will be applied to each URL from the sitemap. If one of the patterns match a URL, this link will be scraped. If no regular expression is defined, the start_urls will be taken as a pattern.
|
||||
|
||||
### `force_sitemap_urls_crawling` _Optional_
|
||||
Specifies if the matched URLs should not respect the same rules as the crawled hyperlink. If set to true, each URL will be scraped even if it does comply with the `start_urls` or `stop_urls`. Default is `force_sitemap_urls_crawling` set to `false`
|
||||
|
||||
### Example
|
||||
```json
|
||||
{
|
||||
[...]
|
||||
"sitemap_urls": [
|
||||
"https://www.mySite.com/sitemap.xml"
|
||||
],
|
||||
"sitemap_urls_regexs": [
|
||||
"/doc/"
|
||||
],
|
||||
"force_sitemap_urls_crawling": true,
|
||||
[...]
|
||||
}
|
||||
```
|
||||
Given this configuration, every webpage of the sitemap whose URL contains '/doc/' will be scraped even if they don't comply with `start_urls` or `stop_urls`.
|
||||
|
||||
### `allowed_domains` _Optional_
|
||||
|
||||
You can pass an array of strings. This is the whitelist of
|
||||
domains the crawler will browse. If a link targets a page that is not in the
|
||||
whitelist, the crawler will not follow it.
|
||||
|
||||
Default is the domain of the first elements in the `start_urls`.
|
||||
|
||||
### `min_indexed_level` _Optional_
|
||||
|
||||
Lets you define the minimum level at which you want a record to be indexed. For
|
||||
example, with a `min_indexed_level: 1`, you will only index records that have at
|
||||
least a `lvl0` and a `lvl1` field.
|
||||
|
||||
This is especially useful when the documentation is split into several pages and
|
||||
all pages duplicate the main title or introduction (see [this issue][1]).
|
||||
With `min_indexed_level`, you can ignore the duplicated title.
|
||||
|
||||
Default is `0`
|
||||
|
||||
### `only_content_level` _Optional_
|
||||
|
||||
This attribute allows you to only index records which match the `text` selectors. Every other record will be skipped.
|
||||
|
||||
Default is `false`
|
||||
|
||||
### `js_render` _Optional_
|
||||
|
||||
The HTML code that we crawl is sometimes generated using Javascript. In those
|
||||
cases, the `js_render` option must be set to `true`. It will enable our
|
||||
internal proxy (Selenium) to render pages before crawling them.
|
||||
|
||||
We highly recommend avoiding client-side rendering. It mainly decreases [the performance of your website](https://medium.com/walmartlabs/the-benefits-of-server-side-rendering-over-client-side-rendering-5d07ff2cefe8).
|
||||
|
||||
Default is `false`
|
||||
|
||||
### `js_wait` _Optional_
|
||||
|
||||
When `js_render` is set to `true`, the `js_wait` parameter lets you change the default waiting time (in seconds) to render the
|
||||
webpage with the Selenium emulator.
|
||||
|
||||
Default is `0`s
|
||||
|
||||
### `use_anchors` _Optional_
|
||||
|
||||
The `use_anchors` needs to be set to True for a javascript doc when the hash is
|
||||
used to route the query. Internally, this will disable the canonicalize feature that
|
||||
is removing the hash from the url.
|
||||
|
||||
This parameter is optional and is set to `false` by default.
|
||||
|
||||
### `strip_chars` _Optional_
|
||||
|
||||
A list of characters to remove from the indexed text.
|
||||
|
||||
You can also override the default `strip_chars` per level
|
||||
|
||||
```json
|
||||
"selectors": {
|
||||
|
|
@ -165,100 +326,27 @@ You can override the default `strip_chars` per level
|
|||
}
|
||||
```
|
||||
|
||||
### `allowed_domains`
|
||||
|
||||
You can pass an array of strings. This is the whitelist of
|
||||
domains the crawler will browse. If a link targets a page that is not in the
|
||||
whitelist, the crawler will not follow it.
|
||||
|
||||
Default is the domain of the first element in the `start_urls`
|
||||
|
||||
### `stop_urls`
|
||||
|
||||
This is the blacklist of urls on which the crawler should stop. If a link in
|
||||
a crawled webpage targets one the elements in the `stop_urls` list, the crawler
|
||||
will not follow the link.
|
||||
|
||||
Note that you can use regexps as well as plain urls.
|
||||
|
||||
Note: It is sometimes needed to add `http://www.example.com/index.html` pages to
|
||||
the `stop_urls` list if you set `http://www.example.com` as a `start_urls`, to
|
||||
avoid duplicated content.
|
||||
|
||||
### `selectors_exclude`
|
||||
|
||||
By default, the `selectors` search is applied page-wide. If there are some parts
|
||||
of the page that you do not want to include (like a header, sidebar or footer),
|
||||
you can add them to the `selectors_exclude` key.
|
||||
|
||||
### `custom_settings`
|
||||
|
||||
This object is any custom Algolia settings you would like to pass to the index
|
||||
settings.
|
||||
|
||||
### `min_indexed_level`
|
||||
|
||||
Lets you define the minimum level at which you want records to be indexed. For
|
||||
example, with a `min_indexed_level: 1`, you will only index records that have at
|
||||
least a `lvl1` field.
|
||||
|
||||
This is especially useful when the documentation is split into several pages,
|
||||
but all pages duplicates the main title (see [this issue][1]).
|
||||
|
||||
### `js_render`
|
||||
|
||||
The HTML code that we crawl is sometimes generated using Javascript. In those
|
||||
cases, the `js_render` option must be set to `true`. It will enable our
|
||||
internal proxy (Selenium) to render pages before crawling them.
|
||||
|
||||
This parameter is optional and is set to `false` by default.
|
||||
|
||||
### `js_wait`
|
||||
|
||||
The `js_wait` parameter lets you change the default waiting time to render the
|
||||
webpage with the Selenium proxy.
|
||||
|
||||
This parameter is optional and is set to `0`s by default.
|
||||
|
||||
### `use_anchors`
|
||||
|
||||
The `use_anchors` is need to be set to True for javascript doc when the hash is
|
||||
used to route the query. Internally this will disable the canonicalize feature that
|
||||
is removing the hash from the url.
|
||||
|
||||
This parameter is optional and is set to False by default.
|
||||
|
||||
### `strip_chars`
|
||||
|
||||
A list of character to remove from the text that is indexed.
|
||||
|
||||
Default is `" .,;:§¶"`
|
||||
|
||||
### `scrape_start_urls`
|
||||
### `nb_hits` _Mandatory_
|
||||
|
||||
Default is `false`
|
||||
|
||||
### `remove_get_params`
|
||||
|
||||
Default is `false`
|
||||
|
||||
### `strict_redirect`
|
||||
|
||||
Default is `false`
|
||||
|
||||
### `nb_hits`
|
||||
|
||||
The number of object that should be indexed. Only used by the [`checker`](#checker).
|
||||
Each time the configuration is locally run, this attribute is set to the number of records indexed.
|
||||
|
||||
This attribute is used for purposed monitoring. We keep a track of its evolution in order to detect main changes.
|
||||
Default is `0`.
|
||||
|
||||
### `custom_settings` _Optional_
|
||||
|
||||
This object is [any custom Algolia settings](https://www.algolia.com/doc/api-client/settings/#the-scope-of-settings-and-parameters) you would like to pass to the index
|
||||
settings. You will [look under the hood of algolia](https://www.algolia.com/doc/).
|
||||
|
||||
## Possible issues
|
||||
|
||||
#### Duplicated content
|
||||
### Duplicated content
|
||||
|
||||
It could happen that the crawled website returned duplicated data. Most of the time, this is because the crawled pages got the same urls with two different schemes.
|
||||
|
||||
If we have URLs like `http://website.com/page` and `http://website.com/page/` (notice the second one ending with `/`), the scrapper will consider them as different. This can be fixed by adding a regex to the `stop_urls` in the `config.json`:
|
||||
If we have URLs like `http://website.com/page` and `http://website.com/page/` (notice the second one ends with `/`), the scraper will consider them as different. This can be fixed by adding a regex to the `stop_urls` in the `config.json`:
|
||||
|
||||
```json
|
||||
"stop_urls": [
|
||||
|
|
@ -274,9 +362,9 @@ In this attribute, you can also list the pages you want to skip:
|
|||
]
|
||||
```
|
||||
|
||||
#### Anchors
|
||||
### Anchors
|
||||
|
||||
The scraper will also consider pages with anchors as different pages. Make sure you remove any hashsign from the urls you put in the stop & start URLs:
|
||||
The scraper will also consider pages with anchors as different pages. Make sure you remove any hash sign from the urls that you put in the stop & start URLs:
|
||||
|
||||
*Bad:*
|
||||
|
||||
|
|
@ -300,4 +388,4 @@ Or :
|
|||
"stop_urls": [
|
||||
"http://website.com/page/"
|
||||
]
|
||||
```
|
||||
```
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ title: Customize autocomplete styles
|
|||
## Attribution
|
||||
|
||||
We're happy to provide DocSearch free of charge for your site, and you're
|
||||
welcome to customise that experience in a way that works for you; all we ask is
|
||||
welcome to customize that experience in a way that works for you; all we ask is
|
||||
that Algolia be attributed within the search context. For example, in the
|
||||
default implementation, we place a small "Search by Algolia" logo in the
|
||||
corner. If you prefer to roll your own UX, you'll need to make sure that this
|
||||
|
|
@ -15,7 +15,7 @@ logo is included in your implementation as well.
|
|||
|
||||
The default colorscheme is white and gray:
|
||||
|
||||
[Default colorscheme](https://community.algolia.com/docsearch/assets/images/default-colorscheme.png)
|
||||
|
||||
|
||||
To update the colors to suit your website, you just need to override a few
|
||||
colors. Here is an example of a CSS file that you can use as a basis to
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
---
|
||||
title: Setup
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
We're scratching our own itch here. As developers, we spend a lot of time
|
||||
reading documentation, and it isn't always easy to find the information we need.
|
||||
|
||||
Not blaming anyone here. Building a good search for a documentation is a complex
|
||||
challenge. We happen to have a lot of experience doing that, and we want to
|
||||
share it with the world. For free.
|
||||
|
||||
Just submit the form on the [website](https://community.algolia.com/docsearch/) and we'll get back to you with what
|
||||
you need to integrate your new search into your website.
|
||||
|
||||
1. We'll crawl your documentation pages,
|
||||
2. We'll configure your search experience,
|
||||
3. You'll need to add a bit of JavaScript and CSS code to your website.
|
||||
|
||||
If you prefer to DIY, you can run the [scraper](https://github.com/algolia/docsearch-scraper) in your own infra.
|
||||
|
||||
## Including the docsearch files in the frontend
|
||||
|
||||
Once we've crawled your documentation website we'll send you the credentials you
|
||||
need to add the following code snippet to your website:
|
||||
|
||||
```html
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" />
|
||||
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script>
|
||||
<script type="text/javascript">
|
||||
docsearch({
|
||||
apiKey: '<API_KEY>',
|
||||
indexName: '<INDEX_NAME>',
|
||||
inputSelector: '<YOUR_INPUT_DOM_SELECTOR>'
|
||||
});
|
||||
</script>
|
||||
```
|
||||
|
||||
You can also install docsearch via `npm`:
|
||||
|
||||
```sh
|
||||
npm install --save docsearch.js
|
||||
```
|
||||
|
|
@ -1,15 +1,14 @@
|
|||
---
|
||||
title: Configuring the search
|
||||
title: Configuring the search results
|
||||
---
|
||||
|
||||
DocSearch is a wrapper around the [autocomplete.js](https://github.com/algolia/autocomplete.js) library that gets its
|
||||
results from the Algolia API. As such, you can use any options provided by
|
||||
[autocomplete.js](https://github.com/algolia/autocomplete.js) and by the Algolia API.
|
||||
results from the Algolia API. As such, you can use any options provided by this project and by [the Algolia API](https://www.algolia.com/doc/api-reference/).
|
||||
|
||||
## Autocomplete options
|
||||
|
||||
You can pass any options to the underlying `autocomplete` instance through
|
||||
the`autocompleteOptions` parameter. You will find all `autocomplete` options in
|
||||
the `autocompleteOptions` parameter. You will find all `autocomplete` options in
|
||||
its [own documentation](https://github.com/algolia/autocomplete.js#options).
|
||||
|
||||
You can also listen to `autocomplete` events through the `.autocomplete`
|
||||
|
|
@ -54,7 +53,7 @@ var search = docsearch({
|
|||
|
||||
### queryHook
|
||||
|
||||
If you want modify the query before it is send to Algolia you can pass the `queryHook` option.
|
||||
If you want modify the query before it is sent to Algolia you can pass the `queryHook` option.
|
||||
|
||||
```javascript
|
||||
var search = docsearch({
|
||||
|
|
@ -102,4 +101,4 @@ docsearch({
|
|||
});
|
||||
```
|
||||
|
||||
You will find all Algolia API options in its [own documentation](https://www.algolia.com/doc/api-reference/api-parameters/).
|
||||
You will find all Algolia API options in its [own documentation](https://www.algolia.com/doc/api-reference/api-parameters/).
|
||||
Loading…
Reference in a new issue