Doc: update content (#657)
* docs: add faq about how many records are created * docs: reorganize the run on your own section to avoid any confusion * docs: remove wrong part of the attribution policy * Update docs/src/faq.md Co-Authored-By: s-pace <sylvain.pace@algolia.com> * Update docs/src/faq.md Co-Authored-By: s-pace <sylvain.pace@algolia.com> * Update docs/src/faq.md Co-Authored-By: s-pace <sylvain.pace@algolia.com> * Update docs/src/run-your-own.md Co-Authored-By: s-pace <sylvain.pace@algolia.com> * Update docs/src/faq.md Co-Authored-By: s-pace <sylvain.pace@algolia.com> * Update docs/src/run-your-own.md Co-Authored-By: s-pace <sylvain.pace@algolia.com> * Update docs/src/run-your-own.md Co-Authored-By: s-pace <sylvain.pace@algolia.com>
This commit is contained in:
Sylvain Pace
GitHub
parent
35a5b4397e
commit
5931b5b16f
3 changed files with 57 additions and 42 deletions
|
|
@ -118,6 +118,17 @@ encourage you to file an issue in their repository explaining how DocSearch
|
|||
could help. Feel free to [send us an email][1] as well, and we'll provide all
|
||||
the help we can.
|
||||
|
||||
## How many records are created by DocSearch?
|
||||
|
||||
The [property `nb_hits`][9] in your configuration keeps track of the number of
|
||||
records that were extracted and indexed by the last DocSearch run. It is updated
|
||||
automatically at each run.
|
||||
|
||||
The DocSearch scraper follows [the recommended atomic-reindexing strategy][10].
|
||||
It creates a brand new temporary index to populate the data scraped from your
|
||||
website. Once the crawl is successfully achieved, this temporary index overwites
|
||||
the old index defined in your configuration with the key `index_name`.
|
||||
|
||||
[1]: mailto:docsearch@algolia.com
|
||||
[2]: https://www.algolia.com/
|
||||
[3]: https://www.algolia.com/pricing
|
||||
|
|
@ -126,3 +137,6 @@ the help we can.
|
|||
[6]: ./run-your-own.html
|
||||
[7]: https://www.algolia.com/doc/api-reference/
|
||||
[8]: ./config-file.html#custom_settings-optional
|
||||
[9]: ./config-file.html#nb_hits-special
|
||||
[10]:
|
||||
https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/in-depth/asynchronicity-and-when-to-wait-for-tasks/#atomic-reindexing
|
||||
|
|
|
|||
|
|
@ -42,17 +42,21 @@ should check the [dedicated configuration documentation][5].
|
|||
docker run -it --env-file=.env -e "CONFIG=$(cat /path/to/your/config.json | jq -r tostring)" algolia/docsearch-scraper
|
||||
```
|
||||
|
||||
## Installation
|
||||
Once the crawl is done, you can [jump to the Integration step][6].
|
||||
|
||||
The scraper is a python tool [based on scrapy][6]. We do recommend to use
|
||||
## Running the crawler from the code base
|
||||
|
||||
### Installation
|
||||
|
||||
The scraper is a python tool [based on scrapy][7]. We do recommend to use
|
||||
[pipenv][8] to install the python environment.
|
||||
|
||||
- [Clone the scraper repository][7].
|
||||
- [Install pipenv][9]
|
||||
- [Clone the scraper repository][9].
|
||||
- [Install pipenv][10]
|
||||
- `pipenv install`
|
||||
- `pipenv shell`
|
||||
|
||||
If you plan to use the browser emulation [(`js_render` set to true)][10], you
|
||||
If you plan to use the browser emulation [(`js_render` set to true)][11], you
|
||||
need to follow this extra step. If you don't, you can dismiss this step.
|
||||
|
||||
### Installing Chrome driver
|
||||
|
|
@ -60,13 +64,32 @@ need to follow this extra step. If you don't, you can dismiss this step.
|
|||
Some websites rendering requires JavaScript. Our crawler rely on a headless
|
||||
chrome emulation. You will need to set up a ChromeDriver.
|
||||
|
||||
- [Install the driver][11] suited to your OS and the version of your Chrome. We
|
||||
- [Install the driver][12] suited to your OS and the version of your Chrome. We
|
||||
do recommend to use the latest version.
|
||||
- Set the environment variable `CHROMEDRIVER_PATH` in your `.env` file. This
|
||||
path must target the downloaded extracted driver.
|
||||
|
||||
You are ready to go.
|
||||
|
||||
### Running the crawler
|
||||
|
||||
|
||||
Running `pipenv shell` will enable your virtual environment. From there, you can
|
||||
run one crawl with the following command:
|
||||
|
||||
```sh
|
||||
$ ./docsearch run /path/to/your/config.json
|
||||
```
|
||||
|
||||
Or from the Docker image:
|
||||
|
||||
```sh
|
||||
$ ./docsearch docker:run /path/to/your/config.json
|
||||
```
|
||||
|
||||
This will start the crawl. It extracts content from parsed pages and push the
|
||||
built records to Algolia.
|
||||
|
||||
## Create a new configuration
|
||||
|
||||
To create a configuration, run `./docsearch bootstrap`. A prompt will ask you
|
||||
|
|
@ -100,35 +123,14 @@ index_name is example [enter to confirm]: <Enter>
|
|||
```
|
||||
|
||||
Create a file from this text into a filename `example.json`, we'll use it later
|
||||
on to start the crawl. You can browse the [list of live configurations][12].
|
||||
|
||||
## Run the crawl from the code base
|
||||
|
||||
Now that you have your environment variables set, you can run the crawler
|
||||
according to your configuration.
|
||||
|
||||
Running `pipenv shell` will enable your virtual environment. From there, you can
|
||||
run one crawl with the following command:
|
||||
|
||||
```sh
|
||||
$ ./docsearch run /path/to/your/config.json
|
||||
```
|
||||
|
||||
Or from the Docker image:
|
||||
|
||||
```sh
|
||||
$ ./docsearch docker:run /path/to/your/config.json
|
||||
```
|
||||
|
||||
This will start the crawl. It extracts content from parsed pages and push the
|
||||
built records to Algolia.
|
||||
on to start the crawl. You can browse the [list of live configurations][13].
|
||||
|
||||
## Testing your results
|
||||
|
||||
You can test your results by running `./docsearch playground`. This will open a
|
||||
web page with a search input. You can do live tests against the indexed results.
|
||||
|
||||
![Playground][13] {mt-2}
|
||||
![Playground][14] {mt-2}
|
||||
|
||||
_Note that if the command fails (it can happen on non-Mac machines), you can get
|
||||
the same result by running a live server in the `./playground` subdirectory.\`_
|
||||
|
|
@ -136,7 +138,7 @@ the same result by running a live server in the `./playground` subdirectory.\`_
|
|||
## Integration
|
||||
|
||||
Once you're satisfied with your config, you can integrate the dropdown menu in
|
||||
your website by following the [instructions here][14].
|
||||
your website by following the [instructions here][15].
|
||||
|
||||
The difference is that you'll also have to add the `appId` key to your
|
||||
`docsearch()` instance. Also don't forget to use a **search** API key here (in
|
||||
|
|
@ -163,12 +165,13 @@ hosted version, so you might not need all the listed commands._
|
|||
[3]: https://www.algolia.com/doc/guides/security/api-keys/#acl
|
||||
[4]: https://github.com/stedolan/jq/wiki/Installation
|
||||
[5]: ./config-file.html
|
||||
[6]: https://scrapy.org/
|
||||
[7]: https://github.com/algolia/docsearch-scraper
|
||||
[6]: #integration
|
||||
[7]: https://scrapy.org/
|
||||
[8]: https://github.com/pypa/pipenv
|
||||
[9]: https://pipenv.readthedocs.io/en/latest/install/#installing-pipenv
|
||||
[10]: ./config-file.html#js_render-optional
|
||||
[11]: http://chromedriver.chromium.org/getting-started
|
||||
[12]: https://github.com/algolia/docsearch-configs/tree/master/configs
|
||||
[13]: ./assets/playground.png
|
||||
[14]: ./dropdown.html
|
||||
[9]: https://github.com/algolia/docsearch-scraper
|
||||
[10]: https://pipenv.readthedocs.io/en/latest/install/#installing-pipenv
|
||||
[11]: ./config-file.html#js_render-optional
|
||||
[12]: http://chromedriver.chromium.org/getting-started
|
||||
[13]: https://github.com/algolia/docsearch-configs/tree/master/configs
|
||||
[14]: ./assets/playground.png
|
||||
[15]: ./dropdown.html
|
||||
|
|
|
|||
|
|
@ -57,10 +57,8 @@ and you're encouraged to style it to fit your own theming. All we ask is that
|
|||
you keep the `search by Algolia` logo and link next to your search results.
|
||||
|
||||
The logo is automatically added in the dropdown with the default styling. It's
|
||||
fine if you hide it through CSS, as long as you re-add it somewhere else on your
|
||||
page close to the search input or search results. It's our way to let more
|
||||
people know about what we do, and how they could also have a fast and relevant
|
||||
search on their website.
|
||||
our way to let more people know about what we do, and how they could benefit
|
||||
from a fast and relevant search on their website.
|
||||
|
||||
If you're using your own [paid Algolia account][2] and [run the crawler
|
||||
yourself][3], you don't have to keep the logo.
|
||||
|
|
|
|||
Loading…
Reference in a new issue