From 5931b5b16f5945b17e940dbf3acbbc0b1a3ea438 Mon Sep 17 00:00:00 2001 From: Sylvain Pace Date: Tue, 23 Apr 2019 10:28:45 +0200 Subject: [PATCH] Doc: update content (#657) * docs: add faq about how many records are created * docs: reorganize the run on your own section to avoid any confusion * docs: remove wrong part of the attribution policy * Update docs/src/faq.md Co-Authored-By: s-pace * Update docs/src/faq.md Co-Authored-By: s-pace * Update docs/src/faq.md Co-Authored-By: s-pace * Update docs/src/run-your-own.md Co-Authored-By: s-pace * Update docs/src/faq.md Co-Authored-By: s-pace * Update docs/src/run-your-own.md Co-Authored-By: s-pace * Update docs/src/run-your-own.md Co-Authored-By: s-pace --- docs/src/faq.md | 14 +++++++ docs/src/run-your-own.md | 79 +++++++++++++++++++++------------------- docs/src/styling.md | 6 +-- 3 files changed, 57 insertions(+), 42 deletions(-) diff --git a/docs/src/faq.md b/docs/src/faq.md index 4db78e38..ee43e64a 100644 --- a/docs/src/faq.md +++ b/docs/src/faq.md @@ -118,6 +118,17 @@ encourage you to file an issue in their repository explaining how DocSearch could help. Feel free to [send us an email][1] as well, and we'll provide all the help we can. +## How many records are created by DocSearch? + +The [property `nb_hits`][9] in your configuration keeps track of the number of +records that were extracted and indexed by the last DocSearch run. It is updated +automatically at each run. + +The DocSearch scraper follows [the recommended atomic-reindexing strategy][10]. +It creates a brand new temporary index to populate the data scraped from your +website. Once the crawl is successfully achieved, this temporary index overwites +the old index defined in your configuration with the key `index_name`. + [1]: mailto:docsearch@algolia.com [2]: https://www.algolia.com/ [3]: https://www.algolia.com/pricing @@ -126,3 +137,6 @@ the help we can. [6]: ./run-your-own.html [7]: https://www.algolia.com/doc/api-reference/ [8]: ./config-file.html#custom_settings-optional +[9]: ./config-file.html#nb_hits-special +[10]: + https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/in-depth/asynchronicity-and-when-to-wait-for-tasks/#atomic-reindexing diff --git a/docs/src/run-your-own.md b/docs/src/run-your-own.md index 5a14899c..ca751630 100644 --- a/docs/src/run-your-own.md +++ b/docs/src/run-your-own.md @@ -42,17 +42,21 @@ should check the [dedicated configuration documentation][5]. docker run -it --env-file=.env -e "CONFIG=$(cat /path/to/your/config.json | jq -r tostring)" algolia/docsearch-scraper ``` -## Installation +Once the crawl is done, you can [jump to the Integration step][6]. -The scraper is a python tool [based on scrapy][6]. We do recommend to use +## Running the crawler from the code base + +### Installation + +The scraper is a python tool [based on scrapy][7]. We do recommend to use [pipenv][8] to install the python environment. -- [Clone the scraper repository][7]. -- [Install pipenv][9] +- [Clone the scraper repository][9]. +- [Install pipenv][10] - `pipenv install` - `pipenv shell` -If you plan to use the browser emulation [(`js_render` set to true)][10], you +If you plan to use the browser emulation [(`js_render` set to true)][11], you need to follow this extra step. If you don't, you can dismiss this step. ### Installing Chrome driver @@ -60,13 +64,32 @@ need to follow this extra step. If you don't, you can dismiss this step. Some websites rendering requires JavaScript. Our crawler rely on a headless chrome emulation. You will need to set up a ChromeDriver. -- [Install the driver][11] suited to your OS and the version of your Chrome. We +- [Install the driver][12] suited to your OS and the version of your Chrome. We do recommend to use the latest version. - Set the environment variable `CHROMEDRIVER_PATH` in your `.env` file. This path must target the downloaded extracted driver. You are ready to go. +### Running the crawler + + +Running `pipenv shell` will enable your virtual environment. From there, you can +run one crawl with the following command: + +```sh +$ ./docsearch run /path/to/your/config.json +``` + +Or from the Docker image: + +```sh +$ ./docsearch docker:run /path/to/your/config.json +``` + +This will start the crawl. It extracts content from parsed pages and push the +built records to Algolia. + ## Create a new configuration To create a configuration, run `./docsearch bootstrap`. A prompt will ask you @@ -100,35 +123,14 @@ index_name is example [enter to confirm]: ``` Create a file from this text into a filename `example.json`, we'll use it later -on to start the crawl. You can browse the [list of live configurations][12]. - -## Run the crawl from the code base - -Now that you have your environment variables set, you can run the crawler -according to your configuration. - -Running `pipenv shell` will enable your virtual environment. From there, you can -run one crawl with the following command: - -```sh -$ ./docsearch run /path/to/your/config.json -``` - -Or from the Docker image: - -```sh -$ ./docsearch docker:run /path/to/your/config.json -``` - -This will start the crawl. It extracts content from parsed pages and push the -built records to Algolia. +on to start the crawl. You can browse the [list of live configurations][13]. ## Testing your results You can test your results by running `./docsearch playground`. This will open a web page with a search input. You can do live tests against the indexed results. -![Playground][13] {mt-2} +![Playground][14] {mt-2} _Note that if the command fails (it can happen on non-Mac machines), you can get the same result by running a live server in the `./playground` subdirectory.\`_ @@ -136,7 +138,7 @@ the same result by running a live server in the `./playground` subdirectory.\`_ ## Integration Once you're satisfied with your config, you can integrate the dropdown menu in -your website by following the [instructions here][14]. +your website by following the [instructions here][15]. The difference is that you'll also have to add the `appId` key to your `docsearch()` instance. Also don't forget to use a **search** API key here (in @@ -163,12 +165,13 @@ hosted version, so you might not need all the listed commands._ [3]: https://www.algolia.com/doc/guides/security/api-keys/#acl [4]: https://github.com/stedolan/jq/wiki/Installation [5]: ./config-file.html -[6]: https://scrapy.org/ -[7]: https://github.com/algolia/docsearch-scraper +[6]: #integration +[7]: https://scrapy.org/ [8]: https://github.com/pypa/pipenv -[9]: https://pipenv.readthedocs.io/en/latest/install/#installing-pipenv -[10]: ./config-file.html#js_render-optional -[11]: http://chromedriver.chromium.org/getting-started -[12]: https://github.com/algolia/docsearch-configs/tree/master/configs -[13]: ./assets/playground.png -[14]: ./dropdown.html +[9]: https://github.com/algolia/docsearch-scraper +[10]: https://pipenv.readthedocs.io/en/latest/install/#installing-pipenv +[11]: ./config-file.html#js_render-optional +[12]: http://chromedriver.chromium.org/getting-started +[13]: https://github.com/algolia/docsearch-configs/tree/master/configs +[14]: ./assets/playground.png +[15]: ./dropdown.html diff --git a/docs/src/styling.md b/docs/src/styling.md index efeab352..874908ef 100644 --- a/docs/src/styling.md +++ b/docs/src/styling.md @@ -57,10 +57,8 @@ and you're encouraged to style it to fit your own theming. All we ask is that you keep the `search by Algolia` logo and link next to your search results. The logo is automatically added in the dropdown with the default styling. It's -fine if you hide it through CSS, as long as you re-add it somewhere else on your -page close to the search input or search results. It's our way to let more -people know about what we do, and how they could also have a fast and relevant -search on their website. +our way to let more people know about what we do, and how they could benefit +from a fast and relevant search on their website. If you're using your own [paid Algolia account][2] and [run the crawler yourself][3], you don't have to keep the logo.