Rewriting the Run Your Own page
This commit is contained in:
Pixelastic
parent
b5a1d1c7a7
commit
d62ce0a9ba
5 changed files with 138 additions and 165 deletions
|
|
@ -76,8 +76,8 @@
|
|||
"url": "config-file.html"
|
||||
},
|
||||
{
|
||||
"title": "Docker image",
|
||||
"url": "docker.html"
|
||||
"title": "Run your own",
|
||||
"url": "run-your-own.html"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
BIN
docs/src/assets/playground.png
Normal file
BIN
docs/src/assets/playground.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
|
|
@ -393,12 +393,12 @@ pages.
|
|||
### `nb_hits` _Special_
|
||||
|
||||
`nb_hits` automatically updated by DocSearch every time it runs your config. It
|
||||
is set to the number of records that were extracted and indexed. We monitor this
|
||||
is set to the number of records that were extracted and indexed. We check this
|
||||
key internally to keep track of any unintended spike or drop that could reveal
|
||||
a misconfiguration.
|
||||
|
||||
You don't have to set nor modify this key, we're only documenting it here in
|
||||
case you were wondering what this was about.
|
||||
You don't have to touch, we're documenting it here in case you were
|
||||
wondering what this was about.
|
||||
|
||||
### `only_content_level` _Optional_
|
||||
|
||||
|
|
@ -515,9 +515,8 @@ By default DocSearch expect websites to have server-side rendering, meaning that
|
|||
HTML source is returned directly by the server. If your content is generated by the
|
||||
front-end, you have to tell DocSearch to emulate a browser through Selenium.
|
||||
|
||||
_As client-side crawling is orders of magnitude slower than server-side
|
||||
crawling, we highly encourage you to update your website to enable server-side
|
||||
rendering._
|
||||
_As client-side crawling is way slower than server-side crawling, we highly
|
||||
encourage you to update your website to enable server-side rendering._
|
||||
|
||||
### `js_render` _Optional_
|
||||
|
||||
|
|
@ -536,7 +535,7 @@ If your website is slow to load, you can use `js_wait` to tell DocSearch to wait
|
|||
a specific amount of time (in seconds) for the page to load before extracting
|
||||
its content.
|
||||
|
||||
Note that this option might have a very large impact on the time required
|
||||
Note that this option might have a large impact on the time required
|
||||
to crawl your website and we would encourage you to enable server-side rendering
|
||||
on your website instead.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,156 +0,0 @@
|
|||
---
|
||||
layout: two-columns
|
||||
title: DocSearch on your own infrastructure
|
||||
---
|
||||
|
||||
// TODO
|
||||
|
||||
When we entirely host your DocSearch implementation, you don't have to bother, we **handle the whole stack**.
|
||||
|
||||
In some specific use cases and situations, you may need to look under the hood and DIY.
|
||||
This is welcome with DocSearch since every tool is open source. Feel free to PR your features!
|
||||
|
||||
DocSearch is basically composed of 3 different projects:
|
||||
* [The search-UI of DocSearch](https://github.com/algolia/docsearch)
|
||||
* [The scraper](https://github.com/algolia/docsearch-scraper), which browses & indexes web pages
|
||||
* [The configuration files repo](https://github.com/algolia/docsearch-configs) for the scraper.
|
||||
|
||||
## The DocSearch backend: [our scraper](https://github.com/algolia/docsearch-scraper):
|
||||
|
||||
This project is a collection of submodules, each one in its own directory:
|
||||
* cli: A command line tool to manage DocSearch. Run `./docsearch` and follow the steps
|
||||
* deployer: Tool used by Algolia to deploy the configuration in our Apache Mesos infrastructure. You do not need this.
|
||||
* doctor: A monitoring tool to check if the indices built by the scraper are in good shape. We mostly know every issue with your website. You do not need this.
|
||||
* playground: An HTML page to easily test your DocSearch indices.
|
||||
* scraper: The core of the scraper. It reads the configuration file, fetches the web pages and indexes them in Algolia.
|
||||
|
||||
## Install
|
||||
|
||||
The DocSearch scraper is based on [Scrapy](https://scrapy.org), a famous python-based web scraper. Because it might need some JavaScript to render the pages it crawls, the scraper is also depending on [selenium](http://www.seleniumhq.org).
|
||||
|
||||
To ease the setup process, a Docker container is provided to help you run the scraper.
|
||||
|
||||
### Environment:
|
||||
|
||||
- Install `python` & `pip`
|
||||
- `brew install python # will install pip`
|
||||
- `apt-get install python`
|
||||
- Or any other way
|
||||
- `git clone git@github.com:algolia/docsearch-scraper.git`
|
||||
- `cd docsearch-scraper`
|
||||
- `pip install --user -r requirements.txt`
|
||||
|
||||
### With docker:
|
||||
|
||||
- Build the underlying Docker image: `./docsearch docker:build`
|
||||
|
||||
## Configure DocSearch
|
||||
|
||||
You need to create an [Algolia account](https://www.algolia.com/users/sign_up) to get the `APPLICATION_ID` and (admin) `API_KEY` credentials the scraper will use to create the underlying indices.
|
||||
|
||||
Create a file named `.env` file at the root of the project containing the following keys:
|
||||
|
||||
```
|
||||
APPLICATION_ID=
|
||||
API_KEY=
|
||||
```
|
||||
|
||||
And run the CLI to see the available commands:
|
||||
|
||||
```sh
|
||||
$ ./docsearch
|
||||
Docsearch CLI
|
||||
|
||||
Usage:
|
||||
./docsearch command [options] [arguments]
|
||||
|
||||
Options:
|
||||
--help Display help message
|
||||
|
||||
Available commands:
|
||||
bootstrap Bootstrap a DocSearch config
|
||||
test Run tests
|
||||
playground Launch the playground
|
||||
run Run a config
|
||||
deploy Deploy configs
|
||||
batch # You do not need these commands
|
||||
batch:sync_helpdesk update conversation for the missing config (conversation ID)
|
||||
connector # You do not need these commands
|
||||
connector:reindex Reindex a connector
|
||||
connector:enable Disable a connector
|
||||
connector:disable Disable a connector
|
||||
connector:logs Reindex a connector
|
||||
deploy # You do not need these commands
|
||||
deploy:scraper Deploy docker scraper images
|
||||
docker
|
||||
docker:build Build scraper images (dev, prod, test)
|
||||
docker:run Run a config using docker
|
||||
emails # You do not need these commands
|
||||
emails:update Add or update contact emails
|
||||
emails:delete Delete contact emails
|
||||
generate # You do not need these commands
|
||||
generate:email Generate the email for a docsearch
|
||||
```
|
||||
|
||||
## Use DocSearch
|
||||
|
||||
### Create a config
|
||||
|
||||
To use DocSearch, the first thing you need is to create a configuration. For more details about configs, check out [our parameters descritption](https://community.algolia.com/docsearch/documentation/docsearch-scraper/config-options/), you'll have a list of options you can use and a lot of live and working examples.
|
||||
|
||||
### Crawl the website
|
||||
|
||||
#### With docker:
|
||||
|
||||
```sh
|
||||
$ ./docsearch docker:run /path/to/your/config
|
||||
```
|
||||
|
||||
#### Without docker
|
||||
|
||||
```sh
|
||||
$ ./docsearch run /path/to/your/config
|
||||
```
|
||||
|
||||
### Try it with our playground
|
||||
|
||||
You can open the included **Playground** to test your DocSearch index.
|
||||
|
||||
```sh
|
||||
$ ./docsearch playground
|
||||
```
|
||||
|
||||
Enter your credentials and the `index_name` mentioned in the config, then try the search!
|
||||
|
||||
### Integrate DocSearch to your website
|
||||
|
||||
To add the DocSearch dropdown menu to your website, add the following snippet to your website:
|
||||
|
||||
```html
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" />
|
||||
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script>
|
||||
<script>
|
||||
var search = docsearch({
|
||||
appId: '<APP_ID>', // the application ID containing your DocSearch index. If we host it for you, please remove this line
|
||||
apiKey: '<API_KEY>', // use a SEARCH-ONLY api key here
|
||||
indexName: '<INDEX_NAME>',
|
||||
inputSelector: '<YOUR_INPUT_DOM_SELECTOR>', // CSS selector targeting the element to replace with the dropdown
|
||||
debug: false // set to `true` if you want to inspect the dropdown menu's CSS
|
||||
});
|
||||
</script>
|
||||
```
|
||||
|
||||
And you are good to go!
|
||||
|
||||
### Specify your own appId
|
||||
|
||||
If you are running the scraper on your own, you will need to tell the widget about your Algolia application ID via the `appId` parameter.
|
||||
|
||||
```javascript
|
||||
var search = docsearch({
|
||||
appId: '<APP_ID>', // the application ID containing your DocSearch data
|
||||
... // other parameters as above
|
||||
});
|
||||
```
|
||||
|
||||
If Algolia is handling the crawling of your site, you do not need to specify `appId`.
|
||||
130
docs/src/run-your-own.md
Normal file
130
docs/src/run-your-own.md
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
---
|
||||
layout: two-columns
|
||||
title: Run your own
|
||||
---
|
||||
|
||||
The version of DocSearch we provide for free is one hosted on our own servers,
|
||||
running every 24 hours. If you need to update your results more often than that,
|
||||
or need to index content sitting behind a firewall, you might want to run the
|
||||
crawler yourself.
|
||||
|
||||
The code of DocSearch is Open-Source, and we packaged it as a Docker image to
|
||||
make this even easier for you to use.
|
||||
|
||||
## Installation
|
||||
|
||||
Start by cloning [the repo][1] and then running `./docsearch docker:build` to
|
||||
create the local image.
|
||||
|
||||
Even if not recommended, you can run DocSearch directly from you host. For
|
||||
that, you'll need to have `python` and `pip` installed, and then run `pip
|
||||
install --user -r requirements.txt`.
|
||||
|
||||
## Configuration
|
||||
|
||||
You'll need to set your Algolia application ID and admin API key as environment
|
||||
variables. If you don't have an Algolia account, you should [create one][2].
|
||||
|
||||
- `APPLICATION_ID` should be set to your Application ID
|
||||
- `API_KEY` should be set to your API Key. Make sure to use an API key with
|
||||
**write** access to your index.
|
||||
|
||||
For convenience, you can create a `.env` file in the repository root with the
|
||||
following format and DocSearch will use those values.
|
||||
|
||||
```
|
||||
APPLICATION_ID=YOUR_APP_ID
|
||||
API_KEY=YOUR_API_KEY
|
||||
```
|
||||
|
||||
## Creating a new config
|
||||
|
||||
To create your config, run `./docsearch bootstrap`. A prompt will ask you for
|
||||
a some information and will then output a JSON config you can use as a base.
|
||||
|
||||
```sh
|
||||
$ ./docsearch bootstrap
|
||||
# Enter your documentation url
|
||||
start url: http://www.example.com/docs/
|
||||
# You most probably don't need variables
|
||||
Does the start_urls require variables ? [y/n]: n
|
||||
# Pick another name, or press enter
|
||||
index_name is example [enter to confirm]: <Enter>
|
||||
|
||||
=================
|
||||
{
|
||||
"index_name": "example",
|
||||
"start_urls": [
|
||||
"http://www.example.com/docs/"
|
||||
],
|
||||
"stop_urls": [],
|
||||
"selectors": {
|
||||
"lvl0": "FIXME h1",
|
||||
"lvl1": "FIXME h2",
|
||||
"lvl2": "FIXME h3",
|
||||
"lvl3": "FIXME h4",
|
||||
"lvl4": "FIXME h5",
|
||||
"text": "FIXME p, FIXME li"
|
||||
}
|
||||
}
|
||||
=================
|
||||
```
|
||||
|
||||
Copy-paste the content into a file name `example.json`, we'll use it later to
|
||||
start the crawling. You can find the complete list of available options in [our
|
||||
documentation][3], or browse the [list of live configs][4].
|
||||
|
||||
## Running your config
|
||||
|
||||
Now that you have your environment variables set, you can run the crawler
|
||||
according to your config.
|
||||
|
||||
```sh
|
||||
$ ./docsearch docker:run /path/to/your/config.json
|
||||
```
|
||||
|
||||
This will crawl all pages, extract content from them and then push it to
|
||||
Algolia.
|
||||
|
||||
## Testing your results
|
||||
|
||||
You can test your results by running `./docsearch playground`. This will open
|
||||
a web page with a search input where you can do live tests against the indexed
|
||||
results.
|
||||
|
||||
<img src="./assets/playground.png" alt="Playground" class="mt-2"/>
|
||||
|
||||
_Note that if the command fails (it can happen on non-Mac machines), you can get
|
||||
the same result by running a live server in the `./playground` subdirectory.`_
|
||||
|
||||
## Adding it to your website
|
||||
|
||||
Once you're satisfied with your config, you can integrate the dropdown menu in
|
||||
your website by following the [instructions here][5].
|
||||
|
||||
The difference is that you'll also have to add the `appId` key to your
|
||||
`docsearch()` instance. Also don't forget to use a **search** API key here (ie.
|
||||
not the **write** API key you used for the crawling).
|
||||
|
||||
|
||||
```javascript
|
||||
docsearch({
|
||||
appId: '<APP_ID>', // Add your own Application ID
|
||||
apiKey: '<API_KEY>', // Set it to your own search API key
|
||||
[…] // Other settings are identical
|
||||
});
|
||||
```
|
||||
|
||||
## Help
|
||||
|
||||
You can run `./docsearch` without any argument to see the list of all available
|
||||
commands.
|
||||
|
||||
Note that we use this CLI tool internally at Algolia to run the free hosted
|
||||
version, so you might not need all the listed commands.
|
||||
|
||||
[1]: https://github.com/algolia/docsearch-scraper
|
||||
[2]: https://www.algolia.com/users/sign_up
|
||||
[3]: ./config-file.html
|
||||
[4]: https://github.com/algolia/docsearch-configs/tree/master/configs
|
||||
[5]: ./dropdown.html
|
||||
Loading…
Reference in a new issue