docsearch/docs/source/documentation/2-docsearch-scraper/1-overview.html.md.erb

---
title: DocSearch on your own infrastructure
---

When we entirely host your DocSearch implementation, you don't have to bother, we **handle the whole stack**.

In some specific use cases and situations, you may need to look under the hood and DIY.
This is welcome with DocSearch since every tool is open source. Feel free to PR your features!

DocSearch is basically composed of 3 different projects:
* [The search-UI of DocSearch](https://github.com/algolia/docsearch)
* [The scraper](https://github.com/algolia/docsearch-scraper), which browses & indexes web pages
* [The configuration files repo](https://github.com/algolia/docsearch-configs) for the scraper.

## The DocSearch backend: [our scraper](https://github.com/algolia/docsearch-scraper):

This project is a collection of submodules, each one in its own directory:
* cli: A command line tool to manage DocSearch. Run `./docsearch` and follow the steps
* deployer: Tool used by Algolia to deploy the configuration in our Apache Mesos infrastructure. You do not need this.
* doctor: A monitoring tool to check if the indices built by the scraper are in good shape. We mostly know every issue with your website. You do not need this.
* playground: An HTML page to easily test your DocSearch indices.
* scraper: The core of the scraper. It reads the configuration file, fetches the web pages and indexes them in Algolia.

## Install

The DocSearch scraper is based on [Scrapy](https://scrapy.org), a famous python-based web scraper. Because it might need some JavaScript to render the pages it crawls, the scraper is also depending on [selenium](http://www.seleniumhq.org).

To ease the setup process, a Docker container is provided to help you run the scraper.

### Environment:

- Install `python` & `pip`
  - `brew install python # will install pip`
  - `apt-get install python`
  - Or any other way
- `git clone git@github.com:algolia/docsearch-scraper.git`
- `cd docsearch-scraper`
- `pip install --user -r requirements.txt`

### With docker:

- Build the underlying Docker image: `./docsearch docker:build`

## Configure DocSearch

You need to create an [Algolia account](https://www.algolia.com/users/sign_up) to get the `APPLICATION_ID` and (admin) `API_KEY` credentials the scraper will use to create the underlying indices.

Create a file named `.env` file at the root of the project containing the following keys:

```
APPLICATION_ID=
API_KEY=
```

And run the CLI to see the available commands:

```sh
$ ./docsearch
Docsearch CLI

Usage:
  ./docsearch command [options] [arguments]

Options:
  --help    Display help message

Available commands:
  bootstrap            Bootstrap a DocSearch config
  test                 Run tests
  playground           Launch the playground
  run                  Run a config
  deploy               Deploy configs
 batch # You do not need these commands
  batch:sync_helpdesk  update conversation for the missing config (conversation ID)
 connector # You do not need these commands
  connector:reindex    Reindex a connector
  connector:enable     Disable a connector
  connector:disable    Disable a connector
  connector:logs       Reindex a connector
 deploy # You do not need these commands
  deploy:scraper       Deploy docker scraper images
 docker
  docker:build         Build scraper images (dev, prod, test)
  docker:run           Run a config using docker
 emails # You do not need these commands
  emails:update        Add or update contact emails
  emails:delete        Delete contact emails
 generate # You do not need these commands
  generate:email       Generate the email for a docsearch
```

## Use DocSearch

### Create a config

To use DocSearch, the first thing you need is to create a configuration. For more details about configs, check out [our parameters descritption](https://community.algolia.com/docsearch/documentation/docsearch-scraper/config-options/), you'll have a list of options you can use and a lot of live and working examples.

### Crawl the website

#### With docker:

```sh
$ ./docsearch docker:run /path/to/your/config
```

#### Without docker

```sh
$ ./docsearch run /path/to/your/config
```

### Try it with our playground

You can open the included **Playground** to test your DocSearch index.

```sh
$ ./docsearch playground
```

Enter your credentials and the `index_name` mentioned in the config, then try the search!

### Integrate DocSearch to your website

To add the DocSearch dropdown menu to your website, add the following snippet to your website:

```html
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" />
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script>
<script>
  var search = docsearch({
    appId: '<APP_ID>', // the application ID containing your DocSearch index. If we host it for you, please remove this line
    apiKey: '<API_KEY>', // use a SEARCH-ONLY api key here
    indexName: '<INDEX_NAME>',
    inputSelector: '<YOUR_INPUT_DOM_SELECTOR>', // CSS selector targeting the element to replace with the dropdown
    debug: false // set to `true` if you want to inspect the dropdown menu's CSS
  });
</script>
```

And you are good to go!

### Specify your own appId

If you are running the scraper on your own, you will need to tell the widget about your Algolia application ID via the `appId` parameter.

```javascript
  var search = docsearch({
    appId: '<APP_ID>', // the application ID containing your DocSearch data
    ... // other parameters as above
  });
```

If Algolia is handling the crawling of your site, you do not need to specify `appId`.