docsearch/packages/website/docs/crawler.mdx

---
title: DocSearch x Algolia Crawler
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

If you're not finding the answer to your question on this website, this page will help you. If you're still unsure, don't hesitate to connect with us on [Discord][1] or let our [support][3] team know.

You can also read our [Crawler FAQ](https://www.algolia.com/doc/tools/crawler/troubleshooting/crawl-status/), to understand how it behaves:

- [One of my pages wasn't crawled](https://www.algolia.com/doc/tools/crawler/troubleshooting/extraction-issues/#a-page-wasnt-crawled)
- [Why are my pages skipped?](https://www.algolia.com/doc/tools/crawler/troubleshooting/fetching-issues/)

For questions related to the DocSearch program, please see our [DocSearch program FAQ](/docs/docsearch-program).

## How often will you crawl my website?

Crawls are scheduled at a random time once a week. You can [configure this schedule from the config file](https://www.algolia.com/doc/tools/crawler/apis/configuration/schedule/) or trigger one manually from [the Crawler interface][2].

## Why do I have duplicate content in my results?

This can happen when you have more than one URL pointing to the same content, for example with `./docs`, `./docs/` and `./docs/index.html`.

We recommend configuring canonical URLs on your website, you can read more on the ["Consolidate duplicate URLs" guide by Google](https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls).

Ultimately, it is possible to set set the [`exclusionPatterns`](https://www.algolia.com/doc/tools/crawler/apis/configuration/exclusion-patterns/) to all the patterns you want to exclude.

## Are the [`docsearch-scraper`](https://github.com/algolia/docsearch-scraper) and [`docsearch-configs`](https://github.com/algolia/docsearch-configs) repository still maintained?

We've deprecated our legacy infrastructure, but you can still use it to [run your own instance](/docs/legacy/run-your-own) and plug it to [DocSearch v3](/docs/v3/docsearch)!

## How to migrate

> Every owners should have received a migration email from Algolia with the details. If you were not part of the previous `index` owners, or the maintainer has changed, you can request access via [our support page](https://www.algolia.com/support/).

All the steps are detailed in the email you've received, but in order to use the new infrastructure you need to:

- Join the Algolia application with the invite included in the email
- Update your frontend integration with the credentials received in the email.

<Tabs
  groupId="language"
  defaultValue="js"
  values={[
    { label: 'JavaScript', value: 'js', },
    { label: 'React', value: 'react', }
  ]
}>
<TabItem value="js">

```js app.js
docsearch({
  container: '#docsearch',
  appId: 'YOUR_NEW_ALGOLIA_APP_ID',
  apiKey: 'YOUR_NEW_ALGOLIA_SEARCH_API_KEY',
  indexName: 'YOUR_INDEX_NAME', // it does not change
});
```

</TabItem>

<TabItem value="react">

```jsx App.js
<DocSearch
  appId="YOUR_NEW_ALGOLIA_APP_ID"
  apiKey="YOUR_NEW_ALGOLIA_SEARCH_API_KEY"
  indexName="YOUR_INDEX_NAME" // it does not change
/>
```

</TabItem>

</Tabs>

## What should I do with my legacy config and credentials?

You can forget about them, we will do the cleaning once all of our users have migrated to the new infrastructure!

You should use [the dedicated web interface][2] to make any changes to your index.

## Why do I see two Algolia apps in my dashboard?

We did not remove access to the legacy DocSearch application (`BH4D9OD16A`) to give you the time to get familiar with our new infrastructure. `BH4D9OD16A` will remain available until the migration has been completed for all the DocSearch users.

## Search yield no results

If your search does not yield any results, but there is no error in [your browser developer tools](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools), there might be an issue with your index.

Make sure that:

1. [Your Crawler config](/docs/record-extractor) matches your website structure

We provide [config templates](/docs/templates) for many website generators, but you can also use them as a base. To debug your selectors, we recommend using [the URL tester](/docs/manage-your-crawls/#url-tester).

2. Your index settings are up to date (you'll see a banner in [the search preview](/docs/manage-your-crawls/#search-preview) if not)

The Crawler only applies `index settings` at index creation index, to keep the Algolia dashboard as the source of truth. If you have drastically changed your config, or moved to a website generator, we recommend you to delete your index from the Algolia dashboard before starting a new crawl.

## Can I delete my crawler?

No. Well, you can but once you do things will not work correctly. We automatically create a default crawler that is associated with your DocSearch application and deleting it with the intention of creating a new one will not work as expected.

## What if I delete my DocSearch Crawler?

The fastest way will be to connect with us on our [Discord](https://alg.li/discord). Alternatively, email us at the address below and we will get to it as soon as we can.

## Can I use the Crawler on password protected sites?

The Crawler as used with DocSearch applications cannot be used for password protected sites that require a login. If you need this functionality, you need to utilize a regular Algolia plan https://www.algolia.com/pricing and add a crawler to it. Note that while it is free to add a pay-as-you-go crawler, the free tier does have limitations.

## Links related to the migration

- [Docusaurus blog post](https://docusaurus.io/blog/2021/11/21/algolia-docsearch-migration)
- [Algolia Dev chat 11-23-2021](https://www.youtube.com/watch?v=htsjpojpKtc&t=2404s)

[1]: https://alg.li/discord
[2]: https://dashboard.algolia.com/crawler
[3]: https://support.algolia.com/