docsearch/packages/website/docs/crawler.mdx

---
title: DocSearch x Algolia Crawler
description: Resolve common DocSearch crawler questions and migration tasks.
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

Use this page to resolve common crawler questions. If you need more help, contact us on [Discord][1] or through [Algolia support][3].

For more troubleshooting guidance, read the [Crawler FAQ][4]:

- [One of my pages wasn't crawled][5]
- [Why are my pages skipped?][6]

For questions related to the DocSearch program, please see our [DocSearch program FAQ](/docs/docsearch-program).

## How often will you crawl my website?

Crawls are scheduled at a random time once a week. You can [configure this schedule from the config file](https://www.algolia.com/doc/tools/crawler/apis/configuration/schedule/) or trigger one manually from [the Crawler interface][2].

## Why do I have duplicate content in my results?

This can happen when you have more than one URL pointing to the same content, for example with `./docs`, `./docs/` and `./docs/index.html`.

Configure canonical URLs on your website. For more information, read Google's [Consolidate duplicate URLs][7] guide.

Use [`exclusionPatterns`][8] to exclude duplicate URL patterns.

## Are the [`docsearch-scraper`][9] and [`docsearch-configs`][10] repositories still maintained?

The legacy scraper infrastructure is deprecated. You can still [run a legacy scraper instance][12] and use its index with the historical [DocSearch v3 UI][13]. For current frontend integrations, choose a [DocSearch v5 package][14]. The scraper or crawler version doesn't determine the frontend package version.

## How do I migrate?

> Every owner should have received a migration email from Algolia with the details. If you were not part of the previous `index` owners, or the maintainer has changed, you can request access via [our support page](https://www.algolia.com/support/).

Use the migration email to complete these steps:

- Join the Algolia application with the invite included in the email
- Update your frontend integration with the credentials received in the email. For v5, follow the [JavaScript][15] or [React][16] guide.

<Tabs
  groupId="language"
  aria-label="Programming language"
  defaultValue="react"
  values={[
    { label: 'React', value: 'react', },
    { label: 'JavaScript', value: 'js', },
  ]
}>
<TabItem value="react">

```jsx title="Search.tsx"
<DocSearch
  appId="YOUR_NEW_ALGOLIA_APP_ID"
  apiKey="YOUR_NEW_ALGOLIA_SEARCH_API_KEY"
  indices={['YOUR_INDEX_NAME']} // The index name doesn't change.
/>
```

</TabItem>

<TabItem value="js">

```js title="search.js"
docsearch({
  container: '#docsearch',
  appId: 'YOUR_NEW_ALGOLIA_APP_ID',
  apiKey: 'YOUR_NEW_ALGOLIA_SEARCH_API_KEY',
  indices: ['YOUR_INDEX_NAME'], // The index name doesn't change.
});
```

</TabItem>
</Tabs>

## What should I do with my legacy config and credentials?

Use the credentials for your current Algolia application. Don't use legacy credentials for new frontend integrations.

You should use [the dedicated web interface][2] to make any changes to your index.

## Why do I see two Algolia apps in my dashboard?

We did not remove access to the legacy DocSearch application (`BH4D9OD16A`) to give you the time to get familiar with our new infrastructure. `BH4D9OD16A` will remain available until the migration has been completed for all the DocSearch users.

## Search yields no results

If your search does not yield any results, but there is no error in [your browser developer tools](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools), there might be an issue with your index.

Check the following settings:

1. Confirm that [your crawler configuration][17] matches your website structure.

   Use a [configuration template][18] for your website generator as a starting point. Test your selectors with [the URL tester][19].

2. Confirm that your index settings are current. If they aren't, [the search preview][20] displays a banner.

   The crawler applies `index settings` only when it creates an index, which keeps the Algolia dashboard as the source of truth. If you changed your configuration significantly or moved to a different website generator, delete the index from the Algolia dashboard before starting a new crawl.

## Can I delete my crawler?

Don't delete the default crawler associated with your DocSearch application. Creating a replacement might not preserve the program configuration.

## What if I delete my DocSearch Crawler?

Contact us on [Discord][1] or email us at the address below.

## Can I use the crawler on password-protected sites?

The crawler included with DocSearch applications can't access password-protected sites that require a login. To crawl these sites, choose an [Algolia plan][11] and add a crawler. Pay-as-you-go plans include a crawler, but the free tier has usage limits.

## Links related to the migration

- [Docusaurus blog post](https://docusaurus.io/blog/2021/11/21/algolia-docsearch-migration)
- [Algolia Dev chat 11-23-2021](https://www.youtube.com/watch?v=htsjpojpKtc&t=2404s)

[1]: https://alg.li/discord
[2]: https://dashboard.algolia.com/crawler
[3]: https://support.algolia.com/
[4]: https://www.algolia.com/doc/tools/crawler/troubleshooting/crawl-status/
[5]: https://www.algolia.com/doc/tools/crawler/troubleshooting/extraction-issues/#a-page-wasnt-crawled
[6]: https://www.algolia.com/doc/tools/crawler/troubleshooting/fetching-issues/
[7]: https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls
[8]: https://www.algolia.com/doc/tools/crawler/apis/configuration/exclusion-patterns/
[9]: https://github.com/algolia/docsearch-scraper
[10]: https://github.com/algolia/docsearch-configs
[11]: https://www.algolia.com/pricing
[12]: /docs/legacy/run-your-own
[13]: /docs/v3/docsearch
[14]: /docs/packages/overview
[15]: /docs/packages/js/getting-started
[16]: /docs/packages/react/getting-started
[17]: /docs/record-extractor
[18]: /docs/templates
[19]: /docs/manage-your-crawls/#url-tester
[20]: /docs/manage-your-crawls/#search-preview