19c1ef9192
* feat(v4): add new UI (#2555) * feat: add new footer ui * feat: add new search box ui * feat: add searchbox actions * feat: add start screen * feat: add no results screen * feat: add new card ui * feat: add new icons * feat: add new key press class * fix: cypress tests * fix: circleci * fix: circleci * fix: close aria label * fix: circleci * fix: remove unused classes * fix: circleci * feat: update version * feat: add new dark mode ui * fix: colors * fix: design review * fix: hit title length * fix: improve accessibility * fix: ci * chore: increase bundle size threshold * fix: css * feat(v4): ask-ai foundations ✨ (#2574) * feat(v4): docsearch askAI context (#2587) * fix(v4): ask ai updates (#2654) * feat(v4): update beta documentation (#2659) * chore: release v4.0.0-beta.0 (#2660) * fix: make release run on branches like v4 * chore: release v4.0.0-beta.0 * hotfix: closing on askai error * chore: release v4.0.0-beta.1 (#2661) * fix: add a section on models * fix: update to `<package>@beta` * feat(docsearch-website): Updated docs (#2662) * feat(v4): update the landing page (#2665) * fix: beta in readme * feat: added glow around the search bar and a little copy above the keyboard * feat: add glow around the search bar and copy above the keyboard * fix: Add more styling * fix: polishing v4 (#2667) * chore: release v4.0.0-beta.2 (#2668) * fix: update docusaurus tarballs * fix(website): update docusaurus tarballs * fix: mobile search bar --------- Co-authored-by: Vasco Bettencourt <32492444+vascobettencourt@users.noreply.github.com> Co-authored-by: Natan Yagudayev <natanyagudayev@gmail.com>
78 lines
No EOL
3.4 KiB
Text
78 lines
No EOL
3.4 KiB
Text
---
|
||
title: Create a New Crawler
|
||
---
|
||
|
||
import useBaseUrl from '@docusaurus/useBaseUrl';
|
||
|
||
# Create a New Crawler
|
||
|
||
:::info
|
||
New DocSearch apps created after **July 2nd, 2024** can now use the Algolia Crawler UI to set up and manage their crawls. This guide walks you through the process of adding your domain, verifying ownership, creating a crawler, and running your first test crawl. You can find the new Crawler UI at [dashboard.algolia.com/crawler](https://dashboard.algolia.com/crawler).
|
||
|
||
If you signed up before July 2nd, 2024, you can still use the Crawler UI, but creating and managing a Crawler is more streamlined for users who joined after that date.
|
||
|
||
Learn more about the [New Crawler UI/UX features](./crawler-configuration-visual).
|
||
:::
|
||
|
||
## Add domains
|
||
|
||
1. Sign in to the [Algolia dashboard](https://dashboard.algolia.com/crawler).
|
||
2. In the left sidebar, select **Data sources**.
|
||
3. Select **Crawler**:
|
||
- Click **Add your domain** and enter the domains or subdomains you want to crawl (e.g., `example.com`, `www.example.com`).
|
||
- If you’ve already added a domain, click the **Domains** tab.
|
||
4. Click **Add domain**.
|
||
|
||
<div className="uil-ta-center">
|
||
<img
|
||
src={useBaseUrl('img/assets/docsearch-domains.png')}
|
||
alt="DocSearch Domains Management"
|
||
/>
|
||
</div>
|
||
|
||
> **Note:** You must verify your domain within a 7-day grace period after adding it. Additionally, your domain must be approved for use by the DocSearch team before you can proceed with crawling.
|
||
|
||
## Verify your domain
|
||
|
||
You must verify ownership of each domain you want to crawl. The default method is email verification, but you can also use a meta tag, HTML file, robots.txt, or DNS record.
|
||
|
||
### Meta tag
|
||
1. In the **Meta tag** tab, click **Copy** to copy the verification tag.
|
||
2. Add the tag to your site's `<head>` section.
|
||
3. Publish your site and click **Verify now** in the Crawler dashboard.
|
||
|
||
### HTML file
|
||
1. In the **HTML file** tab, click **Copy** to copy the verification file content.
|
||
2. Save it as a new HTML file and upload it to your web server.
|
||
3. Add the file’s URL in the dashboard and click **Verify now**.
|
||
|
||
### robots.txt
|
||
1. In the **Robots.txt** tab, click **Copy** to copy the verification code.
|
||
2. Paste it into your site's `robots.txt` file.
|
||
3. Publish and click **Verify now**.
|
||
|
||
### DNS
|
||
1. In the **DNS** tab, copy the provided DNS TXT record.
|
||
2. Add it to your DNS provider’s settings.
|
||
3. Click **Verify now** after the record propagates (may take up to 72 hours).
|
||
|
||
## Create a new crawler
|
||
|
||
Once your domain is verified and approved by our DocSearch team:
|
||
1. Go to the **Crawler** page in the dashboard.
|
||
2. Click **New Crawler** and fill in:
|
||
- **Crawler name** (descriptive)
|
||
- **App ID** (your Algolia application ID)
|
||
- **Start URL** (usually your home page)
|
||
- **Crawler template** (choose a template or default)
|
||
3. Click **Create** to finish and run a test crawl.
|
||
|
||
## Run the test crawl
|
||
|
||
The initial crawl will visit up to 100 URLs to test access and extraction. You can monitor progress in the **Overview** page. After completion, review the extracted records in the Algolia dashboard.
|
||
|
||
## Next steps
|
||
|
||
- Edit your crawler configuration for scheduled crawls, inclusion/exclusion rules, and extraction settings.
|
||
- Use the Crawler’s suggestions for further optimization.
|
||
- For more details, see the [official Algolia documentation](https://www.algolia.com/doc/tools/crawler/getting-started/create-crawler/). |