feat(docs): Enhance AskAI with markdown indexes (#2683)
* feat(docs): enhance AskAI with markdown indexes * fix(docs): linting * feat(docs): added string vs object setup example * feat(docs): added small note * feat(docs): copy change * feat(docs): enhance the documentation steps for markdown * fix(docs): copy changes * fix(docs): copy changes
This commit is contained in:
Natan Yagudayev
GitHub
parent
4210e3b503
commit
4bcdedf8f1
4 changed files with 365 additions and 4 deletions
|
|
@ -189,9 +189,12 @@ Or use our new dedicated [DocSearch Playground](https://community.algolia.com/do
|
|||
|
||||
### Using DocSearch with AskAI
|
||||
|
||||
DocSearch v4 introduces support for AskAI, Algolia’s advanced, AI-powered search capability. AskAI enhances the user experience by providing contextually relevant and intelligent responses directly from your documentation
|
||||
DocSearch v4 introduces support for AskAI, Algolia’s advanced, AI-powered search capability. AskAI enhances the user experience by providing contextually relevant and intelligent responses directly from your documentation.
|
||||
|
||||
To enable AskAI, add your Algolia Assistant ID to your DocSearch configuration:
|
||||
To enable AskAI, you can add your Algolia Assistant ID as a string, or use an object for more advanced configuration (such as specifying a different index, credentials, or search parameters):
|
||||
|
||||
<Tabs groupId="askai-format" defaultValue="string" values={[{ label: 'String', value: 'string' }, { label: 'Object', value: 'object' }]}>
|
||||
<TabItem value="string">
|
||||
|
||||
```js
|
||||
docsearch({
|
||||
|
|
@ -202,6 +205,32 @@ docsearch({
|
|||
});
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="object">
|
||||
|
||||
```js
|
||||
docsearch({
|
||||
appId: 'YOUR_APP_ID',
|
||||
indexName: 'YOUR_INDEX_NAME',
|
||||
apiKey: 'YOUR_SEARCH_API_KEY',
|
||||
askAi: {
|
||||
indexName: 'YOUR_MARKDOWN_INDEX', // Optional: use a different index for AskAI
|
||||
apiKey: 'YOUR_SEARCH_API_KEY', // Optional: use a different API key for AskAI
|
||||
appId: 'YOUR_APP_ID', // Optional: use a different App ID for AskAI
|
||||
assistantId: 'YOUR_ALGOLIA_ASSISTANT_ID',
|
||||
searchParameters: {
|
||||
facetFilters: ['language:en', 'version:1.0.0'], // Optional: filter AskAI context
|
||||
},
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
- Use the string form for a simple setup.
|
||||
- Use the object form to customize which index, credentials, or filters AskAI uses.
|
||||
|
||||
### Filtering search results
|
||||
|
||||
#### Keyword search
|
||||
|
|
|
|||
323
packages/website/docs/v4/askai-markdown-indexing.mdx
Normal file
323
packages/website/docs/v4/askai-markdown-indexing.mdx
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
---
|
||||
title: Improving Answer Quality with Markdown Indexing
|
||||
---
|
||||
|
||||
To deliver more accurate, context-rich answers at scale, AskAI benefits from cleanly structured content. One of the most effective ways to achieve this is by using a Markdown-based indexing helper in your Algolia Crawler configuration. This setup ensures AskAI can access well-formed, content-focused records—especially important for larger documentation sites where metadata, navigation elements, or layout artifacts might otherwise dilute the quality of generative responses.
|
||||
|
||||
:::info
|
||||
These steps are especially valuable for large-scale sites using DocSearch-generated indices, but also apply to custom or smaller setups: you can manually create and upload a Markdown-based index tailored to AskAI.
|
||||
|
||||
**Note:** For more integration examples (Docusaurus, VitePress, Astro/Starlight, and generic setups), see the section at the bottom of this page.
|
||||
:::
|
||||
|
||||
## Overview
|
||||
To maximize the quality of AskAI responses, configure your Crawler to create a dedicated index for Markdown content. This approach enables AskAI to work with structured, chunked records sourced from your documentation, support content, or any Markdown-based material—resulting in significantly more relevant and precise answers.
|
||||
The steps below walk through how to set up your Crawler to index Markdown files specifically for AskAI.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Update your existing DocSearch Crawler configuration
|
||||
|
||||
- In your Crawler config, add the following to your `actions: [ ... ]` array:
|
||||
|
||||
```js
|
||||
// actions: [ ...,
|
||||
{
|
||||
indexName: "my-markdown-index",
|
||||
pathsToMatch: ["https://example.com/docs/**"],
|
||||
recordExtractor: ({ $, url, helpers }) => {
|
||||
// Extract language or other attributes as needed. Optional
|
||||
const language = $("html").attr("lang") || "en";
|
||||
|
||||
return helpers.splitTextIntoRecords({
|
||||
text: helpers.markdown("main"), // Change "main" to match your content tag (e.g., "main", "article", etc.)
|
||||
baseRecord: {
|
||||
objectID: url,
|
||||
lang: language, // Add more attributes as needed
|
||||
},
|
||||
maxRecordBytes: 100000, // Higher = fewer, larger records. Lower = more, smaller records.
|
||||
// Note: Increasing this value may increase the token count for LLMs, which can affect context size and cost.
|
||||
orderingAttributeName: "part",
|
||||
});
|
||||
},
|
||||
},
|
||||
// ...],
|
||||
```
|
||||
|
||||
- Then, add the following to your `initialIndexSettings: { ... }` object:
|
||||
|
||||
```js
|
||||
// initialIndexSettings: { ...,
|
||||
"my-markdown-index": {
|
||||
attributesForFaceting: ["lang"], // Add more if you extract more attributes
|
||||
ignorePlurals: true,
|
||||
minProximity: 4,
|
||||
removeStopWords: true,
|
||||
searchableAttributes: ["unordered(title)", "unordered(text)"],
|
||||
},
|
||||
// ...},
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Run the DocSearch crawler to create a new Ask AI optimized index
|
||||
|
||||
After updating your Crawler configuration:
|
||||
|
||||
1. **Publish your configuration** in the Algolia Crawler dashboard to save and activate it.
|
||||
2. **Run the Crawler** to index your markdown content and create the new index.
|
||||
|
||||
The Crawler will process your content using the markdown extraction helper and populate your new index with clean, structured records optimized for AskAI.
|
||||
|
||||
> **Tip:** Monitor the crawl progress in your dashboard to ensure all pages are processed correctly. You can view the indexed records in your Algolia index to verify the structure and content.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Integrate your new index with Ask AI
|
||||
|
||||
Once your Crawler and index are configured, set up your frontend to use both your main keyword index and your markdown index for AskAI. Here’s how you might configure DocSearch to use your main keyword index for search and your markdown index for AskAI:
|
||||
|
||||
```js
|
||||
docsearch({
|
||||
indexName: 'YOUR_INDEX_NAME', // Main DocSearch keyword index
|
||||
apiKey: 'YOUR_SEARCH_API_KEY',
|
||||
appId: 'YOUR_APP_ID',
|
||||
askAi: {
|
||||
indexName: 'YOUR_INDEX_NAME-markdown', // Markdown index for AskAI
|
||||
apiKey: 'YOUR_SEARCH_API_KEY', // (or a different key if needed)
|
||||
appId: 'YOUR_APP_ID',
|
||||
assistantId: 'YOUR_ALGOLIA_ASSISTANT_ID',
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
- `indexName`: Your main DocSearch index for keyword search.
|
||||
- `askAi.indexName`: The markdown index you created for AskAI context.
|
||||
- `assistantId`: The ID of your configured AskAI assistant.
|
||||
|
||||
> **Tip:** Keep both indexes updated as your documentation evolves to ensure the best search and AI answer quality.
|
||||
|
||||
---
|
||||
|
||||
## Best Practices & Tips
|
||||
|
||||
- **Use clear, consistent titles in your markdown files** for better searchability.
|
||||
- **Test your index** with AskAI to ensure relevant answers are returned.
|
||||
- **Adjust `maxRecordBytes`** if you notice answers are too broad or too fragmented.
|
||||
- **Note:** Increasing `maxRecordBytes` may increase the token count for LLMs, which can affect the size of the context window and the cost of each AskAI response.
|
||||
- **Keep your markdown well-structured** (use headings, lists, etc.) for optimal chunking.
|
||||
- **Add attributes** like `lang`, `version`, or `tags` to your records and `attributesForFaceting` if you want to filter or facet in your search UI or AskAI.
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Why use a separate markdown index?**
|
||||
A: It allows AskAI to access content in a format optimized for LLMs, improving answer quality.
|
||||
|
||||
**Q: Can I use this with other content types?**
|
||||
A: Yes, but markdown is especially well-suited for chunking and context extraction.
|
||||
|
||||
**Q: What if I have very large markdown files?**
|
||||
A: Lower the `maxRecordBytes` value to split content into smaller, more focused records.
|
||||
|
||||
---
|
||||
|
||||
For more details, see the [AskAI documentation](./askai.mdx) or contact support if you need help configuring your Crawler.
|
||||
|
||||
---
|
||||
|
||||
## Crawler Configuration Examples by Integration
|
||||
|
||||
Below are example configurations for setting up your markdown index with different documentation platforms. Each shows how to extract facets (like language, version, tags) and configure the Crawler for your specific integration:
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
<Tabs groupId="integration">
|
||||
|
||||
<TabItem value="generic" label="Non-DocSearch (Generic)">
|
||||
|
||||
**Generic Example:**
|
||||
|
||||
```js
|
||||
// In your Crawler config:
|
||||
|
||||
// actions: [ ...,
|
||||
{
|
||||
indexName: "my-markdown-index",
|
||||
pathsToMatch: ["https://example.com/**"],
|
||||
recordExtractor: ({ $, url, helpers }) => {
|
||||
// Customize selectors or meta extraction as needed. Optional
|
||||
const language = $("html").attr("lang") || "en";
|
||||
|
||||
return helpers.splitTextIntoRecords({
|
||||
text: helpers.markdown("main"), // Change "main" to match your content tag (e.g., "main", "article", etc.)
|
||||
baseRecord: {
|
||||
objectID: url,
|
||||
// Add more optional attributes to the record
|
||||
lang: language
|
||||
},
|
||||
maxRecordBytes: 100000, // Higher = fewer, larger records. Lower = more, smaller records.
|
||||
// Note: Increasing this value may increase the token count for LLMs, which can affect context size and cost.
|
||||
orderingAttributeName: "part",
|
||||
});
|
||||
},
|
||||
},
|
||||
// ...],
|
||||
|
||||
// initialIndexSettings: { ...,
|
||||
"my-markdown-index": {
|
||||
attributesForFaceting: ["lang"], // Recommended if you add more attributes outside of objectID
|
||||
ignorePlurals: true,
|
||||
minProximity: 4,
|
||||
removeStopWords: true,
|
||||
searchableAttributes: ["unordered(title)", "unordered(text)"],
|
||||
},
|
||||
// ...},
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="docusaurus" label="Docusaurus">
|
||||
|
||||
**Docusaurus Example:**
|
||||
|
||||
```js
|
||||
// In your Crawler config:
|
||||
|
||||
// actions: [ ...,
|
||||
{
|
||||
indexName: "my-markdown-index",
|
||||
pathsToMatch: ["https://example.com/docs/**"],
|
||||
recordExtractor: ({ $, url, helpers }) => {
|
||||
// Extract meta tag values. These are required for Docusaurus
|
||||
const language =
|
||||
$('meta[name="docsearch:language"]').attr("content") || "en";
|
||||
const version =
|
||||
$('meta[name="docsearch:version"]').attr("content") || "latest";
|
||||
const docusaurus_tag =
|
||||
$('meta[name="docsearch:docusaurus_tag"]').attr("content") || "";
|
||||
|
||||
return helpers.splitTextIntoRecords({
|
||||
text: helpers.markdown("main"), // Change "main" to match your content tag (e.g., "main", "article", etc.)
|
||||
baseRecord: {
|
||||
objectID: url,
|
||||
lang: language, // Required for Docusaurus
|
||||
language, // Required for Docusaurus
|
||||
version: version.split(","), // in case there are multiple versions. Required for Docusaurus
|
||||
docusaurus_tag: docusaurus_tag // Required for Docusaurus
|
||||
.split(",")
|
||||
.map((tag) => tag.trim())
|
||||
.filter(Boolean),
|
||||
},
|
||||
maxRecordBytes: 100000, // Higher = fewer, larger records. Lower = more, smaller records.
|
||||
// Note: Increasing this value may increase the token count for LLMs, which can affect context size and cost.
|
||||
orderingAttributeName: "part",
|
||||
});
|
||||
},
|
||||
},
|
||||
// ...],
|
||||
|
||||
// initialIndexSettings: { ...,
|
||||
"my-markdown-index": {
|
||||
attributesForFaceting: ["lang", "language", "version", "docusaurus_tag"], // Required for Docusaurus
|
||||
ignorePlurals: true,
|
||||
minProximity: 4,
|
||||
removeStopWords: true,
|
||||
searchableAttributes: ["unordered(title)", "unordered(text)"],
|
||||
},
|
||||
// ...},
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="vitepress" label="VitePress">
|
||||
|
||||
**VitePress Example:**
|
||||
|
||||
```js
|
||||
// In your Crawler config:
|
||||
|
||||
// actions: [ ...,
|
||||
{
|
||||
indexName: "my-markdown-index",
|
||||
pathsToMatch: ["https://example.com/docs/**"],
|
||||
recordExtractor: ({ $, url, helpers }) => {
|
||||
// Extract meta tag values. These are required for VitePress
|
||||
const language = $("html").attr("lang") || "en";
|
||||
|
||||
|
||||
return helpers.splitTextIntoRecords({
|
||||
text: helpers.markdown("main"), // Change "main" to match your content tag (e.g., "main", "article", etc.)
|
||||
baseRecord: {
|
||||
objectID: url,
|
||||
lang: language, // Required for VitePress
|
||||
},
|
||||
maxRecordBytes: 100000, // Higher = fewer, larger records. Lower = more, smaller records.
|
||||
// Note: Increasing this value may increase the token count for LLMs, which can affect context size and cost.
|
||||
orderingAttributeName: "part",
|
||||
});
|
||||
},
|
||||
},
|
||||
// ...],
|
||||
|
||||
// initialIndexSettings: { ...,
|
||||
"my-markdown-index": {
|
||||
attributesForFaceting: ["lang"], // Required for VitePress
|
||||
ignorePlurals: true,
|
||||
minProximity: 4,
|
||||
removeStopWords: true,
|
||||
searchableAttributes: ["unordered(title)", "unordered(text)"],
|
||||
},
|
||||
// ...},
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="astro" label="Astro / Starlight">
|
||||
|
||||
**Astro / Starlight Example:**
|
||||
|
||||
```js
|
||||
// In your Crawler config:
|
||||
|
||||
// actions: [ ...,
|
||||
{
|
||||
indexName: "my-markdown-index",
|
||||
pathsToMatch: ["https://example.com/docs/**"],
|
||||
recordExtractor: ({ $, url, helpers }) => {
|
||||
// Extract meta tag values. These are required for Astro/StarLight
|
||||
const language = $("html").attr("lang") || "en";
|
||||
|
||||
|
||||
return helpers.splitTextIntoRecords({
|
||||
text: helpers.markdown("main"), // Change "main" to match your content tag (e.g., "main", "article", etc.)
|
||||
baseRecord: {
|
||||
objectID: url,
|
||||
lang: language, // Required for Astro/StarLight
|
||||
},
|
||||
maxRecordBytes: 100000, // Higher = fewer, larger records. Lower = more, smaller records.
|
||||
// Note: Increasing this value may increase the token count for LLMs, which can affect context size and cost.
|
||||
orderingAttributeName: "part",
|
||||
});
|
||||
},
|
||||
},
|
||||
// ...],
|
||||
|
||||
// initialIndexSettings: { ...,
|
||||
"my-markdown-index": {
|
||||
attributesForFaceting: ["lang"], // Required for Astro/StarLight
|
||||
ignorePlurals: true,
|
||||
minProximity: 4,
|
||||
removeStopWords: true,
|
||||
searchableAttributes: ["unordered(title)", "unordered(text)"],
|
||||
},
|
||||
// ...},
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
</Tabs>
|
||||
|
||||
> Each example shows how to extract common facets and configure your markdown index for AskAI. Adjust selectors and meta tag names as needed for your site.
|
||||
|
|
@ -65,7 +65,10 @@ export default {
|
|||
placeholder: 'Search or ask AI',
|
||||
appId: 'PMZUYBQDAK',
|
||||
apiKey: '24b09689d5b4223813d9b8e48563c8f6',
|
||||
askAi: 'askAIDemo',
|
||||
askAi: {
|
||||
indexName: 'docsearch-markdown',
|
||||
assistantId: 'askAIDemo',
|
||||
},
|
||||
indexName: 'docsearch',
|
||||
contextualSearch: true,
|
||||
translations: {
|
||||
|
|
|
|||
|
|
@ -24,7 +24,13 @@ export default {
|
|||
{
|
||||
type: 'category',
|
||||
label: 'Algolia AskAI - Beta',
|
||||
items: ['v4/askai', 'v4/askai-prompts', 'v4/askai-whitelisted-domains', 'v4/askai-models'],
|
||||
items: [
|
||||
'v4/askai',
|
||||
'v4/askai-prompts',
|
||||
'v4/askai-whitelisted-domains',
|
||||
'v4/askai-models',
|
||||
'v4/askai-markdown-indexing',
|
||||
],
|
||||
},
|
||||
{
|
||||
type: 'category',
|
||||
|
|
|
|||
Loading…
Reference in a new issue