astro/astro-discovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge reference documentation

Browse Source 
Complete API reference for all configuration options
- 11 reference pages covering entire API surface
- TypeScript type definitions
- Default values and examples
- RFC compliance documentation
This commit is contained in:
Ryan Malloy 2025-11-08 23:40:30 -07:00
commit 07ce65cf9e
11 changed files with 3132 additions and 223 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

304
docs/src/content/docs/reference/api.md
View File

@ -1,31 +1,295 @@
---
title: API Reference
description: API and programmatic interface reference
description: Programmatic API reference
---
Complete API reference for programmatic usage of @astrojs/discovery.
Programmatic API reference for `@astrojs/discovery`.
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## Integration Function
## Coming Soon
### `discovery(config?)`
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
Main integration function for Astro.
## Related Pages
```typescript
function discovery(config?: DiscoveryConfig): AstroIntegration
```
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
**Parameters:**
- `config` (optional): Discovery configuration object
## Need Help?
**Returns:** Astro integration object
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
**Example:**
```typescript
import { defineConfig } from 'astro/config';
import discovery from '@astrojs/discovery';
export default defineConfig({
site: 'https://example.com',
integrations: [
discovery({
robots: { crawlDelay: 2 },
llms: { description: 'My site' }
})
]
});
```
## Generator Functions
These functions are used internally but can be imported for custom usage.
### `generateRobotsTxt(config, siteURL)`
Generate robots.txt content.
```typescript
function generateRobotsTxt(
config: RobotsConfig,
siteURL: URL
): string
```
**Example:**
```typescript
import { generateRobotsTxt } from '@astrojs/discovery/generators';
const robotsTxt = generateRobotsTxt(
{ crawlDelay: 2, allowAllBots: true },
new URL('https://example.com')
);
```
### `generateLLMsTxt(config, siteURL)`
Generate llms.txt content (async).
```typescript
function generateLLMsTxt(
config: LLMsConfig,
siteURL: URL
): Promise<string>
```
**Example:**
```typescript
import { generateLLMsTxt } from '@astrojs/discovery/generators';
const llmsTxt = await generateLLMsTxt(
{ description: 'My site', keyFeatures: ['Feature 1'] },
new URL('https://example.com')
);
```
### `generateHumansTxt(config)`
Generate humans.txt content.
```typescript
function generateHumansTxt(
config: HumansConfig
): string
```
**Example:**
```typescript
import { generateHumansTxt } from '@astrojs/discovery/generators';
const humansTxt = generateHumansTxt({
team: [{ name: 'Alice', role: 'Developer' }]
});
```
### `generateSecurityTxt(config, siteURL)`
Generate security.txt content.
```typescript
function generateSecurityTxt(
config: SecurityConfig,
siteURL: URL
): string
```
**Example:**
```typescript
import { generateSecurityTxt } from '@astrojs/discovery/generators';
const securityTxt = generateSecurityTxt(
{ contact: 'security@example.com', expires: 'auto' },
new URL('https://example.com')
);
```
### `generateCanaryTxt(config, siteURL)`
Generate canary.txt content.
```typescript
function generateCanaryTxt(
config: CanaryConfig,
siteURL: URL
): string
```
**Example:**
```typescript
import { generateCanaryTxt } from '@astrojs/discovery/generators';
const canaryTxt = generateCanaryTxt(
{ organization: 'Example Corp', frequency: 'monthly' },
new URL('https://example.com')
);
```
### `generateWebFingerJRD(config, resource, rels, siteURL, getCollectionData?)`
Generate WebFinger JRD response (async).
```typescript
function generateWebFingerJRD(
config: WebFingerConfig,
requestedResource: string,
requestedRels: string[] | undefined,
siteURL: URL,
getCollectionData?: (collectionName: string) => Promise<any[]>
): Promise<string | null>
```
**Returns:** JRD JSON string or `null` if resource not found
## Validator Functions
### `validateConfig(userConfig)`
Validate and merge configuration with defaults.
```typescript
function validateConfig(
userConfig?: DiscoveryConfig
): DiscoveryConfig
```
**Example:**
```typescript
import { validateConfig } from '@astrojs/discovery/validators';
const config = validateConfig({
robots: { crawlDelay: 2 }
});
// Returns merged config with defaults
```
## Default Values
### Default Robots Config
```typescript
const DEFAULT_ROBOTS_CONFIG = {
enabled: true,
crawlDelay: 1,
allowAllBots: true,
llmBots: {
enabled: true
}
};
```
### Default LLM Bots
```typescript
const DEFAULT_LLM_BOTS = [
'Anthropic-AI',
'Claude-Web',
'GPTBot',
'ChatGPT-User',
'cohere-ai',
'Google-Extended',
'PerplexityBot',
'Applebot-Extended'
];
```
### Default Cache Durations
```typescript
const DEFAULT_CACHING_CONFIG = {
robots: 3600, // 1 hour
llms: 3600, // 1 hour
humans: 86400, // 24 hours
security: 86400, // 24 hours
canary: 3600, // 1 hour
webfinger: 3600, // 1 hour
sitemap: 3600 // 1 hour
};
```
## Custom Templates
You can provide custom template functions:
```typescript
discovery({
templates: {
robots: (config, siteURL) => {
return `User-agent: *\nAllow: /\nSitemap: ${siteURL}/sitemap.xml`;
},
llms: async (config, siteURL) => {
const content = await fetchDynamicContent();
return `# ${siteURL.hostname}\n\n${content}`;
},
humans: (config, siteURL) => {
return `/* TEAM */\n\n Developer: ${config.team[0].name}`;
},
security: (config, siteURL) => {
return `Contact: ${config.contact}\nExpires: ${config.expires}`;
},
canary: (config, siteURL) => {
return `Organization: ${config.organization}\nIssued: ${new Date().toISOString()}`;
}
}
})
```
## Programmatic File Generation
Generate files programmatically:
```typescript
import {
generateRobotsTxt,
generateLLMsTxt,
generateHumansTxt,
generateSecurityTxt,
generateCanaryTxt
} from '@astrojs/discovery/generators';
const siteURL = new URL('https://example.com');
// Generate all files
const robots = generateRobotsTxt({ crawlDelay: 2 }, siteURL);
const llms = await generateLLMsTxt({ description: 'My site' }, siteURL);
const humans = generateHumansTxt({ team: [{ name: 'Alice' }] });
const security = generateSecurityTxt({ contact: 'security@example.com' }, siteURL);
const canary = generateCanaryTxt({ organization: 'Example' }, siteURL);
// Write to files
await fs.writeFile('public/robots.txt', robots);
await fs.writeFile('public/llms.txt', llms);
await fs.writeFile('public/humans.txt', humans);
await fs.writeFile('public/.well-known/security.txt', security);
await fs.writeFile('public/.well-known/canary.txt', canary);
```
## Notes
- All generator functions are pure and have no side effects
- Validation happens automatically when using the integration
- Custom templates override default generators
- All async functions return Promises
- WebFinger is dynamic and requires runtime query handling

187
docs/src/content/docs/reference/cache.md
View File

@ -1,31 +1,178 @@
---
title: Cache Options
title: Cache Configuration
description: HTTP caching configuration reference
---
Configure cache control headers for all discovery files.
Configure HTTP cache control headers for all discovery files.
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## CachingConfig
## Coming Soon
```typescript
interface CachingConfig {
robots?: number;
llms?: number;
humans?: number;
security?: number;
canary?: number;
webfinger?: number;
sitemap?: number;
}
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Properties
## Related Pages
All properties are cache durations in seconds.
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
| Property | Default | Description |
|----------|---------|-------------|
| `robots` | `3600` | robots.txt cache duration (1 hour) |
| `llms` | `3600` | llms.txt cache duration (1 hour) |
| `humans` | `86400` | humans.txt cache duration (24 hours) |
| `security` | `86400` | security.txt cache duration (24 hours) |
| `canary` | `3600` | canary.txt cache duration (1 hour) |
| `webfinger` | `3600` | WebFinger cache duration (1 hour) |
| `sitemap` | `3600` | Sitemap cache duration (1 hour) |
## Need Help?
**Valid range:** 0 to 31536000 seconds (1 year)
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
## Examples
### Custom cache durations
```typescript
discovery({
caching: {
robots: 7200, // 2 hours
llms: 1800, // 30 minutes
humans: 172800, // 48 hours
security: 43200, // 12 hours
canary: 1800, // 30 minutes (check frequently)
webfinger: 7200, // 2 hours
sitemap: 3600 // 1 hour
}
})
```
### Aggressive caching
```typescript
discovery({
caching: {
robots: 86400, // 24 hours
llms: 86400, // 24 hours
humans: 604800, // 1 week
security: 604800 // 1 week
}
})
```
### Minimal caching (development)
```typescript
discovery({
caching: {
robots: 60, // 1 minute
llms: 60, // 1 minute
humans: 300, // 5 minutes
security: 300, // 5 minutes
canary: 60 // 1 minute
}
})
```
### No caching
```typescript
discovery({
caching: {
robots: 0,
llms: 0,
humans: 0,
security: 0,
canary: 0,
webfinger: 0,
sitemap: 0
}
})
```
## Common Patterns
### Environment-based caching
```typescript
discovery({
caching: {
robots: import.meta.env.PROD ? 3600 : 60,
llms: import.meta.env.PROD ? 3600 : 60,
humans: import.meta.env.PROD ? 86400 : 300
}
})
```
### Canary-focused caching
```typescript
discovery({
caching: {
canary: 1800, // 30 minutes - check frequently
security: 86400, // 24 hours - stable
humans: 604800, // 1 week - rarely changes
robots: 3600, // 1 hour - moderate
llms: 3600 // 1 hour - moderate
}
})
```
## Cache-Control Headers
The integration sets these HTTP headers:
```
Cache-Control: public, max-age={duration}
```
Where `{duration}` is the configured cache duration in seconds.
### Example Response
```http
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Cache-Control: public, max-age=3600
```
## Best Practices
1. **Balance freshness vs load:** Longer caches reduce server load but may show stale content
2. **Canary should be short:** Check warrant canaries frequently (30 min to 1 hour)
3. **Humans.txt can be long:** Team info rarely changes (24 hours to 1 week)
4. **Security.txt moderate:** Balance between updates and load (12-24 hours)
5. **Development vs production:** Use short caches in dev, longer in prod
6. **Consider your update frequency:** Match cache to actual content update rate
## Time Conversion Reference
| Duration | Seconds |
|----------|---------|
| 1 minute | 60 |
| 5 minutes | 300 |
| 15 minutes | 900 |
| 30 minutes | 1800 |
| 1 hour | 3600 |
| 2 hours | 7200 |
| 6 hours | 21600 |
| 12 hours | 43200 |
| 24 hours (1 day) | 86400 |
| 48 hours (2 days) | 172800 |
| 1 week | 604800 |
| 1 month (30 days) | 2592000 |
| 1 year | 31536000 |
## Notes
- Caching is applied via `Cache-Control` headers
- CDNs and browsers respect these directives
- Set to `0` to disable caching
- Maximum allowed: 31536000 seconds (1 year)
- Validation warning if outside 0-31536000 range

335
docs/src/content/docs/reference/canary.md
View File

@ -1,31 +1,324 @@
---
title: canary.txt Options
description: Configuration reference for canary.txt
title: canary.txt Configuration
description: Configuration reference for canary.txt (warrant canary)
---
Complete reference for warrant canary configuration options.
Configuration reference for `/.well-known/canary.txt` generation.
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## CanaryConfig
## Coming Soon
```typescript
interface CanaryConfig {
enabled?: boolean;
organization?: string;
contact?: string;
frequency?: 'daily' | 'weekly' | 'monthly' | 'quarterly' | 'yearly';
expires?: string | 'auto';
statements?: CanaryStatement[] | (() => CanaryStatement[]);
additionalStatement?: string;
verification?: string;
previousCanary?: string;
blockchainProof?: {
network: string;
address: string;
txHash?: string;
timestamp?: string;
};
personnelStatement?: boolean;
}
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Properties
## Related Pages
### enabled
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
- **Type:** `boolean`
- **Default:** `true`
- **Description:** Enable or disable canary.txt generation
## Need Help?
### organization
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
- **Type:** `string`
- **Default:** `undefined`
- **Description:** Organization name
### contact
- **Type:** `string`
- **Default:** `undefined`
- **Description:** Contact email for canary inquiries
- **Note:** Email addresses automatically get `mailto:` prefix
### frequency
- **Type:** `'daily' | 'weekly' | 'monthly' | 'quarterly' | 'yearly'`
- **Default:** `'monthly'`
- **Description:** How often the canary is updated
**Auto-calculated expiration based on frequency:**
- `daily`: 2 days
- `weekly`: 10 days
- `monthly`: 35 days
- `quarterly`: 100 days
- `yearly`: 380 days
### expires
- **Type:** `string | 'auto'`
- **Default:** `'auto'` (calculated from frequency)
- **Description:** Expiration date in ISO 8601 format
### statements
- **Type:** `CanaryStatement[] | (() => CanaryStatement[])`
- **Default:** Default statements (NSL, FISA, gag orders, surveillance, backdoors)
- **Description:** Statements about what has NOT been received
**CanaryStatement interface:**
```typescript
interface CanaryStatement {
type: 'nsl' | 'fisa' | 'gag' | 'surveillance' | 'backdoor' | 'encryption' | 'other';
description: string;
received: boolean;
}
```
**Important:** Only statements with `received: false` appear in the canary.
**Default statements:**
```typescript
[
{ type: 'nsl', description: 'National Security Letters (NSLs)', received: false },
{ type: 'fisa', description: 'FISA court orders', received: false },
{ type: 'gag', description: 'Gag orders preventing disclosure', received: false },
{ type: 'surveillance', description: 'Secret government requests for user data', received: false },
{ type: 'backdoor', description: 'Requests to install surveillance capabilities', received: false }
]
```
**Custom statements:**
```typescript
discovery({
canary: {
organization: 'Example Corp',
statements: [
{ type: 'nsl', description: 'National Security Letters', received: false },
{ type: 'gag', description: 'Gag orders', received: false },
{ type: 'other', description: 'Requests for user encryption keys', received: false }
]
}
})
```
**Dynamic statements:**
```typescript
discovery({
canary: {
organization: 'Example Corp',
statements: () => {
// Generate statements dynamically at build time
return [
{ type: 'nsl', description: 'NSLs', received: false },
{ type: 'gag', description: 'Gag orders', received: false }
];
}
}
})
```
### additionalStatement
- **Type:** `string`
- **Default:** `undefined`
- **Description:** Additional free-form statement text
### verification
- **Type:** `string`
- **Default:** `undefined`
- **Description:** URL to PGP signature or other verification method
### previousCanary
- **Type:** `string`
- **Default:** `undefined`
- **Description:** URL to previous canary for continuity verification
### blockchainProof
- **Type:** `{ network: string; address: string; txHash?: string; timestamp?: string }`
- **Default:** `undefined`
- **Description:** Blockchain proof for tamper-evident verification
**Example:**
```typescript
discovery({
canary: {
organization: 'Example Corp',
blockchainProof: {
network: 'Ethereum',
address: '0x1234...5678',
txHash: '0xabcd...ef01',
timestamp: '2025-11-08T12:00:00Z'
}
}
})
```
### personnelStatement
- **Type:** `boolean`
- **Default:** `false`
- **Description:** Include statement about key personnel being free from duress
**Generated text when `true`:**
```
Key Personnel Statement: All key personnel with access to
infrastructure remain free and under no duress.
```
## Generated Output
**Minimal configuration:**
```typescript
discovery({
canary: {
organization: 'Example Corp',
contact: 'canary@example.com'
}
})
```
**Output:**
```
Canonical-URL: https://example.com/.well-known/canary.txt
Issued: 2025-11-08T12:00:00.000Z
Expires: 2025-12-13T12:00:00.000Z
Organization: Example Corp
Contact: mailto:canary@example.com
Frequency: monthly
Statement: As of 2025-11-08, Example Corp has NOT received:
- National Security Letters (NSLs)
- FISA court orders
- Gag orders preventing disclosure
- Secret government requests for user data
- Requests to install surveillance capabilities
This canary will be updated monthly. Absence of an update
within 35 days should be considered significant.
---
This warrant canary follows the proposed canary.txt specification.
See: https://github.com/withastro/astro-discovery/blob/main/CANARY_SPEC.md
```
**Full configuration:**
```typescript
discovery({
canary: {
organization: 'Example Corp',
contact: 'canary@example.com',
frequency: 'weekly',
statements: [
{ type: 'nsl', description: 'National Security Letters', received: false },
{ type: 'gag', description: 'Gag orders', received: false }
],
additionalStatement: 'We are committed to transparency and user privacy.',
verification: 'https://example.com/canary.txt.asc',
previousCanary: 'https://example.com/canary/2025-11-01.txt',
blockchainProof: {
network: 'Ethereum',
address: '0x1234567890abcdef',
txHash: '0xabcdef1234567890'
},
personnelStatement: true
}
})
```
## Common Patterns
### Monthly canary
```typescript
discovery({
canary: {
organization: 'Example Corp',
contact: 'canary@example.com',
frequency: 'monthly'
}
})
```
### With blockchain verification
```typescript
discovery({
canary: {
organization: 'Example Corp',
frequency: 'monthly',
blockchainProof: {
network: 'Bitcoin',
address: 'bc1q...',
txHash: process.env.CANARY_TX_HASH
}
}
})
```
### With PGP signature
```typescript
discovery({
canary: {
organization: 'Example Corp',
contact: 'canary@example.com',
frequency: 'monthly',
verification: 'https://example.com/canary.txt.asc',
previousCanary: 'https://example.com/canary/previous.txt'
}
})
```
### Custom statements only
```typescript
discovery({
canary: {
organization: 'Example Corp',
frequency: 'quarterly',
statements: [
{ type: 'other', description: 'Demands for customer data', received: false },
{ type: 'other', description: 'Requests to weaken encryption', received: false }
],
personnelStatement: true
}
})
```
## Best Practices
1. **Update regularly:** Match your actual update capability to frequency setting
2. **Verify authenticity:** Use PGP signatures or blockchain proofs
3. **Maintain continuity:** Link to previous canaries
4. **Be specific:** Customize statements to your threat model
5. **Automate updates:** Integrate into CI/CD to ensure regular updates
6. **Monitor expiration:** Set alerts before expiration dates
7. **Document process:** Make canary updates part of your security procedures
## Important Notes
1. **Legal implications:** Warrant canaries have unclear legal status in some jurisdictions
2. **Absence is significant:** A missing or expired canary may signal issues
3. **Not a guarantee:** Canaries can be compelled to continue
4. **Supplement, don't replace:** Use alongside other transparency measures
5. **Update discipline:** Missing an update defeats the purpose
## Output Location
- **File:** `/.well-known/canary.txt`
- **URL:** `https://example.com/.well-known/canary.txt`
- **Cache-Control:** `public, max-age=3600` (1 hour, configurable via [caching](/reference/cache/))
- **Specification:** See [CANARY_SPEC.md](https://github.com/withastro/astro-discovery/blob/main/CANARY_SPEC.md)

101
docs/src/content/docs/reference/configuration.md
View File

@ -3,29 +3,92 @@ title: Configuration Options
description: Complete reference for all configuration options
---
Comprehensive reference documentation for all available configuration options.
Complete configuration reference for the `@astrojs/discovery` integration.
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## DiscoveryConfig
## Coming Soon
Main configuration interface passed to the `discovery()` integration function.
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
```typescript
interface DiscoveryConfig {
robots?: RobotsConfig;
llms?: LLMsConfig;
humans?: HumansConfig;
security?: SecurityConfig;
canary?: CanaryConfig;
webfinger?: WebFingerConfig;
sitemap?: SitemapConfig;
caching?: CachingConfig;
templates?: TemplateConfig;
}
```
## Related Pages
### Type Parameters
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
All configuration sections are optional. If omitted, defaults are used.
## Need Help?
| Property | Type | Required | Default |
|----------|------|----------|---------|
| `robots` | `RobotsConfig` | No | `{ enabled: true, crawlDelay: 1, allowAllBots: true }` |
| `llms` | `LLMsConfig` | No | `{ enabled: true }` |
| `humans` | `HumansConfig` | No | `{ enabled: true }` |
| `security` | `SecurityConfig` | No | `undefined` (disabled) |
| `canary` | `CanaryConfig` | No | `undefined` (disabled) |
| `webfinger` | `WebFingerConfig` | No | `undefined` (disabled) |
| `sitemap` | `SitemapConfig` | No | `{}` |
| `caching` | `CachingConfig` | No | See [Cache Reference](/reference/cache/) |
| `templates` | `TemplateConfig` | No | `undefined` |
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
## Complete Example
```typescript
import { defineConfig } from 'astro/config';
import discovery from '@astrojs/discovery';
export default defineConfig({
site: 'https://example.com',
integrations: [
discovery({
robots: {
crawlDelay: 2,
allowAllBots: true,
llmBots: {
enabled: true,
agents: ['CustomBot']
}
},
llms: {
description: 'Site description',
keyFeatures: ['Feature 1', 'Feature 2']
},
humans: {
team: [
{ name: 'Developer', role: 'Creator' }
]
},
security: {
contact: 'security@example.com',
expires: 'auto'
},
caching: {
robots: 3600,
llms: 1800
}
})
]
});
```
## Configuration Sections
Detailed configuration for each section:
- [Robots Configuration](/reference/robots/) - robots.txt generation
- [LLMs Configuration](/reference/llms/) - llms.txt generation
- [Humans Configuration](/reference/humans/) - humans.txt generation
- [Security Configuration](/reference/security/) - security.txt generation
- [Canary Configuration](/reference/canary/) - canary.txt generation
- [WebFinger Configuration](/reference/webfinger/) - WebFinger discovery
- [Sitemap Configuration](/reference/sitemap/) - Sitemap generation
- [Cache Configuration](/reference/cache/) - HTTP caching
- [TypeScript Types](/reference/typescript/) - Complete type definitions

374
docs/src/content/docs/reference/humans.md
View File

@ -1,31 +1,363 @@
---
title: humans.txt Options
description: Configuration reference for humans.txt
title: humans.txt Configuration
description: Configuration reference for humans.txt generation
---
Full reference for humans.txt configuration and formatting options.
Configuration reference for `/humans.txt` generation.
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## HumansConfig
## Coming Soon
```typescript
interface HumansConfig {
enabled?: boolean;
team?: TeamMember[];
thanks?: string[];
site?: SiteInfo;
story?: string;
funFacts?: string[];
philosophy?: string[];
customSections?: Record<string, string>;
}
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Properties
## Related Pages
### enabled
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
- **Type:** `boolean`
- **Default:** `true`
- **Description:** Enable or disable humans.txt generation
## Need Help?
### team
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
- **Type:** `TeamMember[]`
- **Default:** `undefined`
- **Description:** Team members who built the site
**TeamMember interface:**
```typescript
interface TeamMember {
name: string;
role?: string;
contact?: string;
location?: string;
twitter?: string;
github?: string;
}
```
**Example:**
```typescript
discovery({
humans: {
team: [
{
name: 'Alice Developer',
role: 'Lead Developer',
contact: 'alice@example.com',
location: 'New York, NY',
twitter: '@alice_dev',
github: 'alice-dev'
},
{
name: 'Bob Designer',
role: 'UI/UX Designer',
contact: 'bob@example.com',
location: 'San Francisco, CA'
}
]
}
})
```
### thanks
- **Type:** `string[]`
- **Default:** `undefined`
- **Description:** Thank you notes and acknowledgments
**Example:**
```typescript
discovery({
humans: {
thanks: [
'The Astro team for an amazing framework',
'Our amazing community contributors',
'Stack Overflow (obviously)',
'Coffee, lots of coffee'
]
}
})
```
### site
- **Type:** `SiteInfo`
- **Default:** `undefined`
- **Description:** Site technical information
**SiteInfo interface:**
```typescript
interface SiteInfo {
lastUpdate?: string | 'auto';
language?: string;
doctype?: string;
ide?: string;
techStack?: string[];
standards?: string[];
components?: string[];
software?: string[];
}
```
**Example:**
```typescript
discovery({
humans: {
site: {
lastUpdate: 'auto', // Auto-uses current date
language: 'English',
doctype: 'HTML5',
ide: 'VS Code',
techStack: ['Astro', 'TypeScript', 'React', 'Tailwind CSS'],
standards: ['HTML5', 'CSS3', 'ES2023', 'WCAG 2.1'],
components: ['@astrojs/react', '@astrojs/tailwind'],
software: ['Docker', 'GitHub Actions', 'Vercel']
}
}
})
```
**Note:** `lastUpdate: 'auto'` automatically uses the current build date.
### story
- **Type:** `string`
- **Default:** `undefined`
- **Description:** Project story or history (multi-line supported)
**Example:**
```typescript
discovery({
humans: {
story: `
This project started when we realized there was no good way to track
sustainable products online. After months of research and development,
we built a platform that not only helps consumers make better choices
but also rewards companies for their environmental efforts.
Built with love during nights and weekends over 6 months.
`.trim()
}
})
```
### funFacts
- **Type:** `string[]`
- **Default:** `undefined`
- **Description:** Fun facts about the project
**Example:**
```typescript
discovery({
humans: {
funFacts: [
'Built entirely on a mechanical keyboard',
'Fueled by 347 cups of coffee',
'Started at a 48-hour hackathon',
'The first commit was on a flight to Tokyo',
'Tested by 1,000+ beta users before launch'
]
}
})
```
### philosophy
- **Type:** `string[]`
- **Default:** `undefined`
- **Description:** Development philosophy statements
**Example:**
```typescript
discovery({
humans: {
philosophy: [
'Simple is better than complex',
'Make it work, make it right, make it fast',
'User experience over developer convenience',
'Open source by default',
'Leave the web better than we found it'
]
}
})
```
### customSections
- **Type:** `Record<string, string>`
- **Default:** `undefined`
- **Description:** Custom sections to add
**Example:**
```typescript
discovery({
humans: {
customSections: {
'CREDITS': `
Special thanks to:
- Photography by Jane Smith
- Icons by FontAwesome
- Illustrations by UnDraw
`.trim(),
'CONTACT': `
Questions or feedback?
Email us at: hello@example.com
`.trim()
}
}
})
```
## Generated Output Structure
```
/* TEAM */
Name: Alice Developer
Role: Lead Developer
Contact: alice@example.com
From: New York, NY
Twitter: @alice_dev
GitHub: alice-dev
Name: Bob Designer
Role: UI/UX Designer
Contact: bob@example.com
From: San Francisco, CA
/* THANKS */
The Astro team for an amazing framework
Our amazing community contributors
Stack Overflow (obviously)
Coffee, lots of coffee
/* SITE */
Last update: 2025-11-08
Language: English
Doctype: HTML5
IDE: VS Code
Tech Stack: Astro, TypeScript, React, Tailwind CSS
Standards: HTML5, CSS3, ES2023, WCAG 2.1
Components: @astrojs/react, @astrojs/tailwind
Software: Docker, GitHub Actions, Vercel
/* THE STORY */
This project started when we realized there was no good way to track
sustainable products online. After months of research and development,
we built a platform that not only helps consumers make better choices
but also rewards companies for their environmental efforts.
Built with love during nights and weekends over 6 months.
/* FUN FACTS */
Built entirely on a mechanical keyboard
Fueled by 347 cups of coffee
Started at a 48-hour hackathon
/* PHILOSOPHY */
"Simple is better than complex"
"Make it work, make it right, make it fast"
"User experience over developer convenience"
/* CUSTOM SECTION */
Custom content here
```
## Common Patterns
### Minimal team information
```typescript
discovery({
humans: {
team: [
{
name: 'Development Team',
contact: 'dev@example.com'
}
],
thanks: [
'Open source community',
'Early adopters'
]
}
})
```
### Full site details
```typescript
discovery({
humans: {
team: [
{ name: 'Alice', role: 'Developer', github: 'alice' },
{ name: 'Bob', role: 'Designer', twitter: '@bob' }
],
site: {
lastUpdate: 'auto',
language: 'English',
doctype: 'HTML5',
techStack: ['Astro', 'React', 'TypeScript'],
standards: ['HTML5', 'WCAG 2.1']
},
story: 'Built with passion over 6 months',
funFacts: [
'First commit was on a plane',
'500+ cups of coffee consumed'
]
}
})
```
### Solo developer
```typescript
discovery({
humans: {
team: [
{
name: 'Jane Hacker',
role: 'Solo Developer & Designer',
contact: 'jane@example.com',
location: 'Remote',
github: 'jane-hacker'
}
],
thanks: [
'My cat for moral support',
'Stack Overflow',
'The Astro community'
],
philosophy: [
'Ship it',
'Iterate quickly',
'Listen to users'
]
}
})
```
## Output Location
- **File:** `/humans.txt`
- **URL:** `https://example.com/humans.txt`
- **Cache-Control:** `public, max-age=86400` (24 hours, configurable via [caching](/reference/cache/))

413
docs/src/content/docs/reference/llms.md
View File

@ -1,31 +1,402 @@
---
title: llms.txt Options
description: Configuration reference for llms.txt
title: llms.txt Configuration
description: Configuration reference for llms.txt generation
---
Complete reference for llms.txt configuration options and structure.
Configuration reference for `/llms.txt` generation.
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## LLMsConfig
## Coming Soon
```typescript
interface LLMsConfig {
enabled?: boolean;
description?: string | (() => string);
keyFeatures?: string[];
importantPages?: ImportantPage[] | (() => Promise<ImportantPage[]>);
instructions?: string;
apiEndpoints?: APIEndpoint[];
techStack?: TechStack;
brandVoice?: string[];
customSections?: Record<string, string>;
}
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Properties
## Related Pages
### enabled
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
- **Type:** `boolean`
- **Default:** `true`
- **Description:** Enable or disable llms.txt generation
## Need Help?
```typescript
discovery({
llms: {
enabled: false // Disable llms.txt
}
})
```
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
### description
- **Type:** `string | (() => string)`
- **Default:** `undefined`
- **Description:** Site description for AI assistants (can be dynamic function)
**Static example:**
```typescript
discovery({
llms: {
description: 'E-commerce platform for sustainable products'
}
})
```
**Dynamic example:**
```typescript
discovery({
llms: {
description: () => {
const pkg = JSON.parse(fs.readFileSync('./package.json', 'utf-8'));
return `${pkg.name} - ${pkg.description}`;
}
}
})
```
### keyFeatures
- **Type:** `string[]`
- **Default:** `undefined`
- **Description:** Key features of the site
**Example:**
```typescript
discovery({
llms: {
keyFeatures: [
'AI-powered product recommendations',
'Carbon footprint calculator',
'Subscription management',
'Real-time inventory tracking'
]
}
})
```
### importantPages
- **Type:** `ImportantPage[] | (() => Promise<ImportantPage[]>)`
- **Default:** `undefined`
- **Description:** Important pages for AI to know about
**ImportantPage interface:**
```typescript
interface ImportantPage {
name: string;
path: string;
description?: string;
}
```
**Static example:**
```typescript
discovery({
llms: {
importantPages: [
{
name: 'API Documentation',
path: '/docs/api',
description: 'Complete API reference'
},
{
name: 'Getting Started',
path: '/docs/getting-started'
}
]
}
})
```
**Dynamic example with content collections:**
```typescript
import { getCollection } from 'astro:content';
discovery({
llms: {
importantPages: async () => {
const docs = await getCollection('docs');
return docs
.filter(doc => doc.data.featured)
.map(doc => ({
name: doc.data.title,
path: `/docs/${doc.slug}`,
description: doc.data.description
}));
}
}
})
```
### instructions
- **Type:** `string`
- **Default:** `undefined`
- **Description:** Instructions for AI assistants when helping users
**Example:**
```typescript
discovery({
llms: {
instructions: `
When helping users with this site:
1. Check API documentation first at /docs/api
2. Use the /api/search endpoint for product queries
3. Format responses in markdown
4. Include relevant links to documentation
5. Suggest sustainable alternatives when appropriate
`.trim()
}
})
```
### apiEndpoints
- **Type:** `APIEndpoint[]`
- **Default:** `undefined`
- **Description:** API endpoints available
**APIEndpoint interface:**
```typescript
interface APIEndpoint {
path: string;
method?: string; // Default: 'GET'
description: string;
}
```
**Example:**
```typescript
discovery({
llms: {
apiEndpoints: [
{
path: '/api/products',
method: 'GET',
description: 'List all products with filters'
},
{
path: '/api/search',
method: 'POST',
description: 'Search products and documentation'
},
{
path: '/api/calculate-footprint',
method: 'POST',
description: 'Calculate carbon footprint for cart'
}
]
}
})
```
### techStack
- **Type:** `TechStack`
- **Default:** `undefined`
- **Description:** Technology stack information
**TechStack interface:**
```typescript
interface TechStack {
frontend?: string[];
backend?: string[];
ai?: string[];
other?: string[];
}
```
**Example:**
```typescript
discovery({
llms: {
techStack: {
frontend: ['Astro', 'React', 'TypeScript', 'Tailwind CSS'],
backend: ['Node.js', 'PostgreSQL', 'Redis'],
ai: ['OpenAI GPT-4', 'Anthropic Claude'],
other: ['Docker', 'Stripe', 'SendGrid']
}
}
})
```
### brandVoice
- **Type:** `string[]`
- **Default:** `undefined`
- **Description:** Brand voice guidelines for AI assistants
**Example:**
```typescript
discovery({
llms: {
brandVoice: [
'Professional but friendly',
'Technical but accessible',
'Focus on sustainability',
'Use concrete examples',
'Avoid jargon when possible'
]
}
})
```
### customSections
- **Type:** `Record<string, string>`
- **Default:** `undefined`
- **Description:** Custom sections to add to llms.txt
**Example:**
```typescript
discovery({
llms: {
customSections: {
'Pricing Information': `
All products include:
- Free shipping on orders over $50
- 30-day return policy
- Carbon offset for all shipments
`.trim(),
'Support Channels': `
- Email: support@example.com
- Chat: Available 9am-5pm EST
- Forum: https://example.com/forum
`.trim()
}
}
})
```
## Generated Output Structure
```
# example.com
> Site description goes here
---
## Site Information
- **URL**: https://example.com
- **Description**: Site description
## Key Features
- Feature 1
- Feature 2
## Important Pages
- **[Page Name](https://example.com/path)**
Description of the page
## Instructions for AI Assistants
Instructions text...
## API Endpoints
- `GET /api/endpoint`
Description
Full URL: https://example.com/api/endpoint
## Technical Stack
- **Frontend**: Astro, React
- **Backend**: Node.js, PostgreSQL
- **AI/ML**: OpenAI GPT-4
## Brand Voice & Guidelines
- Guideline 1
- Guideline 2
## Custom Section Title
Custom section content...
---
**Last Updated**: 2025-11-08
*This file was generated by [@astrojs/discovery](https://github.com/withastro/astro-discovery)*
```
## Common Patterns
### Basic site information
```typescript
discovery({
llms: {
description: 'Documentation site for our API',
keyFeatures: [
'Interactive API explorer',
'Code examples in multiple languages',
'Live playground'
]
}
})
```
### With content collections
```typescript
discovery({
llms: {
importantPages: async () => {
const [docs, guides] = await Promise.all([
getCollection('docs'),
getCollection('guides')
]);
return [
...docs.map(d => ({
name: d.data.title,
path: `/docs/${d.slug}`,
description: d.data.description
})),
...guides.map(g => ({
name: g.data.title,
path: `/guides/${g.slug}`
}))
];
}
}
})
```
### Full API documentation
```typescript
discovery({
llms: {
apiEndpoints: [
{ path: '/api/users', method: 'GET', description: 'List users' },
{ path: '/api/users', method: 'POST', description: 'Create user' },
{ path: '/api/users/:id', method: 'GET', description: 'Get user' },
{ path: '/api/users/:id', method: 'PUT', description: 'Update user' },
{ path: '/api/users/:id', method: 'DELETE', description: 'Delete user' }
]
}
})
```
## Output Location
- **File:** `/llms.txt`
- **URL:** `https://example.com/llms.txt`
- **Cache-Control:** `public, max-age=3600` (1 hour, configurable via [caching](/reference/cache/))

302
docs/src/content/docs/reference/robots.md
View File

@ -1,31 +1,291 @@
---
title: robots.txt Options
description: Configuration reference for robots.txt
title: robots.txt Configuration
description: Configuration reference for robots.txt generation
---
Detailed reference for all robots.txt configuration options and behaviors.
Configuration reference for `/robots.txt` generation.
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## RobotsConfig
## Coming Soon
```typescript
interface RobotsConfig {
enabled?: boolean;
crawlDelay?: number;
allowAllBots?: boolean;
llmBots?: {
enabled?: boolean;
agents?: string[];
};
additionalAgents?: Array<{
userAgent: string;
allow?: string[];
disallow?: string[];
}>;
customRules?: string;
}
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Properties
## Related Pages
### enabled
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
- **Type:** `boolean`
- **Default:** `true`
- **Description:** Enable or disable robots.txt generation
## Need Help?
```typescript
discovery({
robots: {
enabled: false // Disable robots.txt
}
})
```
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
### crawlDelay
- **Type:** `number`
- **Default:** `1`
- **Description:** Crawl delay in seconds for polite crawlers
```typescript
discovery({
robots: {
crawlDelay: 2 // Wait 2 seconds between requests
}
})
```
### allowAllBots
- **Type:** `boolean`
- **Default:** `true`
- **Description:** Include `User-agent: *` with `Allow: /` directive
**When `true` (default):**
```
User-agent: *
Allow: /
```
**When `false`:**
```
# No default allow rule
```
```typescript
discovery({
robots: {
allowAllBots: false // No default allow
}
})
```
### llmBots
LLM-specific bot configuration.
#### llmBots.enabled
- **Type:** `boolean`
- **Default:** `true`
- **Description:** Include LLM bot section referencing `/llms.txt`
#### llmBots.agents
- **Type:** `string[]`
- **Default:**
```typescript
[
'Anthropic-AI',
'Claude-Web',
'GPTBot',
'ChatGPT-User',
'cohere-ai',
'Google-Extended',
'PerplexityBot',
'Applebot-Extended'
]
```
- **Description:** LLM bot user agents to list
**Example:**
```typescript
discovery({
robots: {
llmBots: {
enabled: true,
agents: ['CustomAI', 'AnotherBot']
}
}
})
```
**Generated output:**
```
# LLM-specific resources
# AI assistants can find additional context at /llms.txt
# See: https://github.com/anthropics/llm-txt
User-agent: CustomAI
User-agent: AnotherBot
Allow: /llms.txt
Allow: /llms-full.txt
```
### additionalAgents
- **Type:** `Array<{ userAgent: string; allow?: string[]; disallow?: string[] }>`
- **Default:** `undefined`
- **Description:** Custom agent-specific rules
**Example:**
```typescript
discovery({
robots: {
additionalAgents: [
{
userAgent: 'BadBot',
disallow: ['/']
},
{
userAgent: 'GoodBot',
allow: ['/api/public'],
disallow: ['/api/private', '/admin']
}
]
}
})
```
**Generated output:**
```
# Custom agent rules
User-agent: BadBot
Disallow: /
User-agent: GoodBot
Allow: /api/public
Disallow: /api/private
Disallow: /admin
```
### customRules
- **Type:** `string`
- **Default:** `undefined`
- **Description:** Raw robots.txt content appended to end of file
**Example:**
```typescript
discovery({
robots: {
customRules: `
# Custom section
User-agent: SpecialBot
Crawl-delay: 10
Request-rate: 1/5
`.trim()
}
})
```
## Default Output
With default configuration:
```
# robots.txt
# Generated by @astrojs/discovery for example.com
User-agent: *
Allow: /
# Sitemaps
Sitemap: https://example.com/sitemap-index.xml
# LLM-specific resources
# AI assistants can find additional context at /llms.txt
# See: https://github.com/anthropics/llm-txt
User-agent: Anthropic-AI
User-agent: Claude-Web
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: cohere-ai
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: Applebot-Extended
Allow: /llms.txt
Allow: /llms-full.txt
# Crawl delay (be nice to our server)
Crawl-delay: 1
```
## Common Patterns
### Block all bots from admin area
```typescript
discovery({
robots: {
additionalAgents: [
{
userAgent: '*',
disallow: ['/admin', '/api/private']
}
]
}
})
```
### Disable LLM bot access
```typescript
discovery({
robots: {
llmBots: {
enabled: false
}
}
})
```
### Custom LLM bot list
```typescript
discovery({
robots: {
llmBots: {
enabled: true,
agents: [
'Anthropic-AI',
'Claude-Web',
'MyCustomBot'
]
}
}
})
```
### Block specific bad bot
```typescript
discovery({
robots: {
additionalAgents: [
{
userAgent: 'BadBot',
disallow: ['/']
}
]
}
})
```
## Output Location
- **File:** `/robots.txt`
- **URL:** `https://example.com/robots.txt`
- **Cache-Control:** `public, max-age=3600` (1 hour, configurable via [caching](/reference/cache/))

343
docs/src/content/docs/reference/security.md
View File

@ -1,31 +1,334 @@
---
title: security.txt Options
title: security.txt Configuration
description: Configuration reference for security.txt (RFC 9116)
---
RFC 9116 compliant security.txt configuration reference.
Configuration reference for `/.well-known/security.txt` generation (RFC 9116).
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## SecurityConfig
## Coming Soon
```typescript
interface SecurityConfig {
enabled?: boolean;
contact: string | string[]; // REQUIRED
expires?: string | 'auto';
encryption?: string | string[];
acknowledgments?: string;
preferredLanguages?: string[];
canonical?: string;
policy?: string;
hiring?: string;
}
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Properties
## Related Pages
### enabled
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
- **Type:** `boolean`
- **Default:** `true`
- **Description:** Enable or disable security.txt generation
## Need Help?
### contact (Required)
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
- **Type:** `string | string[]`
- **Required:** **Yes**
- **Description:** Contact email or URL for security issues
- **RFC Requirement:** Required field per RFC 9116
**Email example:**
```typescript
discovery({
security: {
contact: 'security@example.com' // Auto-converts to mailto:
}
})
```
**Multiple contacts:**
```typescript
discovery({
security: {
contact: [
'security@example.com',
'https://example.com/security/report'
]
}
})
```
**Generated output:**
```
Contact: mailto:security@example.com
Contact: https://example.com/security/report
```
### expires
- **Type:** `string | 'auto'`
- **Default:** `'auto'` (1 year from generation)
- **Description:** Expiration date in ISO 8601 format
- **RFC Requirement:** Required field per RFC 9116
**Auto-expiration (recommended):**
```typescript
discovery({
security: {
contact: 'security@example.com',
expires: 'auto' // Sets to 1 year from build date
}
})
```
**Manual expiration:**
```typescript
discovery({
security: {
contact: 'security@example.com',
expires: '2026-12-31T23:59:59Z'
}
})
```
### encryption
- **Type:** `string | string[]`
- **Default:** `undefined`
- **Description:** URL to encryption key (PGP public key)
**Single key:**
```typescript
discovery({
security: {
contact: 'security@example.com',
encryption: 'https://example.com/pgp-key.txt'
}
})
```
**Multiple keys:**
```typescript
discovery({
security: {
contact: 'security@example.com',
encryption: [
'https://example.com/pgp-key.txt',
'https://keys.openpgp.org/vks/v1/by-fingerprint/ABC123'
]
}
})
```
### acknowledgments
- **Type:** `string`
- **Default:** `undefined`
- **Description:** URL to security acknowledgments/hall of fame page
**Example:**
```typescript
discovery({
security: {
contact: 'security@example.com',
acknowledgments: 'https://example.com/security/hall-of-fame'
}
})
```
### preferredLanguages
- **Type:** `string[]`
- **Default:** `undefined`
- **Description:** Preferred languages for security reports (ISO 639-1 codes)
**Example:**
```typescript
discovery({
security: {
contact: 'security@example.com',
preferredLanguages: ['en', 'es', 'fr']
}
})
```
**Generated output:**
```
Preferred-Languages: en, es, fr
```
### canonical
- **Type:** `string`
- **Default:** `https://{site}/.well-known/security.txt`
- **Description:** Canonical URL for security.txt location
**Example:**
```typescript
discovery({
security: {
contact: 'security@example.com',
canonical: 'https://example.com/.well-known/security.txt'
}
})
```
### policy
- **Type:** `string`
- **Default:** `undefined`
- **Description:** URL to security policy or disclosure policy
**Example:**
```typescript
discovery({
security: {
contact: 'security@example.com',
policy: 'https://example.com/security/disclosure-policy'
}
})
```
### hiring
- **Type:** `string`
- **Default:** `undefined`
- **Description:** URL for security job postings
**Example:**
```typescript
discovery({
security: {
contact: 'security@example.com',
hiring: 'https://example.com/careers/security'
}
})
```
## Generated Output
**Minimal configuration:**
```typescript
discovery({
security: {
contact: 'security@example.com'
}
})
```
**Output:**
```
Canonical: https://example.com/.well-known/security.txt
Contact: mailto:security@example.com
Expires: 2026-11-08T00:00:00.000Z
```
**Full configuration:**
```typescript
discovery({
security: {
contact: [
'security@example.com',
'https://example.com/security/report'
],
expires: 'auto',
encryption: 'https://example.com/pgp-key.txt',
acknowledgments: 'https://example.com/security/hall-of-fame',
preferredLanguages: ['en', 'es'],
policy: 'https://example.com/security/policy',
hiring: 'https://example.com/careers/security'
}
})
```
**Output:**
```
Canonical: https://example.com/.well-known/security.txt
Contact: mailto:security@example.com
Contact: https://example.com/security/report
Expires: 2026-11-08T00:00:00.000Z
Encryption: https://example.com/pgp-key.txt
Acknowledgments: https://example.com/security/hall-of-fame
Preferred-Languages: en, es
Policy: https://example.com/security/policy
Hiring: https://example.com/careers/security
```
## RFC 9116 Compliance
This implementation follows RFC 9116 requirements:
1. **Required fields:** `Contact` and `Expires` are mandatory
2. **Location:** Served at `/.well-known/security.txt`
3. **Format:** Plain text with field-value pairs
4. **Email handling:** Automatically adds `mailto:` prefix
5. **Canonical URL:** Defaults to correct `.well-known` location
6. **Field ordering:** Canonical first, then required fields
## Common Patterns
### Minimal setup
```typescript
discovery({
security: {
contact: 'security@example.com'
}
})
```
### With PGP encryption
```typescript
discovery({
security: {
contact: 'security@example.com',
encryption: 'https://example.com/pgp-key.txt',
preferredLanguages: ['en']
}
})
```
### Full security program
```typescript
discovery({
security: {
contact: [
'security@example.com',
'https://hackerone.com/example'
],
expires: 'auto',
encryption: 'https://example.com/security.asc',
acknowledgments: 'https://example.com/security/thanks',
preferredLanguages: ['en', 'es', 'fr'],
policy: 'https://example.com/security/disclosure',
hiring: 'https://example.com/careers/security-engineer'
}
})
```
## Best Practices
1. **Always set contact:** This is required by RFC 9116
2. **Use auto-expiration:** Let the integration manage expiration dates
3. **Provide encryption:** Offer PGP keys for secure communication
4. **Multiple contact methods:** Email + bug bounty platform
5. **Acknowledge researchers:** Link to your hall of fame
6. **Document policy:** Clear disclosure timelines and expectations
7. **Monitor expiration:** Security.txt should never expire
## Output Location
- **File:** `/.well-known/security.txt`
- **URL:** `https://example.com/.well-known/security.txt`
- **Cache-Control:** `public, max-age=86400` (24 hours, configurable via [caching](/reference/cache/))
- **RFC:** [RFC 9116](https://datatracker.ietf.org/doc/html/rfc9116)

239
docs/src/content/docs/reference/sitemap.md
View File

@ -1,31 +1,230 @@
---
title: Sitemap Options
title: Sitemap Configuration
description: Configuration reference for sitemap generation
---
Reference for sitemap configuration options (passed to @astrojs/sitemap).
Configuration reference for sitemap generation (passed to `@astrojs/sitemap`).
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## SitemapConfig
## Coming Soon
```typescript
interface SitemapConfig {
filter?: (page: string) => boolean;
customPages?: string[];
changefreq?: 'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never';
priority?: number;
i18n?: {
defaultLocale: string;
locales: Record<string, string>;
};
lastmod?: Date;
serialize?: (item: SitemapItem) => SitemapItem | undefined;
[key: string]: any;
}
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Properties
## Related Pages
All options are passed directly to `@astrojs/sitemap`. See [Astro Sitemap documentation](https://docs.astro.build/en/guides/integrations-guide/sitemap/) for complete details.
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
### filter
## Need Help?
- **Type:** `(page: string) => boolean`
- **Default:** `undefined`
- **Description:** Filter function to exclude pages
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
**Example:**
```typescript
discovery({
sitemap: {
filter: (page) => {
return !page.includes('/admin') &&
!page.includes('/draft') &&
!page.includes('/private');
}
}
})
```
### customPages
- **Type:** `string[]`
- **Default:** `undefined`
- **Description:** Custom pages to include in sitemap
**Example:**
```typescript
discovery({
sitemap: {
customPages: [
'https://example.com/external-page',
'https://example.com/another-page'
]
}
})
```
### changefreq
- **Type:** `'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never'`
- **Default:** `undefined`
- **Description:** Change frequency hint for search engines
**Example:**
```typescript
discovery({
sitemap: {
changefreq: 'weekly'
}
})
```
### priority
- **Type:** `number` (0.0 - 1.0)
- **Default:** `undefined`
- **Description:** Priority hint for search engines
**Example:**
```typescript
discovery({
sitemap: {
priority: 0.8
}
})
```
### i18n
- **Type:** `{ defaultLocale: string; locales: Record<string, string> }`
- **Default:** `undefined`
- **Description:** Internationalization configuration
**Example:**
```typescript
discovery({
sitemap: {
i18n: {
defaultLocale: 'en',
locales: {
en: 'en-US',
es: 'es-ES',
fr: 'fr-FR'
}
}
}
})
```
### lastmod
- **Type:** `Date`
- **Default:** `undefined`
- **Description:** Last modification date for all pages
### serialize
- **Type:** `(item: SitemapItem) => SitemapItem | undefined`
- **Default:** `undefined`
- **Description:** Custom serialization function
**Example:**
```typescript
discovery({
sitemap: {
serialize: (item) => {
// Skip draft pages
if (item.url.includes('/draft')) {
return undefined;
}
// Set higher priority for docs
if (item.url.includes('/docs')) {
item.priority = 0.9;
item.changefreq = 'daily';
}
return item;
}
}
})
```
## Common Patterns
### Exclude admin and private pages
```typescript
discovery({
sitemap: {
filter: (page) =>
!page.includes('/admin') &&
!page.includes('/private') &&
!page.includes('/_')
}
})
```
### Set change frequency
```typescript
discovery({
sitemap: {
changefreq: 'daily',
priority: 0.7
}
})
```
### Multilingual sitemap
```typescript
discovery({
sitemap: {
i18n: {
defaultLocale: 'en',
locales: {
en: 'en-US',
es: 'es-ES',
fr: 'fr-FR',
de: 'de-DE'
}
}
}
})
```
### Custom page priorities
```typescript
discovery({
sitemap: {
serialize: (item) => {
if (item.url.includes('/docs')) {
item.priority = 0.9;
item.changefreq = 'weekly';
} else if (item.url.includes('/blog')) {
item.priority = 0.7;
item.changefreq = 'daily';
} else {
item.priority = 0.5;
}
return item;
}
}
})
```
## Output Location
- **File:** `/sitemap-index.xml` (or `/sitemap-0.xml` if small)
- **URL:** `https://example.com/sitemap-index.xml`
- **Cache-Control:** `public, max-age=3600` (1 hour, configurable via [caching](/reference/cache/))
- **Documentation:** [Astro Sitemap Guide](https://docs.astro.build/en/guides/integrations-guide/sitemap/)
## Notes
- Automatically included by `@astrojs/discovery`
- No need to install `@astrojs/sitemap` separately
- Referenced in `robots.txt` automatically
- Supports both static and server-rendered pages

373
docs/src/content/docs/reference/typescript.md
View File

@ -1,31 +1,364 @@
---
title: TypeScript Types
description: TypeScript type definitions and interfaces
description: Complete TypeScript type definitions
---
Complete TypeScript type reference for configuration and APIs.
Complete TypeScript type reference for `@astrojs/discovery`.
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## Import Types
## Coming Soon
```typescript
import type {
DiscoveryConfig,
RobotsConfig,
LLMsConfig,
HumansConfig,
SecurityConfig,
CanaryConfig,
WebFingerConfig,
SitemapConfig,
CachingConfig,
TemplateConfig,
ImportantPage,
APIEndpoint,
TechStack,
TeamMember,
SiteInfo,
CanaryStatement,
WebFingerResource,
WebFingerLink
} from '@astrojs/discovery';
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Main Configuration
## Related Pages
```typescript
interface DiscoveryConfig {
robots?: RobotsConfig;
llms?: LLMsConfig;
humans?: HumansConfig;
security?: SecurityConfig;
canary?: CanaryConfig;
webfinger?: WebFingerConfig;
sitemap?: SitemapConfig;
caching?: CachingConfig;
templates?: TemplateConfig;
}
```
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
## File Configuration Types
## Need Help?
### RobotsConfig
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
```typescript
interface RobotsConfig {
enabled?: boolean;
crawlDelay?: number;
allowAllBots?: boolean;
llmBots?: {
enabled?: boolean;
agents?: string[];
};
additionalAgents?: Array<{
userAgent: string;
allow?: string[];
disallow?: string[];
}>;
customRules?: string;
}
```
### LLMsConfig
```typescript
interface LLMsConfig {
enabled?: boolean;
description?: string | (() => string);
keyFeatures?: string[];
importantPages?: ImportantPage[] | (() => Promise<ImportantPage[]>);
instructions?: string;
apiEndpoints?: APIEndpoint[];
techStack?: TechStack;
brandVoice?: string[];
customSections?: Record<string, string>;
}
```
### HumansConfig
```typescript
interface HumansConfig {
enabled?: boolean;
team?: TeamMember[];
thanks?: string[];
site?: SiteInfo;
story?: string;
funFacts?: string[];
philosophy?: string[];
customSections?: Record<string, string>;
}
```
### SecurityConfig
```typescript
interface SecurityConfig {
enabled?: boolean;
contact: string | string[];
expires?: string | 'auto';
encryption?: string | string[];
acknowledgments?: string;
preferredLanguages?: string[];
canonical?: string;
policy?: string;
hiring?: string;
}
```
### CanaryConfig
```typescript
interface CanaryConfig {
enabled?: boolean;
organization?: string;
contact?: string;
frequency?: 'daily' | 'weekly' | 'monthly' | 'quarterly' | 'yearly';
expires?: string | 'auto';
statements?: CanaryStatement[] | (() => CanaryStatement[]);
additionalStatement?: string;
verification?: string;
previousCanary?: string;
blockchainProof?: {
network: string;
address: string;
txHash?: string;
timestamp?: string;
};
personnelStatement?: boolean;
}
```
### WebFingerConfig
```typescript
interface WebFingerConfig {
enabled?: boolean;
resources?: WebFingerResource[];
collections?: {
name: string;
resourceTemplate: string;
subjectTemplate?: string;
linksBuilder?: (entry: any) => WebFingerLink[];
aliasesBuilder?: (entry: any) => string[];
propertiesBuilder?: (entry: any) => Record<string, string | null>;
}[];
}
```
### SitemapConfig
```typescript
interface SitemapConfig {
filter?: (page: string) => boolean;
customPages?: string[];
changefreq?: 'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never';
priority?: number;
i18n?: {
defaultLocale: string;
locales: Record<string, string>;
};
lastmod?: Date;
[key: string]: any;
}
```
### CachingConfig
```typescript
interface CachingConfig {
robots?: number;
llms?: number;
humans?: number;
security?: number;
canary?: number;
webfinger?: number;
sitemap?: number;
}
```
### TemplateConfig
```typescript
interface TemplateConfig {
robots?: (config: RobotsConfig, siteURL: URL) => string;
llms?: (config: LLMsConfig, siteURL: URL) => string | Promise<string>;
humans?: (config: HumansConfig, siteURL: URL) => string;
security?: (config: SecurityConfig, siteURL: URL) => string;
canary?: (config: CanaryConfig, siteURL: URL) => string;
}
```
## Supporting Types
### ImportantPage
```typescript
interface ImportantPage {
name: string;
path: string;
description?: string;
}
```
### APIEndpoint
```typescript
interface APIEndpoint {
path: string;
method?: string;
description: string;
}
```
### TechStack
```typescript
interface TechStack {
frontend?: string[];
backend?: string[];
ai?: string[];
other?: string[];
}
```
### TeamMember
```typescript
interface TeamMember {
name: string;
role?: string;
contact?: string;
location?: string;
twitter?: string;
github?: string;
}
```
### SiteInfo
```typescript
interface SiteInfo {
lastUpdate?: string | 'auto';
language?: string;
doctype?: string;
ide?: string;
techStack?: string[];
standards?: string[];
components?: string[];
software?: string[];
}
```
### CanaryStatement
```typescript
interface CanaryStatement {
type: 'nsl' | 'fisa' | 'gag' | 'surveillance' | 'backdoor' | 'encryption' | 'other';
description: string;
received: boolean;
}
```
### WebFingerResource
```typescript
interface WebFingerResource {
resource: string;
subject?: string;
aliases?: string[];
properties?: Record<string, string | null>;
links?: WebFingerLink[];
}
```
### WebFingerLink
```typescript
interface WebFingerLink {
rel: string;
href?: string;
type?: string;
titles?: Record<string, string>;
properties?: Record<string, string | null>;
}
```
### SitemapItem
```typescript
interface SitemapItem {
url: string;
lastmod?: Date;
changefreq?: 'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never';
priority?: number;
links?: Array<{
url: string;
lang: string;
}>;
}
```
## Type Guards
### Check if config has security
```typescript
function hasSecurityConfig(config: DiscoveryConfig): config is Required<Pick<DiscoveryConfig, 'security'>> & DiscoveryConfig {
return config.security !== undefined && config.security.contact !== undefined;
}
```
### Check if config has canary
```typescript
function hasCanaryConfig(config: DiscoveryConfig): config is Required<Pick<DiscoveryConfig, 'canary'>> & DiscoveryConfig {
return config.canary !== undefined;
}
```
## Example Usage
```typescript
import { defineConfig } from 'astro/config';
import discovery from '@astrojs/discovery';
import type { DiscoveryConfig, LLMsConfig } from '@astrojs/discovery';
const llmsConfig: LLMsConfig = {
description: 'My awesome site',
keyFeatures: ['Feature 1', 'Feature 2']
};
const discoveryConfig: DiscoveryConfig = {
llms: llmsConfig,
robots: {
crawlDelay: 2
},
caching: {
llms: 1800
}
};
export default defineConfig({
site: 'https://example.com',
integrations: [
discovery(discoveryConfig)
]
});
```
## Notes
- All configuration interfaces are exported from `@astrojs/discovery`
- All properties are optional unless marked "Required"
- Use TypeScript's type inference for better DX
- Enable strict mode for better type safety

384
docs/src/content/docs/reference/webfinger.md
View File

@ -1,31 +1,375 @@
---
title: WebFinger Options
title: WebFinger Configuration
description: Configuration reference for WebFinger (RFC 7033)
---
RFC 7033 compliant WebFinger configuration reference.
Configuration reference for `/.well-known/webfinger` resource discovery (RFC 7033).
:::note[Work in Progress]
This page is currently being developed. Check back soon for complete documentation.
:::
## WebFingerConfig
## Coming Soon
```typescript
interface WebFingerConfig {
enabled?: boolean;
resources?: WebFingerResource[];
collections?: {
name: string;
resourceTemplate: string;
subjectTemplate?: string;
linksBuilder?: (entry: any) => WebFingerLink[];
aliasesBuilder?: (entry: any) => string[];
propertiesBuilder?: (entry: any) => Record<string, string | null>;
}[];
}
```
This section will include:
- Detailed explanations
- Code examples
- Best practices
- Common patterns
- Troubleshooting tips
## Properties
## Related Pages
### enabled
- [Configuration Reference](/reference/configuration/)
- [API Reference](/reference/api/)
- [Examples](/examples/ecommerce/)
- **Type:** `boolean`
- **Default:** `false` (opt-in)
- **Description:** Enable or disable WebFinger generation
## Need Help?
**Note:** WebFinger is opt-in by default because it requires configuration.
- Check our [FAQ](/community/faq/)
- Visit [Troubleshooting](/community/troubleshooting/)
- Open an issue on [GitHub](https://github.com/withastro/astro-discovery/issues)
### resources
- **Type:** `WebFingerResource[]`
- **Default:** `undefined`
- **Description:** Static resources to expose via WebFinger
**WebFingerResource interface:**
```typescript
interface WebFingerResource {
resource: string;
subject?: string;
aliases?: string[];
properties?: Record<string, string | null>;
links?: WebFingerLink[];
}
```
**WebFingerLink interface:**
```typescript
interface WebFingerLink {
rel: string;
href?: string;
type?: string;
titles?: Record<string, string>;
properties?: Record<string, string | null>;
}
```
**Static resource example:**
```typescript
discovery({
webfinger: {
enabled: true,
resources: [
{
resource: 'acct:alice@example.com',
subject: 'acct:alice@example.com',
aliases: [
'https://example.com/@alice',
'https://example.com/users/alice'
],
properties: {
'http://schema.org/name': 'Alice Developer'
},
links: [
{
rel: 'http://webfinger.net/rel/profile-page',
type: 'text/html',
href: 'https://example.com/@alice'
},
{
rel: 'self',
type: 'application/activity+json',
href: 'https://example.com/users/alice'
}
]
}
]
}
})
```
### collections
- **Type:** Collection configuration array
- **Default:** `undefined`
- **Description:** Content collection integration for dynamic WebFinger resources
#### Collection Properties
##### name
- **Type:** `string`
- **Required:** Yes
- **Description:** Astro content collection name
##### resourceTemplate
- **Type:** `string`
- **Required:** Yes
- **Description:** Resource URI template with variables
- **Supported variables:** `{slug}`, `{id}`, `{data.fieldName}`, `{siteURL}`
##### subjectTemplate
- **Type:** `string`
- **Default:** Same as `resourceTemplate`
- **Description:** Subject URI template
##### linksBuilder
- **Type:** `(entry: any) => WebFingerLink[]`
- **Default:** `undefined`
- **Description:** Function to generate links for a collection entry
##### aliasesBuilder
- **Type:** `(entry: any) => string[]`
- **Default:** `undefined`
- **Description:** Function to generate aliases for a collection entry
##### propertiesBuilder
- **Type:** `(entry: any) => Record<string, string | null>`
- **Default:** `undefined`
- **Description:** Function to generate properties for a collection entry
## Common Use Cases
### ActivityPub / Mastodon
Enable federated social network discovery:
```typescript
discovery({
webfinger: {
enabled: true,
collections: [{
name: 'team',
resourceTemplate: 'acct:{slug}@example.com',
linksBuilder: (member) => [
{
rel: 'self',
type: 'application/activity+json',
href: `https://example.com/users/${member.slug}`
},
{
rel: 'http://webfinger.net/rel/profile-page',
type: 'text/html',
href: `https://example.com/@${member.slug}`
},
{
rel: 'http://webfinger.net/rel/avatar',
type: 'image/jpeg',
href: member.data.avatar
}
],
propertiesBuilder: (member) => ({
'http://schema.org/name': member.data.name
})
}]
}
})
```
### OpenID Connect
Provide issuer discovery for authentication:
```typescript
discovery({
webfinger: {
enabled: true,
resources: [
{
resource: 'https://example.com',
links: [
{
rel: 'http://openid.net/specs/connect/1.0/issuer',
href: 'https://example.com'
}
]
}
]
}
})
```
### Team Profiles
Make team members discoverable:
```typescript
discovery({
webfinger: {
enabled: true,
collections: [{
name: 'team',
resourceTemplate: 'acct:{data.email}',
aliasesBuilder: (member) => [
`https://example.com/team/${member.slug}`,
member.data.website
].filter(Boolean),
propertiesBuilder: (member) => ({
'http://schema.org/name': member.data.name,
'http://schema.org/jobTitle': member.data.role,
'http://schema.org/email': member.data.email
}),
linksBuilder: (member) => [
{
rel: 'http://webfinger.net/rel/profile-page',
type: 'text/html',
href: `https://example.com/team/${member.slug}`
},
...(member.data.github ? [{
rel: 'http://webfinger.net/rel/profile-page',
type: 'text/html',
href: `https://github.com/${member.data.github}`,
titles: { en: 'GitHub Profile' }
}] : [])
]
}]
}
})
```
### Blog Authors
Link blog authors to their profiles:
```typescript
discovery({
webfinger: {
enabled: true,
collections: [{
name: 'authors',
resourceTemplate: 'acct:{slug}@example.com',
linksBuilder: (author) => [
{
rel: 'http://webfinger.net/rel/profile-page',
href: `https://example.com/authors/${author.slug}`
},
{
rel: 'http://webfinger.net/rel/avatar',
href: author.data.avatar,
type: 'image/jpeg'
}
],
propertiesBuilder: (author) => ({
'http://schema.org/name': author.data.name,
'http://schema.org/description': author.data.bio
})
}]
}
})
```
## Query Format
WebFinger is queried via HTTP GET with query parameters:
```
GET /.well-known/webfinger?resource=acct:alice@example.com
GET /.well-known/webfinger?resource=acct:alice@example.com&rel=self
```
**Required parameter:**
- `resource`: The resource identifier (e.g., `acct:alice@example.com`)
**Optional parameter:**
- `rel`: Filter links by relation type
## Response Format (JRD)
WebFinger returns JSON Resource Descriptor (JRD):
```json
{
"subject": "acct:alice@example.com",
"aliases": [
"https://example.com/@alice"
],
"properties": {
"http://schema.org/name": "Alice Developer"
},
"links": [
{
"rel": "http://webfinger.net/rel/profile-page",
"type": "text/html",
"href": "https://example.com/@alice"
},
{
"rel": "self",
"type": "application/activity+json",
"href": "https://example.com/users/alice"
}
]
}
```
## Template Variables
### Available Variables
- `{slug}`: Collection entry slug
- `{id}`: Collection entry ID
- `{data.fieldName}`: Access entry data fields
- `{data.nested.field}`: Access nested fields
- `{siteURL}`: Site hostname
### Examples
```typescript
// Basic slug
resourceTemplate: 'acct:{slug}@example.com'
// Result: acct:alice@example.com
// Data field
resourceTemplate: 'acct:{data.email}'
// Result: acct:alice@company.com
// Nested field
resourceTemplate: 'acct:{data.social.mastodon}'
// Result: acct:alice@mastodon.social
// Site URL
resourceTemplate: 'acct:{slug}@{siteURL}'
// Result: acct:alice@example.com
```
## Common Link Relations
- `self`: The resource itself
- `http://webfinger.net/rel/profile-page`: Profile page
- `http://webfinger.net/rel/avatar`: Avatar image
- `http://openid.net/specs/connect/1.0/issuer`: OpenID issuer
- `http://webfinger.net/rel/me`: Personal URL
## Best Practices
1. **Use standard relations:** Stick to IANA-registered or well-known rel values
2. **Include types:** Always specify `type` for links when applicable
3. **Provide aliases:** Help users find resources via multiple identifiers
4. **Use URI properties:** Property names must be URIs (e.g., `http://schema.org/name`)
5. **Enable CORS:** WebFinger responses include `Access-Control-Allow-Origin: *`
6. **Cache appropriately:** Default 1-hour cache is usually sufficient
## Technical Notes
- **Dynamic route:** Not prerendered, handles queries at request time
- **CORS enabled:** Returns `Access-Control-Allow-Origin: *`
- **Media type:** `application/jrd+json`
- **404 handling:** Returns 404 for unknown resources
- **Rel filtering:** Supports `?rel=` parameter for link filtering
## Output Location
- **Endpoint:** `/.well-known/webfinger`
- **URL:** `https://example.com/.well-known/webfinger?resource=acct:user@example.com`
- **Cache-Control:** `public, max-age=3600` (1 hour, configurable via [caching](/reference/cache/))
- **RFC:** [RFC 7033](https://datatracker.ietf.org/doc/html/rfc7033)