astro/astro-discovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
astro-discovery/CHANGELOG.md
Ryan Malloy d25dde4627 feat: initial implementation of @astrojs/discovery integration 
This commit introduces a comprehensive Astro integration that automatically
generates discovery files for websites:

Features:
- robots.txt with LLM bot support (Anthropic-AI, GPTBot, etc.)
- llms.txt for AI assistant context and instructions
- humans.txt for team credits and site information
- Automatic sitemap integration via @astrojs/sitemap

Technical Details:
- TypeScript implementation with full type safety
- Configurable HTTP caching headers
- Custom template support for all generated files
- Sensible defaults with extensive customization options
- Date-based versioning (2025.11.03)

Testing:
- 34 unit tests covering all generators
- Test coverage for robots.txt, llms.txt, and humans.txt
- Integration with Vitest

Documentation:
- Comprehensive README with examples
- API reference documentation
- Contributing guidelines
- Example configurations (minimal and full)
2025-11-03 07:36:39 -07:00

2.0 KiB
Raw Blame History

Changelog

All notable changes to @astrojs/discovery will be documented in this file.

The format is based on Keep a Changelog, and this project uses date-based versioning (YYYY.MM.DD).

[2025.11.03] - 2025-11-03

Added

  • Initial release of @astrojs/discovery
  • Automatic robots.txt generation with LLM bot support
  • Automatic llms.txt generation for AI assistant context
  • Automatic humans.txt generation for team credits
  • Integration with @astrojs/sitemap for sitemap-index.xml
  • Configurable HTTP caching headers
  • Custom template support for all generated files
  • TypeScript type definitions
  • Comprehensive configuration options
  • Example configurations (minimal and full)

Features

  • robots.txt

    • Default allow-all policy
    • LLM-specific bot rules (Anthropic-AI, GPTBot, etc.)
    • Custom agent configurations
    • Crawl delay settings
    • Custom rules support

  • llms.txt

    • Site description and key features
    • Important pages listing
    • AI assistant instructions
    • API endpoint documentation
    • Technology stack information
    • Brand voice guidelines
    • Custom sections

  • humans.txt

    • Team member information
    • Thanks/credits section
    • Site technical information
    • Project story
    • Fun facts
    • Development philosophy
    • Custom sections

  • Configuration

    • Sensible defaults
    • Full customization options
    • Environment-based toggles
    • Dynamic content support
    • Cache control configuration

Documentation

  • Comprehensive README with examples
  • API reference documentation
  • Contributing guidelines
  • Example configurations
  • Integration guides

Future Enhancements

Planned Features

  • security.txt support (RFC 9116)
  • ads.txt support for advertising
  • manifest.json for PWA
  • RSS feed integration
  • OpenGraph tags injection
  • Structured data (JSON-LD)
  • Analytics discovery
  • i18n support for multi-language sites

Testing

  • Unit tests for generators
  • Integration tests
  • E2E tests with real Astro projects

For more information, see README.md