This commit introduces a comprehensive Astro integration that automatically generates discovery files for websites: Features: - robots.txt with LLM bot support (Anthropic-AI, GPTBot, etc.) - llms.txt for AI assistant context and instructions - humans.txt for team credits and site information - Automatic sitemap integration via @astrojs/sitemap Technical Details: - TypeScript implementation with full type safety - Configurable HTTP caching headers - Custom template support for all generated files - Sensible defaults with extensive customization options - Date-based versioning (2025.11.03) Testing: - 34 unit tests covering all generators - Test coverage for robots.txt, llms.txt, and humans.txt - Integration with Vitest Documentation: - Comprehensive README with examples - API reference documentation - Contributing guidelines - Example configurations (minimal and full)
3.9 KiB
Contributing to @astrojs/discovery
Thank you for your interest in contributing to @astrojs/discovery! This guide will help you get started.
Development Setup
-
Clone the repository
git clone https://github.com/withastro/astro-discovery.git cd astro-discovery -
Install dependencies
npm install -
Build the project
npm run build -
Run tests
npm test
Project Structure
@astrojs/discovery/
├── src/
│ ├── index.ts # Main integration entry point
│ ├── types.ts # TypeScript type definitions
│ ├── config-store.ts # Global config management
│ ├── generators/
│ │ ├── robots.ts # robots.txt generator
│ │ ├── llms.ts # llms.txt generator
│ │ └── humans.ts # humans.txt generator
│ ├── routes/
│ │ ├── robots.ts # /robots.txt API route
│ │ ├── llms.ts # /llms.txt API route
│ │ └── humans.ts # /humans.txt API route
│ └── validators/
│ └── config.ts # Configuration validation
├── dist/ # Built output (generated)
├── example/ # Example configurations
└── tests/ # Test files (to be added)
Making Changes
Adding New Features
-
Create a new branch for your feature
git checkout -b feature/your-feature-name -
Make your changes following the existing code style
-
Add tests for your changes
-
Update documentation in README.md
-
Build and test
npm run build npm test
Code Style
- Use TypeScript for all code
- Follow existing naming conventions
- Add JSDoc comments for public APIs
- Keep functions focused and small
- Use meaningful variable names
Commit Messages
Follow conventional commit format:
feat:New featuresfix:Bug fixesdocs:Documentation changestest:Test additions/changesrefactor:Code refactoringchore:Maintenance tasks
Example:
feat: add support for custom LLM bot agents
- Added ability to specify custom LLM bot user agents
- Updated documentation with examples
- Added tests for custom agent configuration
Testing
Running Tests
# Run all tests
npm test
# Run tests in watch mode
npm run test:watch
# Run tests with coverage
npm run test:coverage
Writing Tests
Place tests in the tests/ directory. Follow these patterns:
import { describe, it, expect } from 'vitest';
import { generateRobotsTxt } from '../src/generators/robots';
describe('generateRobotsTxt', () => {
it('generates basic robots.txt', () => {
const result = generateRobotsTxt({}, new URL('https://example.com'));
expect(result).toContain('User-agent: *');
expect(result).toContain('Sitemap: https://example.com/sitemap-index.xml');
});
});
Pull Request Process
-
Update documentation: Ensure README.md reflects any changes
-
Add tests: All new features should have tests
-
Update CHANGELOG: Add your changes to CHANGELOG.md
-
Create a pull request:
- Use a clear, descriptive title
- Reference any related issues
- Describe your changes in detail
- Include screenshots for UI changes
-
Address review feedback: Be responsive to code review comments
Release Process
(For maintainers)
- Update version in package.json using date-based versioning (YYYY.MM.DD)
- Update CHANGELOG.md
- Create a git tag
- Push to npm
npm version 2025.11.03
npm publish
Questions?
- Open an issue for bugs or feature requests
- Start a discussion for questions or ideas
- Join our Discord community
License
By contributing, you agree that your contributions will be licensed under the MIT License.