feat: add security.txt (RFC 9116) and canary.txt support

Implemented two major new discovery file formats:

## security.txt (RFC 9116 - Industry Standard)
- Standardized security contact information
- Required fields: Contact, Expires, Canonical
- Optional fields: Encryption, Acknowledgments, Policy, Hiring, Languages
- Auto-expiration calculation (1 year by default)
- Proper mailto: prefix handling
- Located at /.well-known/security.txt

## canary.txt (NEW SPECIFICATION)
- First standardized format for warrant canaries
- Machine-readable transparency statements
- Auto-expiring based on update frequency
- Support for multiple statement types (NSL, FISA, gag orders, etc.)
- Optional blockchain proof
- Personnel duress statement
- See CANARY_SPEC.md for full specification
- Located at /.well-known/canary.txt

Changes:
- Added SecurityConfig and CanaryConfig type definitions
- Created generators for both formats with smart defaults
- Added API route handlers with virtual module config
- Updated integration to inject /.well-known/ routes
- Added to cache configuration (security: 24h, canary: 1h)
- Exported new types for TypeScript users
- Updated CLAUDE.md with feature priorities
- Created comprehensive CANARY_SPEC.md specification document

Testing: Both features verified in test project with full configuration.
All files generate correctly with proper formatting and validation.
This commit is contained in:
Ryan Malloy 2025-11-03 08:12:42 -07:00
parent c7b47bba5c
commit 6de34f55a9
8 changed files with 682 additions and 1 deletions

272
CANARY_SPEC.md Normal file
View File

@ -0,0 +1,272 @@
# canary.txt Specification
**Version:** 1.0.0-draft
**Status:** Proposal
**Author:** Ryan Malloy
**Date:** 2025-11-03
## Abstract
This document defines `canary.txt`, a standard format for warrant canaries - public statements that organizations publish to indicate they have NOT received secret government requests (national security letters, gag orders, etc.).
## Motivation
Warrant canaries currently exist in various ad-hoc formats across websites, making them:
- Hard to discover
- Difficult to verify programmatically
- Inconsistent in format
- Challenging to monitor at scale
A standardized format enables:
- Automated monitoring services
- Browser extensions that alert users
- Transparency dashboards
- Cryptographic verification
- Historical tracking
## Specification
### Location
The canary file MUST be located at one of:
- `/.well-known/canary.txt` (RECOMMENDED)
- `/canary.txt` (ACCEPTABLE)
### Format
The file MUST be plain text (text/plain; charset=utf-8).
### Required Fields
```
Canonical-URL: https://example.com/.well-known/canary.txt
Issued: 2025-11-03T00:00:00Z
Expires: 2026-11-03T00:00:00Z
```
**Canonical-URL:** The authoritative URL for this canary
**Issued:** ISO 8601 timestamp when this canary was published
**Expires:** ISO 8601 timestamp when this canary expires
### Statement Section
The canary MUST include a clear statement:
```
Statement: As of 2025-11-03, Example Inc has NOT received:
- National Security Letters (NSLs)
- FISA court orders
- Gag orders preventing disclosure
- Secret government requests for user data
- Requests to install surveillance capabilities
```
### Optional Fields
```
Organization: Example Inc
Contact: legal@example.com
Verification: https://example.com/canary-verification.asc
Frequency: monthly
Previous-Canary: https://example.com/.well-known/canary-2025-10.txt
```
**Organization:** Legal entity name
**Contact:** Email for canary-related inquiries
**Verification:** URL to PGP signature or blockchain proof
**Frequency:** Update frequency (daily, weekly, monthly, quarterly)
**Previous-Canary:** Link to previous canary (for continuity)
### PGP Signature (Recommended)
Canaries SHOULD be PGP-signed for authenticity:
```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Canonical-URL: https://example.com/.well-known/canary.txt
Issued: 2025-11-03T00:00:00Z
Expires: 2026-11-03T00:00:00Z
Statement: As of 2025-11-03, Example Inc has NOT received:
- National Security Letters (NSLs)
- FISA court orders
...
-----BEGIN PGP SIGNATURE-----
...
-----END PGP SIGNATURE-----
```
### Blockchain Proof (Optional)
For additional verification, include:
```
Blockchain-Proof: bitcoin:1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa:blockhash
Blockchain-Timestamp: 2025-11-03T00:00:00Z
```
## Example: Minimal Canary
```
Canonical-URL: https://example.com/.well-known/canary.txt
Issued: 2025-11-03T00:00:00Z
Expires: 2026-02-03T00:00:00Z
Organization: Example Inc
Contact: transparency@example.com
Statement: As of 2025-11-03, Example Inc has NOT received any:
- National Security Letters
- FISA court orders
- Gag orders
- Secret government requests
This canary will be updated quarterly. Absence of an update
should be considered significant.
```
## Example: Maximum Security Canary
```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Canonical-URL: https://example.com/.well-known/canary.txt
Issued: 2025-11-03T00:00:00Z
Expires: 2025-12-03T00:00:00Z
Organization: Privacy-First Corp
Contact: canary@privacyfirst.example
Verification: https://privacyfirst.example/pgp-public.asc
Frequency: monthly
Previous-Canary: https://privacyfirst.example/.well-known/canary-2025-10.txt
Blockchain-Proof: bitcoin:1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa:00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048
Blockchain-Timestamp: 2025-11-03T00:05:23Z
Statement: As of 2025-11-03, Privacy-First Corp has NOT:
- Received any National Security Letters (NSLs)
- Received any FISA court orders
- Received any gag orders preventing disclosure
- Been compelled to modify our software or services
- Been compelled to install surveillance capabilities
- Received requests to weaken encryption
- Been prohibited from updating this canary
This statement is updated monthly on the first business day.
Any absence of an update within 35 days should be considered
a removal of this canary.
Key Personnel Statement: All key personnel with access to
infrastructure remain free and under no duress.
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEE...
-----END PGP SIGNATURE-----
```
## Machine-Readable Format (.well-known/canary.json)
For programmatic consumption:
```json
{
"version": "1.0",
"canonical_url": "https://example.com/.well-known/canary.txt",
"issued": "2025-11-03T00:00:00Z",
"expires": "2026-11-03T00:00:00Z",
"organization": "Example Inc",
"contact": "transparency@example.com",
"frequency": "monthly",
"statements": [
{
"type": "nsl",
"received": false,
"description": "National Security Letters"
},
{
"type": "fisa",
"received": false,
"description": "FISA court orders"
},
{
"type": "gag",
"received": false,
"description": "Gag orders"
}
],
"verification": {
"pgp_signature_url": "https://example.com/canary-verification.asc",
"pgp_fingerprint": "1234 5678 9ABC DEF0 1234 5678 9ABC DEF0 1234 5678",
"blockchain": {
"network": "bitcoin",
"address": "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa",
"tx_hash": "blockhash"
}
},
"previous_canary": "https://example.com/.well-known/canary-2025-10.txt"
}
```
## Monitoring & Verification
### Automated Monitoring
Services can:
1. Fetch canary regularly
2. Verify PGP signature
3. Check expiration dates
4. Alert on missing/changed canaries
5. Track historical versions
### Browser Extension Concept
A browser extension could:
- Display canary status icon
- Alert when canary expires/changes
- Show canary history
- Verify signatures
## Legal Considerations
**IMPORTANT:** Consult legal counsel before implementing a warrant canary.
- Effectiveness varies by jurisdiction
- May not be legally tested in all countries
- Should not be sole transparency mechanism
- Consider comprehensive transparency reports
## Security Properties
1. **Passive Statement:** Says what has NOT happened
2. **Absence is Signal:** Missing canary = possible receipt of order
3. **Cryptographic Proof:** PGP prevents forgery
4. **Blockchain Anchoring:** Proves existence at point in time
5. **Continuous Verification:** Automated monitoring detects changes
## Limitations
- Legal effectiveness uncertain in many jurisdictions
- Sophisticated adversaries may compel deceptive updates
- Does not guarantee absence of other surveillance
- Should complement, not replace, other transparency measures
## References
- RFC 8615: Well-Known URIs
- RFC 9116: security.txt
- EFF: Warrant Canary FAQ
- Warrant Canary Watch
## Contributing
This is a draft specification. Feedback welcome at:
- GitHub: [repository URL]
- Email: ryan@supported.systems
## License
This specification is released under CC0 1.0 Universal (Public Domain).
---
**Status:** This is a PROPOSED standard, not yet ratified by any standards body.

164
src/generators/canary.ts Normal file
View File

@ -0,0 +1,164 @@
import type { CanaryConfig, CanaryStatement } from '../types.js';
/**
* Default canary statements (what has NOT been received)
*/
const DEFAULT_STATEMENTS: CanaryStatement[] = [
{
type: 'nsl',
description: 'National Security Letters (NSLs)',
received: false,
},
{
type: 'fisa',
description: 'FISA court orders',
received: false,
},
{
type: 'gag',
description: 'Gag orders preventing disclosure',
received: false,
},
{
type: 'surveillance',
description: 'Secret government requests for user data',
received: false,
},
{
type: 'backdoor',
description: 'Requests to install surveillance capabilities',
received: false,
},
];
/**
* Calculate expiration date based on frequency
*/
function calculateExpiration(frequency: string): string {
const now = new Date();
const expirationMap: Record<string, number> = {
daily: 2, // 2 days
weekly: 10, // 10 days
monthly: 35, // 35 days
quarterly: 100, // ~100 days
yearly: 380, // 380 days
};
const days = expirationMap[frequency] || 35;
const expiration = new Date(now.getTime() + days * 24 * 60 * 60 * 1000);
return expiration.toISOString();
}
/**
* Generate canary.txt content
*
* @param config - Canary configuration
* @param siteURL - Site base URL
* @returns Generated canary.txt content
*/
export function generateCanaryTxt(
config: CanaryConfig,
siteURL: URL
): string {
const lines: string[] = [];
const today = new Date().toISOString().split('T')[0];
// Canonical URL
const canonical = new URL('.well-known/canary.txt', siteURL).href;
lines.push(`Canonical-URL: ${canonical}`);
// Issued date (today)
lines.push(`Issued: ${new Date().toISOString()}`);
// Expires date
const expires = config.expires === 'auto'
? calculateExpiration(config.frequency || 'monthly')
: config.expires || calculateExpiration('monthly');
lines.push(`Expires: ${expires}`);
// Organization (optional)
if (config.organization) {
lines.push(`Organization: ${config.organization}`);
}
// Contact (optional)
if (config.contact) {
const contactValue = config.contact.includes('@') && !config.contact.startsWith('mailto:')
? `mailto:${config.contact}`
: config.contact;
lines.push(`Contact: ${contactValue}`);
}
// Verification URL (optional)
if (config.verification) {
lines.push(`Verification: ${config.verification}`);
}
// Frequency (optional)
if (config.frequency) {
lines.push(`Frequency: ${config.frequency}`);
}
// Previous canary (optional)
if (config.previousCanary) {
lines.push(`Previous-Canary: ${config.previousCanary}`);
}
// Blockchain proof (optional)
if (config.blockchainProof) {
const { network, address, txHash, timestamp } = config.blockchainProof;
const proof = txHash
? `${network}:${address}:${txHash}`
: `${network}:${address}`;
lines.push(`Blockchain-Proof: ${proof}`);
if (timestamp) {
lines.push(`Blockchain-Timestamp: ${timestamp}`);
}
}
lines.push('');
// Statement section
lines.push(`Statement: As of ${today}, ${config.organization || siteURL.hostname} has NOT received:`);
const statements = typeof config.statements === 'function'
? config.statements()
: config.statements || DEFAULT_STATEMENTS;
statements
.filter(s => !s.received)
.forEach(statement => {
lines.push(` - ${statement.description}`);
});
lines.push('');
// Additional statement (optional)
if (config.additionalStatement) {
lines.push(config.additionalStatement.trim());
lines.push('');
}
// Key personnel statement (optional)
if (config.personnelStatement) {
lines.push('Key Personnel Statement: All key personnel with access to');
lines.push('infrastructure remain free and under no duress.');
lines.push('');
}
// Update reminder based on frequency
const frequencyText = config.frequency || 'monthly';
const expirationDays = Math.ceil((new Date(expires).getTime() - Date.now()) / (24 * 60 * 60 * 1000));
lines.push(`This canary will be updated ${frequencyText}. Absence of an update`);
lines.push(`within ${expirationDays} days should be considered significant.`);
lines.push('');
// Footer
lines.push('---');
lines.push('');
lines.push('This warrant canary follows the proposed canary.txt specification.');
lines.push('See: https://github.com/withastro/astro-discovery/blob/main/CANARY_SPEC.md');
return lines.join('\n').trim() + '\n';
}

View File

@ -0,0 +1,75 @@
import type { SecurityConfig } from '../types.js';
/**
* Generate security.txt content per RFC 9116
*
* @param config - Security.txt configuration
* @param siteURL - Site base URL
* @returns Generated security.txt content
*
* @see https://datatracker.ietf.org/doc/html/rfc9116
*/
export function generateSecurityTxt(
config: SecurityConfig,
siteURL: URL
): string {
const lines: string[] = [];
// Canonical URL (recommended)
const canonical = config.canonical || new URL('.well-known/security.txt', siteURL).href;
lines.push(`Canonical: ${canonical}`);
lines.push('');
// Contact (required) - can be email or URL
const contacts = Array.isArray(config.contact) ? config.contact : [config.contact];
contacts.forEach(contact => {
// Add mailto: prefix if it's an email
const contactValue = contact.includes('@') && !contact.startsWith('mailto:')
? `mailto:${contact}`
: contact;
lines.push(`Contact: ${contactValue}`);
});
lines.push('');
// Expires (required by RFC)
const expires = config.expires === 'auto'
? new Date(Date.now() + 365 * 24 * 60 * 60 * 1000).toISOString() // 1 year from now
: config.expires || new Date(Date.now() + 365 * 24 * 60 * 60 * 1000).toISOString();
lines.push(`Expires: ${expires}`);
lines.push('');
// Encryption (optional)
if (config.encryption) {
const encryptions = Array.isArray(config.encryption) ? config.encryption : [config.encryption];
encryptions.forEach(key => {
lines.push(`Encryption: ${key}`);
});
lines.push('');
}
// Acknowledgments (optional)
if (config.acknowledgments) {
lines.push(`Acknowledgments: ${config.acknowledgments}`);
lines.push('');
}
// Preferred-Languages (optional)
if (config.preferredLanguages && config.preferredLanguages.length > 0) {
lines.push(`Preferred-Languages: ${config.preferredLanguages.join(', ')}`);
lines.push('');
}
// Policy (optional)
if (config.policy) {
lines.push(`Policy: ${config.policy}`);
lines.push('');
}
// Hiring (optional)
if (config.hiring) {
lines.push(`Hiring: ${config.hiring}`);
lines.push('');
}
return lines.join('\n').trim() + '\n';
}

View File

@ -103,6 +103,22 @@ export default function discovery(
prerender: true prerender: true
}); });
} }
if (config.security && config.security.enabled !== false) {
injectRoute({
pattern: '/.well-known/security.txt',
entrypoint: '@astrojs/discovery/routes/security',
prerender: true
});
}
if (config.canary && config.canary.enabled !== false) {
injectRoute({
pattern: '/.well-known/canary.txt',
entrypoint: '@astrojs/discovery/routes/canary',
prerender: true
});
}
}, },
'astro:build:done': () => { 'astro:build:done': () => {
@ -118,6 +134,12 @@ export default function discovery(
if (config.humans?.enabled !== false) { if (config.humans?.enabled !== false) {
console.log(' ✅ /humans.txt'); console.log(' ✅ /humans.txt');
} }
if (config.security && config.security.enabled !== false) {
console.log(' ✅ /.well-known/security.txt');
}
if (config.canary && config.canary.enabled !== false) {
console.log(' ✅ /.well-known/canary.txt');
}
console.log(' ✅ /sitemap-index.xml'); console.log(' ✅ /sitemap-index.xml');
console.log(''); console.log('');
@ -132,6 +154,9 @@ export type {
RobotsConfig, RobotsConfig,
LLMsConfig, LLMsConfig,
HumansConfig, HumansConfig,
SecurityConfig,
CanaryConfig,
CanaryStatement,
SitemapConfig, SitemapConfig,
CachingConfig, CachingConfig,
TemplateConfig, TemplateConfig,

30
src/routes/canary.ts Normal file
View File

@ -0,0 +1,30 @@
import type { APIRoute } from 'astro';
import { generateCanaryTxt } from '../generators/canary.js';
// @ts-ignore - Virtual module
import config from 'virtual:@astrojs/discovery/config';
/**
* API route for /.well-known/canary.txt (warrant canary)
*/
export const GET: APIRoute = ({ site }) => {
const canaryConfig = config.canary || {};
const siteURL = site || new URL('http://localhost:4321');
// Use custom template if provided
const content = config.templates?.canary
? config.templates.canary(canaryConfig, siteURL)
: generateCanaryTxt(canaryConfig, siteURL);
// Get cache duration (default: 1 hour - canaries should be checked frequently)
const cacheSeconds = config.caching?.canary ?? 3600;
return new Response(content, {
status: 200,
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': `public, max-age=${cacheSeconds}`,
},
});
};
export const prerender = true;

30
src/routes/security.ts Normal file
View File

@ -0,0 +1,30 @@
import type { APIRoute } from 'astro';
import { generateSecurityTxt } from '../generators/security.js';
// @ts-ignore - Virtual module
import config from 'virtual:@astrojs/discovery/config';
/**
* API route for /.well-known/security.txt (RFC 9116)
*/
export const GET: APIRoute = ({ site }) => {
const securityConfig = config.security || {};
const siteURL = site || new URL('http://localhost:4321');
// Use custom template if provided
const content = config.templates?.security
? config.templates.security(securityConfig, siteURL)
: generateSecurityTxt(securityConfig, siteURL);
// Get cache duration (default: 24 hours)
const cacheSeconds = config.caching?.security ?? 86400;
return new Response(content, {
status: 200,
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': `public, max-age=${cacheSeconds}`,
},
});
};
export const prerender = true;

View File

@ -8,6 +8,10 @@ export interface DiscoveryConfig {
llms?: LLMsConfig; llms?: LLMsConfig;
/** Configuration for humans.txt generation */ /** Configuration for humans.txt generation */
humans?: HumansConfig; humans?: HumansConfig;
/** Configuration for security.txt generation (RFC 9116) */
security?: SecurityConfig;
/** Configuration for canary.txt generation (warrant canary) */
canary?: CanaryConfig;
/** Configuration passed to @astrojs/sitemap */ /** Configuration passed to @astrojs/sitemap */
sitemap?: SitemapConfig; sitemap?: SitemapConfig;
/** HTTP cache control configuration */ /** HTTP cache control configuration */
@ -126,6 +130,10 @@ export interface CachingConfig {
llms?: number; llms?: number;
/** Cache duration for humans.txt (default: 86400) */ /** Cache duration for humans.txt (default: 86400) */
humans?: number; humans?: number;
/** Cache duration for security.txt (default: 86400) */
security?: number;
/** Cache duration for canary.txt (default: 3600) */
canary?: number;
/** Cache duration for sitemap (default: 3600) */ /** Cache duration for sitemap (default: 3600) */
sitemap?: number; sitemap?: number;
} }
@ -140,6 +148,10 @@ export interface TemplateConfig {
llms?: (config: LLMsConfig, siteURL: URL) => string | Promise<string>; llms?: (config: LLMsConfig, siteURL: URL) => string | Promise<string>;
/** Custom humans.txt template */ /** Custom humans.txt template */
humans?: (config: HumansConfig, siteURL: URL) => string; humans?: (config: HumansConfig, siteURL: URL) => string;
/** Custom security.txt template */
security?: (config: SecurityConfig, siteURL: URL) => string;
/** Custom canary.txt template */
canary?: (config: CanaryConfig, siteURL: URL) => string;
} }
/** /**
@ -220,6 +232,75 @@ export interface SiteInfo {
software?: string[]; software?: string[];
} }
/**
* Configuration for security.txt generation (RFC 9116)
*/
export interface SecurityConfig {
/** Enable/disable security.txt generation (default: true) */
enabled?: boolean;
/** Contact email or URL for security issues (REQUIRED by RFC) */
contact: string | string[];
/** Expiration date (ISO 8601 or 'auto' for 1 year) */
expires?: string | 'auto';
/** URL to encryption key (PGP public key) */
encryption?: string | string[];
/** URL to security acknowledgments/hall of fame */
acknowledgments?: string;
/** Preferred languages for security reports */
preferredLanguages?: string[];
/** URL for canonical security.txt */
canonical?: string;
/** Security policy URL */
policy?: string;
/** Hiring/jobs URL for security team */
hiring?: string;
}
/**
* Configuration for canary.txt generation (warrant canary)
*/
export interface CanaryConfig {
/** Enable/disable canary.txt generation (default: true) */
enabled?: boolean;
/** Organization name */
organization?: string;
/** Contact email for canary inquiries */
contact?: string;
/** Update frequency (daily, weekly, monthly, quarterly, yearly) */
frequency?: 'daily' | 'weekly' | 'monthly' | 'quarterly' | 'yearly';
/** Expiration date (ISO 8601 or 'auto' based on frequency) */
expires?: string | 'auto';
/** Statements about what has NOT been received */
statements?: CanaryStatement[] | (() => CanaryStatement[]);
/** Additional free-form statement text */
additionalStatement?: string;
/** URL to PGP signature or verification */
verification?: string;
/** URL to previous canary (for continuity) */
previousCanary?: string;
/** Blockchain proof information */
blockchainProof?: {
network: string;
address: string;
txHash?: string;
timestamp?: string;
};
/** Include key personnel statement */
personnelStatement?: boolean;
}
/**
* Individual canary statement
*/
export interface CanaryStatement {
/** Type of request/order */
type: 'nsl' | 'fisa' | 'gag' | 'surveillance' | 'backdoor' | 'encryption' | 'other';
/** Human-readable description */
description: string;
/** Whether this type of request has been received */
received: boolean;
}
/** /**
* Sitemap item interface (from @astrojs/sitemap) * Sitemap item interface (from @astrojs/sitemap)
*/ */

View File

@ -3,7 +3,7 @@ import type { DiscoveryConfig } from '../types.js';
/** /**
* Default configuration values * Default configuration values
*/ */
const DEFAULT_CONFIG: Required<Omit<DiscoveryConfig, 'templates'>> & { templates?: DiscoveryConfig['templates'] } = { const DEFAULT_CONFIG: Required<Omit<DiscoveryConfig, 'templates' | 'security' | 'canary'>> & { templates?: DiscoveryConfig['templates']; security?: DiscoveryConfig['security']; canary?: DiscoveryConfig['canary'] } = {
robots: { robots: {
enabled: true, enabled: true,
crawlDelay: 1, crawlDelay: 1,
@ -23,6 +23,8 @@ const DEFAULT_CONFIG: Required<Omit<DiscoveryConfig, 'templates'>> & { templates
robots: 3600, // 1 hour robots: 3600, // 1 hour
llms: 3600, // 1 hour llms: 3600, // 1 hour
humans: 86400, // 24 hours humans: 86400, // 24 hours
security: 86400, // 24 hours
canary: 3600, // 1 hour
sitemap: 3600, // 1 hour sitemap: 3600, // 1 hour
}, },
}; };
@ -51,6 +53,8 @@ export function validateConfig(userConfig: DiscoveryConfig = {}): DiscoveryConfi
...DEFAULT_CONFIG.humans, ...DEFAULT_CONFIG.humans,
...userConfig.humans, ...userConfig.humans,
}, },
security: userConfig.security,
canary: userConfig.canary,
sitemap: { sitemap: {
...DEFAULT_CONFIG.sitemap, ...DEFAULT_CONFIG.sitemap,
...userConfig.sitemap, ...userConfig.sitemap,