E-Mail-Versand mit hohem Volumen: Skalierung auf Millionen pro Tag

Das Versenden von Millionen E-Mails pro Tag erfordert eine sorgfältige Architektur. Warteschlangen-Management, Rate Limiting und Infrastruktur-Design beeinflussen Zustellbarkeit und Zuverlässigkeit. So bauen Sie für Skalierung.

Architekturüberblick

Kernkomponenten

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   API/App   │────▶│    Queue    │────▶│   Workers   │
└─────────────┘     └─────────────┘     └─────────────┘
                                               │
                    ┌─────────────┐            │
                    │   Provider  │◀───────────┘
                    │     API     │
                    └─────────────┘

Warteschlangen-Design

interface EmailJob {
  id: string;
  priority: 'critical' | 'high' | 'normal' | 'low';
  payload: {
    to: string;
    from: string;
    subject: string;
    html: string;
  };
  metadata: {
    userId: string;
    campaignId?: string;
    scheduledFor?: Date;
  };
  attempts: number;
  maxAttempts: number;
  createdAt: Date;
}


// Priority queues
const queues = {
  critical: 'email:critical',  // Password resets, 2FA
  high: 'email:high',          // Transactional
  normal: 'email:normal',      // Notifications
  low: 'email:low'             // Marketing, digests
};

Rate Limiting

Provider Rate Limits

class RateLimiter {
  private limits: Map<string, { count: number; resetAt: Date }> = new Map();
  
  async canSend(provider: string): Promise<boolean> {
    const limit = this.limits.get(provider);
    
    if (!limit || limit.resetAt < new Date()) {
      this.limits.set(provider, {
        count: 1,
        resetAt: addSeconds(new Date(), 1)
      });
      return true;
    }
    
    if (limit.count >= PROVIDER_RATE_LIMITS[provider]) {
      return false;
    }
    
    limit.count++;
    return true;
  }
}

ISP-Throttling

const ispLimits = {
  'gmail.com': { perHour: 500, perDay: 5000 },
  'yahoo.com': { perHour: 200, perDay: 2000 },
  'outlook.com': { perHour: 300, perDay: 3000 }
};

async function getThrottledBatch(emails: Email[]): Promise<Email[]> {
  const byDomain = groupBy(emails, e => getDomain(e.to));
  const batch: Email[] = [];
  
  for (const [domain, domainEmails] of Object.entries(byDomain)) {
    const limit = ispLimits[domain] || { perHour: 1000 };
    const sent = await getSentCount(domain, 'hour');
    const available = limit.perHour - sent;
    
    batch.push(...domainEmails.slice(0, available));
  }
  
  return batch;
}

Worker-Skalierung

Horizontale Skalierung

// Worker configuration
const workerConfig = {
  concurrency: 50,           // Emails per worker
  batchSize: 100,            // Fetch from queue
  pollInterval: 100,         // ms between polls
  maxRetries: 3,
  retryDelay: [1000, 5000, 30000] // Exponential backoff
};

async function processQueue(queue: string) {
  while (true) {
    const jobs = await redis.lpop(queue, workerConfig.batchSize);
    
    if (jobs.length === 0) {
      await sleep(workerConfig.pollInterval);
      continue;
    }
    
    await Promise.all(
      jobs.map(job => processJob(JSON.parse(job)))
    );
  }
}

Auto-Scaling-Trigger

const scalingRules = {
  scaleUp: {
    queueDepth: 10000,      // Jobs waiting
    processingTime: 5000,   // ms per job
    errorRate: 0.05         // 5% errors
  },
  scaleDown: {
    queueDepth: 100,
    idleTime: 300000        // 5 minutes idle
  }
};

Batch-Verarbeitung

Effizientes Batching

async function sendBatch(emails: Email[]): Promise<BatchResult> {
  // Group by template for efficiency
  const byTemplate = groupBy(emails, 'templateId');
  const results: SendResult[] = [];
  
  for (const [templateId, templateEmails] of Object.entries(byTemplate)) {
    // Compile template once
    const template = await getCompiledTemplate(templateId);
    
    // Send in parallel with concurrency limit
    const batchResults = await pMap(
      templateEmails,
      email => sendWithTemplate(email, template),
      { concurrency: 50 }
    );
    
    results.push(...batchResults);
  }
  
  return { sent: results.filter(r => r.success).length, results };
}

Monitoring in großem Maßstab

Schlüsselmetriken

const metrics = {
  // Throughput
  emailsPerSecond: gauge('emails_per_second'),
  emailsPerMinute: gauge('emails_per_minute'),
  
  // Latency
  queueLatency: histogram('queue_latency_ms'),
  sendLatency: histogram('send_latency_ms'),
  
  // Errors
  errorRate: gauge('error_rate'),
  bounceRate: gauge('bounce_rate'),
  
  // Queue health
  queueDepth: gauge('queue_depth'),
  oldestJob: gauge('oldest_job_age_seconds')
};

Alarm-Schwellenwerte

const alerts = [
  {
    name: 'high_queue_depth',
    condition: 'queue_depth > 100000',
    severity: 'warning'
  },
  {
    name: 'high_error_rate',
    condition: 'error_rate > 0.05',
    severity: 'critical'
  },
  {
    name: 'slow_processing',
    condition: 'queue_latency_p99 > 60000',
    severity: 'warning'
  }
];

Datenbankaspekte

Effiziente Speicherung

// Partition by date for easy cleanup
const emailLogSchema = {
  tableName: 'email_logs',
  partitionBy: 'RANGE (created_at)',
  indexes: [
    'user_id',
    'campaign_id',
    'status',
    'created_at'
  ],
  retention: '90 days'
};

// Archive old data
async function archiveOldEmails() {
  await db.query(`
    INSERT INTO email_logs_archive
    SELECT * FROM email_logs
    WHERE created_at < NOW() - INTERVAL '90 days'
  `);
  
  await db.query(`
    DELETE FROM email_logs
    WHERE created_at < NOW() - INTERVAL '90 days'
  `);
}

Best Practices

1. —Warteschlangen priorisieren - kritische E-Mails zuerst
2. —Rate Limits beachten - sowohl Provider als auch ISP
3. —Alles monitoren - Warteschlangentiefe, Latenz, Fehler
4. —Auf Ausfälle vorbereiten - Retries, Dead-Letter-Queues
5. —Horizontal skalieren - Worker hinzufügen, nicht größere Server
6. —Effizient batchen - nach Template, Domain gruppieren
7. —Aggressiv archivieren - lassen Sie Logs nicht endlos wachsen

E-Mail mit hohem Volumen dreht sich um konsistente, zuverlässige Zustellung im großen Maßstab. Bauen Sie für das erwartete Volumen, aber entwerfen Sie für 10x Wachstum.