task.md

Task

Implement an Express middleware function called verifyCrawler(). It should identify whether an incoming request is from Google's Mediapartner crawler and attach a verified flag to the request object. A downstream route handler will use this flag to decide what content to return.

Requirements

• Check that the User-Agent header contains the string 'Mediapartners-Google'
• Fetch Google's official crawler IP ranges from https://developers.google.com/static/search/apis/ipranges/special-crawlers.json and verify the request IP is in one of those ranges
• Cache the IP ranges — do not fetch on every request
• Attach req.isCrawler = true if both checks pass, false otherwise
• The middleware must never crash the server — any failure should fail safe (req.isCrawler = false)

verifyCrawler.ts

// verifyCrawler.ts
import { Request, Response, NextFunction } from 'express';

// Google's crawler IP ranges endpoint
const CRAWLER_IP_RANGES_URL =
  'https://developers.google.com/static/search/apis/ipranges/special-crawlers.json';

// TODO: implement caching here

export async function verifyCrawler(
  req: Request,
  res: Response,
  next: NextFunction
): Promise<void> {
  // TODO: implement
  req.isCrawler = false;
  next();
}