Skip to content

Instantly share code, notes, and snippets.

Task

Implement an Express middleware function called verifyCrawler(). It should identify whether an incoming request is from Google's Mediapartner crawler and attach a verified flag to the request object. A downstream route handler will use this flag to decide what content to return.

Requirements

  • Check that the User-Agent header contains the string 'Mediapartners-Google'
  • Fetch Google's official crawler IP ranges from https://developers.google.com/static/search/apis/ipranges/special-crawlers.json and verify the request IP is in one of those ranges
  • Cache the IP ranges — do not fetch on every request
  • Attach req.isCrawler = true if both checks pass, false otherwise
  • The middleware must never crash the server — any failure should fail safe (req.isCrawler = false)
@xijo
xijo / doc-to-md.rb
Last active August 11, 2020 10:20
Ruby script to convert google docs into markdown
# Usage ruby doc-to-md.rb input.html > output.md
# Make sure to install reverse_markdown: gem install reverse_markdown
require 'reverse_markdown'
require 'cgi'
input = ARGF.read
BOLD_CLASSES = input.scan(/\.(\w+)\s*\{[^}]*font-weight:\s*700[^}]*\}/).flatten
@xijo
xijo / html_entities.json
Last active May 28, 2019 17:56
javascript / JSON mapping from common HTML entites to their encoded version
{
"&": "&",
">": ">",
"<": "&lt;",
"\"": "&quot;",
"´": "&acute;",
"¸": "&cedil;",
"ˆ": "&circ;",
"¯": "&macr;",
"·": "&middot;",
@xijo
xijo / rubiks-cube.svg
Created February 15, 2019 00:06
SVG rubiks cube top view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@xijo
xijo / email_validator.gemspec
Created January 17, 2019 10:15
Simple email validation micro gem
Gem::Specification.new do |s|
s.name = 'email_validator'
s.version = '0.1.0'
s.platform = Gem::Platform::RUBY
s.author = 'Betterplace Developers'
s.email = 'developers@betterplace.org'
s.summary = 'Simple email validation for rails'
s.description = "Stay simple but don't interfere with `mail`"
s.files = ['email_validator.rb']
bin/yarn add less
bin/yarn add less-loader
bin/yarn add semantic-ui-less
import "semantic-ui-less/semantic.less"
@primaryColor : #bad304;
const path = require('path')
const customConfig = {
plugins: [],
resolve: {
alias: {
'../../theme.config': path.join(__dirname, '../../new-theme-name/theme.config')
}
}
}