Skip to content

Instantly share code, notes, and snippets.

@JJetmar
Last active October 23, 2023 09:27
Show Gist options
  • Select an option

  • Save JJetmar/687b070a6032d50651dd745fdc782c40 to your computer and use it in GitHub Desktop.

Select an option

Save JJetmar/687b070a6032d50651dd745fdc782c40 to your computer and use it in GitHub Desktop.
Blocks extra unnecessary requests in PupeteerCrawler / Apify SDK
// ...
const crawler = new PuppeteerCrawler({
// ...
preNavigationHooks: [
/*
// Only waits for the initial HTML page and not related assets like, images, etc (also optimization)
async (gotoOptions) => {
if (!gotoOptions) return;
gotoOptions.waitUntil = 'domcontentloaded';
},
*/
// Block all browsers requests that sucess these patterns
async ({ blockRequests }) => {
await blockRequests({
extraUrlPatterns: [
'gstatic.com/images',
'gstatic.com/favicon',
'_mss',
'fonts.gstatic.com',
'ssl.gstatic.com',
'googleusercontent.com',
'ogs.google.com',
'play.google.com',
'.css',
'.svg',
],
});
},
],
// ...
});
// ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment