How can I pass variable into an evaluate function?

JavascriptWeb ScrapingEvaluatePuppeteer

Javascript Problem Overview


I'm trying to pass a variable into a page.evaluate() function in Puppeteer, but when I use the following very simplified example, the variable evalVar is undefined.

I'm new to Puppeteer and can't find any examples to build on, so I need help passing that variable into the page.evaluate() function so I can use it inside.

const puppeteer = require('puppeteer');

(async() => {

  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();

  const evalVar = 'WHUT??';

  try {

    await page.goto('https://www.google.com.au');
    await page.waitForSelector('#fbar');
    const links = await page.evaluate((evalVar) => {

      console.log('evalVar:', evalVar); // appears undefined

      const urls = [];
      hrefs = document.querySelectorAll('#fbar #fsl a');
      hrefs.forEach(function(el) {
        urls.push(el.href);
      });
      return urls;
    })
    console.log('links:', links);

  } catch (err) {

    console.log('ERR:', err.message);

  } finally {

    // browser.close();

  }

})();

Javascript Solutions


Solution 1 - Javascript

You have to pass the variable as an argument to the pageFunction like this:

const links = await page.evaluate((evalVar) => {

  console.log(evalVar); // 2. should be defined now
  …

}, evalVar); // 1. pass variable as an argument

You can pass in multiple variables by passing more arguments to page.evaluate():

await page.evaluate((a, b c) => { console.log(a, b, c) }, a, b, c)

The arguments must either be serializable as JSON or JSHandles of in-browser objects: https://pptr.dev/#?show=api-pageevaluatepagefunction-args

Solution 2 - Javascript

I encourage you to stick on this style, because it's more convenient and readable.

let name = 'jack';
let age  = 33;
let location = 'Berlin/Germany';

await page.evaluate(({name, age, location}) => {

    console.log(name);
    console.log(age);
    console.log(location);
    
},{name, age, location});

Solution 3 - Javascript

Single Variable:

You can pass one variable to page.evaluate() using the following syntax:

await page.evaluate(example => { /* ... */ }, example);

> Note: You do not need to enclose the variable in (), unless you are going to be passing multiple variables.

Multiple Variables:

You can pass multiple variables to page.evaluate() using the following syntax:

await page.evaluate((example_1, example_2) => { /* ... */ }, example_1, example_2);

> Note: Enclosing your variables within {} is not necessary.

Solution 4 - Javascript

It took me quite a while to figure out that console.log() in evaluate() can't show in node console.

Ref: https://github.com/GoogleChrome/puppeteer/issues/1944

> everything that is run inside the page.evaluate function is done in the context of the browser page. The script is running in the browser not in node.js so if you log it will show in the browsers console which if you are running headless you will not see. You also can't set a node breakpoint inside the function.

Hope this can help.

Solution 5 - Javascript

For pass a function, there are two ways you can do it.

// 1. Defined in evaluationContext
await page.evaluate(() => {
  window.yourFunc = function() {...};
});
const links = await page.evaluate(() => {
  const func = window.yourFunc;
  func();
});


// 2. Transform function to serializable(string). (Function can not be serialized)
const yourFunc = function() {...};
const obj = {
  func: yourFunc.toString()
};
const otherObj = {
  foo: 'bar'
};
const links = await page.evaluate((obj, aObj) => {
   const funStr = obj.func;
   const func = new Function(`return ${funStr}.apply(null, arguments)`)
   func();
   
   const foo = aObj.foo; // bar, for object
   window.foo = foo;
   debugger;
}, obj, otherObj);

You can add devtools: true to the launch options for test

Solution 6 - Javascript

I have a typescript example that could help someone new in typescript.

const hyperlinks: string [] = await page.evaluate((url: string, regex: RegExp, querySelect: string) => {
.........
}, url, regex, querySelect);

Solution 7 - Javascript

Slightly different version from @wolf answer above. Make code much more reusable between different context.

// util functions
export const pipe = (...fns) => initialVal => fns.reduce((acc, fn) => fn(acc), initialVal)
export const pluck = key => obj => obj[key] || null
export const map = fn => item => fn(item)
// these variables will be cast to string, look below at fn.toString()
const updatedAt = await page.evaluate(
  ([selector, util]) => {
    let { pipe, map, pluck } = util
    pipe = new Function(`return ${pipe}`)()
    map = new Function(`return ${map}`)()
    pluck = new Function(`return ${pluck}`)()

    return pipe(
      s => document.querySelector(s),
      pluck('textContent'),
      map(text => text.trim()),
      map(date => Date.parse(date)),
      map(timeStamp => Promise.resolve(timeStamp))
    )(selector)
  },
  [
    '#table-announcements tbody td:nth-child(2) .d-none',
    { pipe: pipe.toString(), map: map.toString(), pluck: pluck.toString() },
  ]
)

Also not that functions inside pipe cant used something like this

// incorrect, which is i don't know why
pipe(document.querySelector) 

// should be 
pipe(s => document.querySelector(s))

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionCat BurstonView Question on Stackoverflow
Solution 1 - JavascriptfloziaView Answer on Stackoverflow
Solution 2 - JavascriptMehdi RaashView Answer on Stackoverflow
Solution 3 - JavascriptGrant MillerView Answer on Stackoverflow
Solution 4 - JavascriptharrrrrrryView Answer on Stackoverflow
Solution 5 - JavascriptwolfView Answer on Stackoverflow
Solution 6 - JavascriptSrinivas Reddy ThatiparthyView Answer on Stackoverflow
Solution 7 - JavascriptAzrizView Answer on Stackoverflow