I have got few node packages which works in node.js environment and also in browser. Now I have got two seperate tests (for each environment). What is the best way to run these tests with just `npm test` command? Also I want to add these packages to travis.

I&#39;m using `mocha` and `mocha-phantomjs`.

**Node test command**

    node ./node_modules/mocha/bin/mocha ./test/node/index.js --reporter spec

**Browser test command**

    node ./node_modules/mocha-phantomjs/bin/mocha-phantomjs ./test/browser/index.html

**What I tried:**

 1. Add these commands into `npm test` script seperated with semicolon
  - *Problem:* When there is an error in first script but no error in second script, command exited with 0 and travis build passed.
 2. Let node command test in `npm test` script and create custom script for browser tests. Than add these two commands (`npm test` and `npm run-script test-browser`) into travis.yml as array.
  - *Problem:* Users have to run manually two independent test scripts.
 3. Let node command test in `npm test` script and add browser tests to `npm posttest` command. Travis.yml will than have got just one script and users will also have to run one script (everyone is happy).
  - *Problem:* It just does&#39;t feel right, so I wanted to know if there is some better way.

How to run mocha and mocha-phantomjs tests from one &quot;npm test&quot; command in node.js?

I have a long script that was not written by me. When i run it I get:

    phantomjs file.js
    SyntaxError: Parse error

i checked out the manual and --help, and the best i could came up with was:

    phantomjs --debug=yes file.js
    (irrelevant debug statement from CookieJar)
    SyntaxError: Parse error

Is there any better way to get at least a line number? or any hint at all?

getting more information from phantomjs &quot;SyntaxError: Parse error&quot; message

phantomjs has config loadImage,

but I want more, 

how can I control phantomjs to skip download some kind of resource, 

such as css etc...


=====

good news:
this feature is added.

https://code.google.com/p/phantomjs/issues/detail?id=230

The gist:

    page.onResourceRequested = function(requestData, request) {
        if ((/http:\/\/.+?\.css/gi).test(requestData[&#39;url&#39;]) || requestData[&#39;Content-Type&#39;] == &#39;text/css&#39;) {
            console.log(&#39;The url of the request is matching. Aborting: &#39; + requestData[&#39;url&#39;]);
            request.abort();
        }
    };

How can I control PhantomJS to skip download some kind of resource?

How can one remove a whole IndexedDB database from JavaScript, as opposed to just an object store? I&#39;m using the [IndexedDB shim](http://nparashuram.com/IndexedDBShim/), which may use WebSQL as its backend.

I&#39;d mainly like to know how to do this for the PhantomJS (headless) browser, although Chrome, Safari (on iPad) and IE10 are other important browsers.

How can I remove a whole IndexedDB database from JavaScript?

I&#39;m not sure why PhantomJS is necessary and I&#39;m hoping someone can help.

Why would a developer ever need PhantomJS as opposed to just using some testing framework?

There is a site I am trying to scrape, that first loads an html/js
modifies the form input fields using js and then POSTs.
How can I get the final html output of the POSTed page?

I tried to do this with phantomjs, but it seems to only have an option to render image files. Googling around suggests it should be possible , but I can&#39;t figure out how. My attempt:

    var page = require(&#39;webpage&#39;).create();
    var fs = require(&#39;fs&#39;);
    page.open(&#39;https://www.somesite.com/page.aspx&#39;, function () {
        page.evaluate(function(){
            
        });
    
        page.render(&#39;export.png&#39;);
        fs.write(&#39;1.html&#39;, page.content, &#39;w&#39;);
        phantom.exit();
    });


This code will be used for a client, I can&#39;t expect him to install too many packages (nodejs , casperjs etc)

Thanks

save html output of page after execution of the page&#39;s javascript

I want to use [PhantomJS][1] in [Python][2]. I googled this problem but couldn&#39;t find proper solutions.

I find `os.popen()`  may be a good choice. But I couldn&#39;t pass some arguments to it.

Using `subprocess.Popen()` may be a proper solution for now. I want to know whether there&#39;s a better solution or not.

Is there a way to use PhantomJS in Python?





  [1]: http://phantomjs.org/
  [2]: http://www.python.org/

Is there a way to use PhantomJS in Python?

I set up PhantomJS and recorded it to video: https://www.dailymotion.com/video/xnizmh_1_webcam

Build instructions: http://phantomjs.org/build.html

Is there anything wrong in my setup?

After I set it up I read the [quick start tutorial][1] and tried to write this code

    phantomjs hello.js 

It gives me &quot;command not found&quot; error. How can I solve this problem?


  [1]: https://github.com/ariya/phantomjs/wiki/Quick-Start

How can I setup &amp; run PhantomJS on Ubuntu?

I&#39;m trying to use phantomJS (what an awesome tool btw!) to submit a form for a page that I have login credentials for, and then output the content of the destination page to stdout. I&#39;m able to access the form and set its values successfully using phantom, but I&#39;m not quite sure what the right syntax is to submit the form and output the content of the subsequent page. What I have so far is:

    var page = new WebPage();
    var url = phantom.args[0];

    page.open(url, function (status) {

      if (status !== &#39;success&#39;) {
          console.log(&#39;Unable to access network&#39;);
      } else {

        console.log(page.evaluate(function () {
          
          var arr = document.getElementsByClassName(&quot;login-form&quot;);
          var i;

          for (i=0; i &lt; arr.length; i++) {

            if (arr[i].getAttribute(&#39;method&#39;) == &quot;POST&quot;) {
              arr[i].elements[&quot;email&quot;].value=&quot;mylogin@somedomain.com&quot;;
              arr[i].elements[&quot;password&quot;].value=&quot;mypassword&quot;;

              // This part doesn&#39;t seem to work. It returns the content
              // of the current page, not the content of the page after 
              // the submit has been executed. Am I correctly instrumenting
              // the submit in Phantom?
              arr[i].submit();
              return document.querySelectorAll(&#39;html&#39;)[0].outerHTML;
            }

          }
          
          return &quot;failed :-(&quot;;

        }));
      }

      phantom.exit();
    }

How to submit a form using PhantomJS

We are using Selenium to automate our `UI` testing. Recently we have seen majority of our users using Chrome. So we wanted to know - pros and cons of using PhantomJS vs Selenium:

 - Is there any real advantage in terms of performance, e.g. time taken to execute the test cases?
 - When should one prefer PhantomJS over Selenium?

Casperjs/PhantomJs vs Selenium

I&#39;ve been working on how to make a SPA crawlable by google based on google&#39;s [instructions](https://developers.google.com/webmasters/ajax-crawling/?hl=iw&amp;csw=1). Even though there are quite a few general explanations I couldn&#39;t find anywhere a more thorough step-by-step tutorial with actual examples.  After having finished this I would like to share my solution so that others may also make use of it and possibly improve it further. &lt;br/&gt; 
I am using `MVC` with `Webapi` controllers, and [Phantomjs](http://phantomjs.org/) on the server side, and [Durandal](http://durandaljs.com/) on the client side with `push-state` enabled; I also use [Breezejs](http://www.breezejs.com/) for client-server data interaction, all of which I strongly recommend, but I&#39;ll try to give a general enough explanation that will also help people using other platforms.&lt;br/&gt;


How to make a SPA SEO crawlable?

I&#39;m using [PhantomJS][1] v1.4.1 to load some web pages. I don&#39;t have access to their server-side, I just getting links pointing to them. I&#39;m using obsolete version of Phantom because I need to support Adobe Flash on that web pages.

The problem is many web-sites are loading their minor content async and that&#39;s why Phantom&#39;s onLoadFinished callback (analogue for onLoad in HTML) fired too early when not everything still has loaded. Can anyone suggest how can I wait for full load of a webpage to make, for example, a screenshot with all dynamic content like ads?

  [1]: http://phantomjs.org/

phantomjs not waiting for &quot;full&quot; page load

I&#39;m using the following code based on loadspeed.js example to open up a https:// site which requires http server authentication as well.

    var page = require(&#39;webpage&#39;).create(), system = require(&#39;system&#39;), t, address;
    
    page.settings.userName = &#39;myusername&#39;;
    page.settings.password = &#39;mypassword&#39;;
    
    if (system.args.length === 1) {
        console.log(&#39;Usage: scrape.js &lt;some URL&gt;&#39;);
        phantom.exit();
    } else {
        t = Date.now();
        address = system.args[1];
        page.open(address, function (status) {
            if (status !== &#39;success&#39;) {
                console.log(&#39;FAIL to load the address&#39;);
            } else {
                t = Date.now() - t;
                console.log(&#39;Page title is &#39; + page.evaluate(function () {
                    return document.title;
                }));
                console.log(&#39;Loading time &#39; + t + &#39; msec&#39;);
            }
            phantom.exit();
        });
    }  

Its failing to load the page all the time. What could be wrong here? Are secured sites to be handled any differently? The site can be accessed successfully from browser though.

I&#39;m just starting with Phantom right now and find it too good to stop playing around even though i&#39;m not moving forward with this issue.

PhantomJS failing to open HTTPS site

How do I click an element in PhantomJS?

    page.evaluate(function() {
        document.getElementById(&#39;idButtonSpan&#39;).click();  
    });

This gives me an error &quot;undefined is not a function...&quot;

If I instead 

     return document.getElementById(&#39;idButtonSpan&#39;);

and then print it,

then it prints [object object], so the element does exist.

The element acts as a button, but it&#39;s actually just a span element, not a submit input.


I was able to get this button click to work with Casper, but Casper had other limitations so I&#39;m back to PhantomJS.

PhantomJS; click an element

I&#39;m trying to put list of possible solutions for browser automatic tests suits and headless browser platforms capable of scraping.

***

**BROWSER TESTING / SCRAPING:**

 - [Selenium][1] - **polyglot** flagship in browser automation, bindings for Python, Ruby,  JavaScript, C#, Haskell and more, IDE for Firefox (as an extension) for faster test deployment. Can act as a Server and has tons of features.
 

**JAVASCRIPT**

 - [PhantomJS][2] - **JavaScript**, headless testing with screen capture and automation, uses **Webkit**. As of version 1.8 Selenium&#39;s WebDriver API is implemented, so you can use any WebDriver binding and tests will be compatible with Selenium
 - [SlimerJS][3] - similar to PhantomJS, uses **Gecko** (Firefox) instead of **WebKit**
 - [CasperJS][4] - **JavaScript**, build on both PhantomJS and SlimerJS, has extra features
 - [Ghost Driver][5] - **JavaScript** implementation of the **WebDriver Wire Protocol** for **PhantomJS**. 
 - ***new*** [PhantomCSS][6] - CSS regression testing. A CasperJS module for automating visual regression testing with PhantomJS and [Resemble.js][7].
 - ***new*** [WebdriverCSS][8] - plugin for [Webdriver.io][9] for automating visual regression testing
 - ***new*** [PhantomFlow][10] - Describe and visualize user flows through tests. An experimental approach to Web user interface testing.
 - ***new*** [trifleJS][11] - ports the PhantomJS API to use the Internet Explorer engine.
 - ***new*** [CasperJS IDE][12] *(commercial)*

**NODE.JS**

 - [Node-phantom][13] - bridges the gap between **PhantomJS** and **node.js**
 - [WebDriverJs][14] - Selenium WebDriver bindings for node.js by Selenium Team
 - [WD.js][15] - node module for WebDriver/Selenium 2
 - [yiewd][16] - WD.js wrapper using latest Harmony generators! Get rid of the callback pyramid with **yield** 
 - [ZombieJs][17] - Insanely fast, headless full-stack testing using **node.js**
 - [NightwatchJs][18] - Node JS based testing solution using Selenium Webdriver
 - [Chimera][19] - Chimera: can do everything what phantomJS does, but in a full JS environment
 - [Dalek.js][20] - Automated cross browser testing with JavaScript through Selenium Webdriver
 - [Webdriver.io][9] - better implementation of WebDriver bindings with predefined 50+ actions
 - [Nightmare][21] - Electron bridge with a high-level API. 
 - [jsdom][22] - Tailored towards web scraping. A very lightweight DOM implemented in Node.js, it supports pages with javascript. 
 - ***new*** [Puppeteer][23] - Node library which provides a high-level API to control Chrome or Chromium. Puppeteer runs headless by default.

**WEB SCRAPING / MINING**

 - [Scrapy][24] - **Python**, mainly a scraper/miner - fast, well documented and, can be linked with [Django Dynamic Scraper][25] for nice mining deployments, or [Scrapy Cloud][26] for PaaS (server-less) deployment, works in terminal or an server stand-alone proces, can be used with **Celery**, built on top of **Twisted**
 - [Snailer][27] - **node.js** module, untested yet. 
 - [Node-Crawler][28] - **node.js** module, untested yet.

**ONLINE TOOLS**

 - ***new*** [Web Scraping Language][29] - Simple syntax to crawl the web

 - ***new*** [Online HTTP client][30] - Dedicated SO answer

 - ***dead*** [CasperBox][31] - Run **CasperJS** scripts online

**Android TOOLS for Automation**

 - ***new*** [Mechanica Browser App][33]

***

**RELATED LINKS &amp; RESOURCES** 

 - [Comparsion of Webscraping software][32]
 - ***new*** [Resemble.js][7] : Image analysis and comparison

**Questions:**

- Any pure Node.js solution or Nodejs to PhanthomJS/CasperJS module that actually works and is documented?

**Answer:** Chimera seems to go in that direction, checkout [Chimera][19]

- Other solutions capable of easier JavaScript injection than Selenium?

- Do you know any pure **ruby** solutions?

**Answer:** Checkout the list created by rjk with ruby based solutions

- Do you know any related tech or solution?

**Feel free to edit this question and add content as you wish!** Thank you for your contributions!


  [1]: http://seleniumhq.org
  [2]: http://phantomjs.org/
  [3]: http://slimerjs.org/
  [4]: http://casperjs.org/
  [5]: https://github.com/detro/ghostdriver
  [6]: https://github.com/Huddle/PhantomCSS
  [7]: http://huddle.github.io/Resemble.js/
  [8]: https://github.com/webdriverjs/webdrivercss
  [9]: http://webdriver.io
  [10]: https://github.com/Huddle/PhantomFlow
  [11]: http://triflejs.org/
  [12]: https://gum.co/casperjside
  [13]: https://github.com/alexscheelmeyer/node-phantom
  [14]: https://code.google.com/p/selenium/wiki/WebDriverJs
  [15]: https://github.com/admc/wd
  [16]: https://github.com/jlipps/yiewd
  [17]: http://zombie.labnotes.org/
  [18]: http://nightwatchjs.org/
  [19]: https://github.com/deanmao/node-chimera
  [20]: http://dalekjs.com
  [21]: https://github.com/segmentio/nightmare
  [22]: https://github.com/tmpvar/jsdom
  [23]: https://github.com/GoogleChrome/puppeteer
  [24]: http://scrapy.org
  [25]: http://django-dynamic-scraper.readthedocs.org/en/latest/
  [26]: http://scrapinghub.com/scrapy-cloud.html
  [27]: https://github.com/matthewhudson/snailer
  [28]: https://github.com/sylvinus/node-crawler.git
  [29]: https://www.npmjs.com/package/wslang
  [30]: https://stackoverflow.com/a/12637452/363573
  [31]: https://www.casperbox.com
  [32]: http://blog.screen-scraper.com/2010/06/28/comparison-of-web-scraping-software/
  [33]: https://play.google.com/store/apps/details?id=tools.automate.browser.mechanica

Headless Browser and scraping - solutions

So I am trying to open websites on new tabs inside my WebDriver. I want to do this, because opening a new WebDriver for each website takes about 3.5secs using PhantomJS, I want more speed...

I&#39;m using a multiprocess python script, and I want to get some elements from each page, so the workflow is like this:


    Open Browser
    
    Loop throught my array
    For element in array -&gt; Open website in new tab -&gt; do my business -&gt; close it

But I can&#39;t find any way to achieve this.



Here&#39;s the code I&#39;m using. It takes forever between websites, I need it to be fast... Other tools are allowed, but I don&#39;t know too many tools for scrapping website content that loads with JavaScript (divs created when some event is triggered on load etc) That&#39;s why I need Selenium... BeautifulSoup can&#39;t be used for some of my pages.

    #!/usr/bin/env python
    import multiprocessing, time, pika, json, traceback, logging, sys, os, itertools, urllib, urllib2, cStringIO, mysql.connector, shutil, hashlib, socket, urllib2, re
    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    from PIL import Image
    from os import listdir
    from os.path import isfile, join
    from bs4 import BeautifulSoup
    from pprint import pprint
    
    def getPhantomData(parameters):
    	try:
    		# We create WebDriver
    		browser = webdriver.Firefox()
    		# Navigate to URL
    		browser.get(parameters[&#39;target_url&#39;])
    		# Find all links by Selector
    		links = browser.find_elements_by_css_selector(parameters[&#39;selector&#39;])
    
    		result = []
    		for link in links:
    			# Extract link attribute and append to our list
    			result.append(link.get_attribute(parameters[&#39;attribute&#39;]))
    		browser.close()
    		browser.quit()
    		return json.dumps({&#39;data&#39;: result})
    	except Exception, err:
    		browser.close()
    		browser.quit()
    		print err
    
    def callback(ch, method, properties, body):
    	parameters = json.loads(body)
    	message = getPhantomData(parameters)
    
    	if message[&#39;data&#39;]:
    	    ch.basic_ack(delivery_tag=method.delivery_tag)
    	else:
    	    ch.basic_reject(delivery_tag=method.delivery_tag, requeue=True)
    
    def consume():
    	credentials = pika.PlainCredentials(&#39;invitado&#39;, &#39;invitado&#39;)
    	rabbit = pika.ConnectionParameters(&#39;localhost&#39;,5672,&#39;/&#39;,credentials)
    	connection = pika.BlockingConnection(rabbit)
    	channel = connection.channel()
    
    	# Conectamos al canal
    	channel.queue_declare(queue=&#39;com.stuff.images&#39;, durable=True)
    	channel.basic_consume(callback,queue=&#39;com.stuff.images&#39;)
    
    	print &#39; [*] Waiting for messages. To exit press CTRL^C&#39;
    	try:
    		channel.start_consuming()
    	except KeyboardInterrupt:
    		pass
    
    workers = 5
    pool = multiprocessing.Pool(processes=workers)
    for i in xrange(0, workers):
    	pool.apply_async(consume)
    
    try:
    	while True:
    		continue
    except KeyboardInterrupt:
    	print &#39; [*] Exiting...&#39;
    	pool.terminate()
    	pool.join()

Open web in new tab Selenium + Python

I&#39;m planning a webservice for my own use internally that takes one argument, a URL, and returns html representing the *resolved* DOM from that URL. By resolved I mean that the webservice will firstly get the page at that URL, then use PhantomJS to &#39;render&#39; the page, and then return the resulting source after all DHTML, AJAX calls etc are executed. However launching phantom on a per-request basis (which I&#39;m doing now) is *way* too sluggish. I would rather have a pool of PhantomJS instances with one always available to serve the latest call to my webservice.

Has any work been done on this kind of thing before? I&#39;d rather base this webservice on the work of others than write a pool manager / http proxy server for myself from scratch.

**More Context**: I&#39;ve listed the 2 similar projects that I&#39;ve seen so far below and why I&#39;ve avoided each one, resulting in this question about managing a pool of PhantomJS instances instead.

jsdom - from what I&#39;ve seen it has great functionality for executing scripts on a page, but it doesn&#39;t attempt to replicate browser behaviour, so if I were use it as a general purpose &quot;DOM resolver&quot; there&#39;d end up being a lot of extra coding to handle all kinds of edges cases, event calling, etc. The first example I saw was having to manually call the onload() function of the body tag for a test app I set up using node. It seemed like the beginning of a deep rabbit hole.

Selenium - It just has soo many more moving parts, so setting up a pool to manage long lived browser instances will just be more complicated than using PhantomJS. I don&#39;t need any of it&#39;s macro recording / scripting benefits. I just want a webservice that is as performant at getting a webpage and resolving it&#39;s DOM as if I were browsing to that URL with a browser (or even faster if I can make it ignore images etc.)

How to manage a &#39;pool&#39; of PhantomJS instances

I am trying to spawn an external process `phantomjs` using node&#39;s `child_process` and then send information to that process after it was initialized, is that possible?

I have the following code:

    var spawn = require(&#39;child_process&#39;).spawn,
        child = spawn(&#39;phantomjs&#39;);

    child.stdin.setEncoding = &#39;utf-8&#39;;
    child.stdout.pipe(process.stdout);

    child.stdin.write(&quot;console.log(&#39;Hello from PhantomJS&#39;)&quot;);

But the only thing I got on the stdout is the initial prompt for phantomjs console.

    phantomjs&gt; 

So it seems the `child.stdin.write` is not making any effect.

I am  not sure I can send additional information to phantomjs ater the initial spawn.

thanks in advance.

Nodejs Child Process: write to stdin from an already initialised process

Just installed phantomjs, mac os x yosemite. Whenever I run /bin/phantomjs, with any parameter, I get `Killed: 9`. Any idea?

PhantomJS: getting &quot;Killed: 9&quot; for anything I&#39;m trying

I&#39;m looking for an example of requesting a webpage, waiting for the JavaScript to render (JavaScript modifies the DOM), and then grabbing the HTML of the page.

This should be a simple example with an obvious use-case for PhantomJS. I can&#39;t find a decent example, the documentation seems to be all about command line use.

Save and render a webpage with PhantomJS and node.js

1. What are the differences between PhantomJS and capybara-webkit?
2. What are the advantages of capybara-webkit over PhantomJS?
3. Which of the two is the most efficient tool?
4. Others ...

How do poltergeist/PhantomJS and capybara-webkit differ?

I&#39;m using PhantomJS page.evaluate() to do some scraping. My problem is that the code I pass to the webkit page is sandboxed, and so has no access to the variables of my main phantom script. This makes it hard make the scraping code generic.

    page.open(url, function() {
      var foo = 42;

      page.evaluate(function() {
        // this code has no access to foo
        console.log(foo);
      });
    }

How could I push arguments into the page?




All Phantomjs Solutions on Gang of Coders

Total of 22 Phantomjs Solutions

How to run mocha and mocha-phantomjs tests from one "npm test" command in node.js?

getting more information from phantomjs "SyntaxError: Parse error" message