Selenium WebDriver: Wait for complex page with JavaScript to load

I have a web application to test with Selenium. There is a lot of JavaScript running on page load.
This JavaScript code is not so well written but I can't change anything. So waiting for an element to appear in the DOM with findElement() method is not an option.
I want to create a generic function in Java to wait for a page to load, a possible solution would be:

• run a JavaScript script form WebDriver and store the result of document.body.innerHTML in a string variable body.
• compare the body variable to the previous version of body. if they are the same then set increment a counter notChangedCount otherwise set notChangedCount to zero.
• wait for a litte time (50 ms for example).
• if the page has not changed for some time (500 ms for example) so notChangedCount >= 10 then exit the loop otherwise loop to the first step.

Do you think it's a valid solution?

If anyone actually knew a general and always-applicable answer, it would have been implemented everywhere ages ago and would make our lives SO much easier.

There are many things you can do, but every single one of them has a problem:

1. As Ashwin Prabhu said, if you know the script well, you can observe its behaviour and track some of its variables on window or document etc. This solution, however, is not for everyone and can be used only by you and only on a limited set of pages.

2. Your solution by observing the HTML code and whether it has or hasn't been changed for some time is not bad (also, there is a method to get the original and not-edited HTML directly by WebDriver), but:

• It takes a long time to actually assert a page and could prolong the test significantly.
• You never know what the right interval is. The script might be downloading something big that takes more than 500 ms. There are several scripts on our company's internal page that take several seconds in IE. Your computer may be temporarily short on resources - say that an antivirus will make your CPU work fully, then 500 ms may be too short even for a noncomplex scripts.
• Some scripts are never done. They call themselves with some delay (setTimeout()) and work again and again and could possibly change the HTML every time they run. Seriously, every "Web 2.0" page does it. Even Stack Overflow. You could overwrite the most common methods used and consider the scripts that use them as completed, but ... you can't be sure.
• What if the script does something other than changing the HTML? It could do thousands of things, not just some innerHTML fun.

3. There are tools to help you on this. Namely Progress Listeners together with nsIWebProgressListener and some others. The browser support for this, however, is horrible. Firefox began to try to support it from FF4 onwards (still evolving), IE has basic support in IE9.

And I guess I could come up with another flawed solution soon. The fact is - there's no definite answer on when to say "now the page is complete" because of the everlasting scripts doing their work. Pick the one that serves you best, but beware of its shortcomings.

Thanks Ashwin !

In my case I should need wait for a jquery plugin execution in some element.. specifically "qtip"

based in your hint, it worked perfectly for me :

wait.until( new Predicate<WebDriver>() {
			public boolean apply(WebDriver driver) {
				return ((JavascriptExecutor)driver).executeScript("return document.readyState").equals("complete");
			}
		}
	);

Note: I'm using Webdriver 2

You need to wait for Javascript and jQuery to finish loading. Execute Javascript to check if jQuery.active is 0 and document.readyState is complete, which means the JS and jQuery load is complete.

public boolean waitForJStoLoad() {

    WebDriverWait wait = new WebDriverWait(driver, 30);

    // wait for jQuery to load
    ExpectedCondition<Boolean> jQueryLoad = new ExpectedCondition<Boolean>() {
      @Override
      public Boolean apply(WebDriver driver) {
        try {
          return ((Long)executeJavaScript("return jQuery.active") == 0);
        }
        catch (Exception e) {
          return true;
        }
      }
    };

    // wait for Javascript to load
    ExpectedCondition<Boolean> jsLoad = new ExpectedCondition<Boolean>() {
      @Override
      public Boolean apply(WebDriver driver) {
        return executeJavaScript("return document.readyState")
            .toString().equals("complete");
      }
    };

  return wait.until(jQueryLoad) && wait.until(jsLoad);
}

If all you need to do is wait for the html on the page to become stable before trying to interact with elements, you can poll the DOM periodically and compare the results, if the DOMs are the same within the given poll time, you're golden. Something like this where you pass in the maximum wait time and the time between page polls before comparing. Simple and effective.

public void waitForJavascript(int maxWaitMillis, int pollDelimiter) {
    double startTime = System.currentTimeMillis();
    while (System.currentTimeMillis() < startTime + maxWaitMillis) {
        String prevState = webDriver.getPageSource();
        Thread.sleep(pollDelimiter); // <-- would need to wrap in a try catch
        if (prevState.equals(webDriver.getPageSource())) {
            return;
        }
    }
}

Does the JS library define/initialize any well known variable on the window?

If so you could wait for the variable to appear. You can use

((JavascriptExecutor)driver).executeScript(String script, Object... args)

to test for this condition (something like: window.SomeClass && window.SomeClass.variable != null) and return a boolean true / false.

Wrap this in a WebDriverWait, and wait until the script returns true.

I had a same issue. This solution works for me from WebDriverDoku:

WebDriverWait wait = new WebDriverWait(driver, 10);
WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.id("someid")));

http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp

The below code works perfectly in my case - my page contains complex java scripts

public void checkPageIsReady() {
  
  JavascriptExecutor js = (JavascriptExecutor)driver;
  
  
  //Initially bellow given if condition will check ready state of page.
  if (js.executeScript("return document.readyState").toString().equals("complete")){ 
   System.out.println("Page Is loaded.");
   return; 
  } 
  
  //This loop will rotate for 25 times to check If page Is ready after every 1 second.
  //You can replace your value with 25 If you wants to Increase or decrease wait time.
  for (int i=0; i<25; i++){ 
   try {
    Thread.sleep(1000);
    }catch (InterruptedException e) {} 
   //To check page ready state.
   if (js.executeScript("return document.readyState").toString().equals("complete")){ 
    break; 
   }   
  }
 }

Source - How To Wait For Page To Load/Ready In Selenium WebDriver

Two conditions can be used to check if the page is loaded before finding any element on the page:

WebDriverWait wait = new WebDriverWait(driver, 50);

Using below readyState will wait till page load

wait.until((ExpectedCondition<Boolean>) wd ->
   ((JavascriptExecutor) wd).executeScript("return document.readyState").equals("complete"));

Below JQuery will wait till data has not been loaded

  int count =0;
    	    if((Boolean) executor.executeScript("return window.jQuery != undefined")){
    	        while(!(Boolean) executor.executeScript("return jQuery.active == 0")){
    	            Thread.sleep(4000);
    	            if(count>4)
    	            	break;
    	            count++;
    	        }
    	    }

After these JavaScriptCode try to findOut webElement.

WebElement we = wait.until(ExpectedConditions.presenceOfElementLocated(by));

I asked my developers to create a JavaScript variable "isProcessing" that I can access (in the "ae" object) that they set when things start running and clear when things are done. I then run it in an accumulator that checks it every 100 ms until it gets five in a row for a total of 500 ms without any changes. If 30 seconds pass, I throw an exception because something should have happened by then. This is in C#.

public static void WaitForDocumentReady(this IWebDriver driver)
{
    Console.WriteLine("Waiting for five instances of document.readyState returning 'complete' at 100ms intervals.");
    IJavaScriptExecutor jse = (IJavaScriptExecutor)driver;
    int i = 0; // Count of (document.readyState === complete) && (ae.isProcessing === false)
    int j = 0; // Count of iterations in the while() loop.
    int k = 0; // Count of times i was reset to 0.
    bool readyState = false;
    while (i < 5)
    {
        System.Threading.Thread.Sleep(100);
        readyState = (bool)jse.ExecuteScript("return ((document.readyState === 'complete') && (ae.isProcessing === false))");
        if (readyState) { i++; }
        else
        {
            i = 0;
            k++;
        }
        j++;
        if (j > 300) { throw new TimeoutException("Timeout waiting for document.readyState to be complete."); }
    }
    j *= 100;
    Console.WriteLine("Waited " + j.ToString() + " milliseconds. There were " + k + " resets.");
}

To do it properly, you need to handle the exceptions.

Here is how I do a wait for an iFrame. This requires that your JUnit test class pass the instance of RemoteWebDriver into the page object :

public class IFrame1 extends LoadableComponent<IFrame1> {

	private RemoteWebDriver driver;

	@FindBy(id = "iFrame1TextFieldTestInputControlID" )
    public WebElement iFrame1TextFieldInput;

	@FindBy(id = "iFrame1TextFieldTestProcessButtonID" )
    public WebElement copyButton;

	public IFrame1( RemoteWebDriver drv ) {
		super();
		this.driver = drv;
		this.driver.switchTo().defaultContent();
		waitTimer(1, 1000);
		this.driver.switchTo().frame("BodyFrame1");
		LOGGER.info("IFrame1 constructor...");
	}

	@Override
	protected void isLoaded() throws Error {    	
		LOGGER.info("IFrame1.isLoaded()...");
		PageFactory.initElements( driver, this );
		try {
			assertTrue( "Page visible title is not yet available.", driver
     .findElementByCssSelector("body form#webDriverUnitiFrame1TestFormID h1")
					.getText().equals("iFrame1 Test") );
		} catch ( NoSuchElementException e) {
			LOGGER.info("No such element." );
			assertTrue("No such element.", false);
		}
	}

	@Override
	protected void load() {
		LOGGER.info("IFrame1.load()...");
		Wait<WebDriver> wait = new FluentWait<WebDriver>( driver )
			    .withTimeout(30, TimeUnit.SECONDS)
			    .pollingEvery(5, TimeUnit.SECONDS)
			    .ignoring( NoSuchElementException.class ) 
			    .ignoring( StaleElementReferenceException.class ) ;
	        wait.until( ExpectedConditions.presenceOfElementLocated( 
			By.cssSelector("body form#webDriverUnitiFrame1TestFormID h1") ) );
	}
....

NOTE: You can see my entire working example here.

For the nodejs Selenium library, I used the following snippet. In my case, I was looking for two objects that are added to the window, which in this example are <SOME PROPERTY>, 10000 is the timeout milliseconds, <NEXT STEP HERE> is what happens after the properties are found on the window.

driver.wait( driver => {
	return driver.executeScript( 'if(window.hasOwnProperty(<SOME PROPERTY>) && window.hasOwnProperty(<SOME PROPERTY>)) return true;' ); }, 10000).then( ()=>{
		<NEXT STEP HERE>
}).catch(err => { 
	console.log("looking for window properties", err);
});

This works for me well with dynamically rendered websites:

1. Wait for complete page to load

WebDriverWait wait = new WebDriverWait(driver, 50); wait.until((ExpectedCondition) wd -> ((JavascriptExecutor) wd).executeScript("return document.readyState").equals("complete"));

2. Make another implicit wait with a dummy condition which would always fail

  try {          
 wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//*[contains(text(),'" + "This text will always fail :)" + "')]"))); // condition you are certain won't be true 
  } 
  catch (TimeoutException te) {
  }

3. Finally, instead of getting the html source - which would in most of one page applications would give you a different result , pull the outerhtml of the first html tag

String script = "return document.getElementsByTagName(\"html\")[0].outerHTML;"; 
content = ((JavascriptExecutor) driver).executeScript(script).toString();

You can write some logic to handle this. I have write a method that will return the WebElement and this method will be called three times or you can increase the time and add a null check for WebElement Here is an example

public static void main(String[] args) {
        WebDriver driver = new FirefoxDriver();
        driver.get("https://www.crowdanalytix.com/#home");
        WebElement webElement = getWebElement(driver, "homekkkkkkkkkkkk");
        int i = 1;
        while (webElement == null && i < 4) {
            webElement = getWebElement(driver, "homessssssssssss");
            System.out.println("calling");
            i++;
        }
        System.out.println(webElement.getTagName());
        System.out.println("End");
        driver.close();
    }

    public static WebElement getWebElement(WebDriver driver, String id) {
        WebElement myDynamicElement = null;
        try {
            myDynamicElement = (new WebDriverWait(driver, 10))
                    .until(ExpectedConditions.presenceOfElementLocated(By
                            .id(id)));
            return myDynamicElement;
        } catch (TimeoutException ex) {
            return null;
        }
    }

Here's from my own code:
Window.setTimeout executes only when browser is idle.
So calling the function recursively (42 times) will take 100ms if there is no activity in the browser and much more if the browser is busy doing something else.

	ExpectedCondition<Boolean> javascriptDone = new ExpectedCondition<Boolean>() {
		public Boolean apply(WebDriver d) {
			try{//window.setTimeout executes only when browser is idle,
				//introduces needed wait time when javascript is running in browser
				return  ((Boolean) ((JavascriptExecutor) d).executeAsyncScript(	
					
						" var callback =arguments[arguments.length - 1]; " +
						" var count=42; " +
						" setTimeout( collect, 0);" +
						" function collect() { " +
							" if(count-->0) { "+
								" setTimeout( collect, 0); " +
							" } "+
							" else {callback(" +
							" 	 true" +							
							" );}"+								
						" } "
					));
			}catch (Exception e) {
				return Boolean.FALSE;
			}
		}
	};
	WebDriverWait w = new WebDriverWait(driver,timeOut);  
	w.until(javascriptDone);
	w=null;

As a bonus the counter can be reset on document.readyState or on jQuery Ajax calls or if any jQuery animations are running (only if your app uses jQuery for ajax calls...)
...

" function collect() { " +
							" if(!((typeof jQuery === 'undefined') || ((jQuery.active === 0) && ($(\":animated\").length === 0))) && (document.readyState === 'complete')){" +
							" 	 count=42;" +
							" 	 setTimeout( collect, 0); " +
							" }" +
							" else if(count-->0) { "+
								" setTimeout( collect, 0); " +
							" } "+ 

...

EDIT: I notice executeAsyncScript doesn't work well if a new page loads and the test might stop responding indefinetly, better to use this on instead.

public static ExpectedCondition<Boolean> documentNotActive(final int counter){ 
	return new ExpectedCondition<Boolean>() {
		boolean resetCount=true;
		@Override
		public Boolean apply(WebDriver d) {
			
			if(resetCount){
				((JavascriptExecutor) d).executeScript(
						"   window.mssCount="+counter+";\r\n" + 
						"	window.mssJSDelay=function mssJSDelay(){\r\n" + 
						"		if((typeof jQuery != 'undefined') && (jQuery.active !== 0 || $(\":animated\").length !== 0))\r\n" + 
						"			window.mssCount="+counter+";\r\n" + 
						"		window.mssCount-->0 &&\r\n" + 
						"		setTimeout(window.mssJSDelay,window.mssCount+1);\r\n" + 
						"	}\r\n" + 
						"	window.mssJSDelay();");
				resetCount=false;
			}
			
			boolean ready=false;
			try{
				ready=-1==((Long) ((JavascriptExecutor) d).executeScript(
						"if(typeof window.mssJSDelay!=\"function\"){\r\n" + 
						"	window.mssCount="+counter+";\r\n" + 
						"	window.mssJSDelay=function mssJSDelay(){\r\n" + 
						"		if((typeof jQuery != 'undefined') && (jQuery.active !== 0 || $(\":animated\").length !== 0))\r\n" + 
						"			window.mssCount="+counter+";\r\n" + 
						"		window.mssCount-->0 &&\r\n" + 
						"		setTimeout(window.mssJSDelay,window.mssCount+1);\r\n" + 
						"	}\r\n" + 
						"	window.mssJSDelay();\r\n" + 
						"}\r\n" + 
						"return window.mssCount;"));
			}
			catch (NoSuchWindowException a){
				a.printStackTrace();
				return true;
			}
			catch (Exception e) {
				e.printStackTrace();
				return false;
			}
			return ready;
		}
		@Override
		public String toString() {
			return String.format("Timeout waiting for documentNotActive script");
		}
	};
}

I like your idea of polling the HTML until it's stable. I may add that to my own solution. The following approach is in C# and requires jQuery.

I'm the developer for a SuccessFactors (SaaS) test project where we have no influence at all over the developers or the characteristics of the DOM behind the web page. The SaaS product can potentially change its underlying DOM design 4 times a year, so the hunt is permanently on for robust, performant ways to test with Selenium (including NOT testing with Selenium where possi ble!)

Here's what I use for "page ready". It works in all my own tests currently. The same approach also worked for a big in-house Java web app a couple of years ago, and had been robust for over a year at the time I left the project.

• Driver is the WebDriver instance that communicates with the browser
• DefaultPageLoadTimeout is a timeout value in ticks (100ns per tick)

---

public IWebDriver Driver { get; private set; }

// ...

const int GlobalPageLoadTimeOutSecs = 10;
static readonly TimeSpan DefaultPageLoadTimeout =
    new TimeSpan((long) (10_000_000 * GlobalPageLoadTimeOutSecs));
Driver = new FirefoxDriver();

In what follows, note the order of waits in method PageReady (Selenium document ready, Ajax, animations), which makes sense if you think about it:

1. load the page containing the code
2. use the code to load the data from somewhere via Ajax
3. present the data, possibly with animations

Something like your DOM comparison approach could be used between 1 and 2 to add another layer of robustness.

---

public void PageReady()
{
    DocumentReady();
    AjaxReady();
    AnimationsReady();
}


private void DocumentReady()
{
    WaitForJavascript(script: "return document.readyState", result: "complete");
}

private void WaitForJavascript(string script, string result)
{
    new WebDriverWait(Driver, DefaultPageLoadTimeout).Until(
        d => ((IJavaScriptExecutor) d).ExecuteScript(script).Equals(result));
}

private void AjaxReady()
{
    WaitForJavascript(script: "return jQuery.active.toString()", result: "0");
}

private void AnimationsReady()
{
    WaitForJavascript(script: "return $(\"animated\").length.toString()", result: "0");
}

Don't know how to do that but in my case, end of page load & rendering match with FAVICON displayed in Firefox tab.

So if we can get the favicon image in the webbrowser, the web page is fully loaded.

But how perform this ....

Using implicit wait works for me.

driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);

Refer to this answer https://stackoverflow.com/questions/6992993/selenium-c-sharp-webdriver-wait-until-element-is-present

Here's how I do it:

new WebDriverWait(driver, 20).until(
       ExpectedConditions.jsReturnsValue(
                   "return document.readyState === 'complete' ? true : false"));