How can we use Selenium Webdriver in colab.research.google.com?
PythonSeleniumSelenium WebdriverGoogle ColaboratoryPython Problem Overview
I want to use Selenium Webdriver of Chrome in colab.research.google.com for fast processing. I was able to install Selenium using !pip install selenium but the webdriver of chrome needs a path to webdriverChrome.exe. How am I suppose to use it?
P.S.- colab.research.google.com is an online platform which provides GPU for fast computational problems related to deep learning. Please refrain from solutions such as webdriver.Chrome(path).
Python Solutions
Solution 1 - Python
You can do it by installing the chromium webdriver and adjusting some options such that it does not crash in google colab:
!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("https://www.webite-url.com")
Solution 2 - Python
this one worked in colab
!pip install selenium
!apt-get update
!apt install chromium-chromedriver
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
Solution 3 - Python
I made my own library to make it easy.
!pip install kora -q
from kora.selenium import wd
wd.get("https://www.website.com")
PS: I forget how I searched and experimented until it worked. But I first wrote and shared it in this gist in Dec 2018.
Solution 4 - Python
Don't have enough repu to comment. :(
However @Thomas answer still works in 06.10.2021, but with just one simple change since right of the bat you'll get DeprecationWarning: use options instead of chrome_options
Working code below:
!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://stackoverflow.com/questions/51046454/how-can-we-use-selenium-webdriver-in-colab-research-google-com")
wd.title
Solution 5 - Python
to use selenium in GOOGLE COLAB do the next steps in the colab notebook
!pip install kora -q
HOW TO USE IT INSIDE COLAB :
from kora.selenium import wd
wd.get("enter any website here")
YOU CAN ALSO USE IT WITH Beautiful Soup
import bs4 as soup
wd.get("enter any website here")
html = soup.BeautifulSoup(wd.page_source)
Solution 6 - Python
You can can rid of using .exe file by using WebDriverManager so instead of this
System.setProperty("webdriver.gecko.driver", "driverpath/.exe");
WebDriver driver = new FirefoxDriver();
you will be writing this
WebDriverManager.firefoxdriver().setup();
WebDriver driver = new FirefoxDriver();
All you need is add the dependecy to the POM file(Im assuming you using maven or some build tool) Please see my full answer about how to use this in this link Using WebdriverManager