BeautifulSoup findAll() given multiple classes?
PythonHtmlBeautifulsoupHtml ParsingPython Problem Overview
I would like to scrape a list of items from a website, and preserve the order that they are presented in. These items are organized in a table, but they can be one of two different classes (in random order).
Is there any way to provide multiple classes and have BeautifulSoup4 find all items which are in any of the given classes?
I need to achieve what this code does, except preserve the order of items as it was in the source code:
items = soup.findAll(True,{'class':'class1'})
items += soup.findAll(True,{'class':'class2'})
Python Solutions
Solution 1 - Python
you can do this
soup.findAll(True, {'class':['class1', 'class2']})
example:
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup('<html><body><div class="class1"></div><div class="class2"></div><div class="class3"></div></body></html>')
>>> soup.findAll(True, {"class":["class1", "class2"]})
[<div class="class1"></div>, <div class="class2"></div>]
Solution 2 - Python
I am new to Python with BeautifulSoup but may be my answer help you. I came across the same situation where I have to find multiple classes of one tag so, I just pass the classes into an array and it works for me. Here is the code snippet
# Search with single Class
find_all("tr", {"class":"abc"})
# Search with multiple classes
find_all("tr", {"class": ["abc", "xyz"]})
Solution 3 - Python
Or this with the more recent version of BeautifulSoup:
find_all('a', class_=['class1', 'class2'])
Using "class" would return an error so they use "class_" instead.
Solution 4 - Python
One way to do it is to use regular expression instead of a class name:
import re
import requests
from bs4 import BeautifulSoup
s = requests.Session()
link = 'https://leaderboards.guildwars2.com/en/na/achievements'
r = s.get(link)
soup = BeautifulSoup(r.text)
for item in soup.findAll(True, {"class": re.compile("^(equal|up)$")}):
if 'achievements' in item.attrs['class'] and 'number' in item.attrs['class']:
print item
Solution 5 - Python
<html>
<body>
<div class="cls1">ok</div>
<div class="cls2">hi</div>
<div class="cls1 cls2">both</div>
</body>
</html>
OR operator
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
divs = soup.find_all('div', class_=['cls1', 'cls2'])
print(divs)
output:
[<div class="cls1">ok</div>, <div class="cls2">hi</div>, <div class="cls1 cls2">both</div>]
AND operator
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
divs = soup.select('div.cls1.cls2')
print(divs)
output:
[<div class="cls1 cls2">both</div>]
Solution 6 - Python
If you're working with an Url as parameter dont forget to pass the headers too. I was fighting for like one hour to get these div elements with 2 classes and it wasnt working for mi till i noticed that i forget to pass the this headers.
header = {
"Accept-Language": "es-ES,es;q=0.9",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
}
url = 'something.com'
response = requests.get(url=url,headers=header)
response.raise_for_status()
data = response.text
soup = BeautifulSoup(data, 'html.parser')
elements = soup.select('div.fde444d7ef._c445487e2')