I want to send a value for `&quot;User-agent&quot;` while requesting a webpage using Python Requests.  I am not sure is if it is okay to send this as a part of the header, as in the code below:

    debug = {&#39;verbose&#39;: sys.stderr}
    user_agent = {&#39;User-agent&#39;: &#39;Mozilla/5.0&#39;}
    response  = requests.get(url, headers = user_agent, config=debug)

The debug information isn&#39;t showing the headers being sent during the request.

Is it acceptable to send this information in the header?  If not, how can I send it?

Sending &quot;User-agent&quot; using Requests library in Python

Does someone know a way to request Google to re-crawl a website? If possible, this shouldn&#39;t last months. My site is showing an old title in Google&#39;s search results. How can I show it with the correct title and description? 


How to request Google to re-crawl my website?

For research purposes I&#39;m trying to crawl the public Docker registry ( https://registry.hub.docker.com/ ) and find out 1) how many layers an average image has and 2) the sizes of these layers to get an idea of the distribution.

However I studied the API and public libraries as well as the details on the github but I cant find any method to:

- retrieve all the public repositories/images (even if those are thousands I still need a starting list to iterate through)
- find all the layers of an image
- find the size for a layer (so not an image but for the individual layer).

Can anyone help me find a way to retrieve this information?

Thank you!

EDIT: is anyone able to verify that searching for &#39;\*&#39; in Docker registry is returning all the repositories and not just anything that mentions &#39;\*&#39; anywhere? https://registry.hub.docker.com/search?q=\*

Finding the layers and layer sizes for each Docker image

I have two machines, speed and mass. speed has a fast Internet connection and is running a crawler which downloads a lot of files to disk. mass has a lot of disk space. I want to move the files from speed to mass after they&#39;re done downloading. Ideally, I&#39;d just run:

    $ rsync --remove-source-files speed:/var/crawldir .

but I worry that rsync will unlink a source file that hasn&#39;t finished downloading yet. (I looked at the source code and I didn&#39;t see anything protecting against this.) Any suggestions?

keep rsync from removing unfinished source files

I am trying to learn how to automatically fetch urls from a page. In the following code I am trying to get the title of the webpage:

    import urllib.request
    import re
    
    url = &quot;http://www.google.com&quot;
    regex = r&#39;&lt;title&gt;(,+?)&lt;/title&gt;&#39;
    pattern  = re.compile(regex)
    
    with urllib.request.urlopen(url) as response:
       html = response.read()
    
    title = re.findall(pattern, html)
    print(title)

And I get this unexpected error:

    Traceback (most recent call last):
      File &quot;path\to\file\Crawler.py&quot;, line 11, in &lt;module&gt;
        title = re.findall(pattern, html)
      File &quot;C:\Python33\lib\re.py&quot;, line 201, in findall
        return _compile(pattern, flags).findall(string)
    TypeError: can&#39;t use a string pattern on a bytes-like object

What am I doing wrong?


TypeError: can&#39;t use a string pattern on a bytes-like object in re.findall()

I want to make a website that shows the comparison between amazon and e-bay product price.
Which of these will work better and why? I am somewhat familiar with **BeautifulSoup** but not so much with **Scrapy crawler**.

Difference between BeautifulSoup and Scrapy crawler?

How can one detect the search engine bots using php?

how to detect search engine bots with php?

I am trying to pass a user defined argument to a scrapy&#39;s spider. Can anyone suggest on how to do that?

I read about a parameter `-a` somewhere but have no idea how to use it.

How to pass a user defined argument in scrapy spider

Is it possible to find all the pages and links on ANY given website? I&#39;d like to enter a URL and produce a directory tree of all links from that site?

I&#39;ve looked at HTTrack but that downloads the whole site and I simply need the directory tree.

How to find all links / pages on a website

I&#39;m deploying a replacement site for a client but they don&#39;t want all their old pages to end in 404s. Keeping the old URL structure wasn&#39;t possible because it was hideous.

So I&#39;m writing a 404 handler that should look for an old page being requested and do a permanent redirect to the new page. Problem is, I need a list of all the old page URLs.

I could do this manually, but I&#39;d be interested if there are any apps that would provide me a list of relative (eg: /page/path, not http:/.../page/path) URLs just given the home page. Like a spider but one that doesn&#39;t care about the content other than to find deeper pages.

Get a list of URLs from a site

*What options are there to detect web-crawlers that do not want to be detected?*

(I know that listing detection techniques will allow the smart stealth-crawler programmer to make a better spider, but I do not think that we will ever be able to block smart stealth-crawlers anyway, only the ones that make mistakes.)

I&#39;m not talking about the nice crawlers such as googlebot and Yahoo! Slurp.
I consider a bot nice if it:

 1. identifies itself as a bot in the user agent string
 2. reads robots.txt (and obeys it)

I&#39;m talking about the *bad* crawlers, hiding behind common user agents, using my bandwidth and never giving me anything in return.

There are some trapdoors that can be constructed *updated list (thanks Chris, gs)*:


 1. Adding a directory only listed (marked as disallow) in the robots.txt,
 2. Adding invisible links (possibly marked as rel=&quot;nofollow&quot;?),
    * style=&quot;display: none;&quot; on link or parent container
    * placed underneath another element with higher z-index
 3. detect who doesn&#39;t understand CaPiTaLiSaTioN,
 4. detect who tries to post replies but always fail the Captcha.
 5. detect GET requests to POST-only resources
 6. detect interval between requests
 7. detect order of pages requested
 8. detect who (consistently) requests https resources over http
 9. detect who does not request image file (this in combination with a list of user-agents of known image capable browsers works surprisingly nice)

Some traps would be triggered by both &#39;good&#39; and &#39;bad&#39; bots.
you could combine those with a whitelist:

 1. It trigger a trap
 2. It request `robots.txt`? 
 3. It doest not trigger another trap because it obeyed `robots.txt`

*One other important thing here is:*  
Please consider blind people using a screen readers: give people a way to contact you, or solve a (non-image) Captcha to continue browsing.
  
  

**What methods are there to automatically detect the web crawlers trying to mask themselves as normal human visitors.**


*Update*  
The question is not: *How do I catch every crawler.* The question is: *How can I maximize the chance of detecting a crawler.*  

Some spiders are really good, and actually parse and understand html, xhtml, css javascript, VB script etc...  
I have no illusions: I won&#39;t be able to beat them.

You would however be surprised how stupid some crawlers are. With the best example  of stupidity (in my opinion) being: cast all URLs to lower case before requesting them.

And then there is a whole bunch of crawlers that are just &#39;not good enough&#39; to avoid the various trapdoors.



Detecting &#39;stealth&#39; web-crawlers

Is there a difference between Crawling and Web-scraping?

If there&#39;s a difference, what&#39;s the best method to use in order to collect some web data to supply a database for later use in a customised search engine?


What is the difference between web-crawling and web-scraping?

tl;dr
-----

Hide email address from bots without using scripts and maintain `mailto:` functionality. Method must also support screen-readers.


----------


Summary
-------


 - Email **obfuscation without** using **scripts** or contact forms

 - Email address needs to be **completely visible** to human viewers and **maintain `mailto:` functionality**

 - Email Address **must not be in image form**. 

 - Email address **must be &quot;completely&quot; hidden from spam-crawlers and spam-bots** and **any other harvester type**



----------





Desired Effect:
---------------







 - **No scripts**, please. There are no scripts used in the project and **I&#39;d like to keep it that way**.  

 - Email address is either **displayed on the page** or can be easily displayed after some sort of user interaction, like opening a modal.

 - The **user can click on on the email address** which in turn would trigger the `mailto:` functionality.
 - Clicking the email will open the user&#39;s email application. 

      

    *In other words, `mailto:` functionality must work.*

 - The email address in not visible or not identified as an email address to bots **(This includes the page source)** 

 - I don&#39;t have an inbox that&#39;s full of spam



----------





What does *NOT* Work
--------------------





 - Adding a contact form - or anything similar - instead of the email address



 ***I hate contact forms**. I rarely fill up a contact form. If there&#39;s no email address, I look for a phone number, and if that&#39;s not there, I start looking for an alternative service. I would only fill up a contact form if I absolutely have to.* 



 - Replacing the address with an image of the address



 This creates a **HUGE** disadvantage to someone using a screenreader (**please remember the visually impaired in your future projects**)



 It also **removes** the `mailto:` functionality unless you make the image clickable and then add the `mailto:` functionality as the `href` for the link, but that **defeats the purpose** and now the email is visible to bots.

 

----------


What might work:
----------------




 - Clever usage of `pseudo-elements` in `CSS`

 - Solutions that make use of `base64` encoding

 - **Breaking up** the email address and spreading the parts across the document then putting them back together in a modal when the user clicks a button (This will probably involve multiple `CSS` classes and the usage of `anchor tags`)

 - Alterting `html` attributes via `CSS` 

 @MortezaAsadi gracefully brought up the possibility in the comments below. This is the link to the full - The article is from 2012:

 [What if We Could Use CSS to Alter HTML Attributes?][1] 

 - Other creative solutions that are beyond my scope of knowledge. 



----------



Similar Questions / Fixes
-------------------------





 - [JavaScript: Protect your email address by Joe Maller][2]



(This a great fix suggested by Joe Maller, it works well but it&#39;s **script based**. Here&#39;s what it looks like;



&lt;!-- begin snippet: js hide: false console: true babel: false --&gt;



&lt;!-- language: lang-html --&gt;



    &lt;SCRIPT TYPE=&quot;text/javascript&quot;&gt;

      emailE = &#39;emailserver.com&#39;

      emailE = (&#39;yourname&#39; + &#39;@&#39; + emailE)

      document.write(&#39;&lt;A href=&quot;mailto:&#39; + emailE + &#39;&quot;&gt;&#39; + emailE + &#39;&lt;/a&gt;&#39;)

    &lt;/script&gt;



    &lt;NOSCRIPT&gt;

      Email address protected by JavaScript

    &lt;/NOSCRIPT&gt;







&lt;!-- end snippet --&gt;



 - [Looking for a php only email address obfuscator function][3]



   (A Clever solution using both `PHP` and `CSS` to first **reverse** the email using PHP then **reverse it** back with CSS) A very promising solution that Works great! But it&#39;s **too easy to solve**. 



 - [Is it worth obfuscating email addresses on the web these days?][4]



  (Javascript fix)







 - [https://stackoverflow.com/questions/748780/best-way-to-obfuscate-an-e-mail-address-on-a-website/748805#748805][5]



  **The selected answer works**. It actually works really well. It involves encoding the email as `html entities`. Can it be improved?

  

 Here&#39;s what it looks like;



 &lt;!-- begin snippet: js hide: false console: true babel: false --&gt;



 &lt;!-- language: lang-html --&gt;



    &lt;A HREF=&quot;mailto:

    &amp;#121;&amp;#111;&amp;#117;&amp;#114;&amp;#110;&amp;#097;&amp;#109;&amp;#101;&amp;#064;&amp;#100;&amp;#111;&amp;#109;&amp;#097;&amp;#105;&amp;#110;&amp;#046;&amp;#099;&amp;#111;&amp;#109;&quot;&gt;

    &amp;#121;&amp;#111;&amp;#117;&amp;#114;&amp;#110;&amp;#097;&amp;#109;&amp;#101;&amp;#064;&amp;#100;&amp;#111;&amp;#109;&amp;#097;&amp;#105;&amp;#110;&amp;#046;&amp;#099;&amp;#111;&amp;#109;

    &lt;/A&gt;



&lt;!-- end snippet --&gt;






 - [Does e-mail address obfuscation actually work?][6] 



  (The selected answer to this SuperUser question is great and it presents a study of the amount of spam received by using different obfuscation methods.



  It seems that manipulating the email address with `CSS` to make it `rtl` does work. This is the same method used in the first question I linked to in this section. 



  I am uncertain what effects adding `mailto:` functionality to the fix would have on the results. 





 - There are also many other questions on [SO][7] which all have similar answers. I have not found anything that fits **my desired effect**



----------



The Question:
-------------





Would it be possible to **increase the efficiency** (ie as little spam as possible) of the email obfuscation methods above by **combining two or more of the fixes (or even adding new fixes)** while:



 **A- Maintaining `mailto:` functionality; and**

 **B- Supporting screen-readers**


----------
**Edit:** 

Many of the **answers and comments below** pose a very good question while indicating the impossibility of doing this without some sort of `js`

The question that&#39;s *asked/implied* is:

&gt; Why not use `js`?

The answer is that I am allergic to `js`

Joking aside though,

The three main reasons I asked this question are:

 - Contact forms are becoming more and more accepted as a replacement
   for providing an email address - which they should not.

 - If it **can be done** without scripting then it **should be done** without
   scripting.

 - **Curiosity:** (as I am in fact using one of the `js` fixes currently) I wanted to see *if discussing the matter would lead to a better way of doing it.*

  [1]: http://andydavies.me/blog/2012/08/13/what-if-we-could-use-css-to-manipulate-html-attributes/
  [2]: http://joemaller.com/js-mailer.shtml
  [3]: https://stackoverflow.com/questions/12592363/looking-for-a-php-only-email-address-obfuscator-function
  [4]: https://stackoverflow.com/questions/4098408/is-it-worth-obfuscating-email-addresses-on-the-web-these-days
  [5]: https://stackoverflow.com/questions/748780/best-way-to-obfuscate-an-e-mail-address-on-a-website/748805#748805
  [6]: https://superuser.com/questions/235937/does-e-mail-address-obfuscation-actually-work
  [7]: https://stackoverflow.com/search?tab=newest&amp;q=Email%20Address%20Obfuscation

Hide Email Address from Bots - Keep mailto:

I have a scrapy project which contains multiple spiders.
Is there any way I can define which pipelines to use for which spider? Not all the pipelines i have defined are applicable for every spider.

Thanks


How can I use different pipelines for different spiders in a single Scrapy project

I&#39;m new to Scrapy and I&#39;m looking for a way to run it from a Python script. I found 2 sources that explain this:

[http://tryolabs.com/Blog/2011/09/27/calling-scrapy-python-script/][1]

[http://snipplr.com/view/67006/using-scrapy-from-a-script/][2]

I can&#39;t figure out where I should put my spider code and how to call it from the main function. Please help. This is the example code:


    # This snippet can be used to run scrapy spiders independent of scrapyd or the scrapy command line tool and use it from a script. 
    # 
    # The multiprocessing library is used in order to work around a bug in Twisted, in which you cannot restart an already running reactor or in this case a scrapy instance.
    # 
    # [Here](http://groups.google.com/group/scrapy-users/browse_thread/thread/f332fc5b749d401a) is the mailing-list discussion for this snippet. 
     
    #!/usr/bin/python
    import os
    os.environ.setdefault(&#39;SCRAPY_SETTINGS_MODULE&#39;, &#39;project.settings&#39;) #Must be at the top before other imports
     
    from scrapy import log, signals, project
    from scrapy.xlib.pydispatch import dispatcher
    from scrapy.conf import settings
    from scrapy.crawler import CrawlerProcess
    from multiprocessing import Process, Queue
     
    class CrawlerScript():
     
        def __init__(self):
    		self.crawler = CrawlerProcess(settings)
    		if not hasattr(project, &#39;crawler&#39;):
    			self.crawler.install()
    		self.crawler.configure()
    		self.items = []
    		dispatcher.connect(self._item_passed, signals.item_passed)
     
        def _item_passed(self, item):
        	self.items.append(item)
     
        def _crawl(self, queue, spider_name):
            spider = self.crawler.spiders.create(spider_name)
            if spider:
                self.crawler.queue.append_spider(spider)
        	self.crawler.start()
            self.crawler.stop()
            queue.put(self.items)
     
        def crawl(self, spider):
    		queue = Queue()
    		p = Process(target=self._crawl, args=(queue, spider,))
    		p.start()
    		p.join()
    		return queue.get(True)
     
    # Usage
    if __name__ == &quot;__main__&quot;:
    	log.start()
     
    	&quot;&quot;&quot;
    	This example runs spider1 and then spider2 three times. 
    	&quot;&quot;&quot;
    	items = list()
    	crawler = CrawlerScript()
    	items.append(crawler.crawl(&#39;spider1&#39;))
    	for i in range(3):
    		items.append(crawler.crawl(&#39;spider2&#39;))
    	print items
     
    # Snippet imported from snippets.scrapy.org (which no longer works)
    # author: joehillen
    # date  : Oct 24, 2010


Thank you.


  [1]: http://tryolabs.com/Blog/2011/09/27/calling-scrapy-python-script/
  [2]: http://snipplr.com/view/67006/using-scrapy-from-a-script/



How to run Scrapy from within a Python script

I have several websites, and I can&#39;t remember where I wrote some lines of code. As my pages are indexed by Google, I would like to know if Google offers a facility to search within the HTML source code/mark-up itself, *instead of* just allowing search within the visual, rendered, part of a page?

Thanks

Search in html source with GOOGLE?

I have come across an interview question &quot;If you were designing a web crawler, how would you avoid getting into infinite loops? &quot; and I am trying to answer it.

How does it all begin from the beginning.
Say Google started with some hub pages say hundreds of them (How these hub pages were found in the first place is a different sub-question).
As Google follows links from a page and so on, does it keep making  a hash table to make sure that it doesn&#39;t follow the earlier visited pages.

What if the same page has 2 names (URLs) say in these days when we have URL shorteners etc..

I have taken Google as an example. Though Google doesn&#39;t leak how its web crawler algorithms and page ranking etc work, but any guesses?


Designing a web crawler

Can somebody distinguish between a crawler and scraper in terms of scope and functionality.

crawler vs scraper

I put [a package on PyPi](http://pypi.python.org/pypi/powerlaw) for the first time ~2 months ago, and have made some version updates since then. I noticed this week the download count recording, and was surprised to see it had been downloaded hundreds of times. Over the next few days, I was more surprised to see the download count increasing by sometimes hundreds *per day*, even though this is a niche statistical test toolbox. In particular, older versions of package are continuing to be downloaded, sometimes at higher rates than the newest version.

What is going on here?

Is there a bug in PyPi&#39;s downloaded counting, or is there an abundance of crawlers grabbing open source code (as mine is)?

PyPi download counts seem unrealistic

I&#39;m looking for a way to pseudo-spider a website. The key is that I don&#39;t actually want the content, but rather a simple list of URIs. I can get reasonably close to this idea with [Wget](http://www.gnu.org/software/wget/) using the `--spider` option, but when piping that output through a `grep`, I can&#39;t seem to find the right magic to make it work:

    wget --spider --force-html -r -l1 http://somesite.com | grep &#39;Saving to:&#39;

The `grep` filter seems to have absolutely no affect on the `wget` output. Have I got something wrong or is there another tool I should try that&#39;s more geared towards providing this kind of limited result set?

**UPDATE**

So I just found out offline that, by default, `wget` writes to stderr. I missed that in the man pages (in fact, I still haven&#39;t found it if it&#39;s in there). Once I piped the return to stdout, I got closer to what I need:

    wget --spider --force-html -r -l1 http://somesite.com 2&gt;&amp;1 | grep &#39;Saving to:&#39;

I&#39;d still be interested in other/better means for doing this kind of thing, if any exist.

Spider a Website and Return URLs Only

I&#39;m half-tempted to write my own, but I don&#39;t really have enough time right now.  I&#39;ve seen the Wikipedia list of [open source crawlers][1] but I&#39;d prefer something written in Python.  I realize that I could probably just use one of the tools on the Wikipedia page and wrap it in Python.  I might end up doing that - if anyone has any advice about any of those tools, I&#39;m open to hearing about them.  I&#39;ve used Heritrix via its web interface and I found it to be quite cumbersome.  I definitely won&#39;t be using a browser API for my upcoming project.

Thanks in advance.  Also, this is my first SO question!


  [1]: http://en.wikipedia.org/wiki/Web_crawler#Open-source_crawlers

Anyone know of a good Python based web crawler that I could use?

while crawling website like https://www.netflix.com, getting Forbidden by robots.txt: &lt;GET https://www.netflix.com/&gt;

ERROR: No response downloaded for: https://www.netflix.com/

getting Forbidden by robots.txt: scrapy

I&#39;m using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking).

I found out that Scrapy can handle forms (like logins) as shown [here][1]. But the problem is that there is no form to fill out, so it&#39;s not exactly what I need.

How can I simply click a button, which then shows the information I need?

Do I have to use an external library like mechanize or lxml?

  [1]: http://doc.scrapy.org/topics/request-response.html?highlight=formrequest#scrapy.http.FormRequest

Click a Button in Scrapy

I have had thoughts of trying to write a simple crawler that might crawl and produce a list of its findings for our NPO&#39;s websites and content.

Does anybody have any thoughts on how to do this? Where do you point the crawler to get started? How does it send back its findings and still keep crawling? How does it know what it finds, etc,etc.

How to write a crawler?

I have a web page with a bunch of links. I want to write a script which would dump all the data contained in those links in a local file.

Has anybody done that with PHP? General guidelines and gotchas would suffice as an answer.

How do I make a simple crawler in PHP?

Consider the case, 
I want to crawl websites frequently, but my IP address got blocked after some day/limit.

So, how can change my IP address dynamically or any other ideas?


Change IP address dynamically?

I am writing python to crawl Twitter space using Twitter-py. I have set the crawler to sleep for a while (2 seconds) between each request to api.twitter.com. However, after some times of running (around 1), when the Twitter&#39;s rate limit not exceeded yet, I got this error.

    [Errno 10054] An existing connection was forcibly closed by the remote host.

What are possible causes of this problem and how to solve this?

I have searched through and found that the Twitter server itself may force to close the connection due to many requests.

Thank you very much in advance.

python: [Errno 10054] An existing connection was forcibly closed by the remote host

We actually have burned static/archived copies of our asp.net websites for customers many times.  We have used [WebZip][1] until now but we have had endless problems with crashes, downloaded pages not being re-linked correctly, etc.

We basically need an application that crawls and downloads static copies of everything on our asp.net website (pages, images, documents, css, etc) and then processes the downloaded pages so that they can be browsed locally without an internet connection (get rid of absolute urls in links, etc).  The more idiot proof the better.  This seems like a pretty common and (relatively) simple process but I have tried a few other applications and have been really unimpressed

Does anyone have archive software they would recommend?  Does anyone have a really simple process they would share?

  [1]: http://www.spidersoft.com/

How do you archive an entire website for offline viewing?

What&#39;s the easiest way to do a JSON call from the command-line? I have a website that does a JSON call to retrieve additional data.

The **Request Payload** as shown in Google Chrome looks like this:

    {&quot;version&quot;: &quot;1.1&quot;, &quot;method&quot;:&quot;progr&quot;,&quot;id&quot;:2,&quot;params&quot;:{&quot;call&quot;:...} }

It&#39;s about doing the call from (preferably) linux command line and retrieving the JSON content, not about parsing the incoming JSON data.



How to do HTTP-request/call with JSON payload from command-line?

I&#39;ve built a crawler that had to run on about 5M pages (by increasing the url ID) and then parses the pages which contain the info&#39; I need.

after using an algorithm which run on the urls (200K) and saved the good and bad results I found that the I&#39;m wasting a lot of time. I could see that there are a a few returning subtrahends which I can use to check the next valid url.

you can see the subtrahends quite fast (a little ex&#39; of the few first &quot;good IDs&quot;) -

    510000011 # +8
    510000029 # +18
    510000037 # +8
    510000045 # +8
    510000052 # +7
    510000060 # +8
    510000078 # +18
    510000086 # +8
    510000094 # +8
    510000102 # +8
    510000110 # etc&#39;
    510000128
    510000136
    510000144
    510000151
    510000169
    510000177
    510000185
    510000193
    510000201

after crawling about 200K urls which gave me only 14K good results I knew I was wasting my time and need to optimize it, so I run some statistics and built a function that will check the urls while increasing the id with 8\18\17\8 (top returning subtrahends ) etc&#39;.

this is the function - 

    def checkNextID(ID):
        global numOfRuns, curRes, lastResult
        while ID &lt; lastResult:
            try:
                numOfRuns += 1
                if numOfRuns % 10 == 0:
                    time.sleep(3) # sleep every 10 iterations
                if isValid(ID + 8):
                    parseHTML(curRes)
                    checkNextID(ID + 8)
                    return 0
                if isValid(ID + 18):
                    parseHTML(curRes)
                    checkNextID(ID + 18)
                    return 0
                if isValid(ID + 7):
                    parseHTML(curRes)
                    checkNextID(ID + 7)
                    return 0
                if isValid(ID + 17):
                    parseHTML(curRes)
                    checkNextID(ID + 17)
                    return 0
                if isValid(ID+6):
                    parseHTML(curRes)
                    checkNextID(ID + 6)
                    return 0
                if isValid(ID + 16):
                    parseHTML(curRes)
                    checkNextID(ID + 16)
                    return 0
                else:
                    checkNextID(ID + 1)
                    return 0
            except Exception, e:
                print &quot;somethin went wrong: &quot; + str(e)

what is basically does is -checkNextID(ID) is getting the first id I know that contain the data minus 8 so the first iteration will match the first &quot;if isValid&quot; clause (isValid(ID + 8) will return True).

**lastResult** is a variable which saves the last known url id, so we&#39;ll run until numOfRuns is

**isValid()** is a function that gets an ID + one of the subtrahends and returns True if the url contains what I need and saves a soup object of the url to a global varibale named - &#39;**curRes**&#39;, it returns False if the url doesn&#39;t contain the data I need.

**parseHTML** is a function that gets the soup object (curRes), parses the data I need and then saves the data to a csv, then returns True.

if isValid() returns True, we&#39;ll call parseHTML() and then try to check the next ID+the subtrahends (by calling checkNextID(ID + subtrahends), if none of them will return what I&#39;m looking for I&#39;ll increase it with 1 and check again until I&#39;ll find the next valid url.

you can see the rest of the code [here][1]

after running the code I got about 950~ good results and suddenly an exception had raised -

    

&gt; &quot;somethin went wrong: maximum recursion depth exceeded while calling a
&gt; Python object&quot;

I could see on WireShark that the scipt stuck on id - 510009541 (I started my script with 510000003), the script tried getting the url with that ID a few times before I noticed the error and stopped it.

I was really exciting to see that I got the same results but 25x-40x times faster then my old script, with fewer HTTP requests, it&#39;s very precise, I have missed only 1 result for 1000 good results, which is find by me, it&#39;s impossible to rum 5M times, I had my old script running for 30 hours and got 14-15K results when my new script gave me 960~ results in 5-10 minutes.


  [1]: http://pastebin.com/Vq9Zwpc9

I read about stack limitations, but there must be a solution for the algorithm I&#39;m trying to implement in Python (I can&#39;t go back to my old *&quot;algorithm&quot;*, it will never end).

Thanks!

Python: maximum recursion depth exceeded while calling a Python object

I am wondering how would I go abouts in detecting search crawlers? The reason I ask is because I want to suppress certain JavaScript calls if the user agent is a bot.

I have found an example of how to to detect a certain browser, but am unable to find examples of how to detect a search crawler: 

`/MSIE (\d+\.\d+);/.test(navigator.userAgent); //test for MSIE x.x`

Example of search crawlers I want to block:

    Google 
    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 
    Googlebot/2.1 (+http://www.googlebot.com/bot.html) 
    Googlebot/2.1 (+http://www.google.com/bot.html) 

    Baidu 
    Baiduspider+(+http://www.baidu.com/search/spider_jp.html) 
    Baiduspider+(+http://www.baidu.com/search/spider.htm) 
    BaiDuSpider 

Detect Search Crawlers via JavaScript

I spend a lot of time searching about this.
At the end of the day I combined a number of answers and it works. I share my answer and I&#39;ll appreciate it if anyone edits it or provides us with an easier way to do this.

1- The answer in https://stackoverflow.com/questions/18657976/disable-images-in-selenium-google-chromedriver works in Java. So we should do the same thing in Python:

    opt = webdriver.ChromeOptions()
    opt.add_extension(&quot;Block-image_v1.1.crx&quot;)
    browser = webdriver.Chrome(chrome_options=opt)

2- But downloading &quot;Block-image_v1.1.crx&quot; is a little bit tricky, because there is no direct way to do that. For this purpose, instead of going to: https://chrome.google.com/webstore/detail/block-image/pehaalcefcjfccdpbckoablngfkfgfgj

you can go to http://chrome-extension-downloader.com/
and paste the extension url there to be able to download the extension file.

3- Then you will be able to use the above mentioned code with the path to the extension file that you have downloaded.


All Web Crawler Solutions on Gang of Coders

Total of 32 Web Crawler Solutions

Sending "User-agent" using Requests library in Python