I have the `item` object and i need to pass that along many pages to store data in single item

LIke my item is

    class DmozItem(Item):
        title = Field()
        description1 = Field()
        description2 = Field()
        description3 = Field()



Now those three description are in three separate pages. i want to do somrething like

Now this works good for `parseDescription1`

 

    def page_parser(self, response):
        sites = hxs.select(&#39;//div[@class=&quot;row&quot;]&#39;)
        items = []
        request =  Request(&quot;http://www.example.com/lin1.cpp&quot;,  callback =self.parseDescription1)
        request.meta[&#39;item&#39;] = item
        return request 
            
    def parseDescription1(self,response):
        item = response.meta[&#39;item&#39;]
        item[&#39;desc1&#39;] = &quot;test&quot;
        return item

But i want something like

    def page_parser(self, response):
        sites = hxs.select(&#39;//div[@class=&quot;row&quot;]&#39;)
        items = []
        request =  Request(&quot;http://www.example.com/lin1.cpp&quot;,  callback =self.parseDescription1)
        request.meta[&#39;item&#39;] = item

        request =  Request(&quot;http://www.example.com/lin1.cpp&quot;,  callback =self.parseDescription2)
        request.meta[&#39;item&#39;] = item

        request =  Request(&quot;http://www.example.com/lin1.cpp&quot;,  callback =self.parseDescription2)
        request.meta[&#39;item&#39;] = item

        return request 
            
    def parseDescription1(self,response):
        item = response.meta[&#39;item&#39;]
        item[&#39;desc1&#39;] = &quot;test&quot;
        return item

    def parseDescription2(self,response):
        item = response.meta[&#39;item&#39;]
        item[&#39;desc2&#39;] = &quot;test2&quot;
        return item

    def parseDescription3(self,response):
        item = response.meta[&#39;item&#39;]
        item[&#39;desc3&#39;] = &quot;test3&quot;
        return item

How can i use multiple requests and pass items in between them in scrapy python

How can I get the request url in Scrapy&#39;s `parse()` function? I have a lot of urls in `start_urls` and some of them redirect my spider to homepage and as result I have an empty item. So I need something like `item[&#39;start_url&#39;] = request.url` to store these urls. I&#39;m using the BaseSpider.

Scrapy get request url in parse

I don&#39;t want to crawl simultaneously and get blocked. I would like to send one request per second.

How to give delay between each requests in scrapy?

Is there a way to trigger a method in a Spider class just before it terminates?  

I can terminate the spider myself, like this:

    class MySpider(CrawlSpider):
        #Config stuff goes here...

        def quit(self):
            #Do some stuff...
            raise CloseSpider(&#39;MySpider is quitting now.&#39;)
        
        def my_parser(self, response):
            if termination_condition:
                self.quit()

            #Parsing stuff goes here...

But I can&#39;t find any information on how to determine when the spider is about to quit naturally.

scrapy: Call a function when a spider quits

How do you use Scrapy to scrape web requests that return JSON? For example, the JSON would look like this:

    {
        &quot;firstName&quot;: &quot;John&quot;,
        &quot;lastName&quot;: &quot;Smith&quot;,
        &quot;age&quot;: 25,
        &quot;address&quot;: {
            &quot;streetAddress&quot;: &quot;21 2nd Street&quot;,
            &quot;city&quot;: &quot;New York&quot;,
            &quot;state&quot;: &quot;NY&quot;,
            &quot;postalCode&quot;: &quot;10021&quot;
        },
        &quot;phoneNumber&quot;: [
            {
                &quot;type&quot;: &quot;home&quot;,
                &quot;number&quot;: &quot;212 555-1234&quot;
            },
            {
                &quot;type&quot;: &quot;fax&quot;,
                &quot;number&quot;: &quot;646 555-4567&quot;
            }
        ]
    }

I would be looking to scrape specific items (e.g. `name` and `fax` in the above) and save to csv.

Scraping a JSON response with Scrapy

I want to install Lxml so I can then install Scrapy.

When I updated my Mac today it wouldn&#39;t let me reinstall lxml, I get the following error:

    

    In file included from src/lxml/lxml.etree.c:314:
    /private/tmp/pip_build_root/lxml/src/lxml/includes/etree_defs.h:9:10: fatal error: &#39;libxml/xmlversion.h&#39; file not found
    #include &quot;libxml/xmlversion.h&quot;
             ^
    1 error generated.
    error: command &#39;cc&#39; failed with exit status 1

I have tried using brew to install libxml2 and libxslt, both installed fine but I still cannot install lxml.

Last time I was installing I needed to enable the developer tools on Xcode but since it&#39;s updated to Xcode 5 it doesn&#39;t give me that option anymore.

Does anyone know what I need to do?

Cannot install Lxml on Mac OS X 10.9

I&#39;m practicing the code from &#39;Web Scraping with Python&#39;, and I keep having this certificate problem:

    from urllib.request import urlopen 
    from bs4 import BeautifulSoup 
    import re

    pages = set()
    def getLinks(pageUrl):
    	global pages
    	html = urlopen(&quot;http://en.wikipedia.org&quot;+pageUrl)
    	bsObj = BeautifulSoup(html)
    	for link in bsObj.findAll(&quot;a&quot;, href=re.compile(&quot;^(/wiki/)&quot;)):
    		if &#39;href&#39; in link.attrs:
    			if link.attrs[&#39;href&#39;] not in pages:
    				#We have encountered a new page
    				newPage = link.attrs[&#39;href&#39;] 
    				print(newPage) 
    				pages.add(newPage) 
    				getLinks(newPage)
    getLinks(&quot;&quot;)

The error is:

      File &quot;/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py&quot;, line 1319, in do_open
        raise URLError(err)
    urllib.error.URLError: &lt;urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1049)&gt;


Btw,I was also practicing scrapy, but kept getting the problem: command not found: scrapy (I tried all sorts of solutions online but none works... really frustrating)

Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org

I&#39;m trying to install Scrapy Python framework in OSX 10.11 (El Capitan) via pip. The installation script downloads the required modules and at some point returns the following error:

    OSError: [Errno 1] Operation not permitted: &#39;/tmp/pip-nIfswi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info&#39;

I&#39;ve tried to deactivate the rootless feature in OSX 10.11 with the command:

    sudo nvram boot-args=&quot;rootless=0&quot;;sudo reboot

but I still get the same error when the machine reboots.

Any clue or idea from my fellow StackExchangers?

If it helps, the full script output is the following:

    sudo -s pip install scrapy
    Collecting scrapy
      Downloading Scrapy-1.0.2-py2-none-any.whl (290kB)
        100% |████████████████████████████████| 290kB 345kB/s 
    Requirement already satisfied (use --upgrade to upgrade): cssselect&gt;=0.9 in /Library/Python/2.7/site-packages (from scrapy)
    Requirement already satisfied (use --upgrade to upgrade): queuelib in /Library/Python/2.7/site-packages (from scrapy)
    Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from scrapy)
    Collecting w3lib&gt;=1.8.0 (from scrapy)
      Downloading w3lib-1.12.0-py2.py3-none-any.whl
    Collecting lxml (from scrapy)
      Downloading lxml-3.4.4.tar.gz (3.5MB)
        100% |████████████████████████████████| 3.5MB 112kB/s 
    Collecting Twisted&gt;=10.0.0 (from scrapy)
      Downloading Twisted-15.3.0.tar.bz2 (4.4MB)
        100% |████████████████████████████████| 4.4MB 94kB/s 
    Collecting six&gt;=1.5.2 (from scrapy)
      Downloading six-1.9.0-py2.py3-none-any.whl
    Requirement already satisfied (use --upgrade to upgrade): zope.interface&gt;=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted&gt;=10.0.0-&gt;scrapy)
    Requirement already satisfied (use --upgrade to upgrade): setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface&gt;=3.6.0-&gt;Twisted&gt;=10.0.0-&gt;scrapy)
    Installing collected packages: six, w3lib, lxml, Twisted, scrapy
      Found existing installation: six 1.4.1
        DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
        Uninstalling six-1.4.1:
    Exception:
    Traceback (most recent call last):
      File &quot;/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/basecommand.py&quot;, line 223, in main
    status = self.run(options, args)
      File &quot;/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/commands/install.py&quot;, line 299, in run
    root=options.root_path,
      File &quot;/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_set.py&quot;, line 640, in install
    requirement.uninstall(auto_confirm=True)
      File &quot;/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_install.py&quot;, line 726, in uninstall
    paths_to_remove.remove(auto_confirm)
      File &quot;/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_uninstall.py&quot;, line 125, in remove
    renames(path, new_path)
      File &quot;/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/utils/__init__.py&quot;, line 314, in renames
    shutil.move(old, new)
      File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py&quot;, line 302, in move
    copy2(src, real_dst)
      File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py&quot;, line 131, in copy2
    copystat(src, dst)
      File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py&quot;, line 103, in copystat
    os.chflags(dst, st.st_flags)
    OSError: [Errno 1] Operation not permitted: &#39;/tmp/pip-nIfswi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info&#39;

&quot;OSError: [Errno 1] Operation not permitted&quot; when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)

I have recently been learning Python and am dipping my hand into building a web-scraper.  It&#39;s nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel.

Most of the issues are solvable and I&#39;m having a good little mess around. However I&#39;m hitting a massive hurdle over one issue. If a site loads a table of horses and lists current betting prices this information is not in any source file. The clue is that this data is live sometimes, with the numbers being updated obviously from some remote server. The HTML on my PC simply has a hole where their servers are pushing through all the interesting data that I need.

Now my experience with dynamic web content is low, so this thing is something I&#39;m having trouble getting my head around. 

I think Java or Javascript is a key, this pops up often. 

The scraper is simply a odds comparison engine.  Some sites have APIs but I need this for those that don&#39;t. I&#39;m using the scrapy library with Python 2.7

I do apologize if this question is too open-ended. In short, my question is: how can scrapy be used to scrape this dynamic data so that I can use it?  So that I can scrape this betting odds data in real-time?

Can scrapy be used to scrape dynamic content from websites that are using AJAX?

I want to make a website that shows the comparison between amazon and e-bay product price.
Which of these will work better and why? I am somewhat familiar with **BeautifulSoup** but not so much with **Scrapy crawler**.

Difference between BeautifulSoup and Scrapy crawler?

I am trying to pass a user defined argument to a scrapy&#39;s spider. Can anyone suggest on how to do that?

I read about a parameter `-a` somewhere but have no idea how to use it.

How to pass a user defined argument in scrapy spider

I am working on Scrapy 0.20 with Python 2.7. I found PyCharm has a good Python debugger. I want to test my Scrapy spiders using it. Anyone knows how to do that please?

&lt;h3&gt; What I have tried&lt;/h3&gt;
Actually I tried to run the spider as a script. As a result, I built that script. Then, I tried to add my Scrapy project to PyCharm as a model like this:

    File-&gt;Setting-&gt;Project structure-&gt;Add content root.

But I don&#39;t know what else I have to do

How to use PyCharm to debug Scrapy projects

I have a scrapy project which contains multiple spiders.
Is there any way I can define which pipelines to use for which spider? Not all the pipelines i have defined are applicable for every spider.

Thanks


How can I use different pipelines for different spiders in a single Scrapy project

I&#39;m trying to scrape product information from a webpage, using scrapy. My to-be-scraped webpage looks like this:

* starts with a product_list page with 10 products
* a click on &quot;next&quot;  button loads the next 10 products (url doesn&#39;t change between the two pages)
* i use LinkExtractor to follow each product link into the product page, and get all the information I need

I tried to replicate the next-button-ajax-call but can&#39;t get working, so I&#39;m giving selenium a try. I can run selenium&#39;s webdriver in a separate script, but I don&#39;t know how to integrate with scrapy. Where shall I put the selenium part in my scrapy spider? 

My spider is pretty standard, like the following:

    class ProductSpider(CrawlSpider):
        name = &quot;product_spider&quot;
        allowed_domains = [&#39;example.com&#39;]
        start_urls = [&#39;http://example.com/shanghai&#39;]
        rules = [
            Rule(SgmlLinkExtractor(restrict_xpaths=&#39;//div[@id=&quot;productList&quot;]//dl[@class=&quot;t2&quot;]//dt&#39;), callback=&#39;parse_product&#39;),
            ]

        def parse_product(self, response):
            self.log(&quot;parsing product %s&quot; %response.url, level=INFO)
            hxs = HtmlXPathSelector(response)
            # actual data follows

Any idea is appreciated. Thank you!

selenium with scrapy for dynamic page

I&#39;m new to Scrapy and I&#39;m looking for a way to run it from a Python script. I found 2 sources that explain this:

[http://tryolabs.com/Blog/2011/09/27/calling-scrapy-python-script/][1]

[http://snipplr.com/view/67006/using-scrapy-from-a-script/][2]

I can&#39;t figure out where I should put my spider code and how to call it from the main function. Please help. This is the example code:


    # This snippet can be used to run scrapy spiders independent of scrapyd or the scrapy command line tool and use it from a script. 
    # 
    # The multiprocessing library is used in order to work around a bug in Twisted, in which you cannot restart an already running reactor or in this case a scrapy instance.
    # 
    # [Here](http://groups.google.com/group/scrapy-users/browse_thread/thread/f332fc5b749d401a) is the mailing-list discussion for this snippet. 
     
    #!/usr/bin/python
    import os
    os.environ.setdefault(&#39;SCRAPY_SETTINGS_MODULE&#39;, &#39;project.settings&#39;) #Must be at the top before other imports
     
    from scrapy import log, signals, project
    from scrapy.xlib.pydispatch import dispatcher
    from scrapy.conf import settings
    from scrapy.crawler import CrawlerProcess
    from multiprocessing import Process, Queue
     
    class CrawlerScript():
     
        def __init__(self):
    		self.crawler = CrawlerProcess(settings)
    		if not hasattr(project, &#39;crawler&#39;):
    			self.crawler.install()
    		self.crawler.configure()
    		self.items = []
    		dispatcher.connect(self._item_passed, signals.item_passed)
     
        def _item_passed(self, item):
        	self.items.append(item)
     
        def _crawl(self, queue, spider_name):
            spider = self.crawler.spiders.create(spider_name)
            if spider:
                self.crawler.queue.append_spider(spider)
        	self.crawler.start()
            self.crawler.stop()
            queue.put(self.items)
     
        def crawl(self, spider):
    		queue = Queue()
    		p = Process(target=self._crawl, args=(queue, spider,))
    		p.start()
    		p.join()
    		return queue.get(True)
     
    # Usage
    if __name__ == &quot;__main__&quot;:
    	log.start()
     
    	&quot;&quot;&quot;
    	This example runs spider1 and then spider2 three times. 
    	&quot;&quot;&quot;
    	items = list()
    	crawler = CrawlerScript()
    	items.append(crawler.crawl(&#39;spider1&#39;))
    	for i in range(3):
    		items.append(crawler.crawl(&#39;spider2&#39;))
    	print items
     
    # Snippet imported from snippets.scrapy.org (which no longer works)
    # author: joehillen
    # date  : Oct 24, 2010


Thank you.


  [1]: http://tryolabs.com/Blog/2011/09/27/calling-scrapy-python-script/
  [2]: http://snipplr.com/view/67006/using-scrapy-from-a-script/



How to run Scrapy from within a Python script

I&#39;m trying to put list of possible solutions for browser automatic tests suits and headless browser platforms capable of scraping.

***

**BROWSER TESTING / SCRAPING:**

 - [Selenium][1] - **polyglot** flagship in browser automation, bindings for Python, Ruby,  JavaScript, C#, Haskell and more, IDE for Firefox (as an extension) for faster test deployment. Can act as a Server and has tons of features.
 

**JAVASCRIPT**

 - [PhantomJS][2] - **JavaScript**, headless testing with screen capture and automation, uses **Webkit**. As of version 1.8 Selenium&#39;s WebDriver API is implemented, so you can use any WebDriver binding and tests will be compatible with Selenium
 - [SlimerJS][3] - similar to PhantomJS, uses **Gecko** (Firefox) instead of **WebKit**
 - [CasperJS][4] - **JavaScript**, build on both PhantomJS and SlimerJS, has extra features
 - [Ghost Driver][5] - **JavaScript** implementation of the **WebDriver Wire Protocol** for **PhantomJS**. 
 - ***new*** [PhantomCSS][6] - CSS regression testing. A CasperJS module for automating visual regression testing with PhantomJS and [Resemble.js][7].
 - ***new*** [WebdriverCSS][8] - plugin for [Webdriver.io][9] for automating visual regression testing
 - ***new*** [PhantomFlow][10] - Describe and visualize user flows through tests. An experimental approach to Web user interface testing.
 - ***new*** [trifleJS][11] - ports the PhantomJS API to use the Internet Explorer engine.
 - ***new*** [CasperJS IDE][12] *(commercial)*

**NODE.JS**

 - [Node-phantom][13] - bridges the gap between **PhantomJS** and **node.js**
 - [WebDriverJs][14] - Selenium WebDriver bindings for node.js by Selenium Team
 - [WD.js][15] - node module for WebDriver/Selenium 2
 - [yiewd][16] - WD.js wrapper using latest Harmony generators! Get rid of the callback pyramid with **yield** 
 - [ZombieJs][17] - Insanely fast, headless full-stack testing using **node.js**
 - [NightwatchJs][18] - Node JS based testing solution using Selenium Webdriver
 - [Chimera][19] - Chimera: can do everything what phantomJS does, but in a full JS environment
 - [Dalek.js][20] - Automated cross browser testing with JavaScript through Selenium Webdriver
 - [Webdriver.io][9] - better implementation of WebDriver bindings with predefined 50+ actions
 - [Nightmare][21] - Electron bridge with a high-level API. 
 - [jsdom][22] - Tailored towards web scraping. A very lightweight DOM implemented in Node.js, it supports pages with javascript. 
 - ***new*** [Puppeteer][23] - Node library which provides a high-level API to control Chrome or Chromium. Puppeteer runs headless by default.

**WEB SCRAPING / MINING**

 - [Scrapy][24] - **Python**, mainly a scraper/miner - fast, well documented and, can be linked with [Django Dynamic Scraper][25] for nice mining deployments, or [Scrapy Cloud][26] for PaaS (server-less) deployment, works in terminal or an server stand-alone proces, can be used with **Celery**, built on top of **Twisted**
 - [Snailer][27] - **node.js** module, untested yet. 
 - [Node-Crawler][28] - **node.js** module, untested yet.

**ONLINE TOOLS**

 - ***new*** [Web Scraping Language][29] - Simple syntax to crawl the web

 - ***new*** [Online HTTP client][30] - Dedicated SO answer

 - ***dead*** [CasperBox][31] - Run **CasperJS** scripts online

**Android TOOLS for Automation**

 - ***new*** [Mechanica Browser App][33]

***

**RELATED LINKS &amp; RESOURCES** 

 - [Comparsion of Webscraping software][32]
 - ***new*** [Resemble.js][7] : Image analysis and comparison

**Questions:**

- Any pure Node.js solution or Nodejs to PhanthomJS/CasperJS module that actually works and is documented?

**Answer:** Chimera seems to go in that direction, checkout [Chimera][19]

- Other solutions capable of easier JavaScript injection than Selenium?

- Do you know any pure **ruby** solutions?

**Answer:** Checkout the list created by rjk with ruby based solutions

- Do you know any related tech or solution?

**Feel free to edit this question and add content as you wish!** Thank you for your contributions!


  [1]: http://seleniumhq.org
  [2]: http://phantomjs.org/
  [3]: http://slimerjs.org/
  [4]: http://casperjs.org/
  [5]: https://github.com/detro/ghostdriver
  [6]: https://github.com/Huddle/PhantomCSS
  [7]: http://huddle.github.io/Resemble.js/
  [8]: https://github.com/webdriverjs/webdrivercss
  [9]: http://webdriver.io
  [10]: https://github.com/Huddle/PhantomFlow
  [11]: http://triflejs.org/
  [12]: https://gum.co/casperjside
  [13]: https://github.com/alexscheelmeyer/node-phantom
  [14]: https://code.google.com/p/selenium/wiki/WebDriverJs
  [15]: https://github.com/admc/wd
  [16]: https://github.com/jlipps/yiewd
  [17]: http://zombie.labnotes.org/
  [18]: http://nightwatchjs.org/
  [19]: https://github.com/deanmao/node-chimera
  [20]: http://dalekjs.com
  [21]: https://github.com/segmentio/nightmare
  [22]: https://github.com/tmpvar/jsdom
  [23]: https://github.com/GoogleChrome/puppeteer
  [24]: http://scrapy.org
  [25]: http://django-dynamic-scraper.readthedocs.org/en/latest/
  [26]: http://scrapinghub.com/scrapy-cloud.html
  [27]: https://github.com/matthewhudson/snailer
  [28]: https://github.com/sylvinus/node-crawler.git
  [29]: https://www.npmjs.com/package/wslang
  [30]: https://stackoverflow.com/a/12637452/363573
  [31]: https://www.casperbox.com
  [32]: http://blog.screen-scraper.com/2010/06/28/comparison-of-web-scraping-software/
  [33]: https://play.google.com/store/apps/details?id=tools.automate.browser.mechanica

Headless Browser and scraping - solutions

I&#39;d like to implement some unit tests in a Scrapy (screen scraper/web crawler).  Since a project is run through the &quot;scrapy crawl&quot; command I can run it through something like nose.  Since scrapy is built on top of twisted can I use its unit testing framework Trial? If so, how? Otherwise I&#39;d like to get *nose* working.

**Update:**

I&#39;ve been talking on [Scrapy-Users](http://groups.google.com/group/scrapy-users/browse_thread/thread/ba1fadfd40916e71) and I guess I am supposed to &quot;build the Response in the test code, and then call the method with the response and assert that [I] get the expected items/requests in the output&quot;.  I can&#39;t seem to get this to work though.

I can build a unit-test test class and in a test:

 - create a response object
 - try to call the parse method of my spider with the response object

However it ends up generating [this](https://gist.github.com/1044924) traceback. Any insight as to why?

Scrapy Unit Testing

I just started programming Python. I want to use scrapy to create a bot，and it showed 
TypeError: Object of type &#39;bytes&#39; is not JSON serializable when I run the project. 



    import json
    import codecs

    class W3SchoolPipeline(object):
    
      def __init__(self):
          self.file = codecs.open(&#39;w3school_data_utf8.json&#39;, &#39;wb&#39;, encoding=&#39;utf-8&#39;)

      def process_item(self, item, spider):
          line = json.dumps(dict(item)) + &#39;\n&#39;
          # print line

          self.file.write(line.decode(&quot;unicode_escape&quot;))
          return item

-------------------------------


    from scrapy.spiders import Spider
    from scrapy.selector import Selector
    from w3school.items import W3schoolItem

    class W3schoolSpider(Spider):
   
        name = &quot;w3school&quot;
        allowed_domains = [&quot;w3school.com.cn&quot;]
    
        start_urls = [
            &quot;http://www.w3school.com.cn/xml/xml_syntax.asp&quot;
        ]

        def parse(self, response):
            sel = Selector(response)
            sites = sel.xpath(&#39;//div[@id=&quot;navsecond&quot;]/div[@id=&quot;course&quot;]/ul[1]/li&#39;)

        items = []
        for site in sites:
            item = W3schoolItem()
            title = site.xpath(&#39;a/text()&#39;).extract()
            link = site.xpath(&#39;a/@href&#39;).extract()
            desc = site.xpath(&#39;a/@title&#39;).extract()

            item[&#39;title&#39;] = [t.encode(&#39;utf-8&#39;) for t in title]
            item[&#39;link&#39;] = [l.encode(&#39;utf-8&#39;) for l in link]
            item[&#39;desc&#39;] = [d.encode(&#39;utf-8&#39;) for d in desc]
            items.append(item)
            return items


Traceback：






    TypeError: Object of type &#39;bytes&#39; is not JSON serializable
    2017-06-23 01:41:15 [scrapy.core.scraper] ERROR: Error processing       {&#39;desc&#39;: [b&#39;\x
    e4\xbd\xbf\xe7\x94\xa8 XSLT \xe6\x98\xbe\xe7\xa4\xba XML&#39;],
     &#39;link&#39;: [b&#39;/xml/xml_xsl.asp&#39;],
     &#39;title&#39;: [b&#39;XML XSLT&#39;]}

    Traceback (most recent call last):
    File  
    &quot;c:\users\administrator\appdata\local\programs\python\python36\lib\site-p
    ackages\twisted\internet\defer.py&quot;, line 653, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
    File &quot;D:\LZZZZB\w3school\w3school\pipelines.py&quot;, line 19, in process_item
        line = json.dumps(dict(item)) + &#39;\n&#39;
    File 
    &quot;c:\users\administrator\appdata\local\programs\python\python36\lib\json\_
    _init__.py&quot;, line 231, in dumps
        return _default_encoder.encode(obj)
    File 
    &quot;c:\users\administrator\appdata\local\programs\python\python36\lib\json\e
    ncoder.py&quot;, line 199, in encode
        chunks = self.iterencode(o, _one_shot=True)
    File  
    &quot;c:\users\administrator\appdata\local\programs\python\python36\lib\json\e
    ncoder.py&quot;, line 257, in iterencode
        return _iterencode(o, 0)
    File      
    &quot;c:\users\administrator\appdata\local\programs\python\python36\lib\
    json\encoder.py&quot;, line 180, in default
        o.__class__.__name__)
      TypeError: Object of type &#39;bytes&#39; is not JSON serializable


  

TypeError: Object of type &#39;bytes&#39; is not JSON serializable

while crawling website like https://www.netflix.com, getting Forbidden by robots.txt: &lt;GET https://www.netflix.com/&gt;

ERROR: No response downloaded for: https://www.netflix.com/

getting Forbidden by robots.txt: scrapy

I&#39;m using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking).

I found out that Scrapy can handle forms (like logins) as shown [here][1]. But the problem is that there is no form to fill out, so it&#39;s not exactly what I need.

How can I simply click a button, which then shows the information I need?

Do I have to use an external library like mechanize or lxml?

  [1]: http://doc.scrapy.org/topics/request-response.html?highlight=formrequest#scrapy.http.FormRequest

Click a Button in Scrapy

In the [Scrapy docs][1], there is the following example to illustrate how to use an authenticated session in Scrapy:

    class LoginSpider(BaseSpider):
        name = &#39;example.com&#39;
        start_urls = [&#39;http://www.example.com/users/login.php&#39;]
    
        def parse(self, response):
            return [FormRequest.from_response(response,
                        formdata={&#39;username&#39;: &#39;john&#39;, &#39;password&#39;: &#39;secret&#39;},
                        callback=self.after_login)]
    
        def after_login(self, response):
            # check login succeed before going on
            if &quot;authentication failed&quot; in response.body:
                self.log(&quot;Login failed&quot;, level=log.ERROR)
                return
    
            # continue scraping with authenticated session...

I&#39;ve got that working, and it&#39;s fine. But **my question is**: What do you have to do to `continue scraping with authenticated session`, as they say in the last line&#39;s comment?

  [1]: https://doc.scrapy.org/en/latest/topics/request-response.html#using-formrequest-from-response-to-simulate-a-user-login

Using Scrapy with authenticated (logged in) user session

Consider the case, 
I want to crawl websites frequently, but my IP address got blocked after some day/limit.

So, how can change my IP address dynamically or any other ideas?


Change IP address dynamically?

I&#39;m a bit confused as to how cookies work with Scrapy, and how you manage those cookies.

This is basically a simplified version of what I&#39;m trying to do:
![enter image description here][1]


----------


The way the website works:
--------------------------

When you visit the website you get a session cookie.

When you make a search, the website remembers what you searched for, so when you do something like going to the next page of results, it knows the search it is dealing with.


----------


My script:
----------

My spider has a start url of searchpage_url

The searchpage is requested by `parse()` and the search form response gets passed to `search_generator()`

`search_generator()` then `yield`s lots of search requests using `FormRequest` and the search form response.

Each of those FormRequests, and subsequent child requests need to have it&#39;s own session, so needs to have it&#39;s own individual cookiejar and it&#39;s own session cookie.


----------


I&#39;ve seen the section of the docs that talks about a meta option that stops cookies from being merged. What does that actually mean? Does it mean the spider that makes the request will have its own cookiejar for the rest of its life?

If the cookies are then on a per Spider level, then how does it work when multiple spiders are spawned? Is it possible to make only the first request generator spawn new spiders and make sure that from then on only that spider deals with future requests?

I assume I have to disable multiple concurrent requests.. otherwise one spider would be making multiple searches under the same session cookie, and future requests will only relate to the most recent search made?

I&#39;m confused, any clarification would be greatly received!


----------


EDIT:
-----

Another options I&#39;ve just thought of is managing the session cookie completely manually, and passing it from one request to the other.

I suppose that would mean disabling cookies.. and then grabbing the session cookie from the search response, and passing it along to each subsequent request.

Is this what you should do in this situation?

  [1]: http://i.stack.imgur.com/nn517.png

All Scrapy Solutions on Gang of Coders

Total of 23 Scrapy Solutions