When I run this command:

    wget --user-agent=&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0&quot;  http://yahoo.com

...I get this result (with nothing else in the file):

    &lt;!-- hw147.fp.gq1.yahoo.com uncompressed/chunked Wed Jun 19 03:42:44 UTC 2013 --&gt;

But when I run `wget http://yahoo.com` with no `--user-agent` option, I get the full page.

The user agent is the same header that my current browser sends. Why does this happen? Is there a way to make sure the user agent doesn&#39;t get blocked when using wget?

Sites not accepting wget user agent header

I know I can use the following 2 commands to download a file:

    curl -O example.com/file.zip
    wget example.com/file.zip

But I want them to go into a specific directory. So I can do the following:

    curl -o mydir/file.zip example.com/file.zip
    wget -O mydir/file.zip example.com/file.zip

Is there a way to not have to specify the filename? Something like this:

    curl -dir mydir example.com/file.zip



How to download a file into a directory using curl or wget?

I know that you can manually set some headers with the `--header` option, but I want to know what headers it sends without interaction.

What headers are automatically sent by wget?

I am trying to use [Wget][1] to download a page, but I cannot get past the login screen.

How do I send the username/password using post data on the login page and then download the actual page as an authenticated user?

  [1]: http://en.wikipedia.org/wiki/Wget


How to get past the login page with Wget?

I tried to do a cron and run a url every 5 mintues.

I tried to use WGET however I dont want to download the files on the server, all I want is just to run it.

This is what I used (crontab):

    */5 * * * * wget http://www.example.com/cronit.php

Is there any other command to use other than wget to just run the url and not downlaod it?

Using WGET to run a cronjob PHP

I cannot get wget to mirror a section of a website (a folder path below root) - it only seems to work from the website homepage.

I&#39;ve tried many options - here is one example

    wget -rkp -l3 -np  http://somewebsite/subpath/down/here/

While I only want to mirror the content links below that URL - I also need to download all the page assets which are not in that path.

It seems to work fine for the homepage (`/`) but I can&#39;t get it going for any sub folders.

How to mirror only a section of a website?

I have an wget command like this in my shell script:

    reponse=&quot;`wget -O- http:localhost:8080/app/index.html`&quot;

I don&#39;t understand the -O- option. I was explained that -O is output to somewhere and - is output to the current stream. I don&#39;t see any explaination of &quot;-&quot; in wget. Is that a standard thing for shell scripting. Where I can find reference to it?

Thanks,

What does &quot;wget -O&quot; mean?

This is simplest example running wget:

    wget http://www.example.com/images/misc/pic.png

but how to make wget skip download if `pic.png`is  already available?

Skip download if files already exist in wget?

There is an online HTTP directory that I have access to. I have tried to download all sub-directories and files via `wget`. But, the problem is that when `wget` downloads sub-directories it downloads the `index.html` file which contains the list of files in that directory without downloading the files themselves.

Is there a way to download the sub-directories and files without depth limit (as if the directory I want to download is just a folder which I want to copy to my computer).

![online HTTP directory][1]


  [1]: http://i.stack.imgur.com/uNOuS.png

How to download HTTP directory with all files and sub-directories as they appear on the online files/folders list?

I want to download something with `wget` using a proxy:

    HTTP Proxy: 127.0.0.1
    Port: 8080

The proxy does not need username and password.

How can I do this?

How to set proxy for wget?

I&#39;m using wget to download website content, but wget downloads the files one by one.

How can I make wget download using 4 simultaneous connections?

Multiple simultaneous downloads using Wget?

I try to install wget in MAC OS 10.11.1 but when I run `./configure --with-ssl=openssl` I get this error:

    configure: error: --with-ssl=openssl was given, but SSL is not available.

How to resolve this problem in OSX 10.11.1?

How to install wget in macOS?

I would like to download a local copy of a web page and get all of the css, images, javascript, etc.  

In previous discussions (e.g. [here][1] and [here][2], both of which are more than two years old), two suggestions are generally put forward: `wget -p` and [httrack][3].  However, these suggestions both fail.  I would very much appreciate help with using either of these tools to accomplish the task; alternatives are also lovely.  
  
&lt;br /&gt;
**Option 1: `wget -p`**

`wget -p` successfully downloads all of the web page&#39;s prerequisites (css, images, js).  However, when I load the local copy in a web browser, the page is unable to load the prerequisites because the paths to those prerequisites haven&#39;t been modified from the version on the web. 

For example:

 * In the page&#39;s html, `&lt;link rel=&quot;stylesheet href=&quot;/stylesheets/foo.css&quot; /&gt;` will need to be corrected to point to the new relative path of `foo.css`
 * In the css file, `background-image: url(/images/bar.png)` will similarly need to be adjusted.

Is there a way to modify `wget -p` so that the paths are correct?

&lt;br /&gt;
**Option 2: [httrack][4]**

`httrack` seems like a great tool for mirroring entire websites, but it&#39;s unclear to me how to use it to create a local copy of a single page.  There is a great deal of discussion in the httrack forums about this topic (e.g. [here][5]) but no one seems to have a bullet-proof solution.

&lt;br /&gt;
**Option 3: another tool?**

Some people have suggested paid tools, but I just can&#39;t believe there isn&#39;t a free solution out there.


  [1]: https://stackoverflow.com/questions/706934/download-an-offline-version-of-a-webpage
  [2]: https://superuser.com/questions/130306/suggestions-for-a-site-ripper
  [3]: http://www.httrack.com
  [4]: http://www.httrack.com
  [5]: http://forum.httrack.com/readmsg/19088/index.html

Download a working local copy of a webpage

For example, running `wget https://www.dropbox.com` results in the following errors:

    ERROR: The certificate of `www.dropbox.com&#39; is not trusted.
    ERROR: The certificate of `www.dropbox.com&#39; hasn&#39;t got a known issuer.


How do I fix certificate errors when running wget on an HTTPS URL in Cygwin?

I need files to be downloaded to /tmp/cron_test/. My wget code is

    wget --random-wait -r -p -nd -e robots=off -A&quot;.pdf&quot; -U mozilla http://math.stanford.edu/undergrad/

So is there some parameter to specify the directory?


How to specify the download location with wget?

How to use `wget` and get all the files from website?

I need all files except the webpage files like HTML, PHP, ASP etc.  

How to download all files (but not HTML) from a website using wget?

I am trying to download the files for a project using `wget`, as the SVN server for that project isn&#39;t running anymore and I am only able to access the files through a browser. The base URLs for all the files is the same like 

&gt; http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/*

How can I use `wget` (or any other similar tool) to download all the files in this repository, where the &quot;tzivi&quot; folder is the root folder and there are several files and sub-folders (upto 2 or 3 levels) under it?

How to download an entire directory and subdirectories using wget?

I want to get the content from [this][1] website. 

If I use a browser like Firefox or Chrome I could get the real website page I want, but if I use the Python requests package (or `wget` command) to get it, it returns a totally different HTML page. 

I thought the developer of the website had made some blocks for this.

## Question ## 

**How do I fake a browser visit by using python requests or command wget?**


  [1]: http://www.ichangtou.com/#company:data_000008.html

How to use Python requests to fake a browser visit a.k.a and generate User Agent?

I am downloading a file using the `wget` command. But when it downloads to my local machine, I want it to be saved as a different filename.

For example: I am downloading a file from `www.examplesite.com/textfile.txt`

I want to use `wget` to save the file `textfile.txt` on my local directory as `newfile.txt`. I am using the `wget` command as follows:
    

    wget www.examplesite.com/textfile.txt


wget command to download a file and save as a different filename

I&#39;m using Wget to make http requests to a fresh web server.  I am doing this to warm the MySQL cache.  I do not want to save the files after they are served.

    wget -nv -do-not-save-file $url

Can I do something like `-do-not-save-file` with wget?

How do I request a file but not save it with Wget?

I need to get the final URL after a page redirect preferably with curl or wget.

For example *http://google.com* may redirect to *http://www.google.com*.

The contents are easy to get(ex. `curl --max-redirs 10 http://google.com -L`), but I&#39;m only interested in the final url (in the former case http://www.google.com).

Is there any way of doing this by using only Linux built-in tools? (command line only)

Get final URL after curl is redirected

I&#39;m trying to output the document body and its headers to STDOUT by doing

    wget -S -O - http://google.com

...but it shows only the HTML document.

**Update:** Got it to work with

    wget --save-headers --output-document - http://google.com

`wget --version` shows my version is `GNU Wget 1.11.4 Red Hat modified`.


Wget output document and headers to STDOUT

I am using wget to download all images from a website and it works fine but it stores the original hierarchy of the site with all the subfolders and so the images are dotted around. Is there a way so that it will just download all the images into a single folder? The syntax I&#39;m using at the moment is:

    wget -r -A jpeg,jpg,bmp,gif,png http://www.somedomain.com

How do I use Wget to download all images into a single folder, from a URL?

I have a web directory where I store some config files. I&#39;d like to use wget to pull those files down and maintain their current structure. For instance, the remote directory looks like:

    http://mysite.com/configs/.vim/

.vim holds multiple files and directories. I want to replicate that on the client using wget. Can&#39;t seem to find the right combo of wget flags to get this done. Any ideas?

Using wget to recursively fetch a directory with arbitrary files in it

When I try to download Java from Oracle I instead end up downloading a page telling me that I need agree to the OTN license terms.

&gt; # Sorry!
&gt; 
&gt; In order to download products from Oracle Technology Network you must agree to the OTN license terms.
&gt; 
&gt; Be sure that...
&gt; 
&gt; - Your browser has &quot;cookies&quot; and JavaScript enabled.
&gt; - You clicked on &quot;Accept License&quot; for the product you wish to download.
&gt; - You attempt the download within 30 minutes of accepting the license.

How can I download and install Java?

Downloading Java JDK on Linux via wget is shown license page instead

Are they the same or not? Can certain things be done with one but not the other? What are those? Or is it, at the end of the day, a matter of familiarity?

What is better, curl or wget?

In Linux how can I fetch an URL and get its contents in a variable in shell script? 

How to get the contents of a webpage in a shell variable?

Let&#39;s say I have a text file of hundreds of URLs in one location, e.g.

    http://url/file_to_download1.gz
    http://url/file_to_download2.gz
    http://url/file_to_download3.gz
    http://url/file_to_download4.gz
    http://url/file_to_download5.gz
    ....

What is the correct way to download each of these files with `wget`? I suspect there&#39;s a command like `wget -flag -flag text_file.txt`

How to `wget` a list of URLs in a text file?

I&#39;m trying to download a Debian package inside a Ubuntu container as follows:

    sudo docker run ubuntu:14.04 wget https://downloads-packages.s3.amazonaws.com/ubuntu-14.04/gitlab_7.8.2-omnibus.1-1_amd64.deb

I get

    exec: &quot;wget&quot;: executable file not found in $PATH

I&#39;ve already installed `wget` with docker as follows:

    run ubuntu:14.04 apt-get install wget

How can I download a file?

How to run wget inside Ubuntu Docker image?

I tried to use cURL but it seems that by default (Debian) is not compiled with HTTPS support and I dont want to build it myself.

`wget` seems to have SSL support but I found no information on how to generate an OPTIONS HTTP request with wget.

How to send a HTTP OPTIONS request from the command line?

I am trying to download a tarball from GitHub using [cURL][1], but it does not seem to be redirecting:

    $ curl --insecure https://github.com/pinard/Pymacs/tarball/v0.24-beta2
    &lt;html&gt;&lt;body&gt;You are being &lt;a href=&quot;https://nodeload.github.com/pinard/Pymacs/tarball/v0.24-beta2&quot;&gt;redirected&lt;/a&gt;.&lt;/body&gt;&lt;/html&gt;

Note: wget works for me:

    $ wget --no-check-certificate https://github.com/pinard/Pymacs/tarball/v0.24-beta2

However I want to use cURL because ultimately I want to untar it inline with something like:

    $ curl --insecure https://github.com/pinard/Pymacs/tarball/v0.24-beta2 | tar zx

I found that the URL after redirecting turned out to be https://download.github.com/pinard-Pymacs-v0.24-beta1-0-gcebc80b.tar.gz, but I would like cURL to be smart enough to figure this out.

  [1]: http://en.wikipedia.org/wiki/CURL


How do I download a tarball from GitHub using cURL?

I&#39;m running a PHP script via cron using Wget, with the following command:

    wget -O - -q -t 1 http://www.example.com/cron/run

The script will take a maximum of 5-6 minutes to do its processing. Will WGet wait for it and give it all the time it needs, or will it time out?

Does WGET timeout?

I&#39;m trying to download a file from google drive in a script, and I&#39;m having a little trouble doing so. The files I&#39;m trying to download are [here](https://drive.google.com/folderview?id=0Bz-w5tutuZIYeHJTWjR5WW1pa1E&amp;usp=sharing).

I&#39;ve looked online extensively and I finally managed to get one of them to download. I got the UIDs of the files and the smaller one (1.6MB) downloads fine, however the larger file (3.7GB) always redirects to a page which asks me whether I want to proceed with the download without a virus scan. Could someone help me get past that screen?

Here&#39;s how I got the first file working - 

    curl -L &quot;https://docs.google.com/uc?export=download&amp;id=0Bz-w5tutuZIYeDU0VDRFWG9IVUE&quot; &gt; phlat-1.0.tar.gz

When I run the same on the other file, 

    curl -L &quot;https://docs.google.com/uc?export=download&amp;id=0Bz-w5tutuZIYY3h5YlMzTjhnbGM&quot; &gt; index4phlat.tar.gz

I get the the following output -
[![enter image description here][1]][1]

I notice on the third-to-last line in the link, there a `&amp;confirm=JwkK` which is a random 4 character string but suggests there&#39;s a way to add a confirmation to my URL. One of the links I visited suggested `&amp;confirm=no_antivirus` but that&#39;s not working.

I hope someone here can help with this!


  [1]: http://i.stack.imgur.com/Szcq2.jpg

wget/curl large file from google drive

I&#39;m trying to create a Python function that does the same thing as this wget command:

    wget -c --read-timeout=5 --tries=0 &quot;$URL&quot;

`-c` - Continue from where you left off if the download is interrupted.

`--read-timeout=5` - If there is no new data coming in for over 5 seconds, give up and try again.  Given `-c` this mean it will try again from where it left off.

`--tries=0` - Retry forever.

Those three arguments used in tandem results in a download that cannot fail.

**I want to duplicate those features in my Python script, but I don&#39;t know where to begin...**

Python equivalent of a given wget command

For example:

```lang-none
wget http://somesite.com/TheFile.jpeg

    downloading: TheFile.tar.gz ...
    --09:30:42--  http://somesite.com/TheFile.jpeg
               =&gt; `/home/me/Downloads/TheFile.jpeg&#39;
    Resolving somesite.co... xxx.xxx.xxx.xxx.
    Connecting to somesite.co|xxx.xxx.xxx.xxx|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 1,614,820 (1.5M) [image/jpeg]

    25% [======&gt;                              ] 614,424      173.62K/s    ETA 00:14

How can I get it to look like the following?

    downloading: TheFile.jpeg ...
    25% [======&gt;                              ] 614,424      173.62K/s    ETA 00:14
```

I know curl can do that. However, I need to get wget to do that job.




How can I show the wget progress bar only?

I don&#39;t want to see any message when I use `wget`. I want to suppress all the output it normally produces on the screen.

How can I do it?

How to hide wget output in Linux?

To download the SOFA Statistics from the server I use the wget command:

    wget -c http://sourceforge.net/projects/sofastatistics/files/latest/download?source=dlp
The filename of downloaded file in this case is `download?source=files`. If I add the `--output-document` option to the command, to rename the output file to `sofastatistics-latest.deb`, the format of downloaded file is not recognized by dpkg package.

    dpkg-deb: error: `sofastatistics-latest.deb&#39; is not a debian format archive

How to rename correctly the downloaded file with wget?

**UPDATE - *Jan 08 &#39;15***

With the provided link the downloaded file always will be a *.tar.gz one. To get it with the real name just add the `--content-disposition` option as this (thanks to @6EQUJ5!):

    wget --content-disposition http://sourceforge.net/projects/sofastatistics/files/latest/download?source=dlp

But I needed a *.deb file, so here was right the @creaktive, I had to search for a *.deb file link.

Thanks to all for the answers!

How to rename the downloaded file with wget?

Can I use wget to check for a 404 and not actually download the resource?
If so how?
Thanks

Can I use wget to check , but not download

A page contains links to a set of .zip files, all of which I want to download. I know this can be done by wget and curl. How is it done?


How to download all links to .zip files on a given web page using wget/curl?

I want to use wget to upload a picture to a distant server, using an authentication token, &#39;AUTH_1624582364932749DFHDD&#39;, to the &#39;test&#39; folder.

This command doesn&#39;t work (authorization failed), and I want to make sure that it&#39;s not about syntax:

    wget --post-file=nature.jpg http://ipadress:8080/v1/AUTH_test/test/ --post-data=&quot;AUTH_1624582364932749DFHDD&quot;

Any suggestions?


Post request with Wget?

I am getting a bunch of relatively small pages from a website and was wondering if I could somehow do it in parallel in Bash. Currently my code looks like this, but it takes a while to execute (I think what is slowing me down is the latency in the connection).

    for i in {1..42}
    do
        wget &quot;https://www.example.com/page$i.html&quot;
    done

I have heard of using xargs, but I don&#39;t know anything about that and the man page is very confusing.  Any ideas? Is it even possible to do this in parallel? Is there another way I could go about attacking this?

Parallel wget in Bash

As a user in a class that runs Jupyter notebooks for assignments, I have access to the assignments via the web interface. I assume the assignments are stored somewhere in my personal space on the server, and so I should be able to download them. How can I download all files that are in my personal user space? (e.g., `wget`)

Here&#39;s the path structure:

`https://urltoserver/user/username`

There are several directories: assignments, data, etc.

`https://urltoserver/user/username/assignments`

`https://urltoserver/user/username/data`

`...`

I want to download all the folders (recursively). Just enough that I can launch whatever I see online locally. If there are some forbidden folders, then ok, skip those and download the rest.

Please specify the command exactly as I couldn&#39;t figure it out myself (I tried `wget`)

Download all files in a path on Jupyter notebook server

I tried &#39;-N&#39; and &#39;--no-clobber&#39; but the only result that I get is to retrieve a new copy of the existing example.exe with number a number added using this synax &#39;example.exe.1&#39;. This is not what I&#39;d like to get. I just need to download and overwrite the file example.exe in the same folder where I already saved a copy of example.com without that wget verifies if the mine is older or newer respect the on example.exe file already present in my download folder. Do you think is i possible or I need to create a script that delete the example.exe file or maybe something that change his modification date etc?

How to force wget to overwrite an existing file ignoring timestamp?

I&#39;m looking for a way to pseudo-spider a website. The key is that I don&#39;t actually want the content, but rather a simple list of URIs. I can get reasonably close to this idea with [Wget](http://www.gnu.org/software/wget/) using the `--spider` option, but when piping that output through a `grep`, I can&#39;t seem to find the right magic to make it work:

    wget --spider --force-html -r -l1 http://somesite.com | grep &#39;Saving to:&#39;

The `grep` filter seems to have absolutely no affect on the `wget` output. Have I got something wrong or is there another tool I should try that&#39;s more geared towards providing this kind of limited result set?

**UPDATE**

So I just found out offline that, by default, `wget` writes to stderr. I missed that in the man pages (in fact, I still haven&#39;t found it if it&#39;s in there). Once I piped the return to stdout, I got closer to what I need:

    wget --spider --force-html -r -l1 http://somesite.com 2&gt;&amp;1 | grep &#39;Saving to:&#39;

I&#39;d still be interested in other/better means for doing this kind of thing, if any exist.

Spider a Website and Return URLs Only

poeple more and more send me big zip files via wetransfer, which is fine
as it works and i don&#39;t need to setup a ftp or upload myself.
but i need to get this data to my ubuntu server that sits on a fast connection
in a serverfarm.

i tried to wget the link that wetransfer sends by mail,
but i only get the html info page that shows up before wetransfer sends the file
through http

i tried a txt-browser (lynx, elinks) but wetrasnfer keeps on mumbling about &quot;outdated browser&quot; and it somehow does not work

i end up downloading to my laptop and uploading through throttled DSL to the server
which takes all night...

does anyone had success with downloading wetransfer links from linux terminal?

thnx
mart

download files via terminal/cmdline through wetransfer.com

I&#39;m trying to `wget` to my own box, and it can&#39;t be an internal address in the wget (so says another developer).

When I wget, I get this:

    wget http://example.com
    --2013-03-01 15:03:30--  http://example.com/
    Resolving example.com... 172.20.0.224
    Connecting to example.com|172.20.0.224|:80... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://www.example.com/ [following]
    --2013-03-01 15:03:30--  https://www.example.com/
    Resolving www.example.com... 172.20.0.224
    Connecting to www.example.com|172.20.0.224|:443... connected.
    OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
    Unable to establish SSL connection.

I believe it is because I do not have the certificate setup properly. Using openssl:

    openssl s_client -connect example.com:443
    CONNECTED(00000003)
    15586:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:588:

While if I do the same command on another site, it shows the entire cert. 

Perhaps the ssl cert was never setup in the conf file on Apache for that domain?

If so, what should I be specifying in the virtualhost? Is there any alternative other than specifying `--no-check-certificate` because I don&#39;t want to do that?

Unable to establish SSL connection, how do I fix my SSL cert?

Calling curl without parameters, i get the page output, even with an http status code = 404:

    $ curl http://www.google.com/linux;
    &lt;!DOCTYPE html&gt;
    &lt;html lang=en&gt;
      &lt;meta charset=utf-8&gt;
      &lt;meta name=viewport content=&quot;initial-scale=1, minimum-scale=1, width=device-width&quot;&gt;
      &lt;title&gt;Error 404 (Not Found)!!1&lt;/title&gt;
      &lt;style&gt;
        *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* &gt; body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/errors/logo_sm_2.png) no-repeat}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/errors/logo_sm_2_hr.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:55px;width:150px}
      &lt;/style&gt;
      &lt;a href=//www.google.com/&gt;&lt;span id=logo aria-label=Google&gt;&lt;/span&gt;&lt;/a&gt;
      &lt;p&gt;&lt;b&gt;404.&lt;/b&gt; &lt;ins&gt;That’s an error.&lt;/ins&gt;
      &lt;p&gt;The requested URL &lt;code&gt;/linux&lt;/code&gt; was not found on this server.  &lt;ins&gt;That’s all we know.&lt;/ins&gt;
    
    $ echo $?;
    0

The status code is 0.

Calling it with --fail will not show the output:

    $ curl --fail http://www.google.com/linux;
    curl: (22) The requested URL returned error: 404 Not Found
    
    $ echo $?;
    22

The status code is 22 now ...

Id&#39; like to get the output even when http status = 404, 500 (like the first curl execution) and, at the same time, get a different system error (like in the second curl execution, $? = 22).
Is it possible with curl? If not, how could I achieve this with another tool (this tool must accept file uploads e post data! wget doesn&#39;t seems to be an alternative ...)

Thanks.

Get page output with curl --fail

I&#39;m wondering if there is a way to check ahead of time the size of a file I might download via `wget`? I know that using the `--spider` option tells me if a file exists or not, but I&#39;m interested in finding the size of that file as well.

get file size of a file to wget before wget-ing it?

How do I prevent *wget* from following redirects?

wget: don&#39;t follow redirects

I&#39;m trying to use wget command: 

    wget -p http://www.example.com 

to fetch all the files on the main page. For some websites it works but in most of the cases, it only download the index.html. I&#39;ve tried the wget -r command but it doesn&#39;t work. Any one knows how to fetch all the files on a page, or just give me a list of files and corresponding urls on the page?

Why does wget only download the index.html for some websites?

I want to mirror a folder via FTP, like this:

    wget --mirror --user=x --password=x ftp://ftp.site.com/folder/subfolder/evendeeper

But I do not want to create a directory structure like this:

&gt;ftp.site.com -&gt; folder -&gt; subfolder -&gt; evendeeper

I just want:

&gt;evendeeper

And anything below it to be the resulting structure. It would also be acceptable for the contents of `evendeeper` to wind up in the current directory as long as subdirectories are created for subdirectories of `evendeeper` on the server.

I am aware of the `-np` option, according to the documentation that just keeps it from following links to parent pages (a non-issue for the binary files I&#39;m mirroring via FTP). I am also aware of the `-nd` option, but this prevents creating any directory structure at all, even for subdirectories of `evendeeper`.

I would consider alternatives as long as they are command-line-based, readily available as Ubuntu packages and easily automated like wget.

How do I mirror a directory with wget without creating parent directories?

I can download a file by url but when I try it from bash I get a html page instead of a file.

How to download file with url redirection (301 Moved Permanently) using curl, wget or something else?

**UPD**

Headers from the url request.

&gt; curl -I http://www.somesite.com/data/file/file.rar

    HTTP/1.1 301 Moved Permanently
    Date: Sat, 07 Dec 2013 10:15:28 GMT
    Server: Apache/2.2.22 (Ubuntu)
    X-Powered-By: PHP/5.3.10-1ubuntu3
    Location: http://www.somesite.com/files/html/archive.html
    Vary: Accept-Encoding
    Content-Type: text/html
    X-Pad: avoid browser bug



All Wget Solutions on Gang of Coders

Total of 52 Wget Solutions