get file size of a file to wget before wget-ing it?

Command LineWget

Command Line Problem Overview


I'm wondering if there is a way to check ahead of time the size of a file I might download via wget? I know that using the --spider option tells me if a file exists or not, but I'm interested in finding the size of that file as well.

Command Line Solutions


Solution 1 - Command Line

Hmm.. for me --spider does display the size:

$ wget --spider http://henning.makholm.net/
Spider mode enabled. Check if remote file exists.
--2011-08-08 19:39:48--  http://henning.makholm.net/
Resolving henning.makholm.net (henning.makholm.net)... 85.81.19.235
Connecting to henning.makholm.net (henning.makholm.net)|85.81.19.235|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9535 (9.3K) [text/html]     <-------------------------
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

$ 

(But beware that not all web servers will inform clients of the length of the data except by closing the connection when it's all been sent.)

If you're concerned about wget changing the format it reports the length in, you might use wget --spider --server-response and look for a Content-Length header in the output.

Solution 2 - Command Line

curl --head URL

Look for "Content-Length:" in the output.

And thanks to Henning Makholm's comment:

wget --spider URL

and look for "Length:" in the output.

Solution 3 - Command Line

I was actually looking for the size of a directory and google got me here. While there is no direct answer here, the accepted answer helped me to build the following command on top of it:

wget --spider -m -np URL-to-dir 2>&1 | sed -n -e /unspecified/d -e '/^Length: /{s///;s/ .*//;p}' | paste -s -d+ | bc

The above runs wget in a spider mode for the entire directory, which ends up logging the length for each file in that directory. The output is then piped to sed to extract a sequence of numbers (byte sizes). The last two components in the pipe simply help sum it up to get the total in bytes.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionDang KhoaView Question on Stackoverflow
Solution 1 - Command Linehmakholm left over MonicaView Answer on Stackoverflow
Solution 2 - Command LineKeith ThompsonView Answer on Stackoverflow
Solution 3 - Command LineharidsvView Answer on Stackoverflow