How to `wget` a list of URLs in a text file?

TextWget

Text Problem Overview


Let's say I have a text file of hundreds of URLs in one location, e.g.

http://url/file_to_download1.gz
http://url/file_to_download2.gz
http://url/file_to_download3.gz
http://url/file_to_download4.gz
http://url/file_to_download5.gz
....

What is the correct way to download each of these files with wget? I suspect there's a command like wget -flag -flag text_file.txt

Text Solutions


Solution 1 - Text

Quick man wget gives me the following:

> [..] > > -i file > > --input-file=file > > Read URLs from a local or external file. If - is specified as file, URLs are read from the standard input. (Use ./- to read from a file literally named -.) > > If this function is used, no URLs need be present on the command line. If there are URLs both on the command line and in an input file, those on the command lines will be the first ones to be retrieved. If --force-html is not specified, then file should consist of a series of URLs, one per line. > > [..]

So: wget -i text_file.txt

Solution 2 - Text

try:

wget -i text_file.txt

(check man wget)

Solution 3 - Text

If you also want to preserve the original file name, try with:

wget --content-disposition --trust-server-names -i list_of_urls.txt

Solution 4 - Text

Run it in parallel with

cat text_file.txt | parallel --gnu "wget {}"

Solution 5 - Text

If you're on OpenWrt or using some old version of wget which doesn't gives you -i option:

#!/bin/bash
input="text_file.txt"
while IFS= read -r line
do
  wget $line
done < "$input"

Furthermore, if you don't have wget, you can use curl or whatever you use for downloading individual files.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionShanZhengYangView Question on Stackoverflow
Solution 1 - TextrandomnicknameView Answer on Stackoverflow
Solution 2 - TextceyquemView Answer on Stackoverflow
Solution 3 - TextilCosmicoView Answer on Stackoverflow
Solution 4 - TextYuseferiView Answer on Stackoverflow
Solution 5 - Text0x48pirajView Answer on Stackoverflow