Is there a simple shell command/script that supports excluding certain files/folders from being archived?

I have a directory that need to be archived with a sub directory that has a number of very large files I do not need to backup.

**Not quite solutions:**

The `tar --exclude=PATTERN` command matches the given pattern and excludes those files, but I need specific files &amp; folders to be ignored (full file path), otherwise valid files might be excluded.

I could also use the find command to create a list of files and exclude the ones I don&#39;t want to archive and pass the list to tar, but that only works with for a small amount of files. I have tens of thousands.

I&#39;m beginning to think the only solution is to create a file with a list of files/folders to be excluded, then use rsync with `--exclude-from=file` to copy all the files to a tmp directory, and then use tar to archive that directory.

Can anybody think of a better/more efficient solution?


EDIT: **Charles Ma**&#39;s solution works well. The big gotcha is that the `--exclude=&#39;./folder&#39;` **MUST** be at the beginning of the tar command. Full command (cd first, so backup is relative to that directory):

    cd /folder_to_backup
    tar --exclude=&#39;./folder&#39; --exclude=&#39;./upload/folder2&#39; -zcvf /backup/filename.tgz .

Shell command to tar directory excluding certain files/folders

I typically do:

    tar -czvf my_directory.tar.gz my_directory

What if I just want to include everything (including any hidden system files) in my\_directory, but not the directory itself? I don&#39;t want:

    my_directory
       --- my_file
       --- my_file
       --- my_file

I want:

    my_file
    my_file
    my_file

How do I tar a directory of files and folders without including the directory itself?

I have the following command in the part of a backup shell script:

    tar -cjf site1.bz2 /var/www/site1/

When I list the contents of the archive, I get:

    tar -tf site1.bz2
    var/www/site1/style.css
    var/www/site1/index.html
    var/www/site1/page2.html
    var/www/site1/page3.html
    var/www/site1/images/img1.png
    var/www/site1/images/img2.png
    var/www/site1/subdir/index.html

But I would like to remove the part `/var/www/site1` from directory and file names within the archive, in order to simplify extraction and avoid useless constant directory structure. Never know, in case I would extract backuped websites in a place where web data weren&#39;t stored under `/var/www`.

For the example above, I would like to have :

    tar -tf site1.bz2
    style.css
    index.html
    page2.html
    page3.html
    images/img1.png
    images/img2.png
    subdir/index.html

So, that when I extract, files are extracted in the current directory and I don&#39;t need to move extracted files afterwards, and so that sub-directory structures is preserved.

There are already many questions about tar and backuping in `stackoverflow` and at other places on the web, but most of them ask for dropping the entire sub-directory structure (flattening), or just add or remove the initial / in the names (I don&#39;t know what it changes exactly when extracting), but no more.

After having read some of the solutions found here and there as well as the manual, I tried :

    tar -cjf site1.bz2 -C . /var/www/site1/
    tar -cjf site1.bz2 -C / /var/www/site1/
    tar -cjf site1.bz2 -C /var/www/site1/ /var/www/site1/
    tar -cjf site1.bz2 --strip-components=3 /var/www/site1/

But none of them worked the way I want. Some do nothing, some others don&#39;t archive sub-directories anymore.

It&#39;s inside a backup shell script launched by a Cron, so I don&#39;t know well, which user runs it, what is the path and the current directory, so always writing absolute path is required for everything, and would prefer not changing current directory to avoid breaking something further in the script (because it doesn&#39;t only backup websites, but also databases, then send all that to FTP etc.)

How to achieve this?

Have I just misunderstood how the option -C works?



Tar a directory, but don&#39;t store full absolute paths in the archive

I have an archive 

&gt; *.tar.gz

How can I uncompress this in a destination directory?

How to uncompress a tar.gz in another directory

I normally compress using `tar zcvf` and decompress using `tar zxvf` (using gzip due to habit). 

I&#39;ve recently gotten a quad core CPU with hyperthreading, so I have 8 logical cores, and I notice that many of the cores are unused during compression/decompression. 

Is there any way I can utilize the unused cores to make it faster?

Utilizing multi core for tar+gzip/bzip compression/decompression

What is the difference between tar and zip?  What are the use cases for each?

What is the difference between tar and zip?

I have a file that contain list of files I want to archive with tar.
Let&#39;s call it `mylist.txt`

It contains:

    file1.txt
    file2.txt
    ...
    file10.txt

Is there a way I can issue TAR command that takes `mylist.txt` as input?
Something like

    tar -cvf allfiles.tar -[someoption?] mylist.txt

So that it is similar as if I issue this command:

    tar -cvf allfiles.tar file1.txt file2.txt file10.txt 

Tar archiving that takes input from a list of files

I am trying to create a `.tar.xz` compressed archive in one command. What is the specific syntax for that?

I have tried `tar cf - file | xz file.tar.xz`, but that does not work.

Create a tar.xz in one command

I created .tgz file with `tar czvf file ` command.then I ended up with a tgz file. I want to know the difference between it and tar.gz.

Are tar.gz and tgz the same thing?

I have a `/public_html/` folder, in that folder there&#39;s a `/tmp/` folder that has like 70gb of files I don&#39;t really need.

Now I am trying to create a `.tar.gz` of `/public_html/` excluding `/tmp/`

This is the command I ran:

    tar -pczf MyBackup.tar.gz /home/user/public_html/ --exclude &quot;/home/user/public_html/tmp/&quot; 

The tar is still being created, and by doing an `ls -sh` I can see that `MyBackup.tar.gz` already has about 30gb, and I know for sure that `/public_html/` without `/tmp/` doesn&#39;t have more than 1GB of files.

What did I do wrong?

Excluding directory when creating a .tar.gz file

I have found the question https://stackoverflow.com/questions/1788236/how-to-determine-if-data-is-valid-tar-file, but I was wondering: is there a ready made command line solution?

How to check if a Unix .tar.gz file is a valid file without uncompressing?

I want to extract an archive named `filename.tar.gz`.

Using `tar -xzvf filename.tar.gz` doesn&#39;t extract the file. it is gives this error:

    gzip: stdin: not in gzip format
    tar: Child returned status 1
    tar: Error exit delayed from previous errors

How to extract filename.tar.gz file

Alright, so simple problem here. I&#39;m working on a simple back up code.  It works fine except if the files have spaces in them.  This is how I&#39;m finding files and adding them to a tar archive:

    find . -type f | xargs tar -czvf backup.tar.gz 

The problem is when the file has a space in the name because tar thinks that it&#39;s a folder.  Basically is there a way I can add quotes around the results from find? Or a different way to fix this?

Find files and tar them (with spaces)

I try to tar.gz a directory and use

    tar -czf workspace.tar.gz *

The resulting tar includes `.svn` directories in subdirs but NOT in the current directory (as `*` gets expanded to only &#39;visible&#39; files before it is passed to tar

I tried to 

`tar -czf workspace.tar.gz .` instead but then I am getting an error because &#39;.&#39; has changed while reading:

    tar: ./workspace.tar.gz: file changed as we read it

Is there a trick so that `*` matches all files (including dot-prefixed) in a directory?

(using bash on Linux SLES-11 (2.6.27.19)

tar: add all files and directories in current directory INCLUDING .svn and so on

I want to tar and all .php and .html files in a directory and its subdirectories. If I use 

`tar -cf my_archive *`
 
it tars all the files, which I don&#39;t want. If I use 

`tar -cf my_archive *.php *.html`

it ignores subdirectories. How can I make it tar recursively but include only two types of files?

How to tar certain file types in all subdirectories?

On my Linux machine, I wish to create a .tar.bz2 file of a certain folder. Once I place myself in that folder (in the terminal), what do I type in the terminal command line to place the compressed folder in the home directory of my machine?

Let&#39;s say I am in the folder /home/user/folder. In the folder &quot;folder&quot; are several files (txt, .c etc). How do I compress that folder of type .tar.bz2 and place it in my /home directory?

In the /home/user/folder, I&#39;ve tried `sudo tar -cvjSf folder.tar.bz2` but get an error:

&gt; tar: Cowardly refusing to create an empty archive

Create a .tar.bz2 file Linux

I&#39;m working on a backup script and want to tar up a file directory:

    tar czf ~/backup.tgz /home/username/drupal/sites/default/files

This tars it up, but when I untar the resulting file, it includes the full file structure: the files are in `home/username/drupal/sites/default/files`.

Is there a way to exclude the parent directories, so that the resulting tar just knows about the last directory (`files`)?

How do I tar a directory without retaining the directory structure?

I have a tar file which has number of files within it.
I need to write a python script which will read the contents of the files and gives the count o total characters, including  total number of letters, spaces, newline characters, everything,  without untarring the tar file.

reading tar file contents without untarring it, in python script

I am trying to make a script for unzipping all the .tar.gz files from folders in one directory. For example, I will have a file which it calls ( testing.tar.gz). Then if I do manually, I can press to &quot;extract here&quot; then the .tar.gz file will create a new file, and it calls testing.tar. Finally, if I repeat the process of pressing &quot;extract here&quot;, the .tar file prodcudes me all the .pdf files.

I wonder that how can I do it, and I have my code here and it seems doesn&#39;t realty work tho.

    import os
    import tarfile
    import zipfile

    def extract_file(path, to_directory=&#39;.&#39;):
        if path.endswith(&#39;.zip&#39;):
            opener, mode = zipfile.ZipFile, &#39;r&#39;
        elif path.endswith(&#39;.tar.gz&#39;) or path.endswith(&#39;.tgz&#39;):
            opener, mode = tarfile.open, &#39;r:gz&#39;
        elif path.endswith(&#39;.tar.bz2&#39;) or path.endswith(&#39;.tbz&#39;):
            opener, mode = tarfile.open, &#39;r:bz2&#39;
        else: 
            raise ValueError, &quot;Could not extract `%s` as no appropriate extractor is found&quot; % path
    
        cwd = os.getcwd()
        os.chdir(to_directory)
    
        try:
            file = opener(path, mode)
            try: file.extractall()
            finally: file.close()
        finally:
            os.chdir(cwd)


I want to create a script for unzip (.tar.gz) file via (Python)

I am using `make` and `tar` to backup. When executing makefile, tar command shows `file changed as we read it`. In this case,

 - the tar package is ok when the warning comes up
 - but it stops the tar command for the following backup
 - the file showing the warning in fact doesn&#39;t change -- it is really strange that the warning comes up
 - the files showing the warning come up randomly, I mean, everytime I run my makefile, the files showing the warning are different
 - `--ignore-failed-read` doesn&#39;t help. I am using tar 1.23 in MinGW
 - I just changed my computer to WIN7 64 bit. The script works well in old WIN7 32 bit. But the tar version is not as new as the 1.23.

How can I stop the tar&#39;s warning to stop my backup following the warning?

-------------------------------------------------------------------
**Edit-2**: it might be the reason

As I said above, the bash shell script worked well in my old computer. Comparing with the old computer, the `msys` version is different. So is the version of tar command. In the old computer, tar is 1.13.19 and it is 1.23 in the new computer. I copied the old tar command without copying its dependency msys-1.0.dll to the new computer and renamed it tar_old. And I also updated the tar command in the shell script and run the script. Then everything is ok. So, it seemed that the problem is the tar command. I am sure that there is no any file changed when taring. Is it a bug for tar command in new version? I don&#39;t know.

-------------------------------------------------------------------
**Edit-1**: add more details

The backup is invoked by a bash shell script. It scans the target directory and builds makefile then invokes make to use tar command for backup. Followed is a typical makefile built by the bash shell script.

    #--------------------------------------------
    # backup VC
    #--------------------------------------------
    # the program for packing
    PACK_TOOL=tar
    
    # the option for packing tool
    PACK_OPTION=cjvf
    
    # M$: C driver
    WIN_C_DIR=c:
    
    # M$: D driver
    WIN_D_DIR=d:
    
    # M$: where the software is
    WIN_PRG_DIR=wuyu/tools
    # WIN_PRG_DIR=
    
    # where to save the backup files
    BAKDIR=/home/Wu.Y/MS_bak_MSYS
    
    VC_FRAMEWORK=/home/Wu.Y/MS_bak_MSYS/tools/VC/VC_framework.tar.bz2
    VC_2010=/home/Wu.Y/MS_bak_MSYS/tools/VC/VC_2010.tar.bz2
    
    .PHONY: all
    
    all: $(VC_FRAMEWORK) $(VC_2010)
    
    $(VC_FRAMEWORK): $(WIN_C_DIR)/$(WIN_PRG_DIR)/VC/Framework/*
    	@$(PACK_TOOL) $(PACK_OPTION) &quot;$@&quot; --ignore-failed-read /c/$(WIN_PRG_DIR)/VC/Framework
    $(VC_2010): $(WIN_C_DIR)/$(WIN_PRG_DIR)/VC/VS2010/*
    	@$(PACK_TOOL) $(PACK_OPTION) &quot;$@&quot; --ignore-failed-read /c/$(WIN_PRG_DIR)/VC/VS2010

As you can see, the tar package is stored in ~/MS_bak_MSYS/tools/VC/VC_2010.tar.bz2. I run the script in ~/qqaa. `~/MS_bak_MSYS` is excluded from tar command. So, the tar file I am creating is not inside a directory I am trying to put into tar file. This is why I felt it strange that the warning came up.

 

tar: file changed as we read it

I have a tar.gz-file with the following structure:

    folder1/img.gif
    folder2/img2.gif
    folder3/img3.gif

I want to extract the image files without the folder hierarchy so the extracted result looks like:

    /img.gif
    /img2.gif
    /img3.gif

I need to do this with a combination of Unix and PHP. Here is what I have so far, it works to extract them to the specified directory but keeps the folder hierarchy:

    exec(&#39;gtar --keep-newer-files -xzf images.tgz -C /home/user/public_html/images/&#39;,$ret);


How do I extract files without folder structure using tar

I know that I can use following command to extract a single file to the current working directory (assume I have a tar file named test.tar and a file named testfile1 and testfile2 are inside it):

    $tar xvf test.tar testfile1

And I can use -C option to extract files to another directory:

    $tar xvf test.tar -C anotherDirectory/

When I incorporate the above two techniques together, I suppose that I can extract a single file to another directory. 

    $ tar xvf test.tar testfile1 -C anotherDirectory/

But the result is I can only extract the testfile1 to the current working directory, rather than the anotherDirectory.

I want to know how can I extract a single file from tar to a different directory?

How to extract a single file from tar to a different directory?

I have a bunch of `tar` files in a directory and I want to extract all the files from them at once.  But this doesn&#39;t seem to do anything:

    $ tar xf *.tar

What&#39;s going on here?  How do I untar a bunch of files at once?

How can you untar more than one file at a time?

How can I pipe information into `tar` specifying the names of the file?

How can I build a tar from stdin?

When I invoke `add()` on a `tarfile` object with a file path, the file is added to the tarball with directory hierarchy associated. In other words, if I unzip the tarfile the directories in the original directories hierarchy are reproduced.

Is there a way to simply adding a plain file without directory info that untarring the resulting tarball produce a flat list of files?

How can files be added to a tarfile with Python, without adding the directory hierarchy?

How can I extract the size of the total uncompressed file data in a .tar.gz file from command line?

Check the total content size of a tar gz file

I wonder how to list the content of a tar file only down to some level? 

I understand `tar tvf mytar.tar` will list all files, but sometimes I would like to only see directories down to some level.

Similarly, for the command `ls`, how do I control the level of subdirectories that will be displayed? By default, it will only show the direct subdirectories, but not go further.

Listing the content of a tar file or a directory only down to some level

Can GNU tar add many files to an archive, deleting each one as it is added?

This is useful when there is not enough disk space to hold both the entire tar archive and the original files - and therefore it is not possible to simply manually delete the files after creating an archive in the usual way.

Deleting files after adding to tar archive

How do I extract a tar (or tar.gz, or tar.bz2) file in Java?

How do I extract a tar file in Java?

I have a Bash script that creates a *.tar.gz* file, encrypts, and then sends it to a drive. However, I cannot open the *.tar.gz* file afterwards. Here is my process...

Bash script that encrypts.

```lang-none
#!/bin/sh

# Tar the automysqlbackup directory
tar -zcf &quot;red-backup-$(date &#39;+%Y-%m-%d&#39;).tar.gz&quot; /var/lib/automysqlbackup/

# Encrypt the tar file
openssl aes-256-cbc -a -salt -in &quot;red-backup-$(date &#39;+%Y-%m-%d&#39;).tar.gz&quot; -out &quot;red-backup-$(date &#39;+%Y-%m-%d&#39;).tar.gz.enc&quot; -pass &#39;pass:MySecretPWD&#39;

# Remove the original tar file
rm -rf &quot;red-backup-$(date &#39;+%Y-%m-%d&#39;).tar.gz&quot;

# Upload to Google Drive
gdrive upload --file &quot;red-backup-$(date &#39;+%Y-%m-%d&#39;).tar.gz.enc&quot; -p &quot;jofhriout849uioejfoiu09&quot;
```

Then I download the file and use

```lang-none
sudo openssl aes-256-cbc -e -in red-backup-2016-09-22.tar.gz.enc -out red-backup-2016-09-22.tar.gz
```

I then enter the passphrase for my file twice and I now get a file called

```lang-none
red-backup-2016-09-22.tar.gz
```

When I then try

```lang-none
sudo tar -zxvf red-backup-2016-09-22.tar.gz
```

I get

```lang-none
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
```

I have also tried renaming the file .tar and also tried

```lang-none
sudo tar xvf red-backup-2016-09-22.tar.gz
```

and

```lang-none
sudo tar xvf red-backup-2016-09-22.tar

tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
```

Where am I going wrong?





gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now

I have a directory that I want to compress to send it by e-mail, I&#39;ve tried this:

    tar -cvf filename.tar.gz directory_to_compress/

But when I try to send it by e-mail, Google says:

    filename.tar.gz contains an executable file. For security reasons, Gmail does not allow you to send this type of file.

How to compress a directory into a `tar.gz` file from command line?

How to send a compressed archive that contains executables so that Google&#39;s attachment filter won&#39;t reject it

How to I extract a subdirectory in a tarball into the current directory?

Example, the tarball from [wordpress][1]:

    wordpress/
    wordpress/wp-trackback.php
    wordpress/wp-config-sample.php
    wordpress/wp-settings.php
    wordpress/wp-rss2.php
    wordpress/readme.html
    wordpress/index.php
    ...

How do I extract everything under `wordpress/` into the current directory?  In otherwords, it will not create a `wordpress` directory.


I&#39;ve tried this with no luck:

    tar xvfz latest.tar.gz wordpress -C ./

I know I can extract it normally and move it back, but I figure there has to be a way to do it in one shot.

  [1]: http://wordpress.org/download/

How do I untar a subdirectory into the current directory?

How to create tar.gz archive of my files in Windows to upload and extract in cPanel?

How to create tar.gz archive file in Windows?

I have a directory e.g. `/var/tmp/my-dir/` that I frequently compress with the following command:

    $ cd /var/tmp/
    $ tar -zcf my-dir.tar.gz my-dir/*
    
Later, when I untar _my-dir.tar.gz_, it&#39;ll create `my-dir/` in the current directory. It sounds like the `my-dir` directory is &quot;wrapped&quot; inside the tarball. Is there a _tar_ option to rename `my-dir` to e.g. `your-dir` ***before*** the actual tarring happens. So that ...

    $ tar -zxf my-dir.tar.gz
    # So that ... this creates your-dir/, instead of my-dir/

Thanks.

Rename Directory Name Before tar Happens

My source tree contains several directories which are using Git source control, and I need to tarball the whole tree excluding any references to the Git metadata or custom log files.

I thought I&#39;d have a go using a combination of find/egrep/xargs/tar, but somehow the tar file contains the .git directories and the *.log files.

This is what I have:

    find -type f . | egrep -v &#39;\.git|\.log&#39; | xargs tar rvf ~/app.tar

Can someone explain my misunderstanding here? Why is tar processing the files that find and egrep are filtering?

I&#39;m open to other techniques as well.


Tarballing without Git metadata

I have a large tar file I `split`. Is it possible to `cat` and untar the file using pipeline.  

Something like:

    cat largefile.tgz.aa largefile.tgz.ab | tar -xz

instead of:

    cat largefile.tgz.aa largfile.tgz.ab &gt; largefile.tgz
    tar -xzf largefile.tgz

I have been looking around and I can&#39;t find the answer. I wanted to see if it was possible.

How to extract tar archive from stdin?

Can anyone show me the correct way to compress and decompress tar.gzip files in java i&#39;ve been searching but the most i can find is either zip or gzip(alone). 

All Tar Solutions on Gang of Coders

Total of 37 Tar Solutions