The compression algorithm used in *zlib* is essentially the same as that in *gzip* and *zip*. What are *gzip* and *zip*? How are they different and how are they same?

How are zlib, gzip and zip related? What do they have in common and how are they different?

I&#39;ve just moved over to Ubuntu 8.10 as my dev box; it&#39;s my first serious foray into Linux as a daily-use OS, and I&#39;m having a hard time getting Rails going.  I have followed a number of tutorials which all seem to work fine, but when I try and use gem install or gem update on anything, I get an error that looks like this:

    /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require&#39;: no such file to load -- zlib (LoadError)
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/spec_fetcher.rb:1
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/commands/update_command.rb:5
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:167:in `load_and_instantiate&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:88:in `[]&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:144:in `find_command&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:131:in `process_args&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:102:in `run&#39;
	from /usr/local/lib/ruby/site_ruby/1.8/rubygems/gem_runner.rb:58:in `run&#39;
	from /usr/local/bin/gem:21

I have (as recommended in the tutorials) installed zlib, zlib1g, and zlib1g-dev.  I&#39;ve even found a few tutorials on this particular problem (all for flavors other than mine, however) but they didn&#39;t help.

I&#39;m on Ubuntu 8.10, ruby version  1.8.8dev gem version 1.3.2

I&#39;ve tried reinstalling ruby and gems each and separately, to no avail.



Ubuntu rails install fails on zlib

Gzip format files (created with the `gzip` program, for example) use the &quot;deflate&quot; compression algorithm, which is the same compression algorithm as what [zlib](http://www.zlib.net/) uses. However, when using zlib to inflate a gzip compressed file, the library returns a `Z_DATA_ERROR`.

How can I use zlib to decompress a gzip file?

How can I decompress a gzip stream with zlib?

Using SQLite3 in Python, I am trying to store a compressed version of a snippet of UTF-8 HTML code.

Code looks like this:

    ...
    c = connection.cursor()
    c.execute(&#39;create table blah (cid integer primary key,html blob)&#39;)
    ...
    c.execute(&#39;insert or ignore into blah values (?, ?)&#39;,(cid, zlib.compress(html)))

At which point at get the error:

    sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

If I use &#39;text&#39; rather than &#39;blob&#39; and don&#39;t compress the HTML snippet, it works all fine (db is to large though). When I use &#39;blob&#39; and compress via Python zlib library, I get the above error message. I looked around but couldn&#39;t find a simple answer for this one.

sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings

---
##UPDATE Feb 10 2012:  
zOompf has completed some very thorough research on this very topic [here](http://zoompf.com/blog/2012/02/lose-the-wait-http-compression). It trumps any findings below.
&lt;br/&gt;

---
##UPDATE Sept 11 2010:  
##A testing platform has been created for this [here](http://www.vervestudios.co/projects/compression-tests/)
&lt;br/&gt;

---

&lt;br/&gt;
HTTP 1.1 definitions of GZIP and DEFLATE (zlib) for some background information:

&quot; &#39;Gzip&#39; is the gzip format, and __&#39;deflate&#39; is the zlib format__. They
should probably have called the second one &#39;zlib&#39; instead to avoid
confusion with the raw deflate compressed data format. While the HTTP
1.1 RFC 2616 correctly points to the zlib specification in RFC 1950
for the &#39;deflate&#39; transfer encoding, __there have been reports of
servers and browsers that incorrectly produce or expect raw deflate
data per the deflate specification in RFC 1951, _most notably
Microsoft products___. So even though the &#39;deflate&#39; transfer encoding using the
zlib format would be the more efficient approach (_and in fact exactly
__what the zlib format was designed for___), using the &#39;gzip&#39; transfer
encoding is probably more reliable due to an unfortunate choice of
name on the part of the HTTP 1.1 authors.&quot; (source: &lt;http://www.gzip.org/zlib/zlib_faq.html&gt;)

So, my question: if I send RAW deflate data with NO zlib wrapper (or gzip,
for that matter) are there any modern browsers (e.g., IE6 and up, FF,
Chrome, Safari, etc) that can NOT understand the raw deflate
compressed data (assuming HTTP request header &quot;Accept-Encoding&quot; contains &quot;deflate&quot;)?

Deflate data will ALWAYS be a few bytes smaller than GZIP.

If all these browsers can successfully decode the data, what
downsides are there to sending RAW deflate instead of zlib?


&lt;br /&gt;&lt;br /&gt;
##UPDATE Sept 11 2010:  
##A testing platform has been created for this [here](http://www.vervestudios.co/projects/compression-tests/)

Deflate compression browser compatibility and advantages over GZIP

In my project I need to know what a `zlib` header looks like. I&#39;ve heard it&#39;s rather simple but I cannot find any description of the zlib header.

For example, does it contain a magic number?


What does a zlib header look like?

How do I unzip a gzipped body in a request&#39;s module response?

I have tried several examples around the web but none of them appear to work.

    request(url, function(err, response, body) {
        if(err) {
            handleError(err)
        } else {
            if(response.headers[&#39;content-encoding&#39;] == &#39;gzip&#39;) {    
                // How can I unzip the gzipped string body variable?
                // For instance, this url:
                // http://highsnobiety.com/2012/08/25/norse-projects-fall-2012-lookbook/
                // Throws error:
                // { [Error: incorrect header check] errno: -3, code: &#39;Z_DATA_ERROR&#39; }
                // Yet, browser displays page fine and debugger shows its gzipped
                // And unzipped by browser fine...
                if(response.headers[&#39;content-encoding&#39;] &amp;&amp; response.headers[&#39;content-encoding&#39;].toLowerCase().indexOf(&#39;gzip&#39;) &gt; -1) {	
                    var body = response.body;                    
                    zlib.gunzip(response.body, function(error, data) {
                        if(!error) {
                            response.body = data.toString();
                        } else {
                            console.log(&#39;Error unzipping:&#39;);
                            console.log(error);
                            response.body = body;
                        }
                    });
                }
            }
        }
    }

How do I ungzip (decompress) a NodeJS request&#39;s module gzip response body?

I am trying to compile software on Blue Gene Q using IBM XL compilers and I got this error message:

    &quot;iostreams/zlib.cpp&quot;, line 19.10: 1540-0836 (S) The #include file &quot;zlib.h&quot; is not found.
    make[3]: *** [zlib.o] Error 1

I have installed a new version of zlib and updated `LD_LIBRARY_PATH` with `$HOME/zlib/include`

Am I missing something?


Compilation error - missing zlib.h

This is similar to a [previous question][1], but the answers there don&#39;t satisfy my needs and my question is slightly different:

I currently use gzip compression for some very large files which contain sorted data.  When the files are not compressed, binary search is a handy and efficient way to support seeking to a location in the sorted data.

But when the files are compressed, things get tricky.  I recently found out about [zlib][2]&#39;s `Z_FULL_FLUSH` option, which can be used during compression to insert &quot;sync points&quot; in the compressed output (`inflateSync()` can then begin reading from various points in the file).  This is OK, though files I already have would have to be recompressed to add this feature (and strangely `gzip` doesn&#39;t have an option for this, but I&#39;m willing to write my own compression program if I must).

It seems from [one source][3] that even `Z_FULL_FLUSH` is not a perfect solution...not only is it not supported by all gzip archives, but the very idea of detecting sync points in archives may produce false positives (either by coincidence with the magic number for sync points, or due to the fact that `Z_SYNC_FLUSH` also produces sync points but they are not usable for random access).

Is there a better solution?  I&#39;d like to avoid having auxiliary files for indexing if possible, and explicit, default support for quasi-random access would be helpful (even if it&#39;s large-grained--like being able to start reading at each 10 MB interval).  Is there another compression format with better support for random reads than gzip?

**Edit**: As I mentioned, I wish to do binary search in the compressed data.  I don&#39;t need to seek to a specific (uncompressed) position--only to seek with some coarse granularity within the compressed file. I just want support for something like &quot;Decompress the data starting roughly 50% (25%, 12.5%, etc.) of the way into this compressed file.&quot;

  [1]: https://stackoverflow.com/questions/236414/
  [2]: http://www.zlib.net/
  [3]: https://web.archive.org/web/20090530165348/http://newsgroups.derkeiler.com/Archive/Comp/comp.compression/2006-02/msg00327.html

Compression formats with good support for random access within archives?

I have a gzip file and I am trying to read it via Python as below:

    import zlib
    
    do = zlib.decompressobj(16+zlib.MAX_WBITS)
    fh = open(&#39;abc.gz&#39;, &#39;rb&#39;)
    cdata = fh.read()
    fh.close()
    data = do.decompress(cdata)

it throws this error:

    zlib.error: Error -3 while decompressing: incorrect header check

How can I overcome it?

zlib.error: Error -3 while decompressing: incorrect header check

I am interfacing with a server that requires that data sent to it is compressed with *Deflate* algorithm (Huffman encoding + LZ77) and also sends data that I need to *Inflate*.  

I know that Python includes Zlib, and that the C libraries in Zlib support calls to *Inflate* and *Deflate*, but these apparently are not provided by the Python Zlib module.  It does provide *Compress* and *Decompress*, but when I make a call such as the following:

    result_data = zlib.decompress( base64_decoded_compressed_string )

I receive the following error:

    Error -3 while decompressing data: incorrect header check

Gzip does no better; when making a call such as:

    result_data = gzip.GzipFile( fileobj = StringIO.StringIO( base64_decoded_compressed_string ) ).read()

I receive the error:

    IOError: Not a gzipped file

which makes sense as the data is a *Deflated* file not a true *Gzipped* file.

Now I know that there is a *Deflate* implementation available (Pyflate), but I do not know of an *Inflate* implementation.

It seems that there are a few options:

1.  **Find an existing implementation (ideal) of *Inflate* and *Deflate* in Python**
2.  Write my own Python extension to the zlib c library that includes *Inflate* and *Deflate* 
3.  Call something else that can be executed from the command line (such as a Ruby script, since *Inflate*/*Deflate* calls in zlib are fully wrapped in Ruby)
4.  ?

I am seeking a solution, but lacking a solution I will be thankful for insights, constructive opinions, and ideas.

**Additional information**:
The result of deflating (and encoding) a string should, for the purposes I need, give the same result as the following snippet of C# code, where the input parameter is an array of UTF bytes corresponding to the data to compress:

    public static string DeflateAndEncodeBase64(byte[] data)
    {
        if (null == data || data.Length &lt; 1) return null;
        string compressedBase64 = &quot;&quot;;
    
        //write into a new memory stream wrapped by a deflate stream
        using (MemoryStream ms = new MemoryStream())
        {
            using (DeflateStream deflateStream = new DeflateStream(ms, CompressionMode.Compress, true))
            {
                //write byte buffer into memorystream
                deflateStream.Write(data, 0, data.Length);
                deflateStream.Close();
    
                //rewind memory stream and write to base 64 string
                byte[] compressedBytes = new byte[ms.Length];
                ms.Seek(0, SeekOrigin.Begin);
                ms.Read(compressedBytes, 0, (int)ms.Length);
                compressedBase64 = Convert.ToBase64String(compressedBytes);
            }
        }
        return compressedBase64;
    }

Running this .NET code for the string &quot;deflate and encode me&quot; gives the result 

    7b0HYBxJliUmL23Ke39K9UrX4HShCIBgEyTYkEAQ7MGIzeaS7B1pRyMpqyqBymVWZV1mFkDM7Z28995777333nvvvfe6O51OJ/ff/z9cZmQBbPbOStrJniGAqsgfP358Hz8iZvl5mbV5mi1nab6cVrM8XeT/Dw==

When &quot;deflate and encode me&quot; is run through the Python Zlib.compress() and then base64 encoded, the result is &quot;eJxLSU3LSSxJVUjMS1FIzUvOT0lVyE0FAFXHB6k=&quot;.  

It is clear that zlib.compress() is not an implementation of the same algorithm as the standard Deflate algorithm.

**More Information**:

The first 2 bytes of the .NET deflate data (&quot;7b0HY...&quot;), after b64 decoding are 0xEDBD, which does not correspond to Gzip data (0x1f8b), BZip2 (0x425A) data, or Zlib (0x789C) data.

The first 2 bytes of the Python compressed data (&quot;eJxLS...&quot;), after b64 decoding are 0x789C.  This is a Zlib header.

**SOLVED**

To handle the raw deflate and inflate, without header and checksum, the following things needed to happen:

On deflate/compress: strip the first two bytes (header) and the last four bytes (checksum).

On inflate/decompress: there is a second argument for window size.  If this value is negative it suppresses headers. here are my methods currently, including the base64 encoding/decoding - and working properly:

    import zlib
    import base64
    
    def decode_base64_and_inflate( b64string ):
        decoded_data = base64.b64decode( b64string )
        return zlib.decompress( decoded_data , -15)
    
    def deflate_and_base64_encode( string_val ):
        zlibbed_str = zlib.compress( string_val )
        compressed_string = zlibbed_str[2:-4]
        return base64.b64encode( compressed_string )



Python: Inflate and Deflate implementations

If I were to AES-encrypt a file, and then ZLIB-compress it, would the compression be less efficient than if I first compressed and then encrypted?

In other words, should I compress first or encrypt first, or does it matter?

When compressing and encrypting, should I compress first, or encrypt first?

I&#39;m trying to compile miniSAT on Kali Linux 64 bits but I keep getting the error message:

&gt; fatal error: zlib.h: no such file or directory

I have read that there might be a linking problem that makes the compiler unable to find the header files, but I&#39;m new to Linux and do not know how to solve that.

How to fix fatal error: zlib.h: no such file or directory?

First, please bear with me. I have hard time telling others my problem and this is a long thread...

I am using pythonbrew to run multiple versions of python in Ubuntu 10.10. 
For installing pythonbrew and how it works, please refers to this link below

http://www.howopensource.com/2011/05/how-to-install-and-manage-different-versions-of-python-in-linux/

After reading a couple stackoverflow threads, I finally found the file called Setup under this directory:  **~/.pythonbrew/pythons/Python-2.7.1/lib/python2.7/config**

    In this Setup file I see 
    # Andrew Kuchling&#39;s zlib module.
    # This require zlib 1.1.3 (or later).
    # See http://www.gzip.org/zlib/
    # zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz

I uncommented the last line, then I ran **python -v** again. However, I received the same error when I tried **import zlib**, so I guess I have to do something to install zlib into the lib. 

But I am clueless about what I need to do. Can someone please direct me in the right direction??? Thank you very much!

I am doing this because I want to use different version of python in different virtualenv I created. 
When I did  **virtualenv -p python2.7** I received no module named zlib. 

    jwxie518@jwxie518-P5E-VM-DO:~$ virtualenv -p python2.7 --no-site-packages testenv

    Running virtualenv with interpreter /home/jwxie518/.pythonbrew/pythons/Python-2.7.1/bin/python2.7

    Traceback (most recent call last):
      File &quot;/usr/local/lib/python2.6/dist-packages/virtualenv.py&quot;, line 17, in &lt;module&gt;
        import zlib

    ImportError: No module named zlib


---------- 
**EDIT**

I have to install 2.7.1 by appending --force.


I am developing Django, and I need some of these missing modules, for example sqlite3, and to create my virtualenv I definitely need zlib. If I just use the system default (2.6.6), I have no problem.

To do this with system default, all I need to do is

    virtualenv --no-site-packages testenv

Thanks! 


----------
**(2nd edit)**

I installed 3.2 also and I tested it without problem, so I guess my problem comes down to how to install the missing module(s).

    jwxie518@jwxie518-P5E-VM-DO:~$ virtualenv -p python3.2  testenv
    Running virtualenv with interpreter /home/jwxie518/.pythonbrew/pythons/Python-3.2/bin/python3.2
    New python executable in testenv/bin/python3.2
    Also creating executable in testenv/bin/python
    Installing distribute..................................................................................................................................................................................................................................................................................................................................done.
    Installing pip...............done.
    jwxie518@jwxie518-P5E-VM-DO:~$ virtualenv -p python3.2 --no-site-packages testenv
    Running virtualenv with interpreter /home/jwxie518/.pythonbrew/pythons/Python-3.2/bin/python3.2
    New python executable in testenv/bin/python3.2
    Also creating executable in testenv/bin/python
    Installing distribute..................................................................................................................................................................................................................................................................................................................................done.
    Installing pip...............done.



no module named zlib

I am using this function to uncompress the body or a HTTP response if it is compressed with gzip, compress or deflate. 

    def uncompress_body(self, compression_type, body):
        if compression_type == &#39;gzip&#39; or compression_type == &#39;compress&#39;:
            return zlib.decompress(body)
        elif compression_type == &#39;deflate&#39;:
            compressor = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS)
            compressed = compressor.compress(body)
            compressed += compressor.flush()
            return base64.b64encode(compressed)

        return body

However python throws this error message.

    TypeError: a bytes-like object is required, not &#39;_io.BytesIO&#39;
on this line:

    return zlib.decompress(body)

Essentially, how do I convert from &#39;_io.BytesIO&#39; to a bytes-like object?





Convert from &#39;_io.BytesIO&#39; to a bytes-like object in python3.6?

Trying to get Ruby 1.9.1 up and running with RVM on a fresh install (fedora). 

After doing `rvm install 1.9.1`, the rubygems error logs show that zlib can&#39;t be located 

&gt; no such file to load -- zlib (LoadError)

However both the zlib runtime and development libraries are installed and up-to-date. 

I&#39;m kinda stumped on this one at the moment.

RVM Ruby 1.9.1 install can&#39;t locate zlib but its runtime and dev library are there

Zlib::GzipReader can take &quot;an IO, or IO-like, object.&quot; as it&#39;s input, as stated in docs. 

    Zlib::GzipReader.open(&#39;hoge.gz&#39;) {|gz|
      print gz.read
    }
    
    File.open(&#39;hoge.gz&#39;) do |f|
      gz = Zlib::GzipReader.new(f)
      print gz.read
      gz.close
    end
How should I ungzip a string?

How to decompress Gzip string in ruby?

# My goal
I am trying to install Python 2.7.5 and 3.6.5 side-by-side on my MBP with with pyenv. 

# `pyenv` Installation
Following https://stackoverflow.com/questions/18671253/how-can-i-use-homebrew-to-install-both-python-2-and-3-on-mac , I tried:


    $ pyenv install 3.6.5

Which erred with:

    python-build: use openssl from homebrew
    python-build: use readline from homebrew
    Downloading Python-3.6.5.tar.xz...
    -&gt; https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz
    Installing Python-3.6.5...
    python-build: use readline from homebrew
    
    BUILD FAILED (OS X 10.13.4 using python-build 20160602)
    
    Inspect or clean up the working tree at /var/folders/d0/t8d3jjp161g2dyrd4k67ypc80000gn/T/python-build.20180426091253.34709
    Results logged to /var/folders/d0/t8d3jjp161g2dyrd4k67ypc80000gn/T/python-build.20180426091253.34709.log
    
    Last 10 log lines:
      File &quot;/private/var/folders/d0/t8d3jjp161g2dyrd4k67ypc80000gn/T/python-build.20180426091253.34709/Python-3.6.5/Lib/ensurepip/__main__.py&quot;, line 5, in &lt;module&gt;
        sys.exit(ensurepip._main())
      File &quot;/private/var/folders/d0/t8d3jjp161g2dyrd4k67ypc80000gn/T/python-build.20180426091253.34709/Python-3.6.5/Lib/ensurepip/__init__.py&quot;, line 204, in _main
        default_pip=args.default_pip,
      File &quot;/private/var/folders/d0/t8d3jjp161g2dyrd4k67ypc80000gn/T/python-build.20180426091253.34709/Python-3.6.5/Lib/ensurepip/__init__.py&quot;, line 117, in _bootstrap
        return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
      File &quot;/private/var/folders/d0/t8d3jjp161g2dyrd4k67ypc80000gn/T/python-build.20180426091253.34709/Python-3.6.5/Lib/ensurepip/__init__.py&quot;, line 27, in _run_pip
        import pip
    zipimport.ZipImportError: can&#39;t decompress data; zlib not available
    make: *** [install] Error 1

# What have I tried
* Installing zlib: `brew install zlib`

# My question
How can I install multiple Python environment with `pyenv` on MBP?

pyenv zlib error on MacOS:

I want to download a zip file from the internet and unzip it in memory without saving to a temporary file. How can I do this?

Here is what I tried:

    var url = &#39;http://bdn-ak.bloomberg.com/precanned/Comdty_Calendar_Spread_Option_20120428.txt.zip&#39;;
        
    var request = require(&#39;request&#39;), fs = require(&#39;fs&#39;), zlib = require(&#39;zlib&#39;);
            
      request.get(url, function(err, res, file) {
         if(err) throw err;
         zlib.unzip(file, function(err, txt) {
            if(err) throw err;
            console.log(txt.toString()); //outputs nothing
         });
      });

[EDIT]
As, suggested, I tried using the adm-zip library and I still cannot make this work:

    var ZipEntry = require(&#39;adm-zip/zipEntry&#39;);
    request.get(url, function(err, res, zipFile) {
            if(err) throw err;
            var zip = new ZipEntry();
            zip.setCompressedData(new Buffer(zipFile.toString(&#39;utf-8&#39;)));
            var text = zip.getData();
            console.log(text.toString()); // fails
        });

How to download and unzip a zip file in memory in NodeJs?

Attempting to install the  zlib-devel on mac os x mavericks with homebrew doesn&#39;t work:

    brew install zlib-devel
    Error: No available formula for zlib-devel 
    Searching taps...

This install 

    brew install zlib

works fine though.

All Zlib Solutions on Gang of Coders

Total of 20 Zlib Solutions