You can determine that it is _likely_ to be one of those formats by looking at the first few bytes.  You should then test to see if it really _is_ one of those, using an integrity check from the associated utility for that format, or by actually proceeding to decompress.

You can find the header formats in the descriptions:

 - [Zip (.zip) format description][1], starts with 0x50, 0x4b, 0x03, 0x04 (unless empty — then the last two are 0x05, 0x06 or 0x06, 0x06)
 - [Gzip (.gz) format description][2], starts with 0x1f, 0x8b, 0x08
 - [xz (.xz) format description][3], starts with 0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00

Others:

 - [zlib (.zz) format description][4], starts with two bytes (in bits) 0aaa1000 bbbccccc, where ccccc is chosen so that the first byte viewed as a int16 times 256 plus the second byte viewed as a int16 is a multiple of 31. e.g: 01111000(bits) = 120(int16), 10011100(bits) = 156(int16), 120 * 256 + 156 = 30876 which is a multiple of 31
 - compress (.Z) starts with 0x1f, 0x9d
 - bzip2 (.bz2) starts with 0x42, 0x5a, 0x68
 - [Zstandard (.zstd) format description][5], frame starts with a 4 byte magic number using *little-endian* format `0xFD2FB528`, a skipable frame starts with `0x184D2A5?` (question mark is any value from 0 to F), and dictionary starts with `0xEC30A437`.
 - A few more formats in the [magic database][6] from the `file` command


  [1]: http://www.pkware.com/documents/casestudies/APPNOTE.TXT
  [2]: http://www.ietf.org/rfc/rfc1952.txt
  [3]: http://tukaani.org/xz/xz-file-format-1.0.4.txt
  [4]: http://www.ietf.org/rfc/rfc1950.txt
  [5]: https://www.ietf.org/rfc/rfc8878.txt
  [6]: https://github.com/file/file/blob/master/magic/Magdir/compress

If you&#39;re on a Linux box just use the &#39;file&#39; command.

http://en.wikipedia.org/wiki/File_(command)

    $ mv foo.zip dink
    $ file dink
    dink: gzip compressed data, from Unix, last modified: Sat Aug  6 08:08:57 2011,
    max compression
    $

As an alternative to inspecting the file header by hand, you could use some utility like TrID. The link points to the cross-platform command line version; for Windows there&#39;s a GUI, too.
 

If you want to determine an algorithm used to compress a linux kernel, there is a script for that, see this question and answer: https://unix.stackexchange.com/a/553192/264065

The title should be pretty self explanetory.

For debugging purposes, I would like express to print the response code and body for every request serviced. Printing the response code is easy enough, but printing the response body is trickier, since it seems the response body is not readily available as a property.

The following does NOT work:

    var express = require(&#39;express&#39;);
    var app = express();

    // define custom logging format
    express.logger.format(&#39;detailed&#39;, function (token, req, res) {                                    
        return req.method + &#39;: &#39; + req.path + &#39; -&gt; &#39; + res.statusCode + &#39;: &#39; + res.body + &#39;\n&#39;;
    });  

    // register logging middleware and use custom logging format
    app.use(express.logger(&#39;detailed&#39;));

    // setup routes
    app.get(..... omitted ...);

    // start server
    app.listen(8080);

Of course, I could easily print the responses at the client who emitted the request, but I would prefer doing at the server side too.

PS: If it helps, all my responses are json, but hopefully there is a solution that works with general responses.

express logging response body

Why is pep8 complaining on the next string in the code?

    import re
    re.compile(&quot;\d{3}&quot;)
The warning I receive:

    ID:W1401  Anomalous backslash in string: &#39;\d&#39;. String constant might be missing an r prefix.
Can you explain what is the meaning of the message? What do I need to change in the code so that the warning *W1401* is passed?

The code passes the tests and runs as expected. Moreover `\d{3}` is a valid regex.

pep8 warning on regex string in Python, Eclipse

How can one detect the type of compression used on the file? (assuming that .zip, .gz, .xz or any other extension is not specified).

Is this information stored somewhere in the header of that file?

How to detect type of compression used on the file? (if no file extension is specified)

<p>How can one detect the type of compression used on the file? (assuming that .zip, .gz, .xz or any other extension is not specified).</p>
<p>Is this information stored somewhere in the header of that file?</p>


Until now I believed that 1024 bytes equals 1 KB (kilobyte) but I was reading on the internet about decimal and binary system.

![enter image description here][1]


  [1]: http://i.stack.imgur.com/sR9WU.png

So, actually 1024 bytes = 1 KB would be the correct way to define or simply there is a general confusion?

Really 1 KB (KiloByte) equals 1024 bytes?

I wrote a &#39;simple&#39; (it took me 30 minutes) program that converts decimal number to binary. I am SURE that there&#39;s a lot simpler way so can you show me?
Here&#39;s the code:

    #include &lt;iostream&gt;
    #include &lt;stdlib.h&gt;
    
    using namespace std;
    int a1, a2, remainder;
    int tab = 0;
    int maxtab = 0;
    int table[0];
    int main()
    {
        system(&quot;clear&quot;);
        cout &lt;&lt; &quot;Enter a decimal number: &quot;;
        cin &gt;&gt; a1;
        a2 = a1; //we need our number for later on so we save it in another variable
    
        while (a1!=0) //dividing by two until we hit 0
        {
            remainder = a1%2; //getting a remainder - decimal number(1 or 0)
            a1 = a1/2; //dividing our number by two
            maxtab++; //+1 to max elements of the table
        }
    
        maxtab--; //-1 to max elements of the table (when dividing finishes it adds 1 additional elemnt that we don&#39;t want and it&#39;s equal to 0)
        a1 = a2; //we must do calculations one more time so we&#39;re gatting back our original number
        table[0] = table[maxtab]; //we set the number of elements in our table to maxtab (we don&#39;t get 10&#39;s of 0&#39;s)
    
        while (a1!=0) //same calculations 2nd time but adding every 1 or 0 (remainder) to separate element in table
        {
            remainder = a1%2; //getting a remainder
            a1 = a1/2; //dividing by 2
            table[tab] = remainder; //adding 0 or 1 to an element
            tab++; //tab (element count) increases by 1 so next remainder is saved in another element
        }
    
        tab--; //same as with maxtab--
        cout &lt;&lt; &quot;Your binary number: &quot;;
    
        while (tab&gt;=0) //until we get to the 0 (1st) element of the table
        {
            cout &lt;&lt; table[tab] &lt;&lt; &quot; &quot;; //write the value of an element (0 or 1)
            tab--; //decreasing by 1 so we show 0&#39;s and 1&#39;s FROM THE BACK (correct way)
        }
    
        cout &lt;&lt; endl;
        return 0;
    }

By the way it&#39;s complicated but I tried my best.

**edit - Here is the solution I ended up using:**

    std::string toBinary(int n)
    {
        std::string r;
        while(n!=0) {r=(n%2==0 ?&quot;0&quot;:&quot;1&quot;)+r; n/=2;}
        return r;
    }

C++ - Decimal to binary converting

I have a binary file with some layout I know. For example let format be like this:

 - 2 bytes (unsigned short) - length of a string
 - 5 bytes (5 x chars) - the string - some id name
 - 4 bytes (unsigned int) - a stride
 - 24 bytes (6 x float - 2 strides of 3 floats each) - float data

The file should look like (I added spaces for readability):

    5 hello 3 0.0 0.1 0.2 -0.3 -0.4 -0.5
Here 5 - is 2 bytes: 0x05 0x00. &quot;hello&quot; - 5 bytes and so on.

Now I want to read this file. Currently I do it so:

 - load file to **ifstream**
 - read this stream to `char buffer[2]` 
 - cast it to unsigned short: `unsigned short len{ *((unsigned short*)buffer) };`. Now I have length of a string.
 - read a stream to `vector&lt;char&gt;` and create a `std::string` from this vector. Now I have string id.
 - the same way read next 4 bytes and cast them to unsigned int. Now I have a stride.
 - while not end of file read floats the same way - create a `char bufferFloat[4]` and cast `*((float*)bufferFloat)` for every float.

This works, but for me it looks ugly. Can I read directly to `unsigned short` or `float` or `string` etc. without `char [x]` creating? If no, what is the way to cast correctly (I read that style I&#39;m using - is an old style)?

P.S.: while I wrote a question, the more clearer explanation raised in my head - how to cast arbitrary number of bytes from arbitrary position in `char [x]`?

Update: I forgot to mention explicitly that string and float data length is not known at compile time and is variable.

Parsing a binary file. What is a modern way?

I don&#39;t understand why

    SELECT UUID();
    
Returns something like:

    3f06af63-a93c-11e4-9797-00505690773f

But if I insert it into a binary(16) field (the UUID() function) with for instance a BEFORE INSERT trigger and run a select, it returns something like:

    0782ef48-a439-11

Note that these two UUIDs are not the same data.

I realize binary and an UUID string doesn&#39;t look identical, but shouldn&#39;t the selected data at least be just as long? Otherwise how can it possibly be equally likely to be unique?

Is it better to store it as char(36)? I just need it to be unique to prevent duplicate inserts. It is never selected or used for joins.

EDIT:

before trigger would be like:

    BEGIN
    
    if NEW.UUID IS NULL THEN
    
    NEW.UUID = UUID();
    
    END IF
    
    END

Inserting and selecting UUIDs as binary(16)

&lt;p&gt;I am new in python programming and i am a bit confused. I try to get the bytes from a string to hash and encrypt but i got&lt;/p&gt; 

    b&#39;...&#39;

&lt;p&gt;b character in front of string just like the below example. Is any way avoid this?.Can anyone give a solution? Sorry for this silly question&lt;/p&gt;

    import hashlib
    
    text = &quot;my secret data&quot;
    pw_bytes = text.encode(&#39;utf-8&#39;)
    print(&#39;print&#39;,pw_bytes)
    m = hashlib.md5()
    m.update(pw_bytes)


OUTPUT:
   

     print b&#39;my secret data&#39;


Remove &#39;b&#39; character do in front of a string literal in Python 3

The compression algorithm used in *zlib* is essentially the same as that in *gzip* and *zip*. What are *gzip* and *zip*? How are they different and how are they same?

How are zlib, gzip and zip related? What do they have in common and how are they different?

Say I have a [single-page application](http://espy.github.io/ubersicht/) that uses a third party API for content. The app’s logic is in-browser only; there is no backend I can write to.

To allow deep-linking into the state of the app, I use `pushState()` to keep track of a few variables that determine the state of the app. (Note that Ubersicht’s public version doesn’t do this yet.) 

* **Variables:** `repos`, `labels`, `milestones`, `username`, `show_open` (bool), `with_comments` (bool), and `without_comments` (bool).
* **URL format:** `?label=label_1,label_2,label_3&amp;repos=repo_1…`. 
* **Values:** the usual suspects. Roughly, `[a-zA-Z][a-zA-Z0-9_-]`, or any boolean indicator.

So far so good. 

Now, since the query string can be a bit long and unwieldy and I would like to be able to pass around URLs like `http://espy.github.io/ubersicht/?state=SOMOPAQUETOKENTHATLOSSLESSLYDECOMPRESSESINTOTHEORIGINALVALUES#hoodiehq`, the shorter the better.

My first attempt was going to be using some zlib-like algorithm for [this](https://github.com/imaya/zlib.js). Then @flipzagging pointed to [antirez/**smaz**](https://github.com/antirez/smaz), which looks more suitable for short strings. ([JavaScript version here]( https://github.com/personalcomputer/smaz.js).)

Since `=` and `&amp;` are not specifically handled in the Javascript version (see [line 9 of the main lib file](https://github.com/personalcomputer/smaz.js/blob/master/lib/smaz.js#L9)), we might be able to tweak things a little there.

Furthermore, there is an option for encoding the values in a fixed table. With this option, the order of arguments is pre-defined and all we need to keep track of is the actual value. Example: turn `a=hamster&amp;b=cat` into `7hamster3cat` (length+chars) or `hamster|cat` (value + `|`), potentially before the smaz compression.

Is there anything else I should be looking for?


How to compress URL parameters

On my Linux machine, I wish to create a .tar.bz2 file of a certain folder. Once I place myself in that folder (in the terminal), what do I type in the terminal command line to place the compressed folder in the home directory of my machine?

Let&#39;s say I am in the folder /home/user/folder. In the folder &quot;folder&quot; are several files (txt, .c etc). How do I compress that folder of type .tar.bz2 and place it in my /home directory?

In the /home/user/folder, I&#39;ve tried `sudo tar -cvjSf folder.tar.bz2` but get an error:

&gt; tar: Cowardly refusing to create an empty archive

Create a .tar.bz2 file Linux

I understand that GZIP is a combination of LZ77 and Huffman coding and can be configured with a level between 1-9 where 1 indicates  the  fastest  compression (less  compression) and 9 indicates the slowest compression method (best  compression).

My question is, does the choice of level **only** impact the compression process or is there an additional cost also incurred in decompression depending on the level used to compress?

I ask because typically many web servers will GZIP responses on the fly if the client supports it, e.g. `Accept-Encoding: gzip`. I appreciate that when doing this on the fly a level such as 6 might be the good choice for the average case, since it gives a good balance between speed and compression. 

However, if I have a bunch of static assets that I can GZIP just once ahead of time - and never need to do this again - would there be any downside to using the highest but slowest compression level? I.e. is there now an additional overhead for the client that would not have been incurred had a lower compression level been used.


Does GZIP Compression Level Have Any Impact On Decompression

I was recently compressing some files, and I noticed that base64-encoded data seems to compress really bad. Here is one example:

   * Original file: **429,7 MiB**
   * compress via `xz -9`:&lt;br/&gt;
  `13,2 MiB / 429,7 MiB = 0,031` `4,9 MiB/s` `1:28`
   * `base64` it and compress via `xz -9`:&lt;br/&gt;
  `26,7 MiB / 580,4 MiB = 0,046` `2,6 MiB/s` `3:47`
   * `base64` the original compressed xz file:&lt;br/&gt;
  `17,8 MiB` in almost no time = the expected `1.33x` increase in size

So what can be observed is that:

   * xz compresses really good ☺
   * base64-encoded data doesn&#39;t compress well, it is 2 times larger than the unencoded compressed file
   * *base64-then-compress* is significantly worse and slower than *compress-then-base64*

How can this be? Base64 is a lossless, reversible algorithm, why does it affect compression so much? (I tried with gzip as well, with similar results).

I know it doesn&#39;t make sense to *base64-then-compress* a file, but most of the time one doesn&#39;t have control over the input files, and I would have thought that since the actual information density (or whatever it is called) of a base64-encoded file would be nearly identical to the non-encoded version, and thus be similarily compressible.

Content Type	Original Author	Original Content on Stackoverflow
Question	22332112	View Question on Stackoverflow
Solution 1 - Binary	Mark Adler	View Answer on Stackoverflow
Solution 2 - Binary	ct_	View Answer on Stackoverflow
Solution 3 - Binary	M.Ali	View Answer on Stackoverflow
Solution 4 - Binary	Ashark	View Answer on Stackoverflow

How to detect type of compression used on the file? (if no file extension is specified)

Binary Problem Overview

Binary Solutions

Solution 1 - Binary

Solution 2 - Binary

Solution 3 - Binary

Solution 4 - Binary

pep8 warning on regex string in Python, Eclipse

express logging response body

Attributions