Surprisingly, I find `startswith` is slower than `in`:

    In [10]: s=&quot;ABCD&quot;*10

    In [11]: %timeit s.startswith(&quot;XYZ&quot;)
    1000000 loops, best of 3: 307 ns per loop
    
    In [12]: %timeit &quot;XYZ&quot; in s
    10000000 loops, best of 3: 81.7 ns per loop

As we all know, the `in` operation needs to search the whole string and `startswith` just needs to check the first few characters, so `startswith` should be more efficient.

When `s` is big enough, `startswith` is faster:

    In [13]: s=&quot;ABCD&quot;*200
    
    In [14]: %timeit s.startswith(&quot;XYZ&quot;)
    1000000 loops, best of 3: 306 ns per loop
    
    In [15]: %timeit &quot;XYZ&quot; in s
    1000000 loops, best of 3: 666 ns per loop

So it seems that calling `startswith` has some overhead which makes it slower when the string is small.

And than I tried to figure out what&#39;s the overhead of the `startswith` call.
    
First, I used an `f` variable to reduce the cost of the dot operation - as mentioned in this [answer][1] - here we can see `startswith` is still slower:

    In [16]: f=s.startswith
    
    In [17]: %timeit f(&quot;XYZ&quot;)
    1000000 loops, best of 3: 270 ns per loop

Further, I tested the cost of an empty function call:

    In [18]: def func(a): pass
    
    In [19]: %timeit func(&quot;XYZ&quot;)
    10000000 loops, best of 3: 106 ns per loop

Regardless of the cost of the dot operation and function call, the time of `startswith` is about (270-106)=164ns, but the `in` operation takes only 81.7ns. It seems there are still some overheads for `startswith`, what&#39;s that?

Add the test result between `startswith` and `__contains__` as suggested by poke and lvc:

    In [28]: %timeit s.startswith(&quot;XYZ&quot;)
    1000000 loops, best of 3: 314 ns per loop
    
    In [29]: %timeit s.__contains__(&quot;XYZ&quot;)
    1000000 loops, best of 3: 192 ns per loop


  [1]: https://stackoverflow.com/a/13271125/552942

Why is string&#39;s startswith slower than in?

When comparing floats to integers, some pairs of values take much longer to be evaluated than other values of a similar magnitude.

For example:

    &gt;&gt;&gt; import timeit
    &gt;&gt;&gt; timeit.timeit(&quot;562949953420000.7 &lt; 562949953421000&quot;) # run 1 million times
    0.5387085462592742

But if the float or integer is made smaller or larger by a certain amount, the comparison runs much more quickly:

    &gt;&gt;&gt; timeit.timeit(&quot;562949953420000.7 &lt; 562949953422000&quot;) # integer increased by 1000
    0.1481498428446173
    &gt;&gt;&gt; timeit.timeit(&quot;562949953423001.8 &lt; 562949953421000&quot;) # float increased by 3001.1
    0.1459577925548956

Changing the comparison operator (e.g. using `==` or `&gt;` instead) does not affect the times in any noticeable way. 

This is not *solely* related to magnitude because picking larger or smaller values can result in faster comparisons, so I suspect it is down to some unfortunate way the bits line up. 

Clearly, comparing these values is more than fast enough for most use cases. I am simply curious as to why Python seems to struggle more with some pairs of values than with others.

Why are some float &lt; integer comparisons four times slower than others?

Everything in Python is an object. So the size of an int in Python will be larger than usual.

    &gt;&gt;&gt; sys.getsizeof(int())
    24

OK, but why does it take 12 more bytes for `2⁶&#179;` compared too `2⁶&#179; - 1` and not just one?

    &gt;&gt;&gt; sys.getsizeof(2**63)
    36
    &gt;&gt;&gt; sys.getsizeof(2**62)
    24

I get that `2⁶&#179;` is a long and `2⁶&#179;-1` an int, but why 12 bytes of difference?

No more intuitive, I tried some other things:

    &gt;&gt;&gt; a = 2**63
    &gt;&gt;&gt; a -= 2**62
    &gt;&gt;&gt; sys.getsizeof(a)
    36

`a` is still stored as a long even if it could be in an int now. So that&#39;s not surprising. But:

    &gt;&gt;&gt; a -= (2**63 - 1)
    &gt;&gt;&gt; a = 2**63
    &gt;&gt;&gt; a -= (2**63 - 1)
    &gt;&gt;&gt; a
    1L
    &gt;&gt;&gt; sys.getsizeof(a)
    28

A new size.

    &gt;&gt;&gt; a = 2**63
    &gt;&gt;&gt; a -= 2**63
    &gt;&gt;&gt; a
    0L
    &gt;&gt;&gt; sys.getsizeof(a)
    24

Back to 24 bytes, but still with a long.

Last thing I got:

    &gt;&gt;&gt; sys.getsizeof(long())
    24

**Question:**

How does the memory storage work in those scenarios?

**Sub-questions:**

Why is there a gap of 12 bytes to add what our intuition tells us is just 1 bit?

Why are `int()` and `long()` 24 bytes, but `long(1)` is already 28 bytes and `int(2⁶&#178;)`?

*NB: Python 3.X is working a bit differently, but not more intuitively. Here I focused on Python 2.7; I did not test on prior versions.*


Why is the size of 2⁶&#179; 36 bytes, but 2⁶&#179;-1 is only 24 bytes?

I&#39;ve seen people say that `set` objects in python have O(1) membership-checking. How are they implemented internally to allow this? What sort of data structure does it use? What other implications does that implementation have?


Every answer here was really enlightening, but I can only accept one, so I&#39;ll go with the closest answer to my original question. Thanks all for the info!

How is set() implemented?

    def main():
	    for i in xrange(10**8):
	    	pass
    main()

This piece of code in Python runs in  (Note: The timing is done with the time function in BASH in Linux.)

    real	0m1.841s
    user	0m1.828s
    sys 	0m0.012s

However, if the for loop isn&#39;t placed within a function, 

    for i in xrange(10**8):
    	pass

then it runs for a much longer time:

    real	0m4.543s
    user	0m4.524s
    sys 	0m0.012s

Why is this?

Why does Python code run faster in a function?

I&#39;ve been hearing a lot about the [PyPy][1] project. They claim it is 6.3 times faster than the [CPython][2] interpreter on [their site][3].

Whenever we talk about dynamic languages like Python, speed is one of the top issues. To solve this, they say PyPy is 6.3 times faster.

The second issue is parallelism, the infamous [Global Interpreter Lock][4] (GIL). For this, PyPy says it [can give GIL-less Python][5].

If PyPy can solve these great challenges, what are its weaknesses that are preventing wider adoption? That is to say, what&#39;s preventing someone like me, a typical Python developer, from switching to PyPy *right now*? 

  [1]: http://en.wikipedia.org/wiki/PyPy
  [2]: http://en.wikipedia.org/wiki/CPython
  [3]: http://speed.pypy.org
  [4]: http://en.wikipedia.org/wiki/Global_Interpreter_Lock
  [5]: http://pypy.org/tmdonate.html


Why shouldn&#39;t I use PyPy over CPython if PyPy is 6.3 times faster?

Apparently `list(a)` doesn&#39;t overallocate, `[x for x in a]` overallocates at some points, and `[*a]` overallocates *all the time*?

[![Sizes up to n=100][1]][1]

Here are sizes n from 0 to 12 and the resulting sizes in bytes for the three methods:
```
0 56 56 56
1 64 88 88
2 72 88 96
3 80 88 104
4 88 88 112
5 96 120 120
6 104 120 128
7 112 120 136
8 120 120 152
9 128 184 184
10 136 184 192
11 144 184 200
12 152 184 208
```
Computed like this, [reproducable at repl.it](https://repl.it/repls/CoordinatedUnsungShockwave), using Python 3.**8**:
```
from sys import getsizeof

for n in range(13):
    a = [None] * n
    print(n, getsizeof(list(a)),
             getsizeof([x for x in a]),
             getsizeof([*a]))
```
**So: How does this work?** How does `[*a]` overallocate? Actually, what mechanism does it use to create the result list from the given input? Does it use an iterator over `a` and use something like `list.append`? Where is the source code?

([Colab with data and code](https://colab.research.google.com/drive/1LOy2rb63wHZ2TSFrnoSlYXz1oOa1VkZ6) that produced the images.)

Zooming in to smaller n:

[![Sizes up to n=40][2]][2]

Zooming out to larger n:

[![Sizes up to n=1000][3]][3]

  [1]: https://i.stack.imgur.com/SOZoe.png
  [2]: https://i.stack.imgur.com/7F3Ba.png
  [3]: https://i.stack.imgur.com/xwhob.png

What causes [*a] to overallocate?

I was playing around with timeit and noticed that doing a simple list comprehension over a small string took longer than doing the same operation on a list of small single character strings. Any explanation? It&#39;s almost 1.35 times as much time.

    &gt;&gt;&gt; from timeit import timeit
    &gt;&gt;&gt; timeit(&quot;[x for x in &#39;abc&#39;]&quot;)
    2.0691067844831528
    &gt;&gt;&gt; timeit(&quot;[x for x in [&#39;a&#39;, &#39;b&#39;, &#39;c&#39;]]&quot;)
    1.5286479570345861

What&#39;s happening on a lower level that&#39;s causing this?

Why is it slower to iterate over a small string than a small list?

&gt; **Possible Duplicate:**  
&gt; [Python &amp;ldquo;is&amp;rdquo; operator behaves unexpectedly with integers](https://stackoverflow.com/questions/306313/python-is-operator-behaves-unexpectedly-with-integers)  

&lt;!-- End of automatically inserted text --&gt;

Today I tried to debug my project and after a few hours of analysing I&#39;d got this:

    &gt;&gt;&gt; (0-6) is -6
    False

but,

    &gt;&gt;&gt; (0-5) is -5
    True

Could you explain to me, why?
Maybe this is some kind of bug or very strange behavior.

    &gt; Python 2.7.3 (default, Apr 24 2012, 00:00:54) [GCC 4.7.0 20120414 (prerelease)] on linux2
    &gt;&gt;&gt; type(0-6) 
    &lt;type &#39;int&#39;&gt;
    &gt;&gt;&gt; type(-6) 
    &lt;type &#39;int&#39;&gt;
    &gt;&gt;&gt; type((0-6) is -6)
    &lt;type &#39;bool&#39;&gt;
    &gt;&gt;&gt; 



Why (0-6) is -6 = False?

What&#39;s all this fuss about Python and CPython *(Jython,IronPython)*, I don&#39;t get it:

[python.org][1] mentions that CPython is:

&gt; *The &quot;traditional&quot; implementation of Python (nicknamed CPython)*

[yet another Stack Overflow question][2] mentions that:

&gt; *CPython is the default byte-code interpreter of Python, which is written in C.*

Honestly I don&#39;t get what both of those explanations practically mean but what I thought was that, *if I use CPython does that mean when I run a sample python code, it compiles it to C language and then executes it as if it were C code*

**So what exactly is CPython and how does it differ when compared with python and should I probably use CPython over Python and if so what are its advantages?**


  [1]: http://www.python.org/
  [2]: https://stackoverflow.com/a/2324217/2425215

Python vs Cpython

It is standard convention to use `if foo is None` rather than `if foo == None` to test if a value is specifically `None`.

If you want to determine whether a value is exactly `True` (not just a true-like value), is there any reason to use `if foo == True` rather than `if foo is True`? Does this vary between implementations such as CPython (2.x and 3.x), Jython, PyPy, etc.?

Example: say `True` is used as a singleton value that you want to differentiate from the value `&#39;bar&#39;`, or any other true-like value:

    if foo is True: # vs foo == True
        ...
    elif foo == &#39;bar&#39;:
        ...

Is there a case where using `if foo is True` would yield different results from `if foo == True`?


NOTE: I am aware of [Python booleans - if x:, vs if x == True, vs if x is True][1]. However, it only addresses whether `if foo`, `if foo == True`, or `if foo is True` should generally be used to determine whether `foo` has a true-like value.



---

UPDATE: According to [PEP 285][2] &#167; Specification:

&gt; The values False and True will be singletons, like None.


  [1]: https://stackoverflow.com/q/20420934/369450
  [2]: https://www.python.org/dev/peps/pep-0285

Boolean identity == True vs is True

Python `int`s are **objects** that encapsulate the actual number value. Can we mess with that value, for example setting the value of the object `1` to 2? So that `1 == 2` becomes `True`?


Can we make 1 == 2 true?

[Given Zero Piraeus&#39; answer to another question](https://stackoverflow.com/a/26116307/1763356), we have that

    x = tuple(set([1, &quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;z&quot;, &quot;f&quot;]))
    y = tuple(set([&quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;z&quot;, &quot;f&quot;, 1]))
    print(x == y)

Prints `True` about 85% of the time with [hash randomization](http://lemire.me/blog/archives/2012/01/17/use-random-hashing-if-you-care-about-security/) enabled. Why 85%?

Why does tuple(set([1,&quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;z&quot;,&quot;f&quot;])) == tuple(set([&quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;z&quot;,&quot;f&quot;,1])) 85% of the time with hash randomization enabled?

I want to access some .NET assemblies written in C# from Python code. 

A little research showed I have two choices:

 - [IronPython][1] with .NET interface capability/support built-in
 - Python with the [Python .NET][2] package

What are the trade-offs between both solutions?

  [1]: http://www.codeplex.com/IronPython
  [2]: https://github.com/pythonnet/pythonnet

IronPython vs. Python .NET

I recently installed `lxml`.

Before that, I had to install all the dependencies for that.
So I tried to install `liblxml2-dev`, `liblxslt1-dev` and `python-dev`
(google-searched for what packages are required for `lxml`)
but even after that, I could not able to install `lxml` by using the command
`pip install lxml`.

However as because I am using Python 3.4.0, I thought that may be there are different version of `python-dev` (thought came due to facing some similar version conflict problem). So I tried to install `python3-dev`.

Then I tried to install `lxml` using the same command said earlier and that worked!!!

I google-searched for those sentences
and I could able to understand the meaning of `lxml` and `xslt`. I found a [link][1] for `python-dev` but I am not still sure what it is actually doing.

OS - Ubuntu

  [1]: https://packages.debian.org/sid/python-dev

What is python-dev package used for

So i was playing with `list` objects and found little strange thing that if `list` is created with `list()` it uses more memory, than list comprehension? I&#39;m using Python 3.5.2

    In [1]: import sys
    In [2]: a = list(range(100))
    In [3]: sys.getsizeof(a)
    Out[3]: 1008
    In [4]: b = [i for i in range(100)]
    In [5]: sys.getsizeof(b)
    Out[5]: 912
    In [6]: type(a) == type(b)
    Out[6]: True
    In [7]: a == b
    Out[7]: True
    In [8]: sys.getsizeof(list(b))
    Out[8]: 1008

From the [docs](https://docs.python.org/3.5/library/stdtypes.html#list):

&gt; Lists may be constructed in several ways:
&gt;
&gt; - Using a pair of square brackets to denote the empty list: `[]`
&gt; - Using square brackets, separating items with commas: `[a]`, `[a, b, c]`
&gt; - Using a list comprehension: `[x for x in iterable]`
&gt; - Using the type constructor: `list()` or `list(iterable)`


But it seems that using `list()` it uses more memory.

And as much `list` is bigger, the gap increases.

[![Difference in memory][1]][1]

Why this happens?

**UPDATE #1**

Test with Python 3.6.0b2:

    Python 3.6.0b2 (default, Oct 11 2016, 11:52:53) 
    [GCC 5.4.0 20160609] on linux
    Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
    &gt;&gt;&gt; import sys
    &gt;&gt;&gt; sys.getsizeof(list(range(100)))
    1008
    &gt;&gt;&gt; sys.getsizeof([i for i in range(100)])
    912

**UPDATE #2**

Test with Python 2.7.12:

    Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
    [GCC 5.4.0 20160609] on linux2
    Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
    &gt;&gt;&gt; import sys
    &gt;&gt;&gt; sys.getsizeof(list(xrange(100)))
    1016
    &gt;&gt;&gt; sys.getsizeof([i for i in xrange(100)])
    920

  [1]: https://i.stack.imgur.com/VVHJL.png

list() uses slightly more memory than list comprehension

I have encountered this weird behavior and failed to explain it. These are the benchmarks:

    py -3 -m timeit &quot;tuple(range(2000)) == tuple(range(2000))&quot;
    10000 loops, best of 3: 97.7 usec per loop
    py -3 -m timeit &quot;a = tuple(range(2000));  b = tuple(range(2000)); a==b&quot;
    10000 loops, best of 3: 70.7 usec per loop

How come comparison with variable assignment is faster than using a one liner with temporary variables by more than 27%?

By the Python docs, garbage collection is disabled during timeit so it can&#39;t be that. Is it some sort of an optimization?

The results may also be reproduced in Python 2.x though to lesser extent.

Running Windows 7/10, CPython 3.5.1 all the way to 3.10.1, Intel i7 3.40 GHz, 64 bit both OS and Python. Seems like a different machine I&#39;ve tried running at Intel i7 3.60 GHz with Python 3.5.0 does not reproduce the results.

----------

Running using the same Python process with `timeit.timeit()` @ 10000 loops produced 0.703 and 0.804 respectively. Still shows although to lesser extent. (~12.5%)

Why is code using intermediate variables faster than code without?

Experimenting with magic methods (`__sizeof__` in particular) on different Python objects I stumbled over the following behaviour:

Python 2.7

    &gt;&gt;&gt; False.__sizeof__()
    24
    &gt;&gt;&gt; True.__sizeof__()
    24

Python 3.x

    &gt;&gt;&gt; False.__sizeof__()
    24
    &gt;&gt;&gt; True.__sizeof__()
    28

What changed in Python 3 that makes the size of `True` greater than the size of `False`?


Different object size of True and False in Python 3

Python has a few different implementations: CPython, Jython, PyPy, etc.  I want to programmatically determine which implementation my code is running on.  How can I do that?

To be specific, write a function called ``get_implementation_name()`` for me:

    impl_name = get_implementation_name()
    if impl_name == &quot;CPython&quot;:
      print &quot;I can abuse CPython implementation details. (I&#39;m a bad, bad man.)&quot;
    elif impl_name == &quot;PyPy&quot;:
      print &quot;Can&#39;t count on reference-counting garbage collection here...&quot;
    else:
      print &quot;I better be careful...&quot;


How can I tell which python implementation I&#39;m using?

Can I extend syntax in python for dict comprehensions for other dicts, like the OrderedDict in `collections` module or my own types which inherit from `dict`?  

Just rebinding the `dict` name obviously doesn&#39;t work, the `{key: value}` comprehension syntax still gives you a plain old dict for comprehensions and literals.  

    &gt;&gt;&gt; from collections import OrderedDict
    &gt;&gt;&gt; olddict, dict = dict, OrderedDict
    &gt;&gt;&gt; {i: i*i for i in range(3)}.__class__
    &lt;type &#39;dict&#39;&gt;

So, if it&#39;s possible how would I go about doing that?  It&#39;s OK if it only works in CPython.  For syntax I guess I would try it with a `O{k: v}` prefix like we have on the `r&#39;various&#39; u&#39;string&#39; b&#39;objects&#39;`.

*note:*  Of course we can use a generator expression instead, but I&#39;m more interested seeing how hackable python is in terms of the grammar.

OrderedDict comprehensions

I am trying to understand how Python works (because I use it all the time!). To my understanding, when you run something like python script.py, the script is converted to bytecode and then the interpreter/VM/CPython–really just a C Program–reads in the python bytecode and executes the program accordingly.

How is this bytecode read in? Is it similar to how a text file is read in C? I am unsure how the Python code is converted to machine code. Is it the case that the Python interpreter (the python command in the CLI) is really just a precompiled C program that is already converted to machine code and then the python bytecode files are just put through that program? In other words, is my Python program never actually converted into machine code? Is the python interpreter already in machine code, so my script never has to be?

All Cpython Solutions on Gang of Coders

Total of 21 Cpython Solutions

Why is string's startswith slower than in?

Why are some float < integer comparisons four times slower than others?

Why is the size of 2⁶³ 36 bytes, but 2⁶³-1 is only 24 bytes?