Index is part of data frame , if the index are different , we should say the dataframes are different , even the value of dfs are same , so , if you want to check the value , using `array_equal` from `numpy`


    d1 = pd.DataFrame([[1,2], [10, 20]], index=[0,2])
    d2 = pd.DataFrame([[1, 2], [10, 20]], index=[0, 1])
    np.array_equal(d1.values,d2.values)
    Out[759]: True

For more info about `assert_frame_equal` in [git][1] 


  [1]: https://github.com/pandas-dev/pandas/issues/19411

If you really don&#39;t care about the index being equal, you can drop the index as follows:

    assert_frame_equal(d1.reset_index(drop=True), d2.reset_index(drop=True))

Since the new update (2018.2) IntelliJ IDEA underlines variables, when they are &quot;unnecessarily&quot; reassigned - this includes, however, each use of &quot;+=&quot;.&lt;br&gt;
Is this a mistake in the IDE or am I getting it wrong?&lt;br&gt;
Please see this basic method as an example. (It just adds a list of numbers.)&lt;br&gt;

[![example image][1]][1]


  [1]: https://i.stack.imgur.com/bxLKB.png

IntelliJ IDEA underlines variables when using += in JAVA

I&#39;m attempting to find an object in an array using `Array.prototype.includes`. Is this possible? I realize there is a difference between shallow and deep comparison. Is that the reason the below code returns false? I could not find a relevant answer for `Array.includes()`.

[![dev tools console][1]][1]

  [1]: https://i.stack.imgur.com/7pkYo.png

Array.includes() to find object in array

I try to compare below two dataframe with &quot;check_index_type&quot; set to False. According to the documentation, if it set to False, it shouldn&#39;t &quot;check the Index class, dtype and inferred_type are identical&quot;. Did I misunderstood the documentation? how to compare ignoring the index and return True for below test?

I know I can reset the index but prefer not to.

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.testing.assert_frame_equal.html




    from pandas.util.testing import assert_frame_equal
    import pandas as pd
    d1 = pd.DataFrame([[1,2], [10, 20]], index=[0,2])
    d2 = pd.DataFrame([[1, 2], [10, 20]], index=[0, 1])
    assert_frame_equal(d1, d2, check_index_type=False)


    AssertionError: DataFrame.index are different
    DataFrame.index values are different (50.0 %)
    [left]:  Int64Index([0, 2], dtype=&#39;int64&#39;)
    [right]: Int64Index([0, 1], dtype=&#39;int64&#39;)

how to ignore index comparison for pandas assert frame equal

I try to compare below two dataframe with "check_index_type" set to False. According to the documentation, if it set to False, it shouldn't "check the Index class, dtype and inferred_type are identical". Did I misunderstood the documentation? how to compare ignoring the index and return True for below test?
I know I can reset the index but prefer not to.
<a href="https://pandas.pydata.org/pandas-docs/stable/generated/pandas.testing.assert_frame_equal.html" target="_blank" rel="noopener noreferrer">https://pandas.pydata.org/pandas-docs/stable/generated/pandas.testing.assert_frame_equal.html</a>
<pre><code class="hljs language-ini">from pandas.util.testing import assert_frame_equal
import pandas as pd
d1 = pd.DataFrame([[1,2], [10, 20]], index=[0,2])
d2 = pd.DataFrame([[1, 2], [10, 20]], index=[0, 1])
assert_frame_equal(d1, d2, check_index_type=False)


AssertionError: DataFrame.index are different
DataFrame.index values are different (50.0 %)
[left]: Int64Index([0, 2], dtype='int64')
[right]: Int64Index([0, 1], dtype='int64')
</code></pre>

I have some geometrically distributed data. When I want to take a look at it, I use

    sns.distplot(data, kde=False, norm_hist=True, bins=100)

which results is a picture:

[![Plot 1a][1]][1]

However, bins heights don&#39;t add up to 1, which means y axis doesn&#39;t show probability, it&#39;s something different. If instead we use

    weights = np.ones_like(np.array(data))/float(len(np.array(data)))
    plt.hist(data, weights=weights, bins = 100)

the y axis shall show probability, as bins heights sum up to 1:

[![Plot 1b][2]][2]

It can be seen more clearly here: suppose we have a list

    l = [1, 3, 2, 1, 3]

We have two 1s, two 3s and one 2, so their respective probabilities are 2/5, 2/5 and 1/5. When we use seaborn histplot with 3 bins:

    sns.distplot(l, kde=False, norm_hist=True, bins=3)

we get:

[![Plot 2a][3]][3]

As you can see, the 1st and the 3rd bin sum up to 0.6+0.6=1.2 which is already greater than 1, so y axis is not a probability. When we use 

    weights = np.ones_like(np.array(l))/float(len(np.array(l)))
    plt.hist(l, weights=weights, bins = 3)

we get:

[![enter image description here][4]][4]

and the y axis is probability, as 0.4+0.4+0.2=1 as expected.

The amount of bins in these 2 cases are is the same for both methods used in each case: 100 bins for geometrically distributed data, 3 bins for small array l with 3 possible values. So bins amount is not the issue.

**My question is:** in seaborn distplot called with norm_hist=True, what is the meaning of y axis?


  [1]: https://i.stack.imgur.com/jE09g.png
  [2]: https://i.stack.imgur.com/6yo35.png
  [3]: https://i.stack.imgur.com/AuzZ1.png
  [4]: https://i.stack.imgur.com/FHpvu.png

What is y axis in seaborn distplot?

Long story short
----------------

[PEP-557][1] introduced data classes into Python standard library, that basically can fill the same role as `collections.namedtuple` and `typing.NamedTuple`. And now I&#39;m wondering how to separate the use cases in which namedtuple is still a better solution.

Data classes advantages over NamedTuple
----------------------------------------

Of course, all the credit goes to `dataclass` if we need:

 - mutable objects
 - inheritance support
 - `property` decorators, manageable attributes
 - generated method definitions out of the box or customizable method definitions

Data classes advantages are briefly explained in the same PEP: [Why not just use namedtuple][2].

Q: In which cases namedtuple is still a better choice?
---------------------------------------------------

But how about an opposite question for namedtuples: why not just use dataclass?
I guess probably namedtuple is better from the performance standpoint but found no confirmation on that yet.

Example
-------

Let&#39;s consider the following situation:

We are going to store pages dimensions in a small container with statically defined fields, type hinting and named access. No further hashing, comparing and so on are needed.

**NamedTuple approach:**

    from typing import NamedTuple
    
    PageDimensions = NamedTuple(&quot;PageDimensions&quot;, [(&#39;width&#39;, int), (&#39;height&#39;, int)])

**DataClass approach:**

    from dataclasses import dataclass
    
    @dataclass
    class PageDimensions:
        width: int
        height: int

Which solution is preferable and why?

P.S. the question isn&#39;t a duplicate of [that one][3] in any way, because here I&#39;m asking about **the cases** in which namedtuple is better, not about **the difference** (I&#39;ve checked docs and sources before asking)


  [1]: https://www.python.org/dev/peps/pep-0557/
  [2]: https://www.python.org/dev/peps/pep-0557/#why-not-just-use-namedtuple
  [3]: https://stackoverflow.com/questions/47955263

Data Classes vs typing.NamedTuple primary use cases

I am a little bit confused about the data augmentation performed in PyTorch. Now, as far as I know, when we are performing data augmentation, we are KEEPING our original dataset, and then adding other versions of it (Flipping, Cropping...etc). But that doesn&#39;t seem like happening in PyTorch. As far as I understood from the references, when we use `data.transforms` in PyTorch, then it applies them one by one. So for example:

&lt;!-- language: lang-python --&gt;

    data_transforms = {
        &#39;train&#39;: transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),
        &#39;val&#39;: transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),
    }

Here , for the training, we are first randomly cropping the image and resizing it to shape `(224,224)`. Then we are taking these `(224,224)` images and horizontally flipping them. Therefore, our dataset is now containing ONLY the horizontally flipped images, so our original images are lost in this case. 

Am I right? Is this understanding correct? If not, then where do we tell PyTorch in this code above (taken from Official Documentation) to keep the original images and resize them to the expected shape `(224,224)`?

Thanks 
 

Data Augmentation in PyTorch

I am trying to run this code:

    import web
    
    urls = (
    	&#39;/&#39;, &#39;index&#39;
    )
    
    if __name__ == &quot;__main__&quot;:
        app = web.application(urls, globals())
        app.run()
But it gives me this error everytime

    C:\Users\aidke\Desktop&gt;python app.py
    Traceback (most recent call last):
      File &quot;C:\Users\aidke\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web\utils.py&quot;, line 526, in take
        yield next(seq)
    StopIteration
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File &quot;app.py&quot;, line 14, in &lt;module&gt;
        app = web.application(urls, globals())
      File &quot;C:\Users\aidke\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web\application.py&quot;, line 62, in __init__
        self.init_mapping(mapping)
      File &quot;C:\Users\aidke\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web\application.py&quot;, line 130, in init_mapping
        self.mapping = list(utils.group(mapping, 2))
      File &quot;C:\Users\aidke\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web\utils.py&quot;, line 531, in group
        x = list(take(seq, size))
    RuntimeError: generator raised StopIteration

I tried someone else&#39;s code and the exact same thing happened.  Additionally I tried reinstalling web.py(experimental) but it still didn&#39;t work.

&quot;RuntimeError: generator raised StopIteration&quot; every time I try to run app

I&#39;m trying to understand how to use the `Optional` type hint. From [PEP-484][1], I know I can use `Optional` for `def test(a: int = None)` either as `def test(a: Union[int, None])` or `def test(a: Optional[int])`. 

But how about following examples?

    def test(a : dict = None):
        #print(a) ==&gt; {&#39;a&#39;: 1234}
        #or
        #print(a) ==&gt; None
        
    def test(a : list = None):
        #print(a) ==&gt; [1,2,3,4, &#39;a&#39;, &#39;b&#39;]
        #or
        #print(a) ==&gt; None

If `Optional[type]` seems to mean the same thing as `Union[type, None]`, why should I use `Optional[]` at all?


  [1]: https://www.python.org/dev/peps/pep-0484/

How should I use the Optional type hint?

I have a function, which returns a dictionary like this:

    {&#39;truth&#39;: 185.179993, &#39;day1&#39;: 197.22307753038834, &#39;day2&#39;: 197.26118010160317, &#39;day3&#39;: 197.19846975345905, &#39;day4&#39;: 197.1490578795196, &#39;day5&#39;: 197.37179265011116}

I am trying to append this dictionary to a dataframe like so:

    output = pd.DataFrame()
    output.append(dictionary, ignore_index=True)
    print(output.head())

Unfortunately, the printing of the dataframe results in an empty dataframe. Any ideas?

append dictionary to data frame

Str.replace method returns an attribute error. 

    dc_listings[&#39;price&#39;].str.replace(&#39;,&#39;, &#39;&#39;)
    AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas


Here are the top 5 rows of my price column. 

[![enter image description here][1]][1]



This stack overflow [thread recommends][2] to check if my column has NAN values but non of the values in my column are NAN. 
[![enter image description here][3]][3] 


  [1]: https://i.stack.imgur.com/DoXiN.png
  [2]: https://stackoverflow.com/questions/33692532/pandas-error-can-only-use-str-accessor-with-string-values
  [3]: https://i.stack.imgur.com/gYOaQ.png

AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas

Pandas has both `isna()` and `isnull()`. I usually use `isnull()` to detect missing values and have never met the case so that I had to use other than that.
So, when to use `isna()`?

Pandas isna() and isnull(), what is the difference?

I have a pandas series with boolean entries.  I would like to get a list of indices where the values are `True`.

For example the input `pd.Series([True, False, True, True, False, False, False, True])`

should yield the output `[0,2,3,7]`.

I can do it with a list comprehension, but is there something cleaner or faster?

Getting a list of indices where pandas boolean series is True

    df = df[~df[&quot;column&quot;].str.contains(&quot;Total&quot;)]

    TypeError: bad operand type for unary ~: &#39;float&#39;

Why does `.str.contains()` return a float? What should I be doing here?

Content Type	Original Author	Original Content on Stackoverflow
Question	Lisa	View Question on Stackoverflow
Solution 1 - Python	BENY	View Answer on Stackoverflow
Solution 2 - Python	The Aelfinn	View Answer on Stackoverflow

how to ignore index comparison for pandas assert frame equal

Python Problem Overview

Python Solutions

Solution 1 - Python

Solution 2 - Python

Array.includes() to find object in array

IntelliJ IDEA underlines variables when using += in JAVA

Attributions