This seems rather obvious, but I can&#39;t seem to figure out how to convert an index of data frame to a column?

For example:
    
    df=
            gi       ptt_loc
     0  384444683      593  
     1  384444684      594 
     2  384444686      596  

To,

    df=
        index1    gi       ptt_loc
     0  0     384444683      593  
     1  1     384444684      594 
     2  2     384444686      596  



How to convert index of a pandas dataframe into a column

I have two Series `s1` and `s2` with the same (non-consecutive) indices. How do I combine `s1` and `s2` to being two columns in a DataFrame and keep one of the indices as a third column?


Combining two Series into a DataFrame in pandas

I use `pandas.to_datetime` to parse the dates in my data. Pandas by default represents the dates with `datetime64[ns]` even though the dates are all daily only.
I wonder whether there is an elegant/clever way to convert the dates to `datetime.date` or `datetime64[D]` so that, when I write the data to CSV, the dates are not appended with `00:00:00`. I know I can convert the type manually element-by-element:

    [dt.to_datetime().date() for dt in df.dates]

But this is really slow since I have many rows and it sort of defeats the purpose of using `pandas.to_datetime`. Is there a way to convert the `dtype` of the entire column at once? Or alternatively, does `pandas.to_datetime` support a precision specification so that I can get rid of the time part while working with daily data?

Keep only date part when using pandas.to_datetime

I have a DataFrame, and I want to replace the values in a particular column that exceed a value with zero. I had thought this was a way of achieving this:

    df[df.my_channel &gt; 20000].my_channel = 0

If I copy the channel into a new data frame it&#39;s simple:

    df2 = df.my_channel 

    df2[df2 &gt; 20000] = 0

This does exactly what I want, but seems not to work with the channel as part of the original DataFrame.

Conditional Replace Pandas

I tried:

    x=pandas.DataFrame(...)
    s = x.take([0], axis=1)

And `s` gets a DataFrame, not a Series.

How to get the first column of a pandas DataFrame as a Series?

Why does Pandas tell me that I have objects, although every item in the selected column is a string — even after explicit conversion.

This is my DataFrame:

    &lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;
    Int64Index: 56992 entries, 0 to 56991
    Data columns (total 7 columns):
    id            56992  non-null values
    attr1         56992  non-null values
    attr2         56992  non-null values
    attr3         56992  non-null values
    attr4         56992  non-null values
    attr5         56992  non-null values
    attr6         56992  non-null values
    dtypes: int64(2), object(5)

Five of them are `dtype object`. I explicitly convert those objects to strings:

    for c in df.columns:
        if df[c].dtype == object:
            print &quot;convert &quot;, df[c].name, &quot; to string&quot;
            df[c] = df[c].astype(str)

Then, `df[&quot;attr2&quot;]` still has `dtype object`, although `type(df[&quot;attr2&quot;].ix[0]` reveals `str`, which is correct.

Pandas distinguishes between `int64` and `float64` and `object`. What is the logic behind it when there is no `dtype str`? Why is a `str` covered by `object`?



Strings in a DataFrame, but dtype is object

I&#39;m somewhat new to pandas. I have a pandas data frame that is 1 row by 23 columns.

I want to convert this into a series? I&#39;m wondering what the most pythonic way to do this is?

I&#39;ve tried `pd.Series(myResults)` but it complains `ValueError: cannot copy sequence with size 23 to array axis with dimension 1`. It&#39;s not smart enough to realize it&#39;s still a &quot;vector&quot; in math terms. 

Thanks!

Convert pandas data frame to series

I have a Pandas series sf:

    email
    email1@email.com    [1.0, 0.0, 0.0]
    email2@email.com    [2.0, 0.0, 0.0]
    email3@email.com    [1.0, 0.0, 0.0]
    email4@email.com    [4.0, 0.0, 0.0]
    email5@email.com    [1.0, 0.0, 3.0]
    email6@email.com    [1.0, 5.0, 0.0]

And I would like to transform it to the following DataFrame:

    index | email             | list
    _____________________________________________
    0     | email1@email.com  | [1.0, 0.0, 0.0]
    1     | email2@email.com  | [2.0, 0.0, 0.0]
    2     | email3@email.com  | [1.0, 0.0, 0.0]
    3     | email4@email.com  | [4.0, 0.0, 0.0]
    4     | email5@email.com  | [1.0, 0.0, 3.0]
    5     | email6@email.com  | [1.0, 5.0, 0.0]

I found a way to do it, but I doubt it&#39;s the more efficient one:

    df1 = pd.DataFrame(data=sf.index, columns=[&#39;email&#39;])
    df2 = pd.DataFrame(data=sf.values, columns=[&#39;list&#39;])
    df = pd.merge(df1, df2, left_index=True, right_index=True)

Convert pandas Series to DataFrame

In Pandas, when I select a label that only has one entry in the index I get back a Series, but when I select an entry that has more then one entry I get back a data frame.

Why is that?  Is there a way to ensure I always get back a data frame?


    In [1]: import pandas as pd
    
    In [2]: df = pd.DataFrame(data=range(5), index=[1, 2, 3, 3, 3])
    
    In [3]: type(df.loc[3])
    Out[3]: pandas.core.frame.DataFrame
    
    In [4]: type(df.loc[1])
    Out[4]: pandas.core.series.Series

Pandas selecting by label sometimes return Series, sometimes returns DataFrame

I use Pandas &#39;ver 0.12.0&#39; with Python 2.7 and have a dataframe as below:

    df = pd.DataFrame({&#39;id&#39; : [123,512,&#39;zhub1&#39;, 12354.3, 129, 753, 295, 610],
                        &#39;colour&#39;: [&#39;black&#39;, &#39;white&#39;,&#39;white&#39;,&#39;white&#39;,
                                &#39;black&#39;, &#39;black&#39;, &#39;white&#39;, &#39;white&#39;],
                        &#39;shape&#39;: [&#39;round&#39;, &#39;triangular&#39;, &#39;triangular&#39;,&#39;triangular&#39;,&#39;square&#39;,
                                            &#39;triangular&#39;,&#39;round&#39;,&#39;triangular&#39;]
                        },  columns= [&#39;id&#39;,&#39;colour&#39;, &#39;shape&#39;])

The `id` Series consists of some integers and strings. Its `dtype` by default is `object`. I want to convert all contents of `id` to strings. I tried `astype(str)`, which produces the output below.

    df[&#39;id&#39;].astype(str)
    0    1
    1    5
    2    z
    3    1
    4    1
    5    7
    6    2
    7    6

**1)** How can I convert all elements of `id` to String? 

**2)** I will eventually use `id` for indexing for dataframes. Would having String indices in a dataframe slow things down, compared to having an integer index?

Pandas: change data type of Series to String

Suppose I have a dataframe with countries that goes as:

    cc | temp
    US | 37.0
    CA | 12.0
    US | 35.0
    AU | 20.0

I know that there is a pd.get_dummies function to convert the countries to &#39;one-hot encodings&#39;. However, I wish to convert them to indices instead such that I will get `cc_index = [1,2,1,3]` instead. 

I&#39;m assuming that there is a faster way than using the get_dummies along with a numpy where clause as shown below:

`[np.where(x) for x in df.cc.get_dummies().values]`

This is somewhat easier to do in R using &#39;factors&#39; so I&#39;m hoping pandas has something similar.


Pandas: convert categories to numbers

Is there any way to access the first element of a Series without knowing its index?

Let&#39;s say I have the following Series:

    import pandas as pd

    key=&#39;MCS096&#39;
    SUBJECTS = pd.DataFrame(
        {
            &quot;ID&quot;: pd.Series([146], index=[145]),
            &quot;study&quot;: pd.Series([&quot;MCS&quot;], index=[145]),
            &quot;center&quot;: pd.Series([&quot;Mag&quot;], index=[145]),
            &quot;initials&quot;: pd.Series([&quot;MCS096&quot;], index=[145]),
        }
    )


Print out `SUBJECTS`:

    print(SUBJECTS[SUBJECTS.initials==key][&#39;ID&#39;])
    &gt;&gt;&gt; 145    146
    &gt;&gt;&gt; Name: ID, dtype: int64

How can I get the value 146 here without using index 145?

Get first element of Series without knowing the index

Why do we use &#39;loc&#39; for pandas dataframes? it seems the following code with or without using loc both compile anr run at a simulular speed

    %timeit df_user1 = df.loc[df.user_id==&#39;5561&#39;]

    100 loops, best of 3: 11.9 ms per loop

or

    %timeit df_user1_noloc = df[df.user_id==&#39;5561&#39;]

    100 loops, best of 3: 12 ms per loop

So why use loc?

**Edit:** This has been flagged as a duplicate question. But although [pandas iloc vs ix vs loc explanation?][1] does mention that *

&gt; you can do column retrieval just by using the data frame&#39;s
&gt; __getitem__:

*

    df[&#39;time&#39;]    # equivalent to df.loc[:, &#39;time&#39;]

it does not say why we use loc, although it does explain lots of features of loc, my specific question is &#39;why not just omit loc altogether&#39;? for which i have accepted a very detailed answer below.

Also that other post the answer (which i do not think is an answer) is very hidden in the discussion and any person searching for what i was looking for would find it hard to locate the information and would be much better served by the answer provided to my question.


  [1]: https://stackoverflow.com/questions/31593201/pandas-iloc-vs-ix-vs-loc-explanation/31593712#31593712

Python: Pandas Series - Why use loc?

I try to solve this problem several times and give up. Now, when I have met him again, I decided to ask for some help.

I have this code for my Legend:

    legend:
    {
        layout: &#39;vertical&#39;,
        align: &#39;right&#39;,
        verticalAlign: &#39;top&#39;,
        x: -10,
        y: 100,
        borderWidth: 0,
        
        
        
        labelFormatter: function() 
        {
            if(this.name!=&#39;Series 1&#39;)
            {
                return this.name;
            }
            else
            {
                return &#39;Legend&#39;;
            }
        }
    }
    
If I change the return from &#39;Legend&#39; to &#39;&#39; the text is not shown but still there is a &#39;dash&#39; on the top of the legend. If I do not use label formater function  I have &#39;Series 1&#39; + &#39;dash&#39; like a first row in my legend. How to hide them?

Please, note my version is : Highcharts-2.0.5

This is a simple view of my legend and the dash I want to remove:

[![Sample Image][1]][1]
&lt;!-- source: http://img72.imageshack.us/img72/8305/45167805.png --&gt;

 [1]: https://i.stack.imgur.com/IubwX.png

HighCharts Hide Series Name from the Legend

I have a pandas series `series`.  If I want to get the element-wise floor or ceiling, is there a built in method or do I have to write the function and use apply?  I ask because the data is big so I appreciate efficiency.  Also this question has not been asked with respect to the Pandas package. 

Floor or ceiling of a pandas series in python?

Say we have used pandas `dataframe[column].value_counts()` which outputs:

     apple   5 
     sausage 2
     banana  2
     cheese  1



How do you extract the values in the order same as shown above from max to min ?

e.g: `[apple,sausage,banana,cheese]`

Extract values in Pandas value_counts()

Is there a way to remove a NaN values from a panda series? I have a series that may or may not have some NaN values in it, and I&#39;d like to return a copy of the series with all the NaNs removed.

Remove NaN from pandas series

I have two series `s1` and `s2` in pandas and want to compute the intersection i.e. where all of the values of the series are common.

How would I use the `concat` function to do this? I have been trying to work it out but have been unable to (I don&#39;t want to compute the intersection on the indices of `s1` and `s2`, but on the values).

Finding the intersection between two series in Pandas

I currently have a pandas `Series` with dtype `Timestamp`, and I want to group it by date (and have many rows with different times in each group).

The seemingly obvious way of doing this would be something similar to

    grouped = s.groupby(lambda x: x.date())

However, pandas&#39; `groupby` groups Series by its index. How can I make it group by value instead?

How to group a Series by values in pandas?

Lets say I have a dataframe like this

        A   B
    0   a   b
    1   c   d
    2   e   f 
    3   g   h

0,1,2,3 are times, a, c, e, g is one time series and b, d, f, h is another time series.
I need to be able to add two columns to the orignal dataframe which is got by computing the differences of consecutive rows for certain columns. 

So i need something like this

        A   B   dA
    0   a   b  (a-c)
    1   c   d  (c-e)
    2   e   f  (e-g)
    3   g   h   Nan

I saw something called diff on the dataframe/series but that does it slightly differently as in first element will become Nan.

Adding a column thats result of difference in consecutive rows in pandas

I have a data series which looks like this:
    
    print mys

    id_L1
    2       NaN
    3       NaN
    4       NaN
    5       NaN
    6       NaN
    7       NaN
    8       NaN

I would like to check is all the values are NaN. 

My attempt:

    pd.isnull(mys).all()

Output:
  
    True

Is this the correct way to do it? 

Pandas - check if ALL values are NaN in Series

I have a pandas series with boolean entries.  I would like to get a list of indices where the values are `True`.

For example the input `pd.Series([True, False, True, True, False, False, False, True])`

should yield the output `[0,2,3,7]`.

I can do it with a list comprehension, but is there something cleaner or faster?

Getting a list of indices where pandas boolean series is True

Is there a way to do this?  I cannot seem an easy way to interface pandas series with plotting a CDF.  

Plotting CDF of a pandas series in python

I have some data I&#39;m trying to organize into a `DataFrame` in `Pandas`.  I was trying to make each row a `Series` and append it to the `DataFrame`.  I found a way to do it by appending the `Series` to an empty `list` and then converting the `list` of `Series` to a `DataFrame` 

e.g. `DF = DataFrame([series1,series2],columns=series1.index)`

This `list` to `DataFrame` step seems to be excessive.  I&#39;ve checked out a few examples on here but none of the `Series` preserved the `Index` labels from the `Series` to use them as column labels.

My long way where columns are id_names and rows are type_names:
[![enter image description here][1]][1] 


**Is it possible to append Series to rows of DataFrame without making a list first?**



    #!/usr/bin/python
    
    DF = DataFrame()
    for sample,data in D_sample_data.items():
        SR_row = pd.Series(data.D_key_value)
        DF.append(SR_row)
    DF.head()
    
    TypeError: Can only append a Series if ignore_index=True or if the Series has a name
    

Then I tried


    DF = DataFrame()
    for sample,data in D_sample_data.items():
        SR_row = pd.Series(data.D_key_value,name=sample)
        DF.append(SR_row)
    DF.head()

Empty DataFrame

Tried https://stackoverflow.com/questions/24284342/insert-a-row-to-pandas-dataframe
Still getting an empty dataframe :/ 

**I am trying to get the Series to be the rows, where the index of the Series becomes the column labels of the DataFrame**


  [1]: http://i.stack.imgur.com/L83C8.png

Is it possible to append Series to rows of DataFrame without making a list first?

I read a csv file into a pandas dataframe, and would like to convert the columns with binary answers from strings of yes/no to integers of 1/0. Below, I show one of such columns (&quot;sampleDF&quot; is the pandas dataframe).

    In [13]: sampleDF.housing[0:10]
    Out[13]:
    0     no
    1     no
    2    yes
    3     no
    4     no
    5     no
    6     no
    7     no
    8    yes
    9    yes
    Name: housing, dtype: object

Help is much appreciated!

Is there a simple way to change a column of yes/no to 1/0 in a Pandas dataframe?

When I am writing code in Python, I often need to remove items from a list or other sequence type based on some criteria. I haven&#39;t found a solution that is elegant and efficient, as removing items from a list you are currently iterating through is bad.  For example, you can&#39;t do this:

    for name in names:
        if name[-5:] == &#39;Smith&#39;:
            names.remove(name)

I usually end up doing something like this:

    toremove = []
    for name in names:
        if name[-5:] == &#39;Smith&#39;:
            toremove.append(name)
    for name in toremove:
        names.remove(name)
    del toremove

This is innefficient, fairly ugly and possibly buggy (how does it handle multiple &#39;John Smith&#39; entries?). Does anyone have a more elegant solution, or at least a more efficient one? 

How about one that works with dictionaries?

Elegant way to remove items from sequence in Python?

I have a data frame and I would like to know how many times a given column has the most frequent value.

I try to do it in the following way:

    items_counts = df[&#39;item&#39;].value_counts()
    max_item = items_counts.max()

As a result I get:

    ValueError: cannot convert float NaN to integer

As far as I understand, with the first line I get series in which the values from a column are used as key and frequency of these values are used as values. So, I just need to find the largest value in the series and, because of some reason, it does not work. Does anybody know how this problem can be solved?

How to get the number of the most frequent value in a column?

I am trying to iterate over the rows of a Python Pandas dataframe. Within each row of the dataframe, I am trying to to refer to each value along a row by its column name. 

Here is what I have:

    import numpy as np
    import pandas as pd
    
    df = pd.DataFrame(np.random.rand(10,4),columns=list(&#39;ABCD&#39;))
    print df
              A         B         C         D
    0  0.351741  0.186022  0.238705  0.081457
    1  0.950817  0.665594  0.671151  0.730102
    2  0.727996  0.442725  0.658816  0.003515
    3  0.155604  0.567044  0.943466  0.666576
    4  0.056922  0.751562  0.135624  0.597252
    5  0.577770  0.995546  0.984923  0.123392
    6  0.121061  0.490894  0.134702  0.358296
    7  0.895856  0.617628  0.722529  0.794110
    8  0.611006  0.328815  0.395859  0.507364
    9  0.616169  0.527488  0.186614  0.278792

I used [this approach](https://stackoverflow.com/a/16476974/4057186) to iterate, but it is only giving me part of the solution - **after selecting a row in each iteration, how do I access row elements by their column name?**

Here is what I am trying to do:

    for row in df.iterrows():
    	print row.loc[0,&#39;A&#39;]
    	print row.A
        print row.index()

My understanding is that the row is a Pandas [series][1]. But I have no way to index into the Series.

Is it possible to use column names while simultaneously iterating over rows?


  [1]: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html


Python Pandas iterate over rows and access column names

I have a dataframe extracted from Kaggle&#39;s San Fransico Salaries: https://www.kaggle.com/kaggle/sf-salaries
and I wish to create a set of the values of a column, for instance &#39;Status&#39;.

This is what I have tried but it brings a list of all the records instead of the set (sf is how I name the data frame).

    a=set(sf[&#39;Status&#39;])
    print a

According to this webpage, this should work.
https://stackoverflow.com/questions/15768757/how-to-construct-a-set-out-of-list-items

Create a set from a series in pandas

What is the method to convert a Python list of strings to a `pd.Series` object?

(pandas Series objects can be converted to list using `tolist()` method--but how to do the reverse conversion?)

Convert Python list to pandas Series

I want to add a Series (`s`) to a Pandas DataFrame (`df`) as a new column. The series has more values than there are rows in the dataframe, so I am using the `concat` method along axis 1.

`df = pd.concat((df, s), axis=1)`

This works, but the new column of the dataframe representing the series is given an arbitrary numerical column name, and I would like this column to have a specific name instead.

*Is there a way to add a series to a dataframe, when the series is longer than the rows of the dataframe, and with a specified column name in the resulting dataframe?*



All Series Solutions on Gang of Coders

Total of 31 Series Solutions