You can use the `left_on` and `right_on` options of [pd.merge](https://pandas.pydata.org/docs/reference/api/pandas.merge.html) as follows:


    pd.merge(frame_1, frame_2, left_on=&#39;county_ID&#39;, right_on=&#39;countyid&#39;)

Or equivalently with [DataFrame.merge](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html):

    frame_1.merge(frame_2, left_on=&#39;county_ID&#39;, right_on=&#39;countyid&#39;)


----------


I was not sure from the question if you only wanted to merge if the key was in the left hand DataFrame. If that is the case then the following will do that (the above will in effect do a many to many merge)

    pd.merge(frame_1, frame_2, how=&#39;left&#39;, left_on=&#39;county_ID&#39;, right_on=&#39;countyid&#39;)

Or

    frame_1.merge(frame_2, how=&#39;left&#39;, left_on=&#39;county_ID&#39;, right_on=&#39;countyid&#39;)

you need to make `county_ID` as index for the right frame:

    frame_2.join ( frame_1.set_index( [ &#39;county_ID&#39; ], verify_integrity=True ),
                   on=[ &#39;countyid&#39; ], how=&#39;left&#39; )

for your information, in pandas left join breaks when the right frame has non unique values on the joining column. see this [bug](https://github.com/pydata/pandas/issues/5391).

so you need to verify integrity before joining by `, verify_integrity=True`





Does the question mark after equals have special meaning? ie:

    scope: {foo: &#39;=?&#39;}

does the above mean &#39;do not raise error if &#39;foo&#39; cannot be resolved?

what&#39;s the meaning of &#39;=?&#39; in angularJS directive isolate scope declaration?

From what I understand the HTML5 spec lets you use IDs that are numbers like this.

    &lt;div id=&quot;1&quot;&gt;&lt;/div&gt;
    &lt;div id=&quot;2&quot;&gt;&lt;/div&gt;

I can access these fine using `getElementById` but not with `querySelector`.  If I try do the following I get **SyntaxError: DOM Exception 12** in the console.

    document.querySelector(&quot;#1&quot;)

I&#39;m just curious why using numbers as IDs does not work `querySelector` when the HTML5 spec says these are valid.  I tried multiple browsers.

Using querySelector with IDs that are numbers

I have two DataFrames with the following column names:

    frame_1:
    event_id, date, time, county_ID
    
    frame_2:
    countyid, state

I would like to get a DataFrame with the following columns by joining (left) on `county_ID = countyid`:

    joined_dataframe
    event_id, date, time, county, state

I cannot figure out how to do it if the columns on which I want to join are not the index.

Joining pandas DataFrames by Column names

I have two DataFrames with the following column names:
<pre><code class="hljs language-vbnet">frame_1:
event_id, date, time, county_ID

frame_2:
countyid, state
</code></pre>
I would like to get a DataFrame with the following columns by joining (left) on <code>county_ID = countyid</code>:
<pre><code class="hljs language-lua">joined_dataframe
event_id, date, time, county, state
</code></pre>
I cannot figure out how to do it if the columns on which I want to join are not the index.

In Pandas, when I select a label that only has one entry in the index I get back a Series, but when I select an entry that has more then one entry I get back a data frame.

Why is that?  Is there a way to ensure I always get back a data frame?


    In [1]: import pandas as pd
    
    In [2]: df = pd.DataFrame(data=range(5), index=[1, 2, 3, 3, 3])
    
    In [3]: type(df.loc[3])
    Out[3]: pandas.core.frame.DataFrame
    
    In [4]: type(df.loc[1])
    Out[4]: pandas.core.series.Series

Pandas selecting by label sometimes return Series, sometimes returns DataFrame

How do I remove a package from Pypi?

I uploaded a package to Pypi several months ago.  The package is now obsolete and I&#39;d like to formally remove it.  I cannot find any documentation on how to *remove* my package.

How to remove a package from Pypi

Example

    s=pd.Series([5,4,3,2,1], index=[1,2,3,4,5])
    print s 
    1    5
    2    4
    3    3
    4    2
    5    1

Is there an efficient way to create a series. e.g. containing in each row the lagged values (in this example up to lag 2)

    3    [3, 4, 5]
    4    [2, 3, 4]
    5    [1, 2, 3]

This corresponds to *s=pd.Series([[3,4,5],[2,3,4],[1,2,3]], index=[3,4,5])*

How can this be done in an efficient way for dataframes with a lot of timeseries which are very long?

Thanks


**Edited after seeing the answers**

ok, at the end I implemented this function:

    def buildLaggedFeatures(s,lag=2,dropna=True):
    &#39;&#39;&#39;
    Builds a new DataFrame to facilitate regressing over all possible lagged features
    &#39;&#39;&#39;
    if type(s) is pd.DataFrame:
        new_dict={}
        for col_name in s:
            new_dict[col_name]=s[col_name]
            # create lagged Series
            for l in range(1,lag+1):
                new_dict[&#39;%s_lag%d&#39; %(col_name,l)]=s[col_name].shift(l)
        res=pd.DataFrame(new_dict,index=s.index)

    elif type(s) is pd.Series:
        the_range=range(lag+1)
        res=pd.concat([s.shift(i) for i in the_range],axis=1)
        res.columns=[&#39;lag_%d&#39; %i for i in the_range]
    else:
        print &#39;Only works for DataFrame or Series&#39;
        return None
    if dropna:
        return res.dropna()
    else:
        return res 

it produces the wished outputs and manages the naming of columns in the resulting DataFrame.

For a Series as input:

    s=pd.Series([5,4,3,2,1], index=[1,2,3,4,5])
    res=buildLaggedFeatures(s,lag=2,dropna=False)
       lag_0  lag_1  lag_2
    1      5    NaN    NaN
    2      4      5    NaN
    3      3      4      5
    4      2      3      4
    5      1      2      3

and for a DataFrame as input:

    s2=s=pd.DataFrame({&#39;a&#39;:[5,4,3,2,1], &#39;b&#39;:[50,40,30,20,10]},index=[1,2,3,4,5])
    res2=buildLaggedFeatures(s2,lag=2,dropna=True)
    
       a  a_lag1  a_lag2   b  b_lag1  b_lag2
    3  3       4       5  30      40      50
    4  2       3       4  20      30      40
    5  1       2       3  10      20      30





How to create a lagged data structure using pandas dataframe

How can I convert a `defaultdict`

    number_to_letter
    defaultdict(&lt;class &#39;list&#39;&gt;, {&#39;2&#39;: [&#39;a&#39;], &#39;3&#39;: [&#39;b&#39;], &#39;1&#39;: [&#39;b&#39;, &#39;a&#39;]})

to be a common `dict`?

    {&#39;2&#39;: [&#39;a&#39;], &#39;3&#39;: [&#39;b&#39;], &#39;1&#39;: [&#39;b&#39;, &#39;a&#39;]}


How to convert defaultdict to dict?

I have a `.txt` file in the same folder as this `.py` file and it has this in it:

    cat\n
    dog\n
    rat\n
    cow\n

How can I save a var (var = &#39;ant&#39;) to the next line of the `.txt` file?




How to write to .txt files in Python 3

Here is my code:

    import pandas as pd

    data = pd.DataFrame({&#39;Odd&#39;:[1,3,5,6,7,9], &#39;Even&#39;:[0,2,4,6,8,10]})

    for i in reversed(data):
        print(data[&#39;Odd&#39;], data[&#39;Even&#39;])

When I run this code, i get the following error:

    Traceback (most recent call last):
      File &quot;C:\Python33\lib\site-packages\pandas\core\generic.py&quot;, line 665, in _get_item_cache
        return cache[item]
    KeyError: 5
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File &quot;C:\Users\*****\Documents\******\********\****.py&quot;, line 5, in &lt;module&gt;
        for i in reversed(data):
      File &quot;C:\Python33\lib\site-packages\pandas\core\frame.py&quot;, line 2003, in __getitem__
        return self._get_item_cache(key)
      File &quot;C:\Python33\lib\site-packages\pandas\core\generic.py&quot;, line 667, in _get_item_cache
        values = self._data.get(item)
      File &quot;C:\Python33\lib\site-packages\pandas\core\internals.py&quot;, line 1656, in get
        _, block = self._find_block(item)
      File &quot;C:\Python33\lib\site-packages\pandas\core\internals.py&quot;, line 1936, in _find_block
        self._check_have(item)
      File &quot;C:\Python33\lib\site-packages\pandas\core\internals.py&quot;, line 1943, in _check_have
        raise KeyError(&#39;no item named %s&#39; % com.pprint_thing(item))
    KeyError: &#39;no item named 5&#39;

**Why am I getting this error?  
How can I fix that?   
What is the right way to reverse &lt;code&gt;pandas.DataFrame&lt;/code&gt;?**

Right way to reverse a pandas DataFrame?

I am creating a matrix from a Pandas dataframe as follows:

    dense_matrix = np.array(df.as_matrix(columns = None), dtype=bool).astype(np.int)

And then into a sparse matrix with:

    sparse_matrix = scipy.sparse.csr_matrix(dense_matrix)

Is there any way to go from a df straight to a sparse matrix?

Thanks in advance.

Convert Pandas dataframe to Sparse Numpy Matrix directly

This seems rather obvious, but I can&#39;t seem to figure out how to convert an index of data frame to a column?

For example:
    
    df=
            gi       ptt_loc
     0  384444683      593  
     1  384444684      594 
     2  384444686      596  

To,

    df=
        index1    gi       ptt_loc
     0  0     384444683      593  
     1  1     384444684      594 
     2  2     384444686      596  



How to convert index of a pandas dataframe into a column

I have a dataframe from which I remove some rows. As a result, I get a dataframe in which index is something like that: `[1,5,6,10,11]` and I would like to reset it to `[0,1,2,3,4]`. How can I do it?

---
The following seems to work:

    df = df.reset_index()
    del df[&#39;index&#39;]

The following does not work:

    df = df.reindex()



How to reset index in a pandas dataframe?

## Background
I just upgraded my Pandas from 0.11 to 0.13.0rc1. Now, the application is popping out many new warnings. One of them like this:

    E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_index,col_indexer] = value instead
      quote_df[&#39;TVol&#39;]   = quote_df[&#39;TVol&#39;]/TVOL_SCALE

I want to know what exactly it means?  Do I need to change something?

How should I suspend the warning if I insist to use `quote_df[&#39;TVol&#39;]   = quote_df[&#39;TVol&#39;]/TVOL_SCALE`?

## The function that gives errors

    def _decode_stock_quote(list_of_150_stk_str):
        &quot;&quot;&quot;decode the webpage and return dataframe&quot;&quot;&quot;
    
        from cStringIO import StringIO
    
        str_of_all = &quot;&quot;.join(list_of_150_stk_str)
    
        quote_df = pd.read_csv(StringIO(str_of_all), sep=&#39;,&#39;, names=list(&#39;ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefg&#39;)) #dtype={&#39;A&#39;: object, &#39;B&#39;: object, &#39;C&#39;: np.float64}
        quote_df.rename(columns={&#39;A&#39;:&#39;STK&#39;, &#39;B&#39;:&#39;TOpen&#39;, &#39;C&#39;:&#39;TPCLOSE&#39;, &#39;D&#39;:&#39;TPrice&#39;, &#39;E&#39;:&#39;THigh&#39;, &#39;F&#39;:&#39;TLow&#39;, &#39;I&#39;:&#39;TVol&#39;, &#39;J&#39;:&#39;TAmt&#39;, &#39;e&#39;:&#39;TDate&#39;, &#39;f&#39;:&#39;TTime&#39;}, inplace=True)
        quote_df = quote_df.ix[:,[0,3,2,1,4,5,8,9,30,31]]
        quote_df[&#39;TClose&#39;] = quote_df[&#39;TPrice&#39;]
        quote_df[&#39;RT&#39;]     = 100 * (quote_df[&#39;TPrice&#39;]/quote_df[&#39;TPCLOSE&#39;] - 1)
        quote_df[&#39;TVol&#39;]   = quote_df[&#39;TVol&#39;]/TVOL_SCALE
        quote_df[&#39;TAmt&#39;]   = quote_df[&#39;TAmt&#39;]/TAMT_SCALE
        quote_df[&#39;STK_ID&#39;] = quote_df[&#39;STK&#39;].str.slice(13,19)
        quote_df[&#39;STK_Name&#39;] = quote_df[&#39;STK&#39;].str.slice(21,30)#.decode(&#39;gb2312&#39;)
        quote_df[&#39;TDate&#39;]  = quote_df.TDate.map(lambda x: x[0:4]+x[5:7]+x[8:10])
        
        return quote_df

## More error messages

    E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_index,col_indexer] = value instead
      quote_df[&#39;TVol&#39;]   = quote_df[&#39;TVol&#39;]/TVOL_SCALE
    E:\FinReporter\FM_EXT.py:450: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_index,col_indexer] = value instead
      quote_df[&#39;TAmt&#39;]   = quote_df[&#39;TAmt&#39;]/TAMT_SCALE
    E:\FinReporter\FM_EXT.py:453: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_index,col_indexer] = value instead
      quote_df[&#39;TDate&#39;]  = quote_df.TDate.map(lambda x: x[0:4]+x[5:7]+x[8:10])

How to deal with SettingWithCopyWarning in Pandas

Given a (pre-existing) data frame that has columns of various types, what is the simplest way to convert all its character columns to factors, without affecting any columns of other types?

Here&#39;s an example `data.frame`:

    df &lt;- data.frame(A = factor(LETTERS[1:5]),
                     B = 1:5, C = as.logical(c(1, 1, 0, 0, 1)),
                     D = letters[1:5],
                     E = paste(LETTERS[1:5], letters[1:5]),
                     stringsAsFactors = FALSE)
    df
    #   A B     C D   E
    # 1 A 1  TRUE a A a
    # 2 B 2  TRUE b B b
    # 3 C 3 FALSE c C c
    # 4 D 4 FALSE d D d
    # 5 E 5  TRUE e E e
    str(df)
    # &#39;data.frame&#39;:  5 obs. of  5 variables:
    #  $ A: Factor w/ 5 levels &quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;D&quot;,..: 1 2 3 4 5
    #  $ B: int  1 2 3 4 5
    #  $ C: logi  TRUE TRUE FALSE FALSE TRUE
    #  $ D: chr  &quot;a&quot; &quot;b&quot; &quot;c&quot; &quot;d&quot; ...
    #  $ E: chr  &quot;A a&quot; &quot;B b&quot; &quot;C c&quot; &quot;D d&quot; ...

I know I can do:
    
    df$D &lt;- as.factor(df$D)
    df$E &lt;- as.factor(df$E)

Is there a way to automate this process a bit more?

Content Type	Original Author	Original Content on Stackoverflow
Question	Alexis Eggermont	View Question on Stackoverflow
Solution 1 - Python	Woody Pride	View Answer on Stackoverflow
Solution 2 - Python	behzad.nouri	View Answer on Stackoverflow

Joining pandas DataFrames by Column names

Python Problem Overview

Python Solutions

Solution 1 - Python

Solution 2 - Python

Using querySelector with IDs that are numbers

what's the meaning of '=?' in angularJS directive isolate scope declaration?

Attributions