Stop Pandas from converting int to float due to an insertion in another column

PythonPandasType ConversionType Inference

Python Problem Overview


I have a DataFrame with two columns: a column of int and a column of str.

  • I understand that if I insert NaN into the int column, Pandas will convert all the int into float because there is no NaN value for an int.
  • However, when I insert None into the str column, Pandas converts all my int to float as well. This doesn't make sense to me - why does the value I put in column 2 affect column 1?

Here's a simple working example):

import pandas as pd
df = pd.DataFrame()
df["int"] = pd.Series([], dtype=int)
df["str"] = pd.Series([], dtype=str)

df.loc[0] = [0, "zero"]
print(df)
print()

df.loc[1] = [1, None]
print(df)

The output is:

   int   str
0    0  zero

   int   str
0  0.0  zero
1  1.0   NaN

Is there any way to make the output the following:

   int   str
0    0  zero

   int   str
0    0  zero
1    1   NaN

without recasting the first column to int.

  • I prefer using int instead of float because the actual data in that column are integers. If there's not workaround, I'll just use float though.

  • I prefer not having to recast because in my actual code, I don't
    store the actual dtype.

  • I also need the data inserted row-by-row.

Python Solutions


Solution 1 - Python

If you set dtype=object, your series will be able to contain arbitrary data types:

df["int"] = pd.Series([], dtype=object)
df["str"] = pd.Series([], dtype=str)
df.loc[0] = [0, "zero"]
print(df)
print()
df.loc[1] = [1, None]
print(df)

   int   str
0    0  zero
1  NaN   NaN

  int   str
0   0  zero
1   1  None

Solution 2 - Python

As of pandas 1.0.0 I believe you have another option, which is to first use convert_dtypes. This converts the dataframe columns to dtypes that support pd.NA, avoiding the issues with NaN/None.

...

df = df.convert_dtypes()
df.loc[1] = [1, None]
print(df)

#   int   str
# 0   0  zero
# 1   1  NaN

Solution 3 - Python

If you use DataFrame.append to add the data, the dtypes are preserved, and you do not have to recast or rely on object:

In [157]: df
Out[157]:
   int   str
0    0  zero

In [159]: df.append(pd.DataFrame([[1, None]], columns=['int', 'str']), ignore_index=True)
Out[159]:
   int   str
0    0  zero
1    1  None

Solution 4 - Python

right after

df = pd.DataFrame()

add the below and it will initialize the entire series to int. This worked for me.

df['int'] = 0

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
Questionuser2570465View Question on Stackoverflow
Solution 1 - PythonmaxymooView Answer on Stackoverflow
Solution 2 - PythontotalhackView Answer on Stackoverflow
Solution 3 - PythonfugledeView Answer on Stackoverflow
Solution 4 - PythonQuentinJSView Answer on Stackoverflow