Replace None with NaN in pandas dataframe

PandasDataframeReplaceNanNonetype

Pandas Problem Overview


I have table x:

        website
0 	http://www.google.com/
1 	http://www.yahoo.com
2 	None

I want to replace python None with pandas NaN. I tried:

x.replace(to_replace=None, value=np.nan)

But I got:

TypeError: 'regex' must be a string or a compiled regular expression or a list or dict of strings or regular expressions, you passed a 'bool'

How should I go about it?

Pandas Solutions


Solution 1 - Pandas

You can use DataFrame.fillna or Series.fillna which will replace the Python object None, not the string 'None'.

import pandas as pd
import numpy as np

For dataframe:

df = df.fillna(value=np.nan)

For column or series:

df.mycol.fillna(value=np.nan, inplace=True)

Solution 2 - Pandas

Here's another option:

df.replace(to_replace=[None], value=np.nan, inplace=True)

Solution 3 - Pandas

The following line replaces None with NaN:

df['column'].replace('None', np.nan, inplace=True)

Solution 4 - Pandas

If you use df.replace([None], np.nan, inplace=True), this changed all datetime objects with missing data to object dtypes. So now you may have broken queries unless you change them back to datetime which can be taxing depending on the size of your data.

If you want to use this method, you can first identify the object dtype fields in your df and then replace the None:

obj_columns = list(df.select_dtypes(include=['object']).columns.values)
df[obj_columns] = df[obj_columns].replace([None], np.nan)

Solution 5 - Pandas

DataFrame['Col_name'].replace("None", np.nan, inplace=True)

Solution 6 - Pandas

Its an old question but here is a solution for multiple columns:

values = {'col_A': 0, 'col_B': 0, 'col_C': 0, 'col_D': 0}
df.fillna(value=values, inplace=True)

For more options, check the docs:

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionAdamNYCView Question on Stackoverflow
Solution 1 - PandasGuillaume JacquenotView Answer on Stackoverflow
Solution 2 - PandasNickolaiView Answer on Stackoverflow
Solution 3 - PandasMax IzadiView Answer on Stackoverflow
Solution 4 - PandasDoubledownView Answer on Stackoverflow
Solution 5 - PandasAshish SharmaView Answer on Stackoverflow
Solution 6 - PandasMitziView Answer on Stackoverflow