Python/pyspark data frame rearrange columns
PythonPysparkSpark DataframePython Problem Overview
I have a data frame in python/pyspark with columns id time city zip and so on......
Now I added a new column name to this data frame.
Now I have to arrange the columns in such a way that the name column comes after id
I have done like below
change_cols = ['id', 'name']
cols = ([col for col in change_cols if col in df]
+ [col for col in df if col not in change_cols])
df = df[cols]
I am getting this error
pyspark.sql.utils.AnalysisException: u"Reference 'id' is ambiguous, could be: id#609, id#1224.;"
Why is this error occuring. How can I rectify this.
Python Solutions
Solution 1 - Python
You can use select to change the order of the columns:
df.select("id","name","time","city")
Solution 2 - Python
If you're working with a large number of columns:
df.select(sorted(df.columns))
Solution 3 - Python
If you just want to reorder some of them, while keeping the rest and not bothering about their order :
def get_cols_to_front(df, columns_to_front) :
original = df.columns
# Filter to present columns
columns_to_front = [c for c in columns_to_front if c in original]
# Keep the rest of the columns and sort it for consistency
columns_other = list(set(original)) - set(columns_to_front))
columns_other.sort()
# Apply the order
df = df.select(*columns_to_front, *columns_other)
return df