Is there a library function for Root mean square error (RMSE) in python?
PythonScikit LearnScipyPython Problem Overview
I know I could implement a root mean squared error function like this:
def rmse(predictions, targets):
return np.sqrt(((predictions - targets) ** 2).mean())
What I'm looking for if this rmse function is implemented in a library somewhere, perhaps in scipy or scikit-learn?
Python Solutions
Solution 1 - Python
sklearn >= 0.22.0
sklearn.metrics
has a mean_squared_error
function with a squared
kwarg (defaults to True
). Setting squared
to False
will return the RMSE.
from sklearn.metrics import mean_squared_error
rms = mean_squared_error(y_actual, y_predicted, squared=False)
sklearn < 0.22.0
sklearn.metrics
has a mean_squared_error
function. The RMSE is just the square root of whatever it returns.
from sklearn.metrics import mean_squared_error
from math import sqrt
rms = sqrt(mean_squared_error(y_actual, y_predicted))
Solution 2 - Python
What is RMSE? Also known as MSE, RMD, or RMS. What problem does it solve?
If you understand RMSE: (Root mean squared error), MSE: (Mean Squared Error) RMD (Root mean squared deviation) and RMS: (Root Mean Squared), then asking for a library to calculate this for you is unnecessary over-engineering. All these metrics are a single line of python code at most 2 inches long. The three metrics rmse, mse, rmd, and rms are at their core conceptually identical.
RMSE answers the question: "How similar, on average, are the numbers in list1
to list2
?". The two lists must be the same size. I want to "wash out the noise between any two given elements, wash out the size of the data collected, and get a single number feel for change over time".
Intuition and ELI5 for RMSE:
Imagine you are learning to throw darts at a dart board. Every day you practice for one hour. You want to figure out if you are getting better or getting worse. So every day you make 10 throws and measure the distance between the bullseye and where your dart hit.
You make a list of those numbers list1
. Use the root mean squared error between the distances at day 1 and a list2
containing all zeros. Do the same on the 2nd and nth days. What you will get is a single number that hopefully decreases over time. When your RMSE number is zero, you hit bullseyes every time. If the rmse number goes up, you are getting worse.
Example in calculating root mean squared error in python:
import numpy as np
d = [0.000, 0.166, 0.333] #ideal target distances, these can be all zeros.
p = [0.000, 0.254, 0.998] #your performance goes here
print("d is: " + str(["%.8f" % elem for elem in d]))
print("p is: " + str(["%.8f" % elem for elem in p]))
def rmse(predictions, targets):
return np.sqrt(((predictions - targets) ** 2).mean())
rmse_val = rmse(np.array(d), np.array(p))
print("rms error is: " + str(rmse_val))
Which prints:
d is: ['0.00000000', '0.16600000', '0.33300000']
p is: ['0.00000000', '0.25400000', '0.99800000']
rms error between lists d and p is: 0.387284994115
The mathematical notation:
Glyph Legend: n
is a whole positive integer representing the number of throws. i
represents a whole positive integer counter that enumerates sum. d
stands for the ideal distances, the list2
containing all zeros in above example. p
stands for performance, the list1
in the above example. superscript 2 stands for numeric squared. di is the i'th index of d
. pi is the i'th index of p
.
The rmse done in small steps so it can be understood:
def rmse(predictions, targets):
differences = predictions - targets #the DIFFERENCEs.
differences_squared = differences ** 2 #the SQUAREs of ^
mean_of_differences_squared = differences_squared.mean() #the MEAN of ^
rmse_val = np.sqrt(mean_of_differences_squared) #ROOT of ^
return rmse_val #get the ^
How does every step of RMSE work:
Subtracting one number from another gives you the distance between them.
8 - 5 = 3 #absolute distance between 8 and 5 is +3
-20 - 10 = -30 #absolute distance between -20 and 10 is +30
If you multiply any number times itself, the result is always positive because negative times negative is positive:
3*3 = 9 = positive
-30*-30 = 900 = positive
Add them all up, but wait, then an array with many elements would have a larger error than a small array, so average them by the number of elements.
But wait, we squared them all earlier to force them positive. Undo the damage with a square root!
That leaves you with a single number that represents, on average, the distance between every value of list1 to it's corresponding element value of list2.
If the RMSE value goes down over time we are happy because [variance][2] is decreasing. "Shrinking the Variance" is a kind of machine learning algorithm.
RMSE isn't the most accurate line fitting strategy, total least squares is:
Root mean squared error measures the vertical distance between the point and the line, so if your data is shaped like a banana, flat near the bottom and steep near the top, then the RMSE will report greater distances to points high, but short distances to points low when in fact the distances are equivalent. This causes a skew where the line prefers to be closer to points high than low.
If this is a problem the total least squares method fixes this: https://mubaris.com/posts/linear-regression
Gotchas that can break this RMSE function:
If there are nulls or infinity in either input list, then output rmse value is is going to not make sense. There are three strategies to deal with nulls / missing values / infinities in either list: Ignore that component, zero it out or add a best guess or a uniform random noise to all timesteps. Each remedy has its pros and cons depending on what your data means. In general ignoring any component with a missing value is preferred, but this biases the RMSE toward zero making you think performance has improved when it really hasn't. Adding random noise on a best guess could be preferred if there are lots of missing values.
In order to guarantee relative correctness of the RMSE output, you must eliminate all nulls/infinites from the input.
RMSE has zero tolerance for outlier data points which don't belong
Root mean squared error squares relies on all data being right and all are counted as equal. That means one stray point that's way out in left field is going to totally ruin the whole calculation. To handle outlier data points and dismiss their tremendous influence after a certain threshold, see Robust estimators that build in a threshold for dismissal of outliers as extreme rare events that don't need to be controlled for: like rogue waves: https://youtu.be/8Zpi9V0_5tw?t=5
[2]: https://en.wikipedia.org/wiki/Variance "variance"
Solution 3 - Python
In scikit-learn 0.22.0 you can pass mean_squared_error()
the argument squared=False
to return the RMSE.
from sklearn.metrics import mean_squared_error
mean_squared_error(y_actual, y_predicted, squared=False)
Solution 4 - Python
This is probably faster?:
n = len(predictions)
rmse = np.linalg.norm(predictions - targets) / np.sqrt(n)
Solution 5 - Python
sklearn's mean_squared_error
itself contains a parameter squared
with default value as True
. If we set it to False
, the same function will return RMSE instead of MSE.
from sklearn.metrics import mean_squared_error
rmse = mean_squared_error(y_true, y_pred , squared=False)
Solution 6 - Python
There is a library ml_metrics
which is available without pre-installation in Kaggle's kernels, pretty lightweight and accessible through pypi
(it can be installed easily and fast with pip install ml_metrics
):
from ml_metrics import rmse
rmse(actual=[0, 1, 2], predicted=[1, 10, 5])
# 5.507570547286102
It has few other interesting metrics which are not available in sklearn
, like mapk
.
References:
Solution 7 - Python
Or by simply using only NumPy functions:
def rmse(y, y_pred):
return np.sqrt(np.mean(np.square(y - y_pred)))
Where:
- y is my target
- y_pred is my prediction
Note that rmse(y, y_pred)==rmse(y_pred, y)
due to the square function.
Solution 8 - Python
Yes it is provided by SKLearn, we just need to mention squared = False
in the arguments
from sklearn.metrics import mean_squared_error
mean_squared_error(y_true, y_pred, squared=False)
Solution 9 - Python
from sklearn import metrics
import numpy as np
print(np.sqrt(metrics.mean_squared_error(y_test,y_predict)))
Solution 10 - Python
from sklearn.metrics import mean_squared_error
rmse = mean_squared_error(y_actual, y_predicted, squared=False)
or
import math
from sklearn.metrics import mean_squared_error
rmse = math.sqrt(mean_squared_error(y_actual, y_predicted))
Solution 11 - Python
Here's an example code that calculates the RMSE between two polygon file formats PLY
. It uses both the ml_metrics
lib and the np.linalg.norm
:
import sys
import SimpleITK as sitk
from pyntcloud import PyntCloud as pc
import numpy as np
from ml_metrics import rmse
if len(sys.argv) < 3 or sys.argv[1] == "-h" or sys.argv[1] == "--help":
print("Usage: compute-rmse.py <input1.ply> <input2.ply>")
sys.exit(1)
def verify_rmse(a, b):
n = len(a)
return np.linalg.norm(np.array(b) - np.array(a)) / np.sqrt(n)
def compare(a, b):
m = pc.from_file(a).points
n = pc.from_file(b).points
m = [ tuple(m.x), tuple(m.y), tuple(m.z) ]; m = m[0]
n = [ tuple(n.x), tuple(n.y), tuple(n.z) ]; n = n[0]
v1, v2 = verify_rmse(m, n), rmse(m,n)
print(v1, v2)
compare(sys.argv[1], sys.argv[2])
Solution 12 - Python
- No, there is a library Scikit Learn for machine learning and it can be easily employed by using Python language. It has the a function for Mean Squared Error which i am sharing the link below:
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html
- The function is named mean_squared_error as given below, where y_true would be real class values for the data tuples and y_pred would be the predicted values, predicted by the machine learning algorithm you are using:
mean_squared_error(y_true, y_pred)
- You have to modify it to get RMSE (by using sqrt function using Python).This process is described in this link: https://www.codeastar.com/regression-model-rmsd/
So, final code would be something like:
from sklearn.metrics import mean_squared_error from math import sqrt
RMSD = sqrt(mean_squared_error(testing_y, prediction))
print(RMSD)
Solution 13 - Python
You might want to add absolute value np.abs
if you are dealing with complex numbers.
import numpy as np
rms = np.sqrt(np.mean(np.abs(x-y)**2))
Note that if you use np.linalg.norm
it already takes care of complex numbers.
import numpy as np
rms = np.linalg.norm(x-y)/np.sqrt(len(x))
Solution 14 - Python
Benchmark
For specific use case that you don't need overhead handler and always expecting numpy array input, the fastest way is to manually write function in numpy
. Even more, you can use numba
to speed it up if you call it frequently.
import numpy as np
from numba import jit
from sklearn.metrics import mean_squared_error
%%timeit
mean_squared_error(y[i],y[j], squared=False)
445 µs ± 90.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
def euclidian_distance(y1, y2):
"""
RMS Euclidean method
"""
return np.sqrt(((y1-y2)**2).mean())
%%timeit
euclidian_distance(y[i],y[j])
28.8 µs ± 2.54 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
@jit(nopython=True)
def jit_euclidian_distance(y1, y2):
"""
RMS Euclidean method
"""
return np.sqrt(((y1-y2)**2).mean())
%%timeit
jit_euclidian_distance(y[i],y[j])
2.1 µs ± 234 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
@jit(nopython=True)
def jit2_euclidian_distance(y1, y2):
"""
RMS Euclidean method
"""
return np.linalg.norm(y1-y2)/np.sqrt(y1.shape[0])
%%timeit
jit2_euclidian_distance(y[i],y[j])
2.67 µs ± 60.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Extra note: In my use case, numba
give slightly different but negligible result on np.sqrt(((y1-y2)**2).mean())
, where without numba
, the result will be equal to scipy
result. Try it yourself.