How to get reproducible results in keras
PythonNumpyTheanoKerasPython Problem Overview
I get different results (test accuracy) every time I run the imdb_lstm.py
example from Keras framework (https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py)
The code contains np.random.seed(1337)
in the top, before any keras imports. It should prevent it from generating different numbers for every run. What am I missing?
UPDATE: How to repro:
- Install Keras (http://keras.io/)
- Execute https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py a few times. It will train the model and output test accuracy.
Expected result: Test accuracy is the same on every run.
Actual result: Test accuracy is different on every run.
UPDATE2: I'm running it on Windows 8.1 with MinGW/msys, module versions:
theano 0.7.0
numpy 1.8.1
scipy 0.14.0c1
UPDATE3: I narrowed the problem down a bit. If I run the example with GPU (set theano flag device=gpu0) then I get different test accuracy every time, but if I run it on CPU then everything works as expected. My graphics card: NVIDIA GeForce GT 635)
Python Solutions
Solution 1 - Python
You can find the answer at the Keras docs: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development.
In short, to be absolutely sure that you will get reproducible results with your python script on one computer's/laptop's CPU then you will have to do the following:
- Set the
PYTHONHASHSEED
environment variable at a fixed value - Set the
python
built-in pseudo-random generator at a fixed value - Set the
numpy
pseudo-random generator at a fixed value - Set the
tensorflow
pseudo-random generator at a fixed value - Configure a new global
tensorflow
session
Following the Keras
link at the top, the source code I am using is the following:
# Seed value
# Apparently you may use different seed values at each stage
seed_value= 0
# 1. Set the `PYTHONHASHSEED` environment variable at a fixed value
import os
os.environ['PYTHONHASHSEED']=str(seed_value)
# 2. Set the `python` built-in pseudo-random generator at a fixed value
import random
random.seed(seed_value)
# 3. Set the `numpy` pseudo-random generator at a fixed value
import numpy as np
np.random.seed(seed_value)
# 4. Set the `tensorflow` pseudo-random generator at a fixed value
import tensorflow as tf
tf.random.set_seed(seed_value)
# for later versions:
# tf.compat.v1.set_random_seed(seed_value)
# 5. Configure a new global `tensorflow` session
from keras import backend as K
session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
K.set_session(sess)
# for later versions:
# session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
# sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
# tf.compat.v1.keras.backend.set_session(sess)
It is needless to say that you do not have to to specify any seed
or random_state
at the numpy
, scikit-learn
or tensorflow
/keras
functions that you are using in your python script exactly because with the source code above we set globally their pseudo-random generators at a fixed value.
Solution 2 - Python
Theano's documentation talks about the difficulties of seeding random variables and why they seed each graph instance with its own random number generator.
> Sharing a random number generator between different {{{RandomOp}}} > instances makes it difficult to producing the same stream regardless > of other ops in graph, and to keep {{{RandomOps}}} isolated. > Therefore, each {{{RandomOp}}} instance in a graph will have its very > own random number generator. That random number generator is an input > to the function. In typical usage, we will use the new features of > function inputs ({{{value}}}, {{{update}}}) to pass and update the rng > for each {{{RandomOp}}}. By passing RNGs as inputs, it is possible to > use the normal methods of accessing function inputs to access each > {{{RandomOp}}}’s rng. In this approach it there is no pre-existing > mechanism to work with the combined random number state of an entire > graph. So the proposal is to provide the missing functionality (the > last three requirements) via auxiliary functions: {{{seed, getstate, > setstate}}}.
They also provide examples on how to seed all the random number generators.
> You can also seed all of the random variables allocated by a > RandomStreams object by that object’s seed method. This seed will be > used to seed a temporary random number generator, that will in turn > generate seeds for each of the random variables. >
>>> srng.seed(902340) # seeds rv_u and rv_n with different seeds each
Solution 3 - Python
I finally got reproducible results with my code. It's a combination of answers I saw around the web. The first thing is doing what @alex says:
- Set
numpy.random.seed
; - Use
PYTHONHASHSEED=0
for Python 3.
Then you have to solve the issue noted by @user2805751 regarding cuDNN by calling your Keras code with the following additional THEANO_FLAGS
:
dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic
And finally, you have to patch your Theano installation as per this comment, which basically consists in:
- replacing all calls to
*_dev20
operator by its regular version intheano/sandbox/cuda/opt.py
.
This should get you the same results for the same seed.
Note that there might be a slowdown. I saw a running time increase of about 10%.
Solution 4 - Python
The problem is now solved in Tensorflow 2.0 ! I had the same issue with TF 1.x (see https://stackoverflow.com/questions/59075244/if-keras-results-are-not-reproducible-whats-the-best-practice-for-comparing-mo ) but
import os
####*IMPORANT*: Have to do this line *before* importing tensorflow
os.environ['PYTHONHASHSEED']=str(1)
import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.layers
import random
import pandas as pd
import numpy as np
def reset_random_seeds():
os.environ['PYTHONHASHSEED']=str(1)
tf.random.set_seed(1)
np.random.seed(1)
random.seed(1)
#make some random data
reset_random_seeds()
NUM_ROWS = 1000
NUM_FEATURES = 10
random_data = np.random.normal(size=(NUM_ROWS, NUM_FEATURES))
df = pd.DataFrame(data=random_data, columns=['x_' + str(ii) for ii in range(NUM_FEATURES)])
y = df.sum(axis=1) + np.random.normal(size=(NUM_ROWS))
def run(x, y):
reset_random_seeds()
model = keras.Sequential([
keras.layers.Dense(40, input_dim=df.shape[1], activation='relu'),
keras.layers.Dense(20, activation='relu'),
keras.layers.Dense(10, activation='relu'),
keras.layers.Dense(1, activation='linear')
])
NUM_EPOCHS = 500
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x, y, epochs=NUM_EPOCHS, verbose=0)
predictions = model.predict(x).flatten()
loss = model.evaluate(x, y) #This prints out the loss by side-effect
#With Tensorflow 2.0 this is now reproducible!
run(df, y)
run(df, y)
run(df, y)
Solution 5 - Python
In Tensorflow 2.0 you can set random seed like this :
import tensorflow as tf
tf.random.set_seed(221)
from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential( [
layers.Dense(2,name = 'one'),
layers.Dense(3,activation = 'sigmoid', name = 'two'),
layers.Dense(2,name = 'three')])
x = tf.random.uniform((12,12))
model(x)
Solution 6 - Python
This works for me:
SEED = 123456
import os
import random as rn
import numpy as np
from tensorflow import set_random_seed
os.environ['PYTHONHASHSEED']=str(SEED)
np.random.seed(SEED)
set_random_seed(SEED)
rn.seed(SEED)
Solution 7 - Python
I have trained and tested Sequential()
kind of neural networks using Keras. I performed non linear regression on noisy speech data. I used the following code to generate random seed :
import numpy as np
seed = 7
np.random.seed(seed)
I get the exact same results of val_loss
each time I train and test on the same data.
Solution 8 - Python
I would like to add something to the previous answers. If you use python 3 and you want to get reproducible results for every run, you have to
- set numpy.random.seed in the beginning of your code
- give PYTHONHASHSEED=0 as a parameter to the python interpreter
Solution 9 - Python
It is easier that it seems. Putting only this, it works:
import numpy as np
import tensorflow as tf
import random as python_random
def reset_seeds():
np.random.seed(123)
python_random.seed(123)
tf.random.set_seed(1234)
reset_seeds()
The KEY of the question, VERY IMPORTANT, is to call the function reset_seeds() every time before running the model. Doing that you will obtain reproducible results as I check in the Google Collab.
Solution 10 - Python
I agree with the previous comment, but reproducible results sometimes needs the same environment(e.g. installed packages, machine characteristics and so on). So that, I recommend to copy your environment to other place in case to have reproducible results. Try to use one of the next technologies:
- Docker. If you have a Linux this very easy to move your environment to other place. Also you can try to use DockerHub.
- Binder. This is a cloud platform for reproducing scientific experiments.
- Everware. This is yet another cloud platform for "reusable science". See the project repository on Github.
Solution 11 - Python
The Conference Paper: Non-Random Weight Initialisation in Deep Learning Networks for Repeatable Determinism, publication date Jun 5, 2019 presented at 10th IEEE International Conference Dependable Systems, Services and Technologies (DESSERT-19) at Leeds Beckett University (LBU), United Kingdom, UK, Ireland and the Ukrainian section of IEEE June 5-7, 2019
https://ieeexplore.ieee.org/document/8770007
shows how to get repeatable results by enforcing critical regions of code.
it has been extended to a Journal Paper: Repeatable Determinism using Non-Random Weight Initialisations in Smart City Applications of Deep Learning publication in The Journal of Reliable Intelligent Environments in a Smart Cities special edition, and uses glorot xavier limts and achieve the same accuracy with perceptron layers but grow the weight in to a linear order which may have an advantage for rule extraction in perceptron layers.
Solution 12 - Python
Unlike what has been said before, only Tensorflow seed has an effect on random generation of weights (latest version Tensorflow 2.6.0 and Keras 2.6.0)
Here is a small test you can run to check the influence of each seed (with np being numpy, tf being tensorflow and random the Python random library):
# Testing how seeds influence results
# -----------------------------------
print("Seed specification")
my_seed = 36
# To vary python hash, numpy random, python random and tensorflow random seeds
a, b, c, d = 0, 0, 0, 0
os.environ['PYTHONHASHSEED'] = str(my_seed+a) # Has no effect
np.random.seed(my_seed+b) # Has no effect
random.seed(my_seed+c) # Has no effect
tf.random.set_seed(my_seed+d) # Has an effect
print("Making ML model")
keras.mixed_precision.set_global_policy('float64')
model = keras.Sequential([
layers.Dense(2, input_shape=input_shape),#, activation='relu'),
layers.Dense(output_nb, activation=None),
])
#
weights_save = model.get_weights()
print("Some weights:", weights_save[0].flatten())
We notice that variables a
, b
, c
have no effect on the results.
Only d
has an effect on the results.
So, in the latest versions of Tensorflow, only tensorflow random seed has an influence on the random choice of weights.