How to return history of validation loss in Keras

PythonNeural NetworkNlpDeep LearningKeras

Python Problem Overview


Using Anaconda Python 2.7 Windows 10.

I am training a language model using the Keras exmaple:

print('Build model...')
model = Sequential()
model.add(GRU(512, return_sequences=True, input_shape=(maxlen, len(chars))))
model.add(Dropout(0.2))
model.add(GRU(512, return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(len(chars)))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

def sample(a, temperature=1.0):
    # helper function to sample an index from a probability array
    a = np.log(a) / temperature
    a = np.exp(a) / np.sum(np.exp(a))
    return np.argmax(np.random.multinomial(1, a, 1))


# train the model, output generated text after each iteration
for iteration in range(1, 3):
    print()
    print('-' * 50)
    print('Iteration', iteration)
    model.fit(X, y, batch_size=128, nb_epoch=1)
    start_index = random.randint(0, len(text) - maxlen - 1)

    for diversity in [0.2, 0.5, 1.0, 1.2]:
        print()
        print('----- diversity:', diversity)

        generated = ''
        sentence = text[start_index: start_index + maxlen]
        generated += sentence
        print('----- Generating with seed: "' + sentence + '"')
        sys.stdout.write(generated)

        for i in range(400):
            x = np.zeros((1, maxlen, len(chars)))
            for t, char in enumerate(sentence):
                x[0, t, char_indices[char]] = 1.

            preds = model.predict(x, verbose=0)[0]
            next_index = sample(preds, diversity)
            next_char = indices_char[next_index]

            generated += next_char
            sentence = sentence[1:] + next_char

            sys.stdout.write(next_char)
            sys.stdout.flush()
        print()

According to Keras documentation, the model.fit method returns a History callback, which has a history attribute containing the lists of successive losses and other metrics.

hist = model.fit(X, y, validation_split=0.2)
print(hist.history)

After training my model, if I run print(model.history) I get the error:

 AttributeError: 'Sequential' object has no attribute 'history'

How do I return my model history after training my model with the above code?

UPDATE

The issue was that:

The following had to first be defined:

from keras.callbacks import History 
history = History()

The callbacks option had to be called

model.fit(X_train, Y_train, nb_epoch=5, batch_size=16, callbacks=[history])

But now if I print

print(history.History)

it returns

{}

even though I ran an iteration.

Python Solutions


Solution 1 - Python

Just an example started from

history = model.fit(X, Y, validation_split=0.33, nb_epoch=150, batch_size=10, verbose=0)

You can use

print(history.history.keys())

to list all data in history.

Then, you can print the history of validation loss like this:

print(history.history['val_loss'])

Solution 2 - Python

It's been solved.

The losses only save to the History over the epochs. I was running iterations instead of using the Keras built in epochs option.

so instead of doing 4 iterations I now have

model.fit(......, nb_epoch = 4)

Now it returns the loss for each epoch run:

print(hist.history)
{'loss': [1.4358016599558268, 1.399221191623641, 1.381293383180471, 1.3758836857303727]}

Solution 3 - Python

The following simple code works great for me:

    seqModel =model.fit(x_train, y_train,
          batch_size      = batch_size,
          epochs          = num_epochs,
          validation_data = (x_test, y_test),
          shuffle         = True,
          verbose=0, callbacks=[TQDMNotebookCallback()]) #for visualization

Make sure you assign the fit function to an output variable. Then you can access that variable very easily

# visualizing losses and accuracy
train_loss = seqModel.history['loss']
val_loss   = seqModel.history['val_loss']
train_acc  = seqModel.history['acc']
val_acc    = seqModel.history['val_acc']
xc         = range(num_epochs)

plt.figure()
plt.plot(xc, train_loss)
plt.plot(xc, val_loss)

Hope this helps. source: https://keras.io/getting-started/faq/#how-can-i-record-the-training-validation-loss-accuracy-at-each-epoch

Solution 4 - Python

The dictionary with histories of "acc", "loss", etc. is available and saved in hist.history variable.

Solution 5 - Python

I have also found that you can use verbose=2 to make keras print out the Losses:

history = model.fit(X, Y, validation_split=0.33, nb_epoch=150, batch_size=10, verbose=2)

And that would print nice lines like this:

Epoch 1/1
 - 5s - loss: 0.6046 - acc: 0.9999 - val_loss: 0.4403 - val_acc: 0.9999

According to their documentation:

verbose: 0, 1, or 2. Verbosity mode. 0 = silent, 1 = progress bar, 2 = one line per epoch.

Solution 6 - Python

For plotting the loss directly the following works:

import matplotlib.pyplot as plt
...    
model_ = model.fit(X, Y, epochs= ..., verbose=1 )
plt.plot(list(model_.history.values())[0],'k-o')

Solution 7 - Python

Another option is CSVLogger: https://keras.io/callbacks/#csvlogger. It creates a csv file appending the result of each epoch. Even if you interrupt training, you get to see how it evolved.

Solution 8 - Python

Actually, you can also do it with the iteration method. Because sometimes we might need to use the iteration method instead of the built-in epochs method to visualize the training results after each iteration.

history = [] #Creating a empty list for holding the loss later
for iteration in range(1, 3):
    print()
    print('-' * 50)
    print('Iteration', iteration)
    result = model.fit(X, y, batch_size=128, nb_epoch=1) #Obtaining the loss after each training
    history.append(result.history['loss']) #Now append the loss after the training to the list.
    start_index = random.randint(0, len(text) - maxlen - 1)
print(history)

This way allows you to get the loss you want while maintaining your iteration method.

Solution 9 - Python

Thanks to Alloush,

Following parameter must be included in model.fit():

validation_data = (x_test, y_test)

If it is not defined, val_acc and val_loss will not be exist at output.

Solution 10 - Python

Those who got still error like me:

Convert model.fit_generator() to model.fit()

Solution 11 - Python

you can get loss and metrics like below: returned history object is dictionary and you can access model loss( val_loss) or accuracy(val_accuracy) like below:

model_hist=model.fit(train_data,train_lbl,epochs=my_epoch,batch_size=sel_batch_size,validation_data=val_data)

acc=model_hist.history['accuracy']

val_acc=model_hist.history['val_accuracy']

loss=model_hist.history['loss']

val_loss=model_hist.history['val_loss']

dont forget that for getting val_loss or val_accuracy you should specify validation data in the "fit" function.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionishidoView Question on Stackoverflow
Solution 1 - PythonJeremy AnifaccView Answer on Stackoverflow
Solution 2 - PythonishidoView Answer on Stackoverflow
Solution 3 - PythonRami AlloushView Answer on Stackoverflow
Solution 4 - PythonMarcin MożejkoView Answer on Stackoverflow
Solution 5 - PythonRoozbeh ZabihollahiView Answer on Stackoverflow
Solution 6 - PythonhorseshoeView Answer on Stackoverflow
Solution 7 - PythonJimmyView Answer on Stackoverflow
Solution 8 - PythonRaven CheukView Answer on Stackoverflow
Solution 9 - PythonCengiz GüngörView Answer on Stackoverflow
Solution 10 - PythonAniket SawaleView Answer on Stackoverflow
Solution 11 - PythonAli karimiView Answer on Stackoverflow