I was running TensorFlow and I happen to have something yielding a NaN. I&#39;d like to know what it is but I do not know how to do this. The main issue is that in a &quot;normal&quot; procedural program I would just write a print statement just before the operation is executed. The issue with TensorFlow is that I cannot do that because I first declare (or define) the graph, so adding print statements to the graph definition does not help. Are there any rules, advice, heuristics, anything to track down what might be causing the NaN?

----

In this case I know more precisely what line to look at because I have the following:

    Delta_tilde = 2.0*tf.matmul(x,W) - tf.add(WW, XX) #note this quantity should always be positive because its pair-wise euclidian distance
    Z = tf.sqrt(Delta_tilde)
    Z = Transform(Z) # potentially some transform, currently I have it to return Z for debugging (the identity)
    Z = tf.pow(Z, 2.0)
    A = tf.exp(Z) 

when this line is present I have it that it returns NaN as declared by my summary writers. Why is this? Is there a way to at least explore what value Z has after its being square rooted?

----

For the specific example I posted, I tried `tf.Print(0,Z)` but with no success it printed nothing. As in:

    Delta_tilde = 2.0*tf.matmul(x,W) - tf.add(WW, XX) #note this quantity should always be positive because its pair-wise euclidian distance
    Z = tf.sqrt(Delta_tilde)
    tf.Print(0,[Z]) # &lt;-------- TF PRINT STATMENT
    Z = Transform(Z) # potentially some transform, currently I have it to return Z for debugging (the identity)
    Z = tf.pow(Z, 2.0)
    A = tf.exp(Z) 

I actually don&#39;t understand what `tf.Print` is suppose to do. Why does it need two arguments? If I want to print 1 tensor why would I need to pass 2? Seems bizarre to me.

----

I was looking at the function [tf.add_check_numerics_ops()][1] but it doesn&#39;t say how to use it (plus the docs seem to not be super helpful). Does anyone know how to use this?

----

Since I&#39;ve had comments addressing the data might be bad, I am using standard MNIST. However, I am computing a quantity that is positive (pair-wise eucledian distance) and then square rooting it. Thus, I wouldn&#39;t see how the data specifically would be an issue.



  [1]: https://www.tensorflow.org/versions/r0.9/api_docs/python/control_flow_ops.html#add_check_numerics_ops

How does one debug NaN values in TensorFlow?

I have searched many places but ALL I get is HOW to install it, not how to verify that it is installed. I can verify my NVIDIA driver is installed, and that CUDA is installed, but I don&#39;t know how to verify CuDNN is installed. Help will be much appreciated, thanks!

PS.  
This is for a caffe implementation. Currently everything is working without CuDNN enabled.

How to verify CuDNN installation?

*The original question was in regard to TensorFlow implementations specifically. However, the answers are for implementations in general. This general answer is also the correct answer for TensorFlow.*

When using batch normalization and dropout in TensorFlow (specifically using the contrib.layers) do I need to be worried about the ordering?

It seems possible that if I use dropout followed immediately by batch normalization there might be trouble. For example, if the shift in the batch normalization trains to the larger scale numbers of the training outputs, but then that same shift is applied to the smaller (due to the compensation for having more outputs) scale numbers without dropout during testing, then that shift may be off. Does the TensorFlow batch normalization layer automatically compensate for this? Or does this not happen for some reason I&#39;m missing?

Also, are there other pitfalls to look out for in when using these two together? For example, assuming I&#39;m using them in the correct order in regards to the above (assuming there *is* a correct order), could there be trouble with using both batch normalization and dropout on multiple successive layers? I don&#39;t immediately see a problem with that, but I might be missing something.

Thank you much!

**UPDATE:**

An experimental test *seems* to suggest that ordering *does* matter. I ran the same network twice with only the batch norm and dropout reverse. When the dropout is before the batch norm, validation loss seems to be going up as training loss is going down. They&#39;re both going down in the other case. But in my case the movements are slow, so things may change after more training and it&#39;s just a single test. A more definitive and informed answer would still be appreciated.

Ordering of batch normalization and dropout?

I&#39;m trying to train a CNN to categorize text by topic. When I use binary cross-entropy I get ~80% accuracy, with categorical cross-entropy I get ~50% accuracy.

I don&#39;t understand why this is. It&#39;s a multiclass problem, doesn&#39;t that mean that I have to use categorical cross-entropy and that the results with binary cross-entropy are meaningless?

&lt;!-- language-all: lang-python --&gt;

    model.add(embedding_layer)
    model.add(Dropout(0.25))
    # convolution layers
    model.add(Conv1D(nb_filter=32,
                        filter_length=4,
                        border_mode=&#39;valid&#39;,
                        activation=&#39;relu&#39;))
    model.add(MaxPooling1D(pool_length=2))
    # dense layers
    model.add(Flatten())
    model.add(Dense(256))
    model.add(Dropout(0.25))
    model.add(Activation(&#39;relu&#39;))
    # output layer
    model.add(Dense(len(class_id_index)))
    model.add(Activation(&#39;softmax&#39;))

Then I compile it either it like this using `categorical_crossentropy` as the loss function:

    model.compile(loss=&#39;categorical_crossentropy&#39;, optimizer=&#39;adam&#39;, metrics=[&#39;accuracy&#39;])

or 

    model.compile(loss=&#39;binary_crossentropy&#39;, optimizer=&#39;adam&#39;, metrics=[&#39;accuracy&#39;])

Intuitively it makes sense why I&#39;d want to use categorical cross-entropy, I don&#39;t understand why I get good results with binary, and poor results with categorical.

Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?

Can anyone please clearly explain the difference between 1D, 2D, and 3D convolutions in convolutional neural networks (in deep learning) with the use of examples?

Intuitive understanding of 1D, 2D, and 3D convolutions in convolutional neural networks

I am trying to understand the **strides** argument in tf.nn.avg_pool, tf.nn.max_pool, tf.nn.conv2d. 

The [documentation][1] repeatedly says 

&gt; strides: A list of ints that has length &gt;= 4. The stride of the sliding window for each dimension of the input tensor.

My questions are:

 1. What do each of the 4+ integers represent?
 2. Why must they have strides\[0\] = strides\[3\] = 1 for convnets?
 3. In [this example][2] we see `tf.reshape(_X,shape=[-1, 28, 28, 1])`. Why -1?

Sadly the examples in the docs for reshape using -1 don&#39;t translate too well to this scenario.

  [1]: https://www.tensorflow.org/versions/master/api_docs/python/nn.html#max_pool
  [2]: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3%20-%20Neural%20Networks/convolutional_network.ipynb

Tensorflow Strides Argument

I&#39;m training a neural network for my project using Keras. Keras has provided a function for early stopping. May I know what parameters should be observed to avoid my neural network from overfitting by using early stopping?

Which parameters should be used for early stopping?

Currently I use the following code:
            
    callbacks = [
    	EarlyStopping(monitor=&#39;val_loss&#39;, patience=2, verbose=0),
    	ModelCheckpoint(kfold_weights_path, monitor=&#39;val_loss&#39;, save_best_only=True, verbose=0),
    ]
    model.fit(X_train.astype(&#39;float32&#39;), Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
    	  shuffle=True, verbose=1, validation_data=(X_valid, Y_valid),
    	  callbacks=callbacks)

It tells Keras to stop training when loss didn&#39;t improve for 2 epochs. But I want to stop training after loss became smaller than some constant &quot;THR&quot;:

    if val_loss &lt; THR:
        break

I&#39;ve seen in documentation there are possibility to make your own callback:
http://keras.io/callbacks/
But nothing found how to stop training process. I need an advice.

How to tell Keras stop training based on loss value?

I am newbie in convolutional neural networks and just have idea about feature maps and how convolution is done on images to extract features. I would be glad to know some details on applying batch normalisation in CNN.

I read this paper https://arxiv.org/pdf/1502.03167v3.pdf and could understand the BN algorithm applied on a data but in the end they mentioned that a slight modification is required when applied to CNN:

&gt; For convolutional layers, we additionally want the normalization to obey the convolutional property – so that different elements of the same feature map, at different locations, are normalized in the same way. To achieve this, we jointly normalize all the activations in a mini- batch, over all locations. In Alg. 1, we let B be the set of all values in a feature map across both the elements of a mini-batch and spatial locations – so for a mini-batch of size m and feature maps of size p &#215; q, we use the effec- tive mini-batch of size m′ = |B| = m &#183; pq. We learn a pair of parameters γ(k) and β(k) per feature map, rather than per activation. Alg. 2 is modified similarly, so that during inference the BN transform applies the same linear transformation to each activation in a given feature map.


I am total confused when they say
**&quot;so that different elements of the same feature map, at different locations, are normalized in the same way&quot;**

I know what feature maps mean and different elements are the weights in every feature map. But I could not understand what location or spatial location means.

 I could not understand the below sentence at all
**&quot;In Alg. 1, we let B be the set of all values in a feature map across both the elements of a mini-batch and spatial locations&quot;**

I would be glad if someone cold elaborate and explain me in much simpler terms 

Batch Normalization in Convolutional Neural Network

I understand that Batch Normalisation helps in faster training by turning the activation towards unit Gaussian distribution and thus tackling vanishing gradients problem. Batch norm acts is applied differently at training(use mean/var from each batch) and test time (use finalized running mean/var from training phase).

Instance normalisation, on the other hand, acts as contrast normalisation as mentioned in this paper https://arxiv.org/abs/1607.08022 . The authors mention that the output stylised images should be not depend on the contrast of the input content image and hence Instance normalisation helps. 

But then should we not also use instance normalisation for image classification where class label should not depend on the contrast of input image. I have not seen any paper using instance normalisation in-place of batch normalisation for classification. What is the reason for that? Also, can and should batch and instance normalisation be used together. I am eager to get an intuitive as well as theoretical understanding of when to use which normalisation. 

Instance Normalisation vs Batch normalisation

I&#39;m using Lasagne to create a CNN for the MNIST dataset. I&#39;m following closely to this example: [Convolutional Neural Networks and Feature Extraction with Python][1].

The CNN architecture I have at the moment, which doesn&#39;t include any dropout layers, is:

    NeuralNet(
        layers=[(&#39;input&#39;, layers.InputLayer),        # Input Layer
                (&#39;conv2d1&#39;, layers.Conv2DLayer),     # Convolutional Layer
                (&#39;maxpool1&#39;, layers.MaxPool2DLayer), # 2D Max Pooling Layer
                (&#39;conv2d2&#39;, layers.Conv2DLayer),     # Convolutional Layer
                (&#39;maxpool2&#39;, layers.MaxPool2DLayer), # 2D Max Pooling Layer
                (&#39;dense&#39;, layers.DenseLayer),        # Fully connected layer
                (&#39;output&#39;, layers.DenseLayer),       # Output Layer
                ],
        # input layer
        input_shape=(None, 1, 28, 28),

        # layer conv2d1
        conv2d1_num_filters=32,
        conv2d1_filter_size=(5, 5),
        conv2d1_nonlinearity=lasagne.nonlinearities.rectify,

        # layer maxpool1
        maxpool1_pool_size=(2, 2),

        # layer conv2d2
        conv2d2_num_filters=32,
        conv2d2_filter_size=(3, 3),
        conv2d2_nonlinearity=lasagne.nonlinearities.rectify,

        # layer maxpool2
        maxpool2_pool_size=(2, 2),


        # Fully Connected Layer
        dense_num_units=256,
        dense_nonlinearity=lasagne.nonlinearities.rectify,

       # output Layer
        output_nonlinearity=lasagne.nonlinearities.softmax,
        output_num_units=10,

        # optimization method params
        update= momentum,
        update_learning_rate=0.01,
        update_momentum=0.9,
        max_epochs=10,
        verbose=1,
        )

This outputs the following Layer Information:

      #  name      size
    ---  --------  --------
      0  input     1x28x28
      1  conv2d1   32x24x24
      2  maxpool1  32x12x12
      3  conv2d2   32x10x10
      4  maxpool2  32x5x5
      5  dense     256
      6  output    10

and outputs the number of learnable parameters as **217,706**

I&#39;m wondering how this number is calculated? I&#39;ve read a number of resources, including this StackOverflow&#39;s [question][2], but none clearly generalizes the calculation.

If possible, _can the calculation of the learnable parameters per layer be generalised?_ 

For example, convolutional layer: number of filters x filter width x filter height.

  [1]: http://blog.christianperone.com/2015/08/convolutional-neural-networks-and-feature-extraction-with-python/
  [2]: https://stackoverflow.com/questions/28232235/how-to-calculate-the-number-of-parameters-of-convolutional-neural-networkscnns

How to calculate the number of parameters for convolutional neural network?

I&#39;m going through the [neural transfer pytorch tutorial][1] and am confused about the use of `retain_variable`(deprecated, now referred to as `retain_graph`). The code example show:

    class ContentLoss(nn.Module):
    
        def __init__(self, target, weight):
            super(ContentLoss, self).__init__()
            self.target = target.detach() * weight
            self.weight = weight
            self.criterion = nn.MSELoss()
    
        def forward(self, input):
            self.loss = self.criterion(input * self.weight, self.target)
            self.output = input
            return self.output
    
        def backward(self, retain_variables=True):
            #Why is retain_variables True??
            self.loss.backward(retain_variables=retain_variables)
            return self.loss

From [the documentation][2]


&gt; retain_graph (bool, optional) – If False, the graph used to compute
&gt; the grad will be freed. Note that in nearly all cases setting this
&gt; option to True is not needed and often can be worked around in a much
&gt; more efficient way. Defaults to the value of create_graph.

So by setting `retain_graph= True`, we&#39;re not freeing the memory allocated for the graph on the backward pass. What is the advantage of keeping this memory around, why do we need it?

  [1]: http://pytorch.org/tutorials/advanced/neural_style_tutorial.html#packages
  [2]: http://pytorch.org/docs/master/autograd.html?highlight=backward#torch.autograd.backward

What does the parameter retain_graph mean in the Variable&#39;s backward() method?

I have tried to build a CNN with one layer, but I have some problem with it.
Indeed, the compilator says me that

&gt; ValueError: Error when checking model input: expected conv1d_1_input
&gt; to have 3 dimensions, but got array with shape (569, 30)

This is the code

```python
import numpy
from keras.models import Sequential
from keras.layers.convolutional import Conv1D

numpy.random.seed(7)

datasetTraining = numpy.loadtxt(&quot;CancerAdapter.csv&quot;,delimiter=&quot;,&quot;)
X = datasetTraining[:,1:31]
Y = datasetTraining[:,0]
datasetTesting = numpy.loadtxt(&quot;CancereEvaluation.csv&quot;,delimiter=&quot;,&quot;)
X_test = datasetTraining[:,1:31]
Y_test = datasetTraining[:,0]

model = Sequential()
model.add(Conv1D(2,2,activation=&#39;relu&#39;,input_shape=X.shape))
model.compile(loss=&#39;binary_crossentropy&#39;, optimizer=&#39;adam&#39;, metrics=[&#39;accuracy&#39;])
model.fit(X, Y, epochs=150, batch_size=5)
scores = model.evaluate(X_test, Y_test)

print(&quot;\n%s: %.2f%%&quot; % (model.metrics_names[1], scores[1]*100))
```

Dimension of shape in conv1D

I am using Keras (with Theano) to train my CNN model. Does anyone has idea how can I use it in my C++ application? Does anyone tried something similar? I have idea to write some python code that will generate a c++ code with network functions - any suggestion on it?

I found a similar question [here][1] how to use Tensorflow Keras model in C++ but without answer.


  [1]: https://stackoverflow.com/questions/36412098/convert-keras-model-to-tensorflow-protobuf

Convert Keras model to C++

How do I calculate the output size in a convolution layer?

For example, I have a 2D convolution layer that takes a 3x128x128 input and has 40 filters of size 5x5.

Calculate the output size in convolution layer

I know that, in the 1D case, the convolution between two vectors, `a` and `b`, can be computed as `conv(a, b)`, but also as the product between the `T_a` and `b`, where `T_a` is the corresponding Toeplitz matrix for `a`.

Is it possible to extend this idea to 2D?

Given `a = [5 1 3; 1 1 2; 2 1 3]` and `b=[4 3; 1 2]`, is it possible to convert `a` in a Toeplitz matrix and compute the matrix-matrix product between `T_a` and `b` as in the 1-D case?

2-D convolution as a matrix-matrix multiplication

I have checked all the solutions, but still, I am facing the same error. My training images shape is `(26721, 32, 32, 1)`, which I believe it is 4 dimension, but I don&#39;t know why error shows it is 5 dimension. 

     model = Sequential()
        
     model.add(Convolution2D(16, 5, 5, border_mode=&#39;same&#39;, input_shape= input_shape ))

So this is how I am defining `model.fit_generator` 

    model.fit_generator(train_dataset, train_labels, nb_epoch=epochs, verbose=1,validation_data=(valid_dataset, valid_labels), nb_val_samples=valid_dataset.shape[0],callbacks=model_callbacks)


All Conv Neural-Network Solutions on Gang of Coders

Total of 17 Conv Neural-Network Solutions