TensorFlow: Blas GEMM launch failed

PythonTensorflowKerasBlas

Python Problem Overview


When I'm trying to use TensorFlow with Keras using the gpu, I'm getting this error message:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py:2: UserWarning: Update your `fit_generator` call to the Keras 2 API: `fit_generator(<keras.pre..., 37800, epochs=2, validation_data=<keras.pre..., validation_steps=4200)`
  from ipykernel import kernelapp as app

Epoch 1/2

InternalError                             Traceback (most recent call last)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1038     try:
-> 1039       return fn(*args)
   1040     except errors.OpError as e:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1020                                  feed_dict, fetch_list, target_list,
-> 1021                                  status, run_metadata)
   1022 

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\contextlib.py in __exit__(self, type, value, traceback)
     65             try:
---> 66                 next(self.gen)
     67             except StopIteration:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
    465           compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466           pywrap_tensorflow.TF_GetCode(status))
    467   finally:

InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
	 [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]

During handling of the above exception, another exception occurred:

InternalError                             Traceback (most recent call last)
<ipython-input-13-2a52d1079a66> in <module>()
      1 history=model.fit_generator(batches, batches.n, nb_epoch=2, 
----> 2                     validation_data=val_batches, nb_val_samples=val_batches.n)

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
   1108                                         workers=workers,
   1109                                         pickle_safe=pickle_safe,
-> 1110                                         initial_epoch=initial_epoch)
   1111 
   1112     @interfaces.legacy_generator_methods_support

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
   1888                     outs = self.train_on_batch(x, y,
   1889                                                sample_weight=sample_weight,
-> 1890                                                class_weight=class_weight)
   1891 
   1892                     if not isinstance(outs, list):

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in train_on_batch(self, x, y, sample_weight, class_weight)
   1631             ins = x + y + sample_weights
   1632         self._make_train_function()
-> 1633         outputs = self.train_function(ins)
   1634         if len(outputs) == 1:
   1635             return outputs[0]

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in __call__(self, inputs)
   2227         session = get_session()
   2228         updated = session.run(self.outputs + [self.updates_op],
-> 2229                               feed_dict=feed_dict)
   2230         return updated[:len(self.outputs)]
   2231 

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
    776     try:
    777       result = self._run(None, fetches, feed_dict, options_ptr,
--> 778                          run_metadata_ptr)
    779       if run_metadata:
    780         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    980     if final_fetches or final_targets:
    981       results = self._do_run(handle, final_targets, final_fetches,
--> 982                              feed_dict_string, options, run_metadata)
    983     else:
    984       results = []

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1030     if handle is None:
   1031       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032                            target_list, options, run_metadata)
   1033     else:
   1034       return self._do_call(_prun_fn, self._session, handle, feed_dict,

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1050         except KeyError:
   1051           pass
-> 1052       raise type(e)(node_def, op, message)
   1053 
   1054   def _extend_graph(self):

InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
	 [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]

Caused by op 'dense_1/MatMul', defined at:
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2683, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2787, in run_ast_nodes
    if self.run_code(code, result):
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2847, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-10-1e7a3b259f23>", line 4, in <module>
    model.add(Dense(10, activation='softmax'))
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py", line 466, in add
    output_tensor = layer(self.outputs[0])
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 585, in __call__
    output = self.call(inputs, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\layers\core.py", line 840, in call
    output = K.dot(inputs, self.kernel)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 936, in dot
    out = tf.matmul(x, y)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1801, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1263, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
	 [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]

When I'm trying to use TensorFlow with Keras using the cpu, I'm getting this error message:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py:5: UserWarning: Update your `fit_generator` call to the Keras 2 API: `fit_generator(<keras.pre..., 37800, validation_steps=4200, validation_data=<keras.pre..., epochs=2)`
Epoch 1/2
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1038     try:
-> 1039       return fn(*args)
   1040     except errors.OpError as e:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1020                                  feed_dict, fetch_list, target_list,
-> 1021                                  status, run_metadata)
   1022 

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\contextlib.py in __exit__(self, type, value, traceback)
     65             try:
---> 66                 next(self.gen)
     67             except StopIteration:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
    465           compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466           pywrap_tensorflow.TF_GetCode(status))
    467   finally:

InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
	 [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
	 [[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

During handling of the above exception, another exception occurred:

InternalError                             Traceback (most recent call last)
<ipython-input-14-f66b4d3d5b88> in <module>()
      3 with tf.device('/cpu:0'):
      4     history=model.fit_generator(batches, batches.n, nb_epoch=2, 
----> 5                     validation_data=val_batches, nb_val_samples=val_batches.n)

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
   1108                                         workers=workers,
   1109                                         pickle_safe=pickle_safe,
-> 1110                                         initial_epoch=initial_epoch)
   1111 
   1112     @interfaces.legacy_generator_methods_support

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
   1888                     outs = self.train_on_batch(x, y,
   1889                                                sample_weight=sample_weight,
-> 1890                                                class_weight=class_weight)
   1891 
   1892                     if not isinstance(outs, list):

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in train_on_batch(self, x, y, sample_weight, class_weight)
   1631             ins = x + y + sample_weights
   1632         self._make_train_function()
-> 1633         outputs = self.train_function(ins)
   1634         if len(outputs) == 1:
   1635             return outputs[0]

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in __call__(self, inputs)
   2227         session = get_session()
   2228         updated = session.run(self.outputs + [self.updates_op],
-> 2229                               feed_dict=feed_dict)
   2230         return updated[:len(self.outputs)]
   2231 

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
    776     try:
    777       result = self._run(None, fetches, feed_dict, options_ptr,
--> 778                          run_metadata_ptr)
    779       if run_metadata:
    780         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    980     if final_fetches or final_targets:
    981       results = self._do_run(handle, final_targets, final_fetches,
--> 982                              feed_dict_string, options, run_metadata)
    983     else:
    984       results = []

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1030     if handle is None:
   1031       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032                            target_list, options, run_metadata)
   1033     else:
   1034       return self._do_call(_prun_fn, self._session, handle, feed_dict,

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1050         except KeyError:
   1051           pass
-> 1052       raise type(e)(node_def, op, message)
   1053 
   1054   def _extend_graph(self):

InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
	 [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
	 [[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op 'dense_1/MatMul', defined at:
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2683, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2787, in run_ast_nodes
    if self.run_code(code, result):
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2847, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-12-1e7a3b259f23>", line 4, in <module>
    model.add(Dense(10, activation='softmax'))
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py", line 466, in add
    output_tensor = layer(self.outputs[0])
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 585, in __call__
    output = self.call(inputs, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\layers\core.py", line 840, in call
    output = K.dot(inputs, self.kernel)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 936, in dot
    out = tf.matmul(x, y)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1801, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1263, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
	 [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
	 [[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

In both cases, the error is with InternalError (see above for traceback): Blas GEMM launch failed Can you tell me how to get Blas GEMM to launch? I installed tensorflow and keras in a 3.5 python anaconda environment where I also installed all needed module (numpy, pandas, scipy, scikit-learn). I have a Windows 10 with a NVIDIA gpu that can use CUDA. I downloaded CUDA and cuDNN. I'm using the Jupyter notebook on Chrome.

Sometimes when I run my code, rather than having this error, I get that it starts running and then it crashes. After the crash, I can't do anything on my jupyter notebook and after some time a pop-up asks me if I want to kill the page. This is an image of what I got after the crash. !(http://www.hostingpics.net/viewer.php?id=647186tensorflowError.png)

P.S. I know my problem is similar as in this question: https://stackoverflow.com/questions/43768498/tensorflow-basic-example-error-cublas-status-not-initialized but it has not been solved there and I'm not sure this question is clear enough or is exactly the same problem as I have so I'm posting it with my own error message. This problem is different of: https://stackoverflow.com/questions/37337728/tensorflow-internalerror-blas-sgemm-launch-failed Since I have a problem with GEMM rather than SGEMM and that my problem is both with gpu and cpu and it is not solved by the answer of this question.

Python Solutions


Solution 1 - Python

This worked for me on TensorFlow 2.1.0 (per: https://www.tensorflow.org/api_docs/python/tf/config/experimental/set_memory_growth)

import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU') 
for device in physical_devices:
    tf.config.experimental.set_memory_growth(device, True)

Solution 2 - Python

It's a simple fix, but it was a nightmare to figure it all out

On Windows I found the Keras install in Anaconda3\Lib\site-packages\keras

sources:

https://www.tensorflow.org/guide/using_gpu

https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py

Find the following in your keras/tensorflow_backend.py file you'll add config.gpu_options.allow_growth= True in both places

if _SESSION is None:
            if not os.environ.get('OMP_NUM_THREADS'):
                config = tf.ConfigProto(allow_soft_placement=True)
                config.gpu_options.allow_growth=True
            else:
                num_thread = int(os.environ.get('OMP_NUM_THREADS'))
                config = tf.ConfigProto(intra_op_parallelism_threads=num_thread,
                                        allow_soft_placement=True)
                config.gpu_options.allow_growth=True
            _SESSION = tf.Session(config=config)
        session = _SESSION

Solution 3 - Python

> Make sure you have no other processes using the GPU running. Run nvidia-smi to check this.

SOURCE: An issue brought up by @reedwm.

Solution 4 - Python

Adding the following lines after imports solved the problem:

configuration = tf.compat.v1.ConfigProto()
configuration.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=configuration)

Solution 5 - Python

This Answer is much related to Tensorflow:

Sometimes Tensorflow fails at creation in Windows.

Restarting the notebook using gpu solves it in most cases

If it doesnt then try restarting the notebook after adding these options in your code.

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.9)

tf.Session(config=tf.ConfigProto(gpu_options=gpu_options,allow_soft_placement=True)

I never had such error while using Keras But try restarting your notebook

Solution 6 - Python

Had the same error. Maybe it is related to the problem that tensorflow is allocating all gpu memory. But the fix recommended there didn't work for me and it is not possible yet to limit tensorflow's gpu memory use via keras.json or commandline. Switching keras' backend to Theano resolved the issue for me (howto can be found here).

Solution 7 - Python

I was getting exactly the same error message. I realized that there was an error with my CUDA installation, specifically with the cuBLAS library.

You can check if yours has the same problem by running the sample program simpleCUBLAS (it comes with the CUDA installation, you will probably find it in the CUDA home folder:$CUDA_HOME\samples\7_CUDALibraries\simpleCUBLAS)

Try running this program. If the test fails, you have a problem with your CUDA installation. You should try to reinstall it. That's how I solved the same problem here.

Renaming cublas64_10.dll to cublas64_100.dll may be a solution.

Solution 8 - Python

For me, closing and restarting my processes using Python worked.

I tried a few things here but they didn't work. For example, the

    os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

line. I think it is because I am using a newer version of Keras and Tensorflow. A lot of what I have read on the internet, include the official Keras tutorial, does not work because of version conflicts.

But I saw a couple of posts about more than one Python process running. So I shut down Jupyter, Anaconda, and PyCharm, and restarted everything. And then the error went away. It may or may not be what fixes is for you, but it is worth a try.

Solution 9 - Python

I have got the same error,lucky,I have got it fixed. my error is: the last time,I open the tensorflow sess = tf.Session(),but I forgot close the session.

so I open the terminal, type command:

ps -aux | grep program_name

find the PID,and type command kill the PID:

kill -9 PID

Ok,the GPU is realase.

Solution 10 - Python

I ran into this problem when trying to run several servers that use a model to serve predictions. As I wasn't training a model but simply using it, the difference between using GPU or CPU was minor. For this specific case, the issue can be avoided by forcing Tensorflow to use the CPU by "hiding" the GPU.

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"  # Force TF to use only the CPU

Solution 11 - Python

For me it was a runaway ipynb script which I thought I had terminated but was actually still running, thus my GPU was in use and this error appeared

Solution 12 - Python

Had the same error (Win10 using Keras and Visual Studio Code). Seems like TensorFlow was still active somehow even after terminating my script. Simply closing VS Code and restarting solved the issue.

Solution 13 - Python

I was stuck with this for days and I was able to get rid of this error eventually.

I had the wrong versions of tensorflow and cuda installed in my PC. Just make sure you have the right versions of tf,cuda and cudnn installed.

https://i.stack.imgur.com/Laiii.png

Use this link for reference.

Solution 14 - Python

I had a similar kind of error while inferencing on a Tensorflow model , Fixed that issue by downgrading Tensorflow from 2.1 to 1.14. Initially, I checked the GPU usage, it took all the GPU memory and couldn't able to perform the inference and found the following exception:

> InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(86494, 257), b.shape=(257, 64), m=86494, n=64, k=257 [[{{node log_mel_features/MatMul}}]]
(1) Internal: Blas GEMM launch failed : a.shape=(86494, 257), b.shape=(257, 64), m=86494, n=64, k=257 [[{{node log_mel_features/MatMul}}]] [[log_mel_features/Log/_769]] Bellow are the commands I used:

pip uninstall tensorflow-gpu
pip install tensorflow-gpu==1.14

Solution 15 - Python

Had python open in Dos window open in Windows 10. When running from my IDE, it gave the above message. Exiting from that Dos instance of python allowed me to get past this error when running from my IDE.

Solution 16 - Python

Try running the sample program simpleCUBLAS (it comes with CUDA) to test your CUBLAS installation and see if it works.

In my case (I am using Ubuntu) I had to reinstall CUDA to solve this issue. After I did that, simpleCUBLAS passed the test.

For some reason I started running into the same issue after a while, and I found that cleaning the directory .nv (inside my home folder) resolved the issue, and simpleCUBLAS test passed again.

Solution 17 - Python

I was using Jupyter Lab, but must have had the GPU locked from another TensofFlow program that I ran earlier. After killing Jupyter Lab and restarting it, the error went away.

Solution 18 - Python

I had the same error in Win10 using PyCharm on Keras 2.4.3 and TensorFlow 2.3.0

It seems like an error related to TensorFlow itself running on Windows. Solved the issue by closing PyCharm and rerun it as administrator.

Solution 19 - Python

The error occurs when your GPU memory is full, so allow memory growth of GPU, and the problem will be fixed. You can use the below code snippet:

physical_devices = tf.config.experimental.list_physical_devices('GPU')

for device in physical_devices:
    tf.config.experimental.set_memory_growth(device, True)

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionNicolasView Question on Stackoverflow
Solution 1 - PythonsocasantaView Answer on Stackoverflow
Solution 2 - PythonLinda MacPhee-CobbView Answer on Stackoverflow
Solution 3 - PythonliwyView Answer on Stackoverflow
Solution 4 - Pythonburhan rashidView Answer on Stackoverflow
Solution 5 - PythonMichael YadidyaView Answer on Stackoverflow
Solution 6 - Pythonfotis jView Answer on Stackoverflow
Solution 7 - PythonTaironeView Answer on Stackoverflow
Solution 8 - PythonPaul J AbernathyView Answer on Stackoverflow
Solution 9 - Pythonxiaxia wangView Answer on Stackoverflow
Solution 10 - PythonEdgar HView Answer on Stackoverflow
Solution 11 - Pythonuser5305519View Answer on Stackoverflow
Solution 12 - PythonPeterView Answer on Stackoverflow
Solution 13 - Pythonenvi zView Answer on Stackoverflow
Solution 14 - PythonAkib KhanView Answer on Stackoverflow
Solution 15 - PythondemongolemView Answer on Stackoverflow
Solution 16 - PythonTaironeView Answer on Stackoverflow
Solution 17 - PythonNikhil GuptaView Answer on Stackoverflow
Solution 18 - PythonharunuzView Answer on Stackoverflow
Solution 19 - PythonPurnendu ShuklaView Answer on Stackoverflow