TensorFlow: Blas GEMM launch failed
PythonTensorflowKerasBlasPython Problem Overview
When I'm trying to use TensorFlow with Keras using the gpu, I'm getting this error message:
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py:2: UserWarning: Update your `fit_generator` call to the Keras 2 API: `fit_generator(<keras.pre..., 37800, epochs=2, validation_data=<keras.pre..., validation_steps=4200)`
from ipykernel import kernelapp as app
Epoch 1/2
InternalError Traceback (most recent call last)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1038 try:
-> 1039 return fn(*args)
1040 except errors.OpError as e:
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1020 feed_dict, fetch_list, target_list,
-> 1021 status, run_metadata)
1022
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\contextlib.py in __exit__(self, type, value, traceback)
65 try:
---> 66 next(self.gen)
67 except StopIteration:
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
465 compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466 pywrap_tensorflow.TF_GetCode(status))
467 finally:
InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
During handling of the above exception, another exception occurred:
InternalError Traceback (most recent call last)
<ipython-input-13-2a52d1079a66> in <module>()
1 history=model.fit_generator(batches, batches.n, nb_epoch=2,
----> 2 validation_data=val_batches, nb_val_samples=val_batches.n)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
86 warnings.warn('Update your `' + object_name +
87 '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88 return func(*args, **kwargs)
89 wrapper._legacy_support_signature = inspect.getargspec(func)
90 return wrapper
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
1108 workers=workers,
1109 pickle_safe=pickle_safe,
-> 1110 initial_epoch=initial_epoch)
1111
1112 @interfaces.legacy_generator_methods_support
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
86 warnings.warn('Update your `' + object_name +
87 '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88 return func(*args, **kwargs)
89 wrapper._legacy_support_signature = inspect.getargspec(func)
90 return wrapper
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
1888 outs = self.train_on_batch(x, y,
1889 sample_weight=sample_weight,
-> 1890 class_weight=class_weight)
1891
1892 if not isinstance(outs, list):
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in train_on_batch(self, x, y, sample_weight, class_weight)
1631 ins = x + y + sample_weights
1632 self._make_train_function()
-> 1633 outputs = self.train_function(ins)
1634 if len(outputs) == 1:
1635 return outputs[0]
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in __call__(self, inputs)
2227 session = get_session()
2228 updated = session.run(self.outputs + [self.updates_op],
-> 2229 feed_dict=feed_dict)
2230 return updated[:len(self.outputs)]
2231
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
776 try:
777 result = self._run(None, fetches, feed_dict, options_ptr,
--> 778 run_metadata_ptr)
779 if run_metadata:
780 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
980 if final_fetches or final_targets:
981 results = self._do_run(handle, final_targets, final_fetches,
--> 982 feed_dict_string, options, run_metadata)
983 else:
984 results = []
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1030 if handle is None:
1031 return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032 target_list, options, run_metadata)
1033 else:
1034 return self._do_call(_prun_fn, self._session, handle, feed_dict,
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1050 except KeyError:
1051 pass
-> 1052 raise type(e)(node_def, op, message)
1053
1054 def _extend_graph(self):
InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
Caused by op 'dense_1/MatMul', defined at:
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py", line 3, in <module>
app.launch_new_instance()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
app.start()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelapp.py", line 477, in start
ioloop.IOLoop.instance().start()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\ioloop.py", line 888, in start
handler_func(fd_obj, events)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 235, in dispatch_shell
handler(stream, idents, msg)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\zmqshell.py", line 533, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2683, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2787, in run_ast_nodes
if self.run_code(code, result):
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2847, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-10-1e7a3b259f23>", line 4, in <module>
model.add(Dense(10, activation='softmax'))
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py", line 466, in add
output_tensor = layer(self.outputs[0])
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 585, in __call__
output = self.call(inputs, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\layers\core.py", line 840, in call
output = K.dot(inputs, self.kernel)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 936, in dot
out = tf.matmul(x, y)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1801, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1263, in _mat_mul
transpose_b=transpose_b, name=name)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()
InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
When I'm trying to use TensorFlow with Keras using the cpu, I'm getting this error message:
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py:5: UserWarning: Update your `fit_generator` call to the Keras 2 API: `fit_generator(<keras.pre..., 37800, validation_steps=4200, validation_data=<keras.pre..., epochs=2)`
Epoch 1/2
---------------------------------------------------------------------------
InternalError Traceback (most recent call last)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1038 try:
-> 1039 return fn(*args)
1040 except errors.OpError as e:
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1020 feed_dict, fetch_list, target_list,
-> 1021 status, run_metadata)
1022
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\contextlib.py in __exit__(self, type, value, traceback)
65 try:
---> 66 next(self.gen)
67 except StopIteration:
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
465 compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466 pywrap_tensorflow.TF_GetCode(status))
467 finally:
InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
[[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
During handling of the above exception, another exception occurred:
InternalError Traceback (most recent call last)
<ipython-input-14-f66b4d3d5b88> in <module>()
3 with tf.device('/cpu:0'):
4 history=model.fit_generator(batches, batches.n, nb_epoch=2,
----> 5 validation_data=val_batches, nb_val_samples=val_batches.n)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
86 warnings.warn('Update your `' + object_name +
87 '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88 return func(*args, **kwargs)
89 wrapper._legacy_support_signature = inspect.getargspec(func)
90 return wrapper
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
1108 workers=workers,
1109 pickle_safe=pickle_safe,
-> 1110 initial_epoch=initial_epoch)
1111
1112 @interfaces.legacy_generator_methods_support
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
86 warnings.warn('Update your `' + object_name +
87 '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88 return func(*args, **kwargs)
89 wrapper._legacy_support_signature = inspect.getargspec(func)
90 return wrapper
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
1888 outs = self.train_on_batch(x, y,
1889 sample_weight=sample_weight,
-> 1890 class_weight=class_weight)
1891
1892 if not isinstance(outs, list):
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in train_on_batch(self, x, y, sample_weight, class_weight)
1631 ins = x + y + sample_weights
1632 self._make_train_function()
-> 1633 outputs = self.train_function(ins)
1634 if len(outputs) == 1:
1635 return outputs[0]
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in __call__(self, inputs)
2227 session = get_session()
2228 updated = session.run(self.outputs + [self.updates_op],
-> 2229 feed_dict=feed_dict)
2230 return updated[:len(self.outputs)]
2231
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
776 try:
777 result = self._run(None, fetches, feed_dict, options_ptr,
--> 778 run_metadata_ptr)
779 if run_metadata:
780 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
980 if final_fetches or final_targets:
981 results = self._do_run(handle, final_targets, final_fetches,
--> 982 feed_dict_string, options, run_metadata)
983 else:
984 results = []
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1030 if handle is None:
1031 return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032 target_list, options, run_metadata)
1033 else:
1034 return self._do_call(_prun_fn, self._session, handle, feed_dict,
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1050 except KeyError:
1051 pass
-> 1052 raise type(e)(node_def, op, message)
1053
1054 def _extend_graph(self):
InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
[[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op 'dense_1/MatMul', defined at:
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py", line 3, in <module>
app.launch_new_instance()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
app.start()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelapp.py", line 477, in start
ioloop.IOLoop.instance().start()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\ioloop.py", line 888, in start
handler_func(fd_obj, events)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 235, in dispatch_shell
handler(stream, idents, msg)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\zmqshell.py", line 533, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2683, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2787, in run_ast_nodes
if self.run_code(code, result):
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2847, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-12-1e7a3b259f23>", line 4, in <module>
model.add(Dense(10, activation='softmax'))
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py", line 466, in add
output_tensor = layer(self.outputs[0])
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 585, in __call__
output = self.call(inputs, **kwargs)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\layers\core.py", line 840, in call
output = K.dot(inputs, self.kernel)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 936, in dot
out = tf.matmul(x, y)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1801, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1263, in _mat_mul
transpose_b=transpose_b, name=name)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()
InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
[[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
In both cases, the error is with InternalError (see above for traceback): Blas GEMM launch failed Can you tell me how to get Blas GEMM to launch? I installed tensorflow and keras in a 3.5 python anaconda environment where I also installed all needed module (numpy, pandas, scipy, scikit-learn). I have a Windows 10 with a NVIDIA gpu that can use CUDA. I downloaded CUDA and cuDNN. I'm using the Jupyter notebook on Chrome.
Sometimes when I run my code, rather than having this error, I get that it starts running and then it crashes. After the crash, I can't do anything on my jupyter notebook and after some time a pop-up asks me if I want to kill the page. This is an image of what I got after the crash. !(http://www.hostingpics.net/viewer.php?id=647186tensorflowError.png)
P.S. I know my problem is similar as in this question: https://stackoverflow.com/questions/43768498/tensorflow-basic-example-error-cublas-status-not-initialized but it has not been solved there and I'm not sure this question is clear enough or is exactly the same problem as I have so I'm posting it with my own error message. This problem is different of: https://stackoverflow.com/questions/37337728/tensorflow-internalerror-blas-sgemm-launch-failed Since I have a problem with GEMM rather than SGEMM and that my problem is both with gpu and cpu and it is not solved by the answer of this question.
Python Solutions
Solution 1 - Python
This worked for me on TensorFlow 2.1.0 (per: https://www.tensorflow.org/api_docs/python/tf/config/experimental/set_memory_growth)
import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
for device in physical_devices:
tf.config.experimental.set_memory_growth(device, True)
Solution 2 - Python
It's a simple fix, but it was a nightmare to figure it all out
On Windows I found the Keras install in Anaconda3\Lib\site-packages\keras
sources:
https://www.tensorflow.org/guide/using_gpu
https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py
Find the following in your keras/tensorflow_backend.py file you'll add config.gpu_options.allow_growth= True in both places
if _SESSION is None:
if not os.environ.get('OMP_NUM_THREADS'):
config = tf.ConfigProto(allow_soft_placement=True)
config.gpu_options.allow_growth=True
else:
num_thread = int(os.environ.get('OMP_NUM_THREADS'))
config = tf.ConfigProto(intra_op_parallelism_threads=num_thread,
allow_soft_placement=True)
config.gpu_options.allow_growth=True
_SESSION = tf.Session(config=config)
session = _SESSION
Solution 3 - Python
> Make sure you have no other processes using the GPU running. Run nvidia-smi to check this.
SOURCE: An issue brought up by @reedwm.
Solution 4 - Python
Adding the following lines after imports solved the problem:
configuration = tf.compat.v1.ConfigProto()
configuration.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=configuration)
Solution 5 - Python
This Answer is much related to Tensorflow:
Sometimes Tensorflow fails at creation in Windows.
Restarting the notebook using gpu solves it in most cases
If it doesnt then try restarting the notebook after adding these options in your code.
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.9)
tf.Session(config=tf.ConfigProto(gpu_options=gpu_options,allow_soft_placement=True)
I never had such error while using Keras But try restarting your notebook
Solution 6 - Python
Had the same error. Maybe it is related to the problem that tensorflow is allocating all gpu memory. But the fix recommended there didn't work for me and it is not possible yet to limit tensorflow's gpu memory use via keras.json or commandline. Switching keras' backend to Theano resolved the issue for me (howto can be found here).
Solution 7 - Python
I was getting exactly the same error message. I realized that there was an error with my CUDA installation, specifically with the cuBLAS
library.
You can check if yours has the same problem by running the sample program simpleCUBLAS
(it comes with the CUDA installation, you will probably find it in the CUDA home folder:$CUDA_HOME\samples\7_CUDALibraries\simpleCUBLAS
)
- If you didn't download the samples or otherwise don't have them, get them on GitHub / see documentation at NVIDIA's website
Try running this program. If the test fails, you have a problem with your CUDA installation. You should try to reinstall it. That's how I solved the same problem here.
Renaming cublas64_10.dll to cublas64_100.dll may be a solution.
Solution 8 - Python
For me, closing and restarting my processes using Python worked.
I tried a few things here but they didn't work. For example, the
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
line. I think it is because I am using a newer version of Keras and Tensorflow. A lot of what I have read on the internet, include the official Keras tutorial, does not work because of version conflicts.
But I saw a couple of posts about more than one Python process running. So I shut down Jupyter, Anaconda, and PyCharm, and restarted everything. And then the error went away. It may or may not be what fixes is for you, but it is worth a try.
Solution 9 - Python
I have got the same error,lucky,I have got it fixed.
my error is: the last time,I open the tensorflow sess = tf.Session()
,but I forgot close the session.
so I open the terminal, type command:
ps -aux | grep program_name
find the PID,and type command kill the PID:
kill -9 PID
Ok,the GPU is realase.
Solution 10 - Python
I ran into this problem when trying to run several servers that use a model to serve predictions. As I wasn't training a model but simply using it, the difference between using GPU or CPU was minor. For this specific case, the issue can be avoided by forcing Tensorflow to use the CPU by "hiding" the GPU.
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1" # Force TF to use only the CPU
Solution 11 - Python
For me it was a runaway ipynb script which I thought I had terminated but was actually still running, thus my GPU was in use and this error appeared
Solution 12 - Python
Had the same error (Win10 using Keras and Visual Studio Code). Seems like TensorFlow was still active somehow even after terminating my script. Simply closing VS Code and restarting solved the issue.
Solution 13 - Python
I was stuck with this for days and I was able to get rid of this error eventually.
I had the wrong versions of tensorflow and cuda installed in my PC. Just make sure you have the right versions of tf,cuda and cudnn installed.
https://i.stack.imgur.com/Laiii.png
Use this link for reference.
Solution 14 - Python
I had a similar kind of error while inferencing on a Tensorflow model , Fixed that issue by downgrading Tensorflow from 2.1 to 1.14. Initially, I checked the GPU usage, it took all the GPU memory and couldn't able to perform the inference and found the following exception:
> InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(86494, 257), b.shape=(257, 64), m=86494, n=64, k=257 [[{{node log_mel_features/MatMul}}]]
(1) Internal: Blas GEMM launch failed : a.shape=(86494, 257), b.shape=(257, 64), m=86494, n=64, k=257
[[{{node log_mel_features/MatMul}}]]
[[log_mel_features/Log/_769]]
Bellow are the commands I used:
pip uninstall tensorflow-gpu
pip install tensorflow-gpu==1.14
Solution 15 - Python
Had python open in Dos window open in Windows 10. When running from my IDE, it gave the above message. Exiting from that Dos instance of python allowed me to get past this error when running from my IDE.
Solution 16 - Python
Try running the sample program simpleCUBLAS (it comes with CUDA) to test your CUBLAS installation and see if it works.
In my case (I am using Ubuntu) I had to reinstall CUDA to solve this issue. After I did that, simpleCUBLAS passed the test.
For some reason I started running into the same issue after a while, and I found that cleaning the directory .nv (inside my home folder) resolved the issue, and simpleCUBLAS test passed again.
Solution 17 - Python
I was using Jupyter Lab, but must have had the GPU locked from another TensofFlow program that I ran earlier. After killing Jupyter Lab and restarting it, the error went away.
Solution 18 - Python
I had the same error in Win10 using PyCharm on Keras 2.4.3 and TensorFlow 2.3.0
It seems like an error related to TensorFlow itself running on Windows. Solved the issue by closing PyCharm and rerun it as administrator.
Solution 19 - Python
The error occurs when your GPU memory is full, so allow memory growth of GPU, and the problem will be fixed. You can use the below code snippet:
physical_devices = tf.config.experimental.list_physical_devices('GPU')
for device in physical_devices:
tf.config.experimental.set_memory_growth(device, True)