TL;DR using relative path as
projector will cause TensorBoard cannot find metadata. The correct way is use FQPN (fully-qualified path name, aka absolute path)
How it happened
Follow code copied from TensorFlow official website https://www.tensorflow.org/versions/r1.3/programmers_guide/embedding, which is also the official approach to use TensorFlow’s embedding projector. This is also most people actually do in their code.
from tensorflow.contrib.tensorboard.plugins import projector # Create randomly initialized embedding weights which will be trained. vocabulary_size = 10000 embedding_size = 200 embedding_var = tf.get_variable('word_embedding', [vocabulary_size, embedding_size]) # Format: tensorflow/tensorboard/plugins/projector/projector_config.proto config = projector.ProjectorConfig() # You can add multiple embeddings. Here we add only one. embedding = config.embeddings.add() embedding.tensor_name = embedding_var.name # Link this tensor to its metadata file (e.g. labels). embedding.metadata_path = os.path.join(LOG_DIR, 'metadata.tsv') # Use the same LOG_DIR where you stored your checkpoint. summary_writer = tf.summary.FileWriter(LOG_DIR) # The next line writes a projector_config.pbtxt in the LOG_DIR. TensorBoard will # read this file during startup. projector.visualize_embeddings(summary_writer, config)
Above code binding tensor
embedding_var to it’s metadata file location by
metadata_path, then the config information will write to disk by
The problem is the path of metadata file assign to
embedding.metadata_path is relative path. If current path is log directory (i.e:
LOG_DIR in source code), program can find the metadata file. But people usually don’t start TensorBoard in log directory, most people like me usually use
TensorBoard --logdir ./log to start TensorBoard, in such condition, TensorBoard read the config and get the location of metadata file, but TensorBoard can not find the file by this location info, this is because location is correct only if your current work path is log directory, which most likely not true. Then TensorBoard cannot find the metadata file, loading process will never be done.
PS Someone also notice this issue and give same solution