Each Answer to this Q is separated by one/two green lines.
The standard way of initializing variables in TensorFlow is
init = tf.initialize_all_variables() sess = tf.Session() sess.run(init)
After running some learning for a while I create a new set of variables but once I initialize them it resets all my existing variables. At the moment my way around this is to save all the variable I need and then reapply them after the tf.initalize_all_variables call. This works but is a bit ugly and clunky. I cannot find anything like this in the docs…
Does anyone know of any good way to just initialize the uninitialized variables?
There is no elegant* way to enumerate the uninitialized variables in a graph. However, if you have access to the new variable objects—let’s call them
v_8—you can selectively initialize them using
init_new_vars_op = tf.initialize_variables([v_6, v_7, v_8]) sess.run(init_new_vars_op)
* A process of trial and error could be used to identify the uninitialized variables, as follows:
uninitialized_vars =  for var in tf.all_variables(): try: sess.run(var) except tf.errors.FailedPreconditionError: uninitialized_vars.append(var) init_new_vars_op = tf.initialize_variables(uninitialized_vars) # ...
…however, I would not condone such behavior :-).
UPDATE: TensorFlow 0.9 has a new method that “fixes” all this but only if you’re using a VariableScope with
reuse set to
True. tf.report_uninitialized_variables which can be used in one line with
sess.run( tf.initialize_variables( list( tf.get_variable(name) for name in sess.run( tf.report_uninitialized_variables( tf.all_variables( ) ) ) ) ) )
or more intelligently through the ability to specify the variables you expect to be initialized:
def guarantee_initialized_variables(session, list_of_variables = None): if list_of_variables is None: list_of_variables = tf.all_variables() uninitialized_variables = list(tf.get_variable(name) for name in session.run(tf.report_uninitialized_variables(list_of_variables))) session.run(tf.initialize_variables(uninitialized_variables)) return unintialized_variables
This is still less ideal than actually knowing which variables are and are not initialized and taking care of that properly, but in the case of misdirection like the
optim classes (see below) it may be hard to avoid.
Also note, tf.initialize_variables cannot evaluate tf.report_uninitialized_variables, so both of them have to be run within the context of the session to work.
There is an inelegant but concise way to do it. Before introducing your new variables run
temp = set(tf.all_variables()) and afterwards run
sess.run(tf.initialize_variables(set(tf.all_variables()) - temp)). These together will only initialize any variables created after the temp value is assigned.
I’ve been playing with transfer learning, so I wanted a quick way to do it too, but this is the best way I could find. Especially when using things like AdamOptimizer, which doesn’t give you easy (or any, I’m not sure) access to the variables it uses. So the following actually shows up in my code. (I initialize the new layer’s variables explicitly, and run it once to show the initial error before transfer learning. Just for a sanity check.)
temp = set(tf.all_variables()) train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy) #I honestly don't know how else to initialize ADAM in TensorFlow. sess.run(tf.initialize_variables(set(tf.all_variables()) - temp))
And it solves all my problems.
optim = tf.train.AdadeltaOptimizer(1e-4) loss = cross_entropy(y,yhat) train_step = optim.minimize(loss) sess.run(tf.initialize_variables([optim.get_slot(loss, name) for name in optim.get_slot_names()])
This however gives me
AttributeError: 'NoneType' object has no attribute 'initializer'. I’ll make edits when I figure out what I did wrong, so you don’t make my mistakes.
TF does not have a function that does exactly what you want, but you can easily write one:
import tensorflow as tf def initialize_uninitialized(sess): global_vars = tf.global_variables() is_initialized = sess.run([tf.is_variable_initialized(var) for var in global_vars]) not_initialized_vars = [v for (v, f) in zip(global_vars, is_initialized) if not f] print [str(i.name) for i in not_initialized_vars] # only for testing if len(not_initialized_vars): sess.run(tf.variables_initializer(not_initialized_vars))
Here I extract all global variables, iterate all of them and check whether they are already initialized. After this I get a list of uninitialized variables which I initialize. I also print variables that I am going to initialize for debugging purposes.
You can easily verify that it works as expected:
a = tf.Variable(3, name="my_var_a") b = tf.Variable(4, name="my_var_b") sess = tf.Session() initialize_uninitialized(sess) initialize_uninitialized(sess) c = tf.Variable(5, name="my_var_a") # the same name, will be resolved to different name d = tf.Variable(6, name="my_var_d") initialize_uninitialized(sess) print '\n\n', sess.run([a, b, c, d])
This will print all the unitialized variables before initializing them and the last sess.run will make sure persuade you that all variables are initialized.
For the case @Poik mentioned, when variables are created by optimizers so that they cannot be accessed directly, a cleaner solution is to use
Some optimizer subclasses, such as
AdagradOptimizer allocate and manage additional variables associated with the variables to train. These are called Slots. You can use
tf.train.Optimizer.get_slot_names() to get all slot names a optimizer has and then use
tf.train.Optimizer.get_slot to get the variable allocated for these slots.
I’ve come up with a method for TensorFlow r0.11:
def get_uninitialized_variables(variables=None): """Get uninitialized variables as a list. Parameters ---------- variables : collections.Iterable[tf.Variable] Return only uninitialized variables within this collection. If not specified, will return all uninitialized variables. Returns ------- list[tf.Variable] """ sess = tf.get_default_session() if variables is None: variables = tf.all_variables() else: variables = list(variables) init_flag = sess.run( tf.pack([tf.is_variable_initialized(v) for v in variables])) return [v for v, f in zip(variables, init_flag) if not f]
Btw, if you want to initialize only a single tensor (for e.g.
tf.Variable) that hasn’t been initialized using
tf.global_variables_initializer(), then you can use
your_tensor.initializer in the
sess.run() as in the following example:
In : weights = tf.Variable(tf.zeros(shape=(3, 4)), name="weights") In : with tf.Session() as sess: ...: sess.run(weights.initializer) ...: print(weights.eval()) ...: # the result [[ 0. 0. 0. 0.] [ 0. 0. 0. 0.] [ 0. 0. 0. 0.]]
I think that the easiest way is to create all the training operators first and initialize variables afterwards.
For example, I solved the problem of layer-wise pretraining with Adam Optimizer in the following way:
# create an optimizer pretrain_optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate) # Make an array of the trainers for all the layers trainers=[pretrain_optimizer.minimize(loss_reconstruction(ae.run_less_layers(ae._input_, i+1), ae.run_less_layers(ae._input_, i+1, is_target=True)), global_step=tf.contrib.framework.get_or_create_global_step(), name="Layer_wise_optimizer_"+str(i)) for i in xrange(len(ae_shape) - 2)] # Initialize all the variables sess.run(tf.global_variables_initializer())