Python 实用宝典

Question 1

I am confused about the method view() in the following code snippet.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool  = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*5*5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

My confusion is regarding the following line.

x = x.view(-1, 16*5*5)

What does tensor.view() function do? I have seen its usage in many places, but I can’t understand how it interprets its parameters.

What happens if I give negative values as parameters to the view() function? For example, what happens if I call, tensor_variable.view(1, 1, -1)?

Can anyone explain the main principle of view() function with some examples?

Question 2

The view function is meant to reshape the tensor.

Say you have a tensor

import torch
a = torch.range(1, 16)

a is a tensor that has 16 elements from 1 to 16(included). If you want to reshape this tensor to make it a 4 x 4 tensor then you can use

a = a.view(4, 4)

Now a will be a 4 x 4 tensor. Note that after the reshape the total number of elements need to remain the same. Reshaping the tensor a to a 3 x 5 tensor would not be appropriate.

What is the meaning of parameter -1?

If there is any situation that you don’t know how many rows you want but are sure of the number of columns, then you can specify this with a -1. (Note that you can extend this to tensors with more dimensions. Only one of the axis value can be -1). This is a way of telling the library: “give me a tensor that has these many columns and you compute the appropriate number of rows that is necessary to make this happen”.

This can be seen in the neural network code that you have given above. After the line x = self.pool(F.relu(self.conv2(x))) in the forward function, you will have a 16 depth feature map. You have to flatten this to give it to the fully connected layer. So you tell pytorch to reshape the tensor you obtained to have specific number of columns and tell it to decide the number of rows by itself.

Drawing a similarity between numpy and pytorch, view is similar to numpy’s reshape function.

Question 3

Let’s do some examples, from simpler to more difficult.

The view method returns a tensor with the same data as the self tensor (which means that the returned tensor has the same number of elements), but with a different shape. For example:

a = torch.arange(1, 17)  # a's shape is (16,)

a.view(4, 4) # output below
  1   2   3   4
  5   6   7   8
  9  10  11  12
 13  14  15  16
[torch.FloatTensor of size 4x4]

a.view(2, 2, 4) # output below
(0 ,.,.) = 
1   2   3   4
5   6   7   8

(1 ,.,.) = 
 9  10  11  12
13  14  15  16
[torch.FloatTensor of size 2x2x4]

Assuming that -1 is not one of the parameters, when you multiply them together, the result must be equal to the number of elements in the tensor. If you do: a.view(3, 3), it will raise a RuntimeError because shape (3 x 3) is invalid for input with 16 elements. In other words: 3 x 3 does not equal 16 but 9.
You can use -1 as one of the parameters that you pass to the function, but only once. All that happens is that the method will do the math for you on how to fill that dimension. For example a.view(2, -1, 4) is equivalent to a.view(2, 2, 4). [16 / (2 x 4) = 2]
Notice that the returned tensor shares the same data. If you make a change in the “view” you are changing the original tensor’s data:
```
b = a.view(4, 4)
b[0, 2] = 2
a[2] == 3.0
False
```
Now, for a more complex use case. The documentation says that each new view dimension must either be a subspace of an original dimension, or only span d, d + 1, …, d + k that satisfy the following contiguity-like condition that for all i = 0, …, k – 1, stride[i] = stride[i + 1] x size[i + 1]. Otherwise, contiguous() needs to be called before the tensor can be viewed. For example:
```
a = torch.rand(5, 4, 3, 2) # size (5, 4, 3, 2)
a_t = a.permute(0, 2, 3, 1) # size (5, 3, 2, 4)

# The commented line below will raise a RuntimeError, because one dimension
# spans across two contiguous subspaces
# a_t.view(-1, 4)

# instead do:
a_t.contiguous().view(-1, 4)

# To see why the first one does not work and the second does,
# compare a.stride() and a_t.stride()
a.stride() # (24, 6, 2, 1)
a_t.stride() # (24, 2, 1, 6)
```
Notice that for a_t, stride[0] != stride[1] x size[1] since 24 != 2 x 3

Question 4

`torch.Tensor.view()`

Simply put, torch.Tensor.view() which is inspired by numpy.ndarray.reshape() or numpy.reshape(), creates a new view of the tensor, as long as the new shape is compatible with the shape of the original tensor.

Let’s understand this in detail using a concrete example.

In [43]: t = torch.arange(18) 

In [44]: t 
Out[44]: 
tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17])

With this tensor t of shape (18,), new views can only be created for the following shapes:

(1, 18) or equivalently (1, -1) or (-1, 18)
(2, 9) or equivalently (2, -1) or (-1, 9)
(3, 6) or equivalently (3, -1) or (-1, 6)
(6, 3) or equivalently (6, -1) or (-1, 3)
(9, 2) or equivalently (9, -1) or (-1, 2)
(18, 1) or equivalently (18, -1) or (-1, 1)

As we can already observe from the above shape tuples, the multiplication of the elements of the shape tuple (e.g. 2*9, 3*6 etc.) must always be equal to the total number of elements in the original tensor (18 in our example).

Another thing to observe is that we used a -1 in one of the places in each of the shape tuples. By using a -1, we are being lazy in doing the computation ourselves and rather delegate the task to PyTorch to do calculation of that value for the shape when it creates the new view. One important thing to note is that we can only use a single -1 in the shape tuple. The remaining values should be explicitly supplied by us. Else PyTorch will complain by throwing a RuntimeError:

RuntimeError: only one dimension can be inferred

So, with all of the above mentioned shapes, PyTorch will always return a new view of the original tensor t. This basically means that it just changes the stride information of the tensor for each of the new views that are requested.

Below are some examples illustrating how the strides of the tensors are changed with each new view.

# stride of our original tensor `t`
In [53]: t.stride() 
Out[53]: (1,)

Now, we will see the strides for the new views:

# shape (1, 18)
In [54]: t1 = t.view(1, -1)
# stride tensor `t1` with shape (1, 18)
In [55]: t1.stride() 
Out[55]: (18, 1)

# shape (2, 9)
In [56]: t2 = t.view(2, -1)
# stride of tensor `t2` with shape (2, 9)
In [57]: t2.stride()       
Out[57]: (9, 1)

# shape (3, 6)
In [59]: t3 = t.view(3, -1) 
# stride of tensor `t3` with shape (3, 6)
In [60]: t3.stride() 
Out[60]: (6, 1)

# shape (6, 3)
In [62]: t4 = t.view(6,-1)
# stride of tensor `t4` with shape (6, 3)
In [63]: t4.stride() 
Out[63]: (3, 1)

# shape (9, 2)
In [65]: t5 = t.view(9, -1) 
# stride of tensor `t5` with shape (9, 2)
In [66]: t5.stride()
Out[66]: (2, 1)

# shape (18, 1)
In [68]: t6 = t.view(18, -1)
# stride of tensor `t6` with shape (18, 1)
In [69]: t6.stride()
Out[69]: (1, 1)

So that’s the magic of the view() function. It just changes the strides of the (original) tensor for each of the new views, as long as the shape of the new view is compatible with the original shape.

Another interesting thing one might observe from the strides tuples is that the value of the element in the 0^th position is equal to the value of the element in the 1^st position of the shape tuple.

In [74]: t3.shape 
Out[74]: torch.Size([3, 6])
                        |
In [75]: t3.stride()    |
Out[75]: (6, 1)         |
          |_____________|

This is because:

In [76]: t3 
Out[76]: 
tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11],
        [12, 13, 14, 15, 16, 17]])

the stride (6, 1) says that to go from one element to the next element along the 0^th dimension, we have to jump or take 6 steps. (i.e. to go from 0 to 6, one has to take 6 steps.) But to go from one element to the next element in the 1^st dimension, we just need only one step (for e.g. to go from 2 to 3).

Thus, the strides information is at the heart of how the elements are accessed from memory for performing the computation.

torch.reshape()

This function would return a view and is exactly the same as using torch.Tensor.view() as long as the new shape is compatible with the shape of the original tensor. Otherwise, it will return a copy.

However, the notes of torch.reshape() warns that:

contiguous inputs and inputs with compatible strides can be reshaped without copying, but one should not depend on the copying vs. viewing behavior.

Question 5

I figured it out that x.view(-1, 16 * 5 * 5) is equivalent to x.flatten(1), where the parameter 1 indicates the flatten process starts from the 1st dimension(not flattening the ‘sample’ dimension) As you can see, the latter usage is semantically more clear and easier to use, so I prefer flatten().

Question 6

What is the meaning of parameter -1?

You can read -1 as dynamic number of parameters or “anything”. Because of that there can be only one parameter -1 in view().

If you ask x.view(-1,1) this will output tensor shape [anything, 1] depending on the number of elements in x. For example:

import torch
x = torch.tensor([1, 2, 3, 4])
print(x,x.shape)
print("...")
print(x.view(-1,1), x.view(-1,1).shape)
print(x.view(1,-1), x.view(1,-1).shape)

Will output:

tensor([1, 2, 3, 4]) torch.Size([4])
...
tensor([[1],
        [2],
        [3],
        [4]]) torch.Size([4, 1])
tensor([[1, 2, 3, 4]]) torch.Size([1, 4])

Question 7

weights.reshape(a, b) will return a new tensor with the same data as weights with size (a, b) as in it copies the data to another part of memory.

weights.resize_(a, b) returns the same tensor with a different shape. However, if the new shape results in fewer elements than the original tensor, some elements will be removed from the tensor (but not from memory). If the new shape results in more elements than the original tensor, new elements will be uninitialized in memory.

weights.view(a, b) will return a new tensor with the same data as weights with size (a, b)

Question 8

I really liked @Jadiel de Armas examples.

I would like to add a small insight to how elements are ordered for .view(…)

For a Tensor with shape (a,b,c), the order of it’s elements are determined by a numbering system: where the first digit has a numbers, second digit has b numbers and third digit has c numbers.
The mapping of the elements in the new Tensor returned by .view(…) preserves this order of the original Tensor.

Question 9

Let’s try to understand view by the following examples:

    a=torch.range(1,16)

print(a)

    tensor([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14.,
            15., 16.])

print(a.view(-1,2))

    tensor([[ 1.,  2.],
            [ 3.,  4.],
            [ 5.,  6.],
            [ 7.,  8.],
            [ 9., 10.],
            [11., 12.],
            [13., 14.],
            [15., 16.]])

print(a.view(2,-1,4))   #3d tensor

    tensor([[[ 1.,  2.,  3.,  4.],
             [ 5.,  6.,  7.,  8.]],

            [[ 9., 10., 11., 12.],
             [13., 14., 15., 16.]]])
print(a.view(2,-1,2))

    tensor([[[ 1.,  2.],
             [ 3.,  4.],
             [ 5.,  6.],
             [ 7.,  8.]],

            [[ 9., 10.],
             [11., 12.],
             [13., 14.],
             [15., 16.]]])

print(a.view(4,-1,2))

    tensor([[[ 1.,  2.],
             [ 3.,  4.]],

            [[ 5.,  6.],
             [ 7.,  8.]],

            [[ 9., 10.],
             [11., 12.]],

            [[13., 14.],
             [15., 16.]]])

-1 as an argument value is an easy way to compute the value of say x provided we know values of y, z or the other way round in case of 3d and for 2d again an easy way to compute the value of say x provided we know values of y or vice versa..

Question 10

I was looking for alternative ways to save a trained model in PyTorch. So far, I have found two alternatives.

torch.save() to save a model and torch.load() to load a model.
model.state_dict() to save a trained model and model.load_state_dict() to load the saved model.

I have come across to this discussion where approach 2 is recommended over approach 1.

My question is, why the second approach is preferred? Is it only because torch.nn modules have those two function and we are encouraged to use them?

Question 11

I’ve found this page on their github repo, I’ll just paste the content here.

Recommended approach for saving a model

There are two main approaches for serializing and restoring a model.

The first (recommended) saves and loads only the model parameters:

torch.save(the_model.state_dict(), PATH)

Then later:

the_model = TheModelClass(*args, **kwargs)
the_model.load_state_dict(torch.load(PATH))

The second saves and loads the entire model:

torch.save(the_model, PATH)

Then later:

the_model = torch.load(PATH)

However in this case, the serialized data is bound to the specific classes and the exact directory structure used, so it can break in various ways when used in other projects, or after some serious refactors.

Question 12

It depends on what you want to do.

Case # 1: Save the model to use it yourself for inference: You save the model, you restore it, and then you change the model to evaluation mode. This is done because you usually have BatchNorm and Dropout layers that by default are in train mode on construction:

torch.save(model.state_dict(), filepath)

#Later to restore:
model.load_state_dict(torch.load(filepath))
model.eval()

Case # 2: Save model to resume training later: If you need to keep training the model that you are about to save, you need to save more than just the model. You also need to save the state of the optimizer, epochs, score, etc. You would do it like this:

state = {
    'epoch': epoch,
    'state_dict': model.state_dict(),
    'optimizer': optimizer.state_dict(),
    ...
}
torch.save(state, filepath)

To resume training you would do things like: state = torch.load(filepath), and then, to restore the state of each individual object, something like this:

model.load_state_dict(state['state_dict'])
optimizer.load_state_dict(state['optimizer'])

Since you are resuming training, DO NOT call model.eval() once you restore the states when loading.

Case # 3: Model to be used by someone else with no access to your code: In Tensorflow you can create a .pb file that defines both the architecture and the weights of the model. This is very handy, specially when using Tensorflow serve. The equivalent way to do this in Pytorch would be:

torch.save(model, filepath)

# Then later:
model = torch.load(filepath)

This way is still not bullet proof and since pytorch is still undergoing a lot of changes, I wouldn’t recommend it.

Question 13

The pickle Python library implements binary protocols for serializing and de-serializing a Python object.

When you import torch (or when you use PyTorch) it will import pickle for you and you don’t need to call pickle.dump() and pickle.load() directly, which are the methods to save and to load the object.

In fact, torch.save() and torch.load() will wrap pickle.dump() and pickle.load() for you.

A state_dict the other answer mentioned deserves just few more notes.

What state_dict do we have inside PyTorch? There are actually two state_dicts.

The PyTorch model is torch.nn.Module has model.parameters() call to get learnable parameters (w and b). These learnable parameters, once randomly set, will update over time as we learn. Learnable parameters are the first state_dict.

The second state_dict is the optimizer state dict. You recall that the optimizer is used to improve our learnable parameters. But the optimizer state_dict is fixed. Nothing to learn in there.

Because state_dict objects are Python dictionaries, they can be easily saved, updated, altered, and restored, adding a great deal of modularity to PyTorch models and optimizers.

Let’s create a super simple model to explain this:

import torch
import torch.optim as optim

model = torch.nn.Linear(5, 2)

# Initialize optimizer
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

print("Model's state_dict:")
for param_tensor in model.state_dict():
    print(param_tensor, "\t", model.state_dict()[param_tensor].size())

print("Model weight:")    
print(model.weight)

print("Model bias:")    
print(model.bias)

print("---")
print("Optimizer's state_dict:")
for var_name in optimizer.state_dict():
    print(var_name, "\t", optimizer.state_dict()[var_name])

This code will output the following:

Model's state_dict:
weight   torch.Size([2, 5])
bias     torch.Size([2])
Model weight:
Parameter containing:
tensor([[ 0.1328,  0.1360,  0.1553, -0.1838, -0.0316],
        [ 0.0479,  0.1760,  0.1712,  0.2244,  0.1408]], requires_grad=True)
Model bias:
Parameter containing:
tensor([ 0.4112, -0.0733], requires_grad=True)
---
Optimizer's state_dict:
state    {}
param_groups     [{'lr': 0.001, 'momentum': 0.9, 'dampening': 0, 'weight_decay': 0, 'nesterov': False, 'params': [140695321443856, 140695321443928]}]

Note this is a minimal model. You may try to add stack of sequential

model = torch.nn.Sequential(
          torch.nn.Linear(D_in, H),
          torch.nn.Conv2d(A, B, C)
          torch.nn.Linear(H, D_out),
        )

Note that only layers with learnable parameters (convolutional layers, linear layers, etc.) and registered buffers (batchnorm layers) have entries in the model’s state_dict.

Non learnable things, belong to the optimizer object state_dict, which contains information about the optimizer’s state, as well as the hyperparameters used.

The rest of the story is the same; in the inference phase (this is a phase when we use the model after training) for predicting; we do predict based on the parameters we learned. So for the inference, we just need to save the parameters model.state_dict().

torch.save(model.state_dict(), filepath)

And to use later model.load_state_dict(torch.load(filepath)) model.eval()

Note: Don’t forget the last line model.eval() this is crucial after loading the model.

Also don’t try to save torch.save(model.parameters(), filepath). The model.parameters() is just the generator object.

On the other side, torch.save(model, filepath) saves the model object itself, but keep in mind the model doesn’t have the optimizer’s state_dict. Check the other excellent answer by @Jadiel de Armas to save the optimizer’s state dict.

Question 14

A common PyTorch convention is to save models using either a .pt or .pth file extension.

Save/Load Entire Model Save:

path = "username/directory/lstmmodelgpu.pth"
torch.save(trainer, path)

Load:

Model class must be defined somewhere

model = torch.load(PATH)
model.eval()

Question 15

If you want to save the model and wants to resume the training later:

Single GPU: Save:

state = {
        'epoch': epoch,
        'state_dict': model.state_dict(),
        'optimizer': optimizer.state_dict(),
}
savepath='checkpoint.t7'
torch.save(state,savepath)

Load:

checkpoint = torch.load('checkpoint.t7')
model.load_state_dict(checkpoint['state_dict'])
optimizer.load_state_dict(checkpoint['optimizer'])
epoch = checkpoint['epoch']

Multiple GPU: Save

state = {
        'epoch': epoch,
        'state_dict': model.module.state_dict(),
        'optimizer': optimizer.state_dict(),
}
savepath='checkpoint.t7'
torch.save(state,savepath)

Load:

checkpoint = torch.load('checkpoint.t7')
model.load_state_dict(checkpoint['state_dict'])
optimizer.load_state_dict(checkpoint['optimizer'])
epoch = checkpoint['epoch']

#Don't call DataParallel before loading the model otherwise you will get an error

model = nn.DataParallel(model) #ignore the line if you want to load on Single GPU

Question 16

I have been using the introductory example of matrix multiplication in TensorFlow.

matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
product = tf.matmul(matrix1, matrix2)

When I print the product, it is displaying it as a Tensor object:

<tensorflow.python.framework.ops.Tensor object at 0x10470fcd0>

But how do I know the value of product?

The following doesn’t help:

print product
Tensor("MatMul:0", shape=TensorShape([Dimension(1), Dimension(1)]), dtype=float32)

I know that graphs run on Sessions, but isn’t there any way I can check the output of a Tensor object without running the graph in a session?

Question 17

The easiest^[A] way to evaluate the actual value of a Tensor object is to pass it to the Session.run() method, or call Tensor.eval() when you have a default session (i.e. in a with tf.Session(): block, or see below). In general^[B], you cannot print the value of a tensor without running some code in a session.

If you are experimenting with the programming model, and want an easy way to evaluate tensors, the tf.InteractiveSession lets you open a session at the start of your program, and then use that session for all Tensor.eval() (and Operation.run()) calls. This can be easier in an interactive setting, such as the shell or an IPython notebook, when it’s tedious to pass around a Session object everywhere. For example, the following works in a Jupyter notebook:

with tf.Session() as sess:  print(product.eval())

This might seem silly for such a small expression, but one of the key ideas in Tensorflow 1.x is deferred execution: it’s very cheap to build a large and complex expression, and when you want to evaluate it, the back-end (to which you connect with a Session) is able to schedule its execution more efficiently (e.g. executing independent parts in parallel and using GPUs).

[A]: To print the value of a tensor without returning it to your Python program, you can use the tf.print() operator, as Andrzej suggests in another answer. According to the official documentation:

To make sure the operator runs, users need to pass the produced op to tf.compat.v1.Session‘s run method, or to use the op as a control dependency for executed ops by specifying with tf.compat.v1.control_dependencies([print_op]), which is printed to standard output.

Also note that:

In Jupyter notebooks and colabs, tf.print prints to the notebook cell outputs. It will not write to the notebook kernel’s console logs.

[B]: You might be able to use the tf.get_static_value() function to get the constant value of the given tensor if its value is efficiently calculable.

Question 18

While other answers are correct that you cannot print the value until you evaluate the graph, they do not talk about one easy way of actually printing a value inside the graph, once you evaluate it.

The easiest way to see a value of a tensor whenever the graph is evaluated (using run or eval) is to use the Print operation as in this example:

# Initialize session
import tensorflow as tf
sess = tf.InteractiveSession()

# Some tensor we want to print the value of
a = tf.constant([1.0, 3.0])

# Add print operation
a = tf.Print(a, [a], message="This is a: ")

# Add more elements of the graph using a
b = tf.add(a, a)

Now, whenever we evaluate the whole graph, e.g. using b.eval(), we get:

I tensorflow/core/kernels/logging_ops.cc:79] This is a: [1 3]

Question 19

Reiterating what others said, its not possible to check the values without running the graph.

A simple snippet for anyone looking for an easy example to print values is as below. The code can be executed without any modification in ipython notebook

import tensorflow as tf

#define a variable to hold normal random values 
normal_rv = tf.Variable( tf.truncated_normal([2,3],stddev = 0.1))

#initialize the variable
init_op = tf.initialize_all_variables()

#run the graph
with tf.Session() as sess:
    sess.run(init_op) #execute init_op
    #print the random values that we sample
    print (sess.run(normal_rv))

Output:

[[-0.16702934  0.07173464 -0.04512421]
 [-0.02265321  0.06509651 -0.01419079]]

Question 20

No, you can not see the content of the tensor without running the graph (doing session.run()). The only things you can see are:

the dimensionality of the tensor (but I assume it is not hard to calculate it for the list of the operations that TF has)
type of the operation that will be used to generate the tensor (transpose_1:0, random_uniform:0)
type of elements in the tensor (float32)

I have not found this in documentation, but I believe that the values of the variables (and some of the constants are not calculated at the time of assignment).

Take a look at this example:

import tensorflow as tf
from datetime import datetime
dim = 7000

The first example where I just initiate a constant Tensor of random numbers run approximately the same time irrespectibly of dim (0:00:00.003261)

startTime = datetime.now()
m1 = tf.truncated_normal([dim, dim], mean=0.0, stddev=0.02, dtype=tf.float32, seed=1)
print datetime.now() - startTime

In the second case, where the constant is actually gets evaluated and the values are assigned, the time clearly depends on dim (0:00:01.244642)

startTime = datetime.now()
m1 = tf.truncated_normal([dim, dim], mean=0.0, stddev=0.02, dtype=tf.float32, seed=1)
sess = tf.Session()
sess.run(m1)
print datetime.now() - startTime

And you can make it more clear by calculating something (d = tf.matrix_determinant(m1), keeping in mind that the time will run in O(dim^2.8))

P.S. I found were it is explained in documentation:

A Tensor object is a symbolic handle to the result of an operation, but does not actually hold the values of the operation’s output.

Question 21

I think you need to get some fundamentals right. With the examples above you have created tensors (multi dimensional array). But for tensor flow to really work you have to initiate a “session” and run your “operation” in the session. Notice the word “session” and “operation”. You need to know 4 things to work with tensorflow:

tensors
Operations
Sessions
Graphs

Now from what you wrote out you have given the tensor, and the operation but you have no session running nor a graph. Tensor (edges of the graph) flow through graphs and are manipulated by operations (nodes of the graph). There is default graph but you can initiate yours in a session.

When you say print , you only access the shape of the variable or constant you defined.

So you can see what you are missing :

 with tf.Session() as sess:     
           print(sess.run(product))
           print (product.eval())

Hope it helps!

Question 22

In Tensorflow 1.x

import tensorflow as tf
tf.enable_eager_execution()
matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
product = tf.matmul(matrix1, matrix2)

#print the product
print(product)         # tf.Tensor([[12.]], shape=(1, 1), dtype=float32)
print(product.numpy()) # [[12.]]

With Tensorflow 2.x, eager mode is enabled by default. so the following code works with TF2.0.

import tensorflow as tf
matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
product = tf.matmul(matrix1, matrix2)

#print the product
print(product)         # tf.Tensor([[12.]], shape=(1, 1), dtype=float32)
print(product.numpy()) # [[12.]]

Question 23

Based on the answers above, with your particular code snippet you can print the product like this:

import tensorflow as tf
#Initialize the session
sess = tf.InteractiveSession()

matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
product = tf.matmul(matrix1, matrix2)

#print the product
print(product.eval())

#close the session to release resources
sess.close()

Question 24

In Tensorflow 2.0+ (or in Eager mode environment) you can call .numpy() method:

import tensorflow as tf

matrix1 = tf.constant([[3., 3.0]])
matrix2 = tf.constant([[2.0],[2.0]])
product = tf.matmul(matrix1, matrix2)

print(product.numpy())

Question 25

tf.keras.backend.eval is useful for evaluating small expressions.

tf.keras.backend.eval(op)

TF 1.x and TF 2.0 compatible.

Minimal Verifiable Example

from tensorflow.keras.backend import eval

m1 = tf.constant([[3., 3.]])
m2 = tf.constant([[2.],[2.]])

eval(tf.matmul(m1, m2))
# array([[12.]], dtype=float32)

This is useful because you do not have to explicitly create a Session or InteractiveSession.

Question 26

You can check the output of a TensorObject without running the graph in a session, by enabling eager execution.

Simply add the following two lines of code: import tensorflow.contrib.eager as tfe tfe.enable_eager_execution()

right after you import tensorflow.

The output of print product in your example will now be: tf.Tensor([[ 12.]], shape=(1, 1), dtype=float32)

Note that as of now (November 2017) you’ll have to install a Tensorflow nightly build to enable eager execution. Pre-built wheels can be found here.

Question 27

Please note that tf.Print() will change the tensor name. If the tensor you seek to print is a placeholder, feeding data to it will fail as the original name will not be found during feeding. For example:

import tensorflow as tf
tens = tf.placeholder(tf.float32,[None,2],name="placeholder")
print(eval("tens"))
tens = tf.Print(tens,[tens, tf.shape(tens)],summarize=10,message="tens:")
print(eval("tens"))
res = tens + tens
sess = tf.Session()
sess.run(tf.global_variables_initializer())

print(sess.run(res))

Output is:

python test.py
Tensor("placeholder:0", shape=(?, 2), dtype=float32)
Tensor("Print:0", shape=(?, 2), dtype=float32)
Traceback (most recent call last):
[...]
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'placeholder' with dtype float

Question 28

You should think of TensorFlow Core programs as consisting of two discrete sections:

Building the computational graph.
Running the computational graph.

So for the code below you just Build the computational graph.

matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
product = tf.matmul(matrix1, matrix2)

You need also To initialize all the variables in a TensorFlow program , you must explicitly call a special operation as follows:

init = tf.global_variables_initializer()

Now you build the graph and initialized all variables ,next step is to evaluate the nodes, you must run the computational graph within a session. A session encapsulates the control and state of the TensorFlow runtime.

The following code creates a Session object and then invokes its run method to run enough of the computational graph to evaluate product :

sess = tf.Session()
// run variables initializer
sess.run(init)

print(sess.run([product]))

Question 29

You can use Keras, one-line answer will be to use eval method like so:

import keras.backend as K
print(K.eval(your_tensor))

Question 30

Try this simple code! (it is self explanatory)

import tensorflow as tf
sess = tf.InteractiveSession() # see the answers above :)
x = [[1.,2.,1.],[1.,1.,1.]]    # a 2D matrix as input to softmax
y = tf.nn.softmax(x)           # this is the softmax function
                               # you can have anything you like here
u = y.eval()
print(u)

Question 31

I didn’t find it easy to understand what is required even after reading all the answers until I executed this. TensofFlow is new to me too.

def printtest():
x = tf.constant([1.0, 3.0])
x = tf.Print(x,[x],message="Test")
init = (tf.global_variables_initializer(), tf.local_variables_initializer())
b = tf.add(x, x)
with tf.Session() as sess:
    sess.run(init)
    print(sess.run(b))
    sess.close()

But still you may need the value returned by executing the session.

def printtest():
    x = tf.constant([100.0])
    x = tf.Print(x,[x],message="Test")
    init = (tf.global_variables_initializer(), tf.local_variables_initializer())
    b = tf.add(x, x)
    with tf.Session() as sess:
        sess.run(init)
        c = sess.run(b)
        print(c)
        sess.close()

Question 32

Basically, in tensorflow when you create a tensor of any sort they are created and stored inside which can be accessed only when you run a tensorflow session. Say you have created a constant tensor
c = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
Without running a session, you can get
– op: An Operation. Operation that computes this tensor.
– value_index: An int. Index of the operation’s endpoint that produces this tensor.
– dtype: A DType. Type of elements stored in this tensor.

To get the values you can run a session with the tensor you require as:

with tf.Session() as sess:
    print(sess.run(c))
    sess.close()

The output will be something like this:

array([[1., 2., 3.], [4., 5., 6.]], dtype=float32)

Question 33

Enable the eager execution which is introduced in tensorflow after version 1.10. It’s very easy to use.

# Initialize session
import tensorflow as tf
tf.enable_eager_execution()


# Some tensor we want to print the value of
a = tf.constant([1.0, 3.0])

print(a)

Question 34

Using tips provided in https://www.tensorflow.org/api_docs/python/tf/print I use the log_d function to print formatted strings.

import tensorflow as tf

def log_d(fmt, *args):
    op = tf.py_func(func=lambda fmt_, *args_: print(fmt%(*args_,)),
                    inp=[fmt]+[*args], Tout=[])
    return tf.control_dependencies([op])


# actual code starts now...

matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
product = tf.matmul(matrix1, matrix2)

with log_d('MAT1: %s, MAT2: %s', matrix1, matrix2): # this will print the log line
    product = tf.matmul(matrix1, matrix2)

with tf.Session() as sess:
    sess.run(product)

Question 35

import tensorflow as tf
sess = tf.InteractiveSession()
x = [[1.,2.,1.],[1.,1.,1.]]    
y = tf.nn.softmax(x)           

matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
product = tf.matmul(matrix1, matrix2)

print(product.eval())
tf.reset_default_graph()
sess.close()

Question 36

tf.Print is now deprecated, here’s how to use tf.print (lowercase p) instead.

While running a session is a good option, it is not always the way to go. For instance, you may want to print some tensor in a particular session.

The new print method returns a print operation which has no output tensors:

print_op = tf.print(tensor_to_print)

Since it has no outputs, you can’t insert it in a graph the same way as you could with tf.Print. Instead, you can you can add it to control dependencies in your session in order to make it print.

sess = tf.compat.v1.Session()
with sess.as_default():
  tensor_to_print = tf.range(10)
  print_op = tf.print(tensor_to_print)
with tf.control_dependencies([print_op]):
  tripled_tensor = tensor_to_print * 3
sess.run(tripled_tensor)

Sometimes, in a larger graph, maybe created partly in subfunctions, it is cumbersome to propagate the print_op to the session call. Then, tf.tuple can be used to couple the print operation with another operation, which will then run with that operation whichever session executes the code. Here’s how that is done:

print_op = tf.print(tensor_to_print)
some_tensor_list = tf.tuple([some_tensor], control_inputs=[print_op])
# Use some_tensor_list[0] instead of any_tensor below.

Question 37

Question: How to print the value of a Tensor object in TensorFlow?

Answer:

import tensorflow as tf

# Variable
x = tf.Variable([[1,2,3]])

# initialize
init = (tf.global_variables_initializer(), tf.local_variables_initializer())

# Create a session
sess = tf.Session()

# run the session
sess.run(init)

# print the value
sess.run(x)

系统	3.6	3.8
Linux CPU		–
Linux GPU		–
Windows CPU/GPU	–	–
Linux(Ppc64le)CPU	–	–
Linux(Ppc64le)GPU	–	–
Linux(Aarch64)CPU

组件	描述
火炬	像NumPy这样的张量器库，具有强大的GPU支持
torch.autograd	基于磁带的自动区分库，支持TORCH中的所有可微分张量操作
torch.jit	从PyTorch代码创建可序列化和可优化模型的编译堆栈(TorchScript)
torch.nn	与Autograd深度集成的神经网络库，旨在实现最大的灵活性
torch.multiprocessing	Python多处理，但具有跨进程共享火炬张量的神奇内存。适用于数据加载和HogWild培训
torch.utils	为方便起见，DataLoader和其他实用程序功能

CUDA版	Xcode版本
10.0	Xcode 9.4
10.1	Xcode 10.1

问题：PyTorch中的“视图”方法如何工作？

回答 0

参数-1是什么意思？

What is the meaning of parameter -1?

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

问题：在PyTorch中保存经过训练的模型的最佳方法？

回答 0

推荐的模型保存方法

Recommended approach for saving a model

回答 1

回答 2

回答 3

模型类必须在某处定义

Model class must be defined somewhere

回答 4

问题：如何在TensorFlow中打印Tensor对象的值？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

回答 13

回答 14

回答 15

回答 16

回答 17

回答 18

回答 19

回答 20

更多关于PyTorch的信息

一种支持GPU的张量库

动态神经网络：基于磁带的自动评分

Python优先

势在必行的经验

快速精益

无痛延长

安装

二进制文件

NVIDIA Jetson平台

来自源

安装依赖项

获取PyTorch源代码

安装PyTorch

调整生成选项(可选)

Docker镜像

使用预构建映像

自己打造形象

构建文档

以前的版本

快速入门

资源

沟通

发布和贡献

团队

许可证

有趣好用的Python教程