Plato is a python package built on top of Theano with two objectives:
1) Simplify the use of Theano.
2) Build a good libary of standard Deep Learning algorithms.
This tutorial takes you throught the Plato API. It's useful but not necessary to have a basic knowledge of Theano to do this tutorial.
In Plato, we have the concept of symbolic functions, which are function that take and return theano symbolic variables. These functions can be compiled to numeric functions which take and return numpy arrays and python ints/floats.
from plato.core import symbolic
@symbolic
def add_two_numbers(x, y):
return x+y
f = add_two_numbers.compile()
print '3+4=%s' % f(3, 4)
Internally, here is what happens: On the first (and in this case, only) call to add_two_numbers, Plato inspects the inputs (3, 4), looks at their type (both scalar integers in this case), and gets Theano to compile a symbolic expression that adds them together. The equivalent code using just theano would be:
import theano
from theano.tensor import scalar
x = scalar(dtype = 'int32')
y = scalar(dtype = 'int32')
z = x+y
f = theano.function(inputs = [x, y], outputs = z)
print '3+4=%s' % f(3, 4)
Thus the first advantage of Plato is that it removes the need to create input variables and make sure their type matches the data that you're going to feed in.
We are also able to create stateful functions. Unlike Theano, we do not pass state-updates in the return value. Instead, we call the function add_update(shared_var, new_value). The following example shows how to make a "function" with some internal state that updates on each call.
from plato.core import symbolic, add_update, create_shared_variable
@symbolic
def counter():
count = create_shared_variable(0) # Create a shared variable, initialized at zero, which stores the count.
new_count = count+1
add_update(count, new_count)
return new_count
f = counter.compile()
print 'I can count to ten. See: %s' % ([int(f()) for _ in xrange(10)])
f2 = counter.compile()
print 'I can too: %s' % ([int(f2()) for _ in xrange(10)])
# Note that we start from scratch when we compile the function a new time,
# because the shared variable is initialized within the function call. If we
# had declaired counter outside the function, the second count would run from
# 11 to 20.
We often have situations where we have a variable that is shared between two functions (e.g. in a classifier, the weights may be modified by the train function and used by the predict function). As a simple example, we will train an online linear-regressor.
from plato.core import create_shared_variable, symbolic, add_update
import theano.tensor as tt
import numpy as np
# Set up parameters
n_in = 20
n_out = 4
n_training_samples = 500
n_test_samples = 500
n_epochs = 2
noise = 0.1
random_seed = 1234
score_report_period = 100
# Create a regression dataset
rng = np.random.RandomState(random_seed)
w_true = rng.randn(n_in, n_out) # (n_in, n_out)
training_data = rng.randn(n_training_samples, n_in) # (n_training_samples, n_in)
training_target = training_data.dot(w_true) + noise*rng.randn(n_training_samples, n_out) # (n_training_samples, n_out)
test_data = rng.randn(n_test_samples, n_in) # (n_test_samples, n_in)
test_target = test_data.dot(w_true) + noise*rng.randn(n_test_samples, n_out) # (n_test_samples, n_out)
# Create a linear regressor
class LinearRegressor:
def __init__(self, n_in, n_out, eta = 0.01):
self.w = create_shared_variable(np.zeros((n_in, n_out)))
self.eta = eta
@symbolic
def train(self, x, targ): # x: (n_samples, n_in), targ: (n_samples, n_out)
y = self.predict(x)
cost = ((targ - y)**2).sum(axis=1).mean(axis=0)
add_update(self.w, self.w - self.eta*tt.grad(cost=cost, wrt=self.w))
@symbolic
def predict(self, x): # x: (n_samples, n_in)
return x.dot(self.w)
# Setup the predictor and compile functions
predictor = LinearRegressor(n_in, n_out)
f_train = predictor.train.compile()
f_predict = predictor.predict.compile()
# Train on one sample at a time and periodically report score.
for i in xrange(n_training_samples*n_epochs+1):
if i % score_report_period == 0:
out = f_predict(test_data)
test_cost = ((test_target-out)**2).sum(axis=1).mean(axis=0)
print 'Test-Cost at epoch %s: %s' % (float(i)/n_training_samples, test_cost)
f_train(training_data[[i % n_training_samples]], training_target[[i % n_training_samples]])
In Python, classes can also act as functions, if they implement a __call__ method. This can be useful when you want to make parameterized functions. Therefore Plato also allows you to decorate callable classes. For example:
from plato.core import symbolic
@symbolic
class MultiplyBySomething:
def __init__(self, what):
self.what = what
def __call__(self, x):
return x*self.what
f = MultiplyBySomething(3).compile()
print '3*4=%s' % f(4)
Unlike Theano, Plato allows you to pass inputs into compiled functions by name. The only requirement is that you are consistent with their usage (if you call the function as f(3, y=4) the first, time, you cannot call it as f(3, 4) the next time, otherwise you will get an error. See the following example:
from plato.core import symbolic
@symbolic
def add_and_div(x, y, z):
return (x+y)//z
f = add_and_div.compile()
print '(2+4)/3 = %s' % f(x=4, y=2, z=3)
print '(1+3)/2 = %s' % f(z=2, y=3, x=1)
try:
print 'Lets try again, but leave x as an unnamed arg...'
f(2, y=4, z=3.)
except KeyError as e:
print 'You were inconsistent - referenced x as a kwarg in the first call but not the second.'
A big advantage of Plato is easier debugging. There are two ways in which Plato helps you debug. The first is what we call "initial values".
Theano allows you to add "test-values" to your symbolic variables (see tutorial). This helps to catch shape-errors when building the graph, instead of at run-time, where it is difficult to find the line of code that caused them. However, it can be a bit of extra work for the programmer, because they have to manually attach test values to their variables. Fortunately, since Plato compiles your functions on the first pass, it can attach test-values "under the hood".
For example, lets look at a matrix multiplication, where we accidently get the shapes of our matrices wrong. Since all inputs are given test values, we can easily track down the error - the traceback will lead back to the correct line. This would not have been possible without test values, because the error would occur in the compiled code, which is no-longer linked to the source code.
Plato attaches the following properties to all symbolic variables:
var.ival - The initial value of the variable (a numpy array or scalar)
var.ishape - The initial shape of the variable
var.indim - The initial number of dimensions of the variable
var.idtype - The initial dtype of the variable
import numpy as np
from plato.core import symbolic
@symbolic
def forward_pass(x, w):
print 'x-shape: %s' % (x.ishape, )
print 'w-shape: %s' % (w.ishape, )
# Note that the above test-values only display on the first iteration.
y = x.dot(w)
print 'Success! y-shape: %s' % (y.ishape, )
return y
f = forward_pass.compile(add_test_values=True)
try:
# The following will cause an error (because second argument should have shape (4, 3))
h = f(np.random.randn(5, 4), np.random.rand(3, 4))
except ValueError as err:
# If you do not catch the error, you get a stacktrace which points to the line at fault.
print str(err
print "Now we try again with the correct shape, and it succeeds."
h = f(np.random.randn(5, 4), np.random.rand(4, 3))
print "Note that if we run again we print nothing because the symbolic function is just run once, on the first pass."
h = f(np.random.randn(5, 4), np.random.rand(4, 3))
Initial values can also be used to initialize shared variables. You may want to initialize a shared variable to be the same size as another variable in the graph. You could than say shared_var = theano.shared(np.zeros(var.ishape)).
We can use var.ival (see part 6, above) and its brothers to access variable values on the first pass, but what if we want to view variable values every time the function is called? Ordinarily in Theano, this would require setting those variables as outputs, and restructuring code to peek at what would normally be an internal variables. Plato does a bit of magic which allows you to print/plot/do anything with internal variables. The following example illustrates this:
import numpy as np
from plato.core import symbolic, tdbprint, create_shared_variable
import theano.tensor as tt
class Layer:
def __init__(self, w):
self.w = create_shared_variable(w)
@symbolic
def forward_pass(self, x):
pre_sigmoid = x.dot(self.w)
tdbprint(pre_sigmoid, name = 'Pre-Sigmoid Activation') # Here we make a trace of an internal variable
y = tt.nnet.sigmoid(pre_sigmoid)
tdbprint(y, name = 'Post-Sigmoid Activation') # Here we make a trace of an internal variable
return y
n_in = 4
n_out = 3
rng = np.random.RandomState(seed=1234)
layer = Layer(rng.randn(n_in, n_out))
fwd_fcn = layer.forward_pass.compile()
for _ in xrange(3):
y = fwd_fcn(rng.randn(n_in))
print '==='
You can also return a dictionary of named outputs. To demonstrate this, we can take the previous example, and instead of printing the pre-sigmoid as a debug value, we return in an output dictionary.
import numpy as np
from plato.core import symbolic, tdbprint, create_shared_variable
import theano.tensor as tt
class Layer:
def __init__(self, w):
self.w = create_shared_variable(w)
@symbolic
def forward_pass(self, x):
pre_sigmoid = x.dot(self.w)
y = tt.nnet.sigmoid(pre_sigmoid)
return {'pre-sigmoid': pre_sigmoid, 'output': y}
n_in = 4
n_out = 3
rng = np.random.RandomState(seed=1234)
layer = Layer(rng.randn(n_in, n_out))
fwd_fcn = layer.forward_pass.compile()
for _ in xrange(3):
result = fwd_fcn(rng.randn(n_in))
for k, v in result.iteritems():
print '%s: %s' % (k, v)
print '===='
If you want to store and retrieve internal variable values, you can use tdb_trace, and get_tdb_traces in plato.core.
You can also create live plots of internal variables using the function tdbplot in plato.tools.tdb_plotting, but this tutorial does not cover it.
In larger programs, it can be useful to enforce interfaces - that is, functions are required to obey a certain contract. This allows function A to use function B without knowing in particular which function it is - just that it takes inputs and returns outputs in some specified format. For instance, you may have some code that iterates through a dataset and trains a predictor, but doesn't necessarily know what kind of predictor it is - just that it has a train function that accepts inputs and targets, updates some internal state variable.
For this reason, we have an extended set of decorators which enforce type-checking on inputs, outputs, and updates.
@symbolic - No format requirements.
@symbolic_simple - Returns a single output variable.
@symbolic_multi - Returns a tuple of output tensors.
@symbolic_stateless - Makes no state updates.
@symbolic_updater - Returns nothing and produces at least one state update.
@symbolic_named_output - Returns a dictionary of named outputs
To make custom function con you can decorate with @SymbolicFunction(input_format, output_format, update_format), where each of the arguments is an IFormat object. See plato.core for examples.
When functions fail to obey the contract specified by their decorators, a SymbolicFormatError is raised.
For example:
from plato.core import symbolic_stateless, symbolic, SymbolicFormatError, add_update, create_shared_variable
@symbolic_stateless # Bad! We decorated with "symbolic_stateless", but we make a state update inside
def running_sum(x):
shared_var = create_shared_variable(0)
y = x + shared_var
add_update(shared_var, y)
return y
f = running_sum.compile()
print 'Trying to run incorrectly-decorated function...'
try:
f(3)
except SymbolicFormatError as err:
print ' %s: %s' % (err.__class__.__name__, err.message)
print 'Lets try again with the correct format....'
@symbolic
def running_sum(x):
shared_var = create_shared_variable(0)
y = x + shared_var
add_update(shared_var, y)
return y
f = running_sum.compile()
print ' cumsum([1,2,3,4]) = %s' % ([int(f(i)) for i in xrange(1, 5)], )
When you use a numpy array on a theano symbolic function, it treats it as a constant. We can use the fixed_args argument to compile() to partially-specify a function. Theano will then compile the function with these arguments as fixed constants. For example:
from plato.core import symbolic
@symbolic
def multiply(x, y):
return x*y
f_mult_by_3 = multiply.compile(fixed_args = dict(x=3))
print '3*2 = %s' % f_mult_by_3(y=2)
print '3*5 = %s' % f_mult_by_3(y=5)
For looping operations in Theano, there is the scan function. In Plato, symbolic functions have a .scan method of their own that you can call to output the result of the given function when called in a loop with updates applied sequentially and the return values piled into an array. The arguments have all the same names and semantics as Theano's scan function, so see Theano's scan documentation for details.
from plato.core import symbolic, create_shared_variable, add_update
import numpy as np
@symbolic
def running_sum(x):
shared_var = create_shared_variable(0)
y = x + shared_var
add_update(shared_var, y)
return y
@symbolic
def running_cumsum(arr):
cumsum = running_sum.scan(sequences = [arr])
return cumsum
f = running_cumsum.compile()
print 'Running Cumsum of [1,2,3,4]: %s' % (f(np.arange(1, 5)), )
print 'Continuing Running Cumsum of [1,2,3,4]: %s' % (f(np.arange(1, 5)), )
Congratulations. You've completed the 12-step program and you're ready to use Plato. You may now want to see examples of various Learning Algorithms in Plato.