Artemis Experiment Tutorial¶
The scientific method can generally involves the following loop:
- Come up with a hypothesis.
- Create an experiment to test this hypothesis
- Run experiment, observe, record results.
- Given results, either celebrate, or return to step 1.
All too often, what we actually do is the following:
- Come up with hypothesis
- Create an experiment to test this hypothesis
- Run, experiment, observe results.
- Be somehow unsatisfied with results, change some parameter in experiment.
- Run experiment again with new parameters.
- Observe results again, remain unsatisfied, revert to step 5 several times.
- Get tired of waiting for experiments to run just to analyze results. Start saving results of experiments, so they can be loaded an analyzed at any time.
- Run several experiments, load and compare results.
- Realize we've forgotten which parameters correspond to which results.
- Try to run all experiments again and save their results, this time keeping track of their parameters, but realize we've forgotten the parameters we used in our first experiments.
- Eventually get to some result, resolve to be more organized the next time around.
- Repeat from step 1.
The Artemis Experiment Framework aims to help organize this process. It does this with the following functionality.
- Recording Experimental Results: The framework captures the console output, plots, and results of an experiment so that we can review them later without having to run the experiment again.
- Creating Variations on Experiments: The framework allows us to define variants on experiments by changing some parameter. This allows up to build and maintain a list of variations on our experiment, and the results of running these variations.
- Comparing Experiments: Using the saved results of each experiment, Artemis allows us to easily go back and compare the results of different experiments.
We will demonstrate these in the following tutorial.
Introduction¶
Suppose we want to run a simple experiment: We kidnap 5 drunks from the local bar, take them to a point in a secluded field or parking lot, then release them, and record their progress. While the ethics committee processes our request, we run a simulation to get some preliminary results. We may code our simulation as follows:
import numpy as np
from matplotlib import pyplot as plt
%matplotlib notebook
def demo_drunkards_walk(n_steps=500, n_drunkards=5, homing_instinct = 0, n_dim=2, seed=1234):
"""
Release several drunkards in a field to randomly stumble around. Record their progress.
"""
rng = np.random.RandomState(seed)
drunkards = np.zeros((n_steps+1, n_drunkards, n_dim))
for t in range(1, n_steps+1):
drunkards[t] = drunkards[t-1]*(1-homing_instinct) + rng.randn(n_drunkards, n_dim)
if t%100==0:
print('Status at step {}: Mean: {}, STD: {}'.format(t, drunkards[t].mean(), drunkards[t].std()))
plt.plot(drunkards[:, :, 0], drunkards[:, :, 1])
plt.grid()
plt.xlabel('$\Delta$ Longitude (arcseconds)')
plt.ylabel('$\Delta$ Latitude (arcseconds)')
plt.show()
demo_drunkards_walk()
Running Experiments¶
Now, suppose our simulation takes a long time to run. We would like to record our results so that we review them later without having to re-run the experiment. We can achieve this by decorating our experiment with the "@experiment_function" decordator. The decorator registers the function demo_drunkards_walk, as an "experiment", which allows us to capture its output when it is run:
import numpy as np
from artemis.experiments import experiment_function
from artemis.experiments.ui import browse_experiments
from matplotlib import pyplot as plt
from artemis.experiments.experiments import clear_all_experiments
clear_all_experiments() # Removes any previous versions of demo_drunkards_walk that have may been registered
%matplotlib notebook
@experiment_function
def demo_drunkards_walk(n_steps=500, n_drunkards=5, homing_instinct = 0, n_dim=2, seed=1234):
"""
Release several drunkards in a field to randomly stumble around. Record their progress.
"""
rng = np.random.RandomState(seed)
drunkards = np.zeros((n_steps+1, n_drunkards, n_dim))
for t in range(1, n_steps+1):
drunkards[t] = drunkards[t-1]*(1-homing_instinct) + rng.randn(n_drunkards, n_dim)
if t%100==0:
print('Status at step {}: Mean: {}, STD: {}'.format(t, drunkards[t].mean(), drunkards[t].std()))
plt.plot(drunkards[:, :, 0], drunkards[:, :, 1])
plt.grid()
plt.xlabel('Step')
plt.ylabel('Drunkard Position')
plt.show()
We can now run this experiment by calling browse_experiments() to open the experiment user interface, and entering run 0, meaning "run experiment 0, record all figures and console output". (We could also do this programatically with demo_drunkards_walk.run()). In the menu, enter run 0
demo_drunkards_walk.browse(close_after=True)
Viewing Results of Experiments¶
Now if we want to go back later and see these results, we can enter the UI again and enter show 0 meaning "show the results of experiment 0". In the meny, unter show 0
%matplotlib notebook
demo_drunkards_walk.browse(close_after=True)
This displays all the output of the experiment, and should show the figure that was created.
Creating Variants¶
We now want to try changing parameters to our experiment. We could of course simply change the default arguments and run again, but then our saved experiment no longer corresponds to the new version of this experiment. We also want to be able to re-run our original experiment whenever we want (without having to write down the parameters it was run with the first time). To keep track of our variats without losing the original experiment, we can use the add_variant method.
Suppose, in the following example, that we want to give our drunkards a "homing instinct" that makes them tend towards the origin. We create two variants of our experiment with different degrees of homing instinct:
import numpy as np
from artemis.experiments import experiment_function
from artemis.experiments.ui import browse_experiments
from matplotlib import pyplot as plt
from artemis.experiments.experiments import clear_all_experiments
clear_all_experiments() # Removes previous versions of demo_drunkards_walk that have been registered
@experiment_function
def demo_drunkards_walk(n_steps=500, n_drunkards=5, homing_instinct = 0, n_dim=2, seed=1234):
"""
Release several drunkards in a field to randomly stumble around. Record their progress.
"""
rng = np.random.RandomState(seed)
drunkards = np.zeros((n_steps+1, n_drunkards, n_dim))
for t in range(1, n_steps+1):
drunkards[t] = drunkards[t-1]*(1-homing_instinct) + rng.randn(n_drunkards, n_dim)
if t%100==0:
print('Status at step {}: Mean: {}, STD: {}'.format(t, drunkards[t].mean(), drunkards[t].std()))
plt.plot(drunkards[:, :, 0], drunkards[:, :, 1])
plt.grid()
plt.xlabel('$\Delta$ Longitude (arcseconds)')
plt.ylabel('$\Delta$ Latitude (arcseconds)')
plt.show()
demo_drunkards_walk.add_variant(homing_instinct = 0.01)
demo_drunkards_walk.add_variant(homing_instinct = 0.1)
We can now open browse_experiments(), and see that our record of experiment 0 is still saved, and we now have two new experiments which have not yet been run. We can run them by entering run 1,2.
demo_drunkards_walk.browse(close_after = True)
Note that we can also could also create variants of our variants if we wanted. For instance, if we wanted to try a drunkard's walk in 3D:
X = demo_drunkards_walk.add_variant(homing_instinct = 0.1)
X.add_variant(n_dim=3)Separating Display and Computation¶
The above is ok if our experiments run quickly and we just want to plot what the drunkards are doing. But we may want to do some other analysis on our results after running the experiment (without having to start again). Or we may simply want to change the way we plot our results, without having to re-run everythign. In these cases, it becomes beneficial to separate plotting from computing the results. We can use the display_function argument to do this. This display_function should accept the return value of your experiment as its first argument.
import numpy as np
from artemis.experiments import experiment_function, ExperimentFunction
from artemis.experiments.ui import browse_experiments
from matplotlib import pyplot as plt
from artemis.experiments.experiments import clear_all_experiments
clear_all_experiments() # Removes previous versions of demo_drunkards_walk that have been registered
%matplotlib notebook
def display_drunkards_walk(record):
print('===== CREATING PLOT OF RECORD {} NOW ===='.format(record.get_id()))
drunkards = record.get_result()
plt.plot(drunkards[:, :, 0], drunkards[:, :, 1])
plt.grid()
plt.xlabel('$\Delta$ Longitude (arcseconds)')
plt.ylabel('$\Delta$ Latitude (arcseconds)')
plt.show()
@ExperimentFunction(show=display_drunkards_walk)
def demo_drunkards_walk(n_steps=500, n_drunkards=5, homing_instinct = 0, n_dim=2, seed=1234):
"""
Release several drunkards in a field to randomly stumble around. Record their progress.
"""
rng = np.random.RandomState(seed)
drunkards = np.zeros((n_steps+1, n_drunkards, n_dim))
for t in range(1, n_steps+1):
drunkards[t] = drunkards[t-1]*(1-homing_instinct) + rng.randn(n_drunkards, n_dim)
if t%100==0:
print('Status at step {}: Mean: {}, STD: {}'.format(t, drunkards[t].mean(), drunkards[t].std()))
return drunkards
demo_drunkards_walk.add_variant(homing_instinct = 0.01)
demo_drunkards_walk.add_variant(homing_instinct = 0.1)
First, since we've changed the code for our experiment, we delete old experiments and run them all again (output not shown):
variants = demo_drunkards_walk.get_all_variants()
for experiment in variants:
for record in experiment.get_records():
record.delete()
experiment.run()
Then browse through our results, and view the results of experiment 1: "demo_drunkards_walk.homing_instinct=0.01", by entering show 1.
demo_drunkards_walk.browse(close_after=True)
Comparing Results across experiments¶
Eventually we want to compare the results of different experiments. For this, we can define the comparison_function argument. This accepts a dictionary, indexed by the experiment name, with values being the return values from saved experiments.
import numpy as np
from artemis.experiments import ExperimentFunction
from artemis.experiments.ui import browse_experiments
from matplotlib import pyplot as plt
from artemis.experiments.experiments import clear_all_experiments
clear_all_experiments() # Removes previous versions of demo_drunkards_walk that have been registered
%matplotlib notebook
def compare_drunkards_walk(records):
plot_handles = []
for i, record in enumerate(records):
drunkards = record.get_result()
plot_handles.append(plt.plot(drunkards[:, :, 0], drunkards[:, :, 1], color='C{}'.format(i)))
plt.grid()
plt.xlabel('$\Delta$ Longitude (arcseconds)')
plt.ylabel('$\Delta$ Latitude (arcseconds)')
plt.legend([p[0] for p in plot_handles], [record.get_experiment().get_id() for record in records])
plt.show()
@ExperimentFunction(compare=compare_drunkards_walk)
def demo_drunkards_walk(n_steps=500, n_drunkards=5, homing_instinct = 0, n_dim=2, seed=1234):
"""
Release several drunkards in a field to randomly stumble around. Record their progress.
"""
rng = np.random.RandomState(seed)
drunkards = np.zeros((n_steps+1, n_drunkards, n_dim))
for t in xrange(1, n_steps+1):
drunkards[t] = drunkards[t-1]*(1-homing_instinct) + rng.randn(n_drunkards, n_dim)
if t%100==0:
print('Status at step {}: Mean: {}, STD: {}'.format(t, drunkards[t].mean(), drunkards[t].std()))
return drunkards
demo_drunkards_walk.add_variant(homing_instinct = 0.01)
demo_drunkards_walk.add_variant(homing_instinct = 0.1)
Now you can compare the results by entering compare all
demo_drunkards_walk.browse(close_after=True)
Conclusion¶
The value of the experiment framework is that it lets you keep track of the things you've tried and the outcomes. This is intended to replace the mish-mash of solutions that people usually use when doing this kind of thing (e.g. saving old commands in terminal, writing results to file and manually loading them later, etc).