Quantitative Big Imaging

Scaling up / Big Data

ETHZ: 227-0966-00L

## Loading required package: knitr

Course Outline

Literature / Useful References

Big Data

Cluster Computing


Cloud Computing



There are three different types of problems that we will run into.

Really big data sets

Many datasets

Exploratory Studies

Example Projects

Zebra fish Full Animal Phenotyping

Full adult animal at cellular resolution 1000s of samples of full adult animals, imaged at 0.74 \(\mu m\) resolution: Images 11500 x 2800 x 628 \(\longrightarrow\) 20-40GVx / sample

Brain Project

Whole brain with cellular resolution 1 \(cm^3\) scanned at 1 \(\mu m\) resolution: Images \(\longrightarrow\) 1000 GVx / sample

What is wrong with usual approaches?

Normally when problems are approached they are solved for a single task as quickly as possible - I need to filter my image with a median filter with a neighborhood of 5 x 5 and a square kernel - then make a threshold of 10 - label the components - then count how many voxels are in each component - save it to a file


You have to rewrite everything, everytime

If you start with a bad approach, it is very difficult to fix, big data and reproducibility must be considered from the beginning

Computer Science Principles

Disclosure : There are entire courses / PhD thesis’s / Companies about this, so this is just a quick introduction

What is parallelism?

Parallelism is when you can divide a task into separate pieces which can then be worked on at the same time.

For example

Some tasks are easy to parallelize while others are very difficult. Rather than focusing on programming, real-life examples are good indicators of difficultly.

What is distributed computing?

Distributed computing is very similar to parallel computing, but a bit more particular. Parallel means you process many tasks at the same time, while distributed means you are no longer on the same CPU, process, or even on the same machine.

The distributed has some important implications since once you are no longer on the same machine the number of variables like network delay, file system issues, and other users becomes a major problem.

Distributed Computing Examples

  1. You have 10 friends who collectively know all the capital cities of the world.
  1. Each friend has some money with them

Resource Contention

The largest issue with parallel / distributed tasks is the need to access the same resources at the same time


Dining Philopher’s Problem - 6 philosophers at the table - 6 forks - Everyone needs two forks to eat - Each philospher takes the fork on his left

Parallel Challenges


Parallel computing requires a significant of coordinating between computers for non-easily parallelizable tasks.


The second major issue is mutability, if you have two cores / computers trying to write the same information at the same it is no longer deterministic (not good)


The simple act of taking turns and waiting for every independent process to take its turn can completely negate the benefits of parallel computing

Distributed Challenges

Inherits all of the problems of parallel programming with a whole variety of new issues.

Sending Instructions / Data Afar

Fault Tolerance

If you have 1000 computers working on solving a problem and one fails, you do not want your whole job to crash

Data Storage

How can you access and process data from many different computers quickly without very expensive infrastructure

Imperative Programming

Directly coordinating tasks on a computer.

Making a soup

  1. Buy vegetables at market
  2. then Buy meat at butcher
  3. then Chop carrots into pieces
  4. then Chop potatos into pieces
  5. then Heat water
  6. then Wait until boiling then add chopped vegetables
  7. then Wait 5 minutes and add meat


Making a soup


They look fairly similar, so what is the difference? The second is needlessly complicated for one person, but what if you have a team, how can several people make an imparitive soup faster (chopping vegetables together?)

Imperative soup

  1. Buy {carrots, peas, tomatoes} at market
  2. then Buy meat at butcher
  3. then Chop carrots into pieces
  4. then Chop potatos into pieces
  5. then Heat water
  6. then Wait until boiling then add chopped vegetables
  7. then Wait 5 minutes and add meat

How can many people make a declarative soup faster? Give everyone a different task (not completely efficient since some tasks have to wait on others)

Declarative soup




Lazy Evaluation


One of the major challenges of scaling up experiments and analysis is keeping all of the results organized in a clear manner. As we have seen in the last lectures, many of the results produced many text files - many files are difficult to organize - Matlab, R are designed for in-memory computation - Datasets can have many parameters and be complicated - Transitioning from Excel to Matlab or R means rewriting everything

Queue Computing

Queue processing systems (like Sun Grid Engine, Oracle Grid Engine, Apple XGrid, Condor, etc) are used to manage - resources (computers, memory, storage) - jobs (tasks to be run) - users Based on a set of rules for how to share the resources to the users to run tasks.




Structure of Cluster

Master (or Name) Node(s)

The node with which every other node communicates, the main address.

Worker Nodes

The nodes where the computation is performed.


The actual process that decides which jobs will run using which resources (worker nodes, memory, bandwidth) at which time


A database is a collection of data stored in the format of tables: a number of columns and rows.


Here we have an table of the animals measured in an experiment and their weight

id Weight
1 100
2 40
3 80


The cells is then an analysis looking at the cellular structures

Animal Type Anisotropy Volume
1 Cancer 0.5 1.00
1 Healthy 1.0 2.00
2 Cancer 0.5 0.95

Relational-databases can store relationships between different tables so the relationship between Animal in table Cells and id in table Animals can be preserved.


SQL (pronounced Sequel) stands for structured query language and is nearly universal for both searching (called querying) and adding (called inserting) data into databases. SQL is used in various forms from Firefox storing its preferences locally (using SQLite) to Facebook storing some of its user information (MySQL and Hive). So refering to the two tables we defined in the last entry, we can use SQL to get information about the tables independently of how they are stored (a single machine, a supercomputer, or in the cloud)

Basic queries

SELECT Volume FROM Cells \[ \rightarrow \begin{bmatrix} 1,2,0.95\end{bmatrix} \]

SELECT AVG(Volume) FROM Cells WHERE Type = "Cancer" \[ \rightarrow 0.975 \]

We could have done these easily without SQL using Excel, Matlab or R

More Advanced SQL

SELECT ATable.Weight,CTable.Volume FROM Animals as ATable 
  INNER JOIN Cells as CTable on (ATable.id=CTable.Animal)

\[ \rightarrow \begin{bmatrix} 1,0.95\end{bmatrix} \]

Networks using SQL

If we expand our SQL example to cellular networks with an additional table explicitly describing the links between cells

Table Representation

id1 id2 No.Juncs
1 2 800
1 3 40
3 1 300

Now to calculate how many connections each cell has

SELECT id,COUNT(*) AS connection_count FROM Cells 
  INNER JOIN Network ON (id=id1 OR id=id2)

\[ \rightarrow \begin{bmatrix} (1 & 3) \\ (2 & 1) \\ (3 & 2)\end{bmatrix} \]

Beyond SQL: NoSQL

Basic networks can be entered and queries using SQL but relatively simple sounding requests can get complicated very quickly

How many cells are within two connections of each cell

SELECT id,COUNT(*) AS connection_count FROM Cells as CellsA
  INNER JOIN Network as NetA ON Where (id=NetA.id1)
  INNER JOIN Network as NetB ON Where (NetA.id2=NetB.id1)

This is still readable but becomes very cumbersome quickly and difficult to manage

NoSQL (Not Only SQL)

A new generation of database software which extends the functionality of SQL to allow for more scalability (MongoDB) or specificity for problems like networks or graphs called generally Graph Databases

Big Data: Definition

Velocity, Volume, Variety

When a ton of heterogeneous is coming in fast.

Performant, scalable, and flexible

When scaling isn’t scary

10X, 100X, 1000X is the same amount of effort

When you are starving for enough data

Director of AMPLab said their rate limiting factor is always enough interesting data

O ‘clicks’ per sample

A brief oversimplified story

Google ran into ‘big data’ and its associated problems years ago: Peta- and exabytes of websites to collect and make sense of. Google uses an algorithm called PageRank(tm) for evaluating the quality of websites. They could have probably used existing tools if page rank were some magic program that could read and determine the quality of a site

for every_site_on_internet

Just divide all the websites into a bunch of groups and have each computer run a group, easy!


While the actual internals of PageRank are not public, the general idea is that sites are ranked based on how many sites link to them

for current_site in every_site_on_internet
  current_pagerank = new SecretPageRankObj(current_site);
  for other_site in every_site_on_internet
    if current_site is_linked_to other_site

How do you divide this task? - Maybe try and divide the sites up: english_sites, chinese_sites, … - Run pagerank and run them separately. - What happens when a chinese_site links to an english_site? - Buy a really big, really fast computer? - On the most-powerful computer in the world, one loop would take months

It gets better

Google’s Solution: MapReduce (part of it)

some people claim to have had the idea before, Google is certainly the first to do it at scale

Several engineers at Google recognized common elements in many of the tasks being performed. They then proceeded to divide all tasks into two classes Map and Reduce


Map is where a function is applied to every element in the list and the function depends only on exactly that element \[ \vec{L} = \begin{bmatrix} 1,2,3,4,5 \end{bmatrix} \] \[ f(x) = x^2 \] \[ map(f \rightarrow \vec{L}) = \begin{bmatrix} 1,4,9,16,25 \end{bmatrix} \]


Reduce is more complicated and involves aggregating a number of different elements and summarizing them. For example the \(\Sigma\) function can be written as a reduce function \[ \vec{L} = \begin{bmatrix} 1,2,3,4,5 \end{bmatrix} \] \[ g(a,b) = a+b \] Reduce then applies the function to the first two elements, and then to the result of the first two with the third and so on until all the elements are done \[ reduce(f \rightarrow \vec{L}) = g(g(g(g(1,2),3),4),5) \]


They designed a framework for handling distributing and running these types of jobs on clusters. So for each job a dataset (\(\vec{L}\)), Map-task (\(f\)), a grouping, and Reduce-task (\(g\)) are specified

All of the steps in between can be written once in a robust, safe manner and then used for every task which can be described using this MapReduce paradigm. These tasks \(\langle \vec{L}, f(x), g(a,b) \rangle\) is refered to as a job.

Key-Value Pairs / Grouping

The initial job was very basic, for more complicated jobs, a new notion of Key-value (KV) pairs must be introduced. A KV pair is made up of a key and value. A key must be comparable / hashable (a number, string, immutable list of numbers, etc) and is used for grouping data. The value is the associated information to this key.

Counting Words

Using MapReduce on a folder full of text-documents. - \[ \vec{L} = \begin{bmatrix} "\textrm{Info}\cdots", "\textrm{Expenses}\cdots",\cdots \end{bmatrix} \] - Map is then a function \(f\) which takes in a long string and returns a list of all of the words (text seperated by spaces) as key-value pairs with the value being the number of times that word appeared - f(x) = [(word,1) for word in x.split(" ")] - Grouping is then performed by keys (group all words together) - Reduce adds up the values for each word

L = ["cat dog car",
  "dog car dog"]

\[ \downarrow \textbf{ Map } : f(x) \]


\[ \downarrow \textrm{ Shuffle / Group} \]

"cat": (1)
"dog": (1,1,1)
"car": (1,1)

\[ \downarrow \textbf{ Reduce } : g(a,b) \]



Hadoop is the opensource version of MapReduce developed by Yahoo and released as an Apache project. It provides underlying infrastructure and filesystem that handles storing and distributing data so each machine stores some of the data locally and processing jobs run where the data is stored. - Non-local data is copied over the network. - Storage is automatically expanded with processing power. - It’s how Amazon, Microsoft, Yahoo, Facebook, … deal with exabytes of data

Spark / Resilient Distributed Datasets

Technical Specifications

Zaharia, M., et. al (2012). Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing

Practical Specification

Cloud Computing

Cloud Computing

Cloud Resources

Near Future Imaging Goals


Would be nice

Tomcat Goals

Tomcat Goals