From microscopes to satellites: Interactive Image Analysis at Scale Using Spark

• Imaging in 2015
• Changing Computing
• Small Data vs Big Data
• The Problem(s)

The Tools

• Our Image Analysis
• 3D Imaging
• Hyperspectral Imaging
• Interactive Analysis / Streaming

The Science

• Academic Projects
• Commercial Projects
• Outlook / Developments

X-Ray

• Swiss Light Source (SRXTM) images at (>1000fps) \( \rightarrow \) 8GB/s, diffraction patterns (cSAXS) at 30GB/s
• Nanoscopium (Soleil), 10TB/day, 10-500GB file sizes, very heterogenous data
• Swiss Radiologists will make 1PB of data year

Optical

• Light-sheet microscopy (see talk of Jeremy Freeman) produces images \( \rightarrow \) 500MB/s
• High-speed confocal images at (>200fps) \( \rightarrow \) 78Mb/s

Geospatial

• New satellite projects (Skybox, etc) will measure >10 petabytes of images a year

Personal

• GoPro 4 Black - 60MB/s (3840 x 2160 x 30fps) for $600
• fps1000 - 400MB/s (640 x 480 x 840 fps) for $400

The figure highlights the exponential growth in both:

• the amount of data
• analysis time

over the last 16 years based on the latest generation detector at the TOMCAT Beamline.

It assumes a manual analysis rate of 1Mpx/second

The breakdown of science into component tasks has changed from a time perspective over the last years and how we expect it to change by 2020.

• Experiment Design (Pink)
• Data Management (Blue)
• Measurement Time (Green)
• Post Processing (Purple)

The primary trends to focus on are the rapid decrease in measurement time and increasing importantance of post-processing.

If you looked at one 1000 x 1000 sized image every second

It would take you
139 hours to browse through a terabyte of data.

Moores Law

\[ \textrm{Transistors} \propto 2^{T/(\textrm{18 months})} \]

Based on data from https://gist.github.com/humberto-ortiz/de4b3a621602b78bf90d

There are now many more transistors inside a single computer but the processing speed hasn't increased. How can this be?

• Multiple Core
  • Many machines have multiple cores for each processor which can perform tasks independently
• Multiple CPUs
  • More than one chip is commonly present
• New modalities
  • GPUs provide many cores which operate at slow speed

Parallel Code is important

The figure shows the range of cloud costs (determined by peak usage) compared to a local workstation with utilization shown as the average number of hours the computer is used each week.

The figure shows the cost of a cloud based solution as a percentage of the cost of buying a single machine. The values below 1 show the percentage as a number. The panels distinguish the average time to replacement for the machines in months

Big Data is a very popular term today. There is a ton of hype and seemingly money is being thrown at any project which includes these words. First to get over the biggest misconception big data isn't about how much data you have, it is how you work with it

Before we can think about Big Data we need to define what it's replacing, Small Data.

Small Data

The 0815 approach, using standard tools and lots of clicking

High-throughput \( \rightarrow \) easy measurement

analysis \( \rightarrow \) complex, multistep, and time-consuming

Genetic studies require hundreds to thousands of samples, in this case the difference between 717 and 1200 samples is the difference between finding the links and finding nothing.

• Hand Identification \( \rightarrow \) 30s / object
• 30-40k objects per sample
• One Sample in 6.25 weeks
• 1300 samples \( \rightarrow \) 120 man-years

Advantages

• Hand verification of every sample, visual identification of errors and problems

Disadvantages

• Biased by user
• Questionable reproducibility
• Time-consuming
• Exploration challenging
• Data versioning is difficult

• ImageJ macro for segmentation
• Python script for shape analysis
• Paraview macro for network and connectivity
• Python script to pool results
• MySQL Database storing results (5 minutes / query)

1.5 man-years

Advantages

• Faster than manual methods
• Less bias
• More reproducible

Disadvantages

• Compartmentalized analysis
• Time-consuming
• Complex interdependent workflow
• Fragile (machine crashes, job failures)
• Expensive - Measure twice, cut once, not exploratory

Correctness

The most important job for any piece of analysis is to be correct.

• A powerful testing framework is essential
• Avoid repetition of code which leads to inconsistencies
• Use compilers to find mistakes rather than users

Easily understood, changed, and used

Almost all image processing tasks require a number of people to evaluate and implement them and are almost always moving targets

• Flexible, modular structure that enables replacing specific pieces

Fast

The last of the major priorities is speed which covers both scalability, raw performance, and development time.

• Long waits for processing discourages exploration
• Manual access to data on separeate disks is a huge speed barrier
• Real-time image processing requires millisecond latencies
• Implementing new ideas can be done quickly

• Rather than building an analysis as quickly as possible and then trying to hack it to scale up to large datasets
  • chose the framework first
  • then start making the necessary tools.
• Google, Amazon, Yahoo, and many other companies have made huge in-roads into these problems
• The real need is a fast, flexible framework for robustly, scalably performing complicated analyses, a sort of Excel for big imaging data.

Apache Spark and Hadoop 2

The two frameworks provide a free out of the box solution for

• scaling to >10000 computers
• storing and processing exabytes of data
• fault tolerance
  • 2/3rds of computers can crash and a request still accurately finishes
• hardware and software platform indpendence (Mac, Windows, Linux)

These frameworks are really cool and Spark has a big vocabulary, but flatMap, filter, aggregate, join, groupBy, and fold still do not sound like anything I want to do to an image.

I want to

• filter out noise, segment, choose regions of interest
• contour, component label
• measure, count, and analyze
• …

Spark Image Layer

• Developed at 4Quant, ETH Zurich, and Paul Scherrer Institut
• The Spark Image Layer is a Domain Specific Language for Microscopy for Spark.
• It converts common imaging tasks into coarse-grained Spark operations

We have developed a number of commands for SIL handling standard image processing tasks

Fully exensible with

For many traditional scientific studies, the sample count is fairly low 10-100, for such low counts, if we look at the costs (estimated from our experience with cortical bone microstructure and rheology)

• Development costs are very high for Big
• Traditionally students are much cheaper than the 150kCHF / year used in this model
• Medium Data is a good compromise

As studies get bigger (>100 samples), we see that these costs begin to radically shift.

• The high development costs in Medium and Big \( \rightarrow \) 0
• The high human costs of Small and Medium \( \propto \) \( Samples \)

We can then visualize these contours easily

or apply them back to the original map

\[ \downarrow \textrm{Perform analysis on a every image} \]

\[ \downarrow \textrm{Filter out the most anisotropic cells} \]

The query is processed by the SQL parsing engine

• sent to the database where it can be combined with locally available private data
• sent to the image layer for all image processing components to be integrated
• sent to the Spark Core so specific map, reduce, fold operations can be generated
• sent to the master node where it can be distributed across the entire cluster / cloud

Pathology

Adult Zebrafish Imaging

• In collaboration with K. Cheng (PMC), …
• Store entire image collection online
• Manually segment hundreds of cells
• Determine feature vector for automatic segmentation

Viral Infection

• In collaboration with M. Gerber (UniZurich)
• Store all images for a time-series
• Identify cells, nucleus, and infected regions
• Track infection progression and colocalize with cells

Nexus Personalized Medicine Platform

• In collaboration with M. Prummel
• Images for large studies stored in OpenBIS
• Run CellProfiler-style analysis on entire collection of images directly in database
• Use approximate results to rapidly, interactively screen

Mouse Brain Structure

• In collaboration with (A. Patera, A. Astolfo, M. Schneider , B. Weber)
• Store all images in one place
• Iteratively determine image offsets using cross-correlation
• Stitch images together as needed for analysis / view generation

In the biological imaging community, the open source tools of ImageJ2 and Fiji are widely accepted and have a large number of readily available plugins and tools.

We can integrate the functionality directly into Spark and perform operations on much larger datasets than a single machine could have in memory. Additionally these analyses can be performed on streaming data.

A spinoff - 4Quant: From images to insight

• Quantitative Search Machine for Images
  • Find all patients with livers larger than 10cm diameter
  • Count the number of highly anisotropic nuclei in myeloma patients
• Custom Analysis Solutions
  • Custom-tailored software to solve your problems
• One Stop Shop
  • Measurement, analysis, and statistical analysis

Education / Training

• Consulting
  • Advice on imaging techniques, analysis possibilities
• Education
  • Workshops on Image Analysis
  • Courses / Training
  • Quantitative Big Imaging
• Contact Us: contact@4quant.com

• Flavio Trolese

• Dr. Prat Das Kanungo

• Dr. Ana Balan

• Prof. Marco Stampanoni

• AIT at PSI and Scientific Computer at ETH
• TOMCAT Group

We are interested in partnerships and collaborations

Learn more at

• 4Quant: From Images to Statistics - http://www.4quant.com
• Quantitative Big Imaging Course at ETH Zurich
• X-Ray Imaging Group at ETH Zurich - http://bit.ly/1gD8wKb

A pairing between spatial information (position) and some other kind of information (value). \[ \vec{x} \rightarrow \vec{f} \]

We are used to seeing images in a grid format where the position indicates the row and column in the grid and the intensity (absorption, reflection, tip deflection, etc) is shown as a different color

The alternative form for this image is as a list of positions and a corresponding value

\[ \hat{I} = (\vec{x},\vec{f}) \]

This representation can be called the feature vector and in this case it only has Intensity

If we use feature vectors to describe our image, we are no longer to worrying about how the images will be displayed, and can focus on the segmentation/thresholding problem from a classification rather than a image-processing stand point.

Example

So we have an image of a cell and we want to identify the membrane (the ring) from the nucleus (the point in the middle).

A simple threshold doesn't work because we identify the point in the middle as well. We could try to use morphological tricks to get rid of the point in the middle, or we could better tune our segmentation to the ring structure.

In this case we add a very simple feature to the image, the distance from the center of the image (distance).

We now have a more complicated image, which we can't as easily visualize, but we can incorporate these two pieces of information together.

Now instead of trying to find the intensity for the ring, we can combine density and distance to identify it

\[ iff (5<\textrm{Distance}<10 \\ \& 0.5<\textrm{Intensity}>1.0) \]

The distance while illustrative is not a commonly used features, more common various filters applied to the image

• Gaussian Filter (information on the values of the surrounding pixels)
• Sobel / Canny Edge Detection (information on edges in the vicinity)
• Entroy (information on variability in vicinity)

The distributions of the features appear very different and can thus likely be used for identifying different parts of the images.

Combine this with our a priori information (called supervised analysis)

Now that the images are stored as feature vectors, they can be easily analyzed with standard Machine Learning tools. It is also much easier to combine with training information.

Want to predict Training from x,y, Absorb, and Scatter \( \rightarrow \) MLLib: Logistic Regression, Random Forest, K-Nearest Neighbors, …

Apache Spark is brilliant platform and utilizing GraphX, MLLib, and other packages there unlimited possibilities

• Scala can be a beautiful but not easy language
• Python is an easier language
• Both suffer from
  • Non-obvious workflows
  • Scripts depending on scripts depending on scripts (can be very fragile)

• Although all analyses can be expressed as a workflow, this is often difficult to see from the code
• Non-technical persons have little ability to understand or make minor adjustments to analysis
• Parameters require recompiling to change
• or GUIs need to be placed on top

Combining many different components together inside of the Spark Shell, IPython or Zeppelin, make it easier to assemble workflows

Web and Image Database Examples

Here we use a KNIME -based workflow and our Spark Imaging Layer extensions to create a workflow without any Scala or programming knowledge and with an easily visible flow from one block to the next without any performance overhead of using other tools.

• Collaboration with A. Astolfo and A. Patera
• Measure a full mouse brain (1cm\( ^3 \)) with cellular resolution (1 \( \mu \) m)
• 10 x 10 x 10 scans at 2560 x 2560 x 2160 \( \rightarrow \) 14 TVoxels
• 0.000004% of the entire dataset

• 14TVoxels = 56TB
• Each scan needs to be registered and aligned together
• There are no computers with 56TB of memory
• Even multithreaded approachs are not feasible and require many logistics
• Analysis of the stitched data is also of interest (segmentation, vessel analysis, distribution and network connectivity)

\[ \textrm{Images}: \textrm{RDD}[((x,y,z),Img[Double])] =\\ \left[(\vec{x},\textrm{Img}),\cdots\right] \]

dispField = Images.
  cartesian(Images).map{
  ((xA,ImgA), (xB,ImgB)) =>
    xcorr(ImgA,ImgB,in=xB-xA)
  }

From the updated information provided by the cross correlations and by applying appropriate smoothing criteria (if necessary).

The stitching itself, rather than rewriting the original data can be done in a lazy fashion as certain regions of the image are read.

def getView(tPos,tSize) =
  stImgs.
  filter(x=>abs(x-tPos)<img.size).
  map { (x,img) =>
   val oImg = new Image(tSize)
   oImg.copy(img,x,tPos)
}.addImages(AVG)

This also ensures the original data is left unaltered and all analysis is reversible.

The attending physician or the general practioner is the integrator, responsible for combining, and interpreting a number of complex, often contradictory inputs.

The centralized storage and analysis of images provides an alternative of backup pathway so that the risk that something is overlooked is lower

Eventually this could lead to the GP getting second opinions automatically from Deep Learning tools or IBM's Watson and even allowing the patient to directly view and interpret some of the results