• Who am I?
• Who are you?
• Why does this workshop exist?
• General Philosophy
• Why is quantitative imaging important?

• Introduction
• Images
• Filtering
• Segmenting
• Statistics

• Kevin Mader (mader@biomed.ee.ethz.ch)
  • Lecturer at ETH Zurich
  • Postdoc in the X-Ray Microscopy Group at ETH Zurich and Swiss Light Source at Paul Scherrer Institute
  • Spin-off 4Quant for Big Data with Images

What have I done with images?

• Quantitative Big Imaging Course
  • kmader.github.io/Quantitative-Big-Imaging-2015/

Projects

• Relationship between structure and genetic background for 1300 bones
• Tracking bubbles in a liquid foam
• Finding throat and neck cancer in 100s of patients

A wide spectrum of backgrounds

• Biologists, Biomedical Engineers, Physicists, Chemists, Mechanical Engineers, …

A wide range of skills

• I think I've heard of Matlab before \( \rightarrow \) I write template C++ code and hand optimize it afterwards

So how will this ever work?

Adaptive assignments

1. Conceptual, graphical assignments with practical examples
   • Emphasis on chosing correct steps and understanding workflow
2. Opportunities to create custom implementations, plugins, and perform more complicated analysis on larger datasets if interested
   • Emphasis on performance, customizing analysis, and scalability

Install this software

• KNIME Setup
• Install the latest Image Processing tools

Software Know-how

• Comfortable with the first few exercises (2,3)
• Able to install KNIME

General Material

• Jean Claude, Morphometry with R
  • Online through ETHZ
  • Buy it
• John C. Russ, “The Image Processing Handbook”,(Boca Raton, CRC Press)
  • Available online within domain ethz.ch (or proxy.ethz.ch / public VPN)
• J. Weickert, Visualization and Processing of Tensor Fields
  • Online

Today

• Imaging
  • ImageJ and SciJava
• Cloud Computing
  • The Case for Energy-Proportional Computing _ Luiz André Barroso, Urs Hölzle, IEEE Computer, December 2007_
  • Concurrency
• Reproducibility
  • Trouble at the lab Scientists like to think of science as self-correcting. To an alarming degree, it is not
  • Why is reproducible research important? The Real Reason Reproducible Research is Important
  • Science Code Manifesto
  • Reproducible Research Class @ Johns Hopkins University

What is the purpose?

• Discover and validate new knowledge

How?

• Use the scientific method as an approach to convince other people
• Build on the results of others so we don't start from the beginning

Important Points

• While qualitative assessment is important, it is difficult to reliably produce and scale
  • Quantitative analysis is far from perfect, but provides metrics which can be compared and regenerated by anyone

Inspired by: imagej-pres

• Images are great for qualitative analyses since our brains can quickly interpret them without large programming investements.

• Proper processing and quantitative analysis is however much more difficult with images.

  • If you measure a temperature, quantitative analysis is easy, \( 50K \), the average temperature also has a meaning
  • If you measure an image it is much more difficult and much more prone to mistakes, subtle setup variations, and confusing analyses

• 83% of Harvard Radiologists thought this scan looked perfectly normal
• http://www.npr.org/sections/health-shots/2013/02/11/171409656/why-even-radiologists-can-miss-a-gorilla-hiding-in-plain-sight
• called inattentional blindness
  • since the focus was on counting cancerous lung nodules
  • standard sanity checks were never made

Human vision system is imperfect

Which center square seems brighter?

Are the intensities constant in the image?

• Count how many cells are in the bone slice
• Ignore the ones that are ‘too big’ or shaped ‘strangely’
• Are there more on the right side or left side?
• Are the ones on the right or left bigger, top or bottom?

• Do it all over again for 96 more samples, this time with 2000 slices instead of just one!

• Now again with 1090 samples!

• Those metrics were quantitative and could be easily visually extracted from the images
• What happens if you have softer metrics

• How aligned are these cells?
• Is the group on the left more or less aligned than the right?
• errr?

Science demands repeatability! and really wants reproducability

• Experimental conditions can change rapidly and are difficult to make consistent
• Animal and human studies are prohibitively time consuming and expensive to reproduce
• Terabyte datasets cannot be easily passed around many different groups
• Privacy concerns can also limit sharing and access to data

• Science is already difficult enough
• Image processing makes it even more complicated
• Many image processing tasks are multistep, have many parameters, use a variety of tools, and consume a very long time

How can we keep track of everything for ourselves and others?

• We can make the data analysis easy to repeat by an independent 3rd party

Easy to follow the list, anyone with the right steps can execute and repeat (if not reproduce) the soup

Simple Soup

1. Buy {carrots, peas, tomatoes} at market
2. then Buy meat at butcher
3. then Chop carrots into pieces
4. then Chop potatos into pieces
5. then Heat water
6. then Wait until boiling then add chopped vegetables
7. then Wait 5 minutes and add meat

More complicated soup

Here it is harder to follow and you need to carefully keep track of what is being performed

Steps 1-4

1. then Mix carrots with potatos \( \rightarrow mix_1 \)
2. then add egg to \( mix_1 \) and fry for 20 minutes
3. then Tenderize meat for 20 minutes
4. then add tomatoes to meat and cook for 10 minutes \( \rightarrow mix_2 \)
5. then Wait until boiling then add \( mix_1 \)
6. then Wait 5 minutes and add \( mix_2 \)

Simple Soup

Complicated Soup

Clearly a linear set of instructions is ill-suited for even a fairly easy soup, it is then even more difficult when there are dozens of steps and different pathsways

Furthermore a clean workflow allows you to better parallelize the task since it is clear which tasks can be performed independently

A very abstract definition: A pairing between spatial information (position) and some other kind of information (value).

In most cases this is a 2 dimensional position (x,y coordinates) and a numeric value (intensity)

This can then be rearranged from a table form into an array form and displayed as we are used to seeing images

The next step is to apply a color map (also called lookup table, LUT) to the image so it is a bit more exciting

Which can be arbitrarily defined based on how we would like to visualize the information in the image

Formally a lookup table is a function which \[ f(\textrm{Intensity}) \rightarrow \textrm{Color} \]

These transformations can also be non-linear as is the case of the graph below where the mapping between the intensity and the color is a \( \log \) relationship meaning the the difference between the lower values is much clearer than the higher ones

On a real image the difference is even clearer

Changing a 'lookup table' can also be called “Normalization”, “Equalization”, “Auto-Enhance” and many other names. It must be used very carefully when displaying scientific results.

• \[ \Downarrow \textrm{After} \]

• 

For a 3D image, the position or spatial component has a 3rd dimension (z if it is a spatial, or t if it is a movie)

This can then be rearranged from a table form into an array form and displayed as a series of slices

Control of in vitro tissue-engineered bone-like structures using human mesenchymal stem cells and porous silk scaffolds in Biomaterials 2007 by Sandra Hofmann, et. al

Hypothesis

tissue engineered bone-like structure resulting from silk fibroin (SF) implants is pre-determined by the scaffolds’ geometry

Experiment

SF scaffolds with different pore diameters were prepared and seeded with human mesenchymal stem cells (hMSC). As compared to static seeding, dynamic cell seeding in spinner flasks resulted in equal cell viability and proliferation, and better cell distribution throughout the scaffold as visualized by histology and confocal microscopy

Natural bone consists of cortical and trabecular morphologies, the latter having variable pore sizes. This study aims at engineering different bone-like structures using scaffolds with small pores (112–224 μm) in diameter on one side and large pores (400–500 μm) on the other, while keeping scaffold porosities constant among groups. We hypothesized that tissue engineered bone-like structure resulting from silk fibroin (SF) implants is pre-determined by the scaffolds’ geometry. To test this hypothesis, SF scaffolds with different pore diameters were prepared and seeded with human mesenchymal stem cells (hMSC). As compared to static seeding, dynamic cell seeding in spinner flasks resulted in equal cell viability and proliferation, and better cell distribution throughout the scaffold as visualized by histology and confocal microscopy, and was, therefore, selected for subsequent differentiation studies. Differentiation of hMSC in osteogenic cell culture medium in spinner flasks for 3 and 5 weeks resulted in increased alkaline phosphatase activity and calcium deposition when compared to control medium. Micro-computed tomography (μCT) detailed the pore structures of the newly formed tissue and suggested that the structure of tissue-engineered bone was controlled by the underlying scaffold geometry.

Qualitative

Quantitative

• uneven illumination

• blur from the optical system

• dirt on the lens

• noise from the camera

The light which is reflected by the object is measured by the camera.

• Bright\( \rightarrow \) very reflective

• Dark \( \downarrow \) not reflective

• Can be quantitative for flat objects

• Not quantitative for uneven surfaces (shadows)

The light which passes through is measured

• Quantitative for single planes
• More difficult with multiple planes

It is important to understand the contrasts well since that determines everything else for our further processing and interpretation of the images. Specifically we can focus on quantitative and qualitative contrasts.

Qualitative (H&E)

• Shows anatomy / structures
• but more intense/darker structures are not more significant
• Automatic thresholding is practical

Quantitative (von Kossa)

• Shows functionality
• Intensity is correlated to a metric
• For X-ray tomography the intensity is proportional to the absorption coefficient which is related to the calcification density
• Rescaling / shifting is not allowed
• Automatic thresholding is not practical

Segmentation and all the steps leading up to it are really a specialized type of learning problem.

Returning to the ring image we had before, we start with our knowledge or ground-truth of the ring

Which we want to identify the from the following image by using image processing tools

What does identify mean?

• Classify the pixels in the ring as Foreground
• Classify the pixels outside of the ring as Background

• True Positive values in the ring that are classified as Foreground
• True Negative values outside the ring that are classified as Background
• False Positive values outside the ring that are classified as Foreground
• False Negative values in the ring that are classified as Background

We can then apply a threshold to the image to determine the number of points in each category

Try a number of different threshold values on the image and compare them to the original classification

• Recall (sensitivity)= \( TP/(TP+FN) \)
• Precision = \( TP/(TP+FP) \)

ROC Curve

Reciever Operating Characteristic (first developed for WW2 soldiers detecting objects in battlefields using radar). The ideal is the top-right (identify everything and miss nothing)

We can then use this ROC curve to compare different filters (or even entire workflows), if the area is higher the approach is better.

Different approaches can be compared by area under the curve

Another way of showing the ROC curve (more common for machine learning rather than medical diagnosis) is using the True positive rate and False positive rate

• True Positive Rate (recall)= \( TP/(TP+FN) \)
• False Positive Rate = \( FP/(FP+TN) \)

These show very similar information with the major difference being the goal is to be in the upper left-hand corner. Additionally random guesses can be shown as the slope 1 line. Therefore for a system to be useful it must lie above the random line.

While finding a ring might be didactic, it is not really a relevant problem and these terms are much more meaningful when applied to medical images where every False Positives and False Negative can be mean life-threatening surgery or the lack thereof. (Data courtesy of Zhentian Wang)

From these images, an expert labeled the calcifications by hand, so we have ground truth data on where they are:

We can perform the same analysis on an image like this one, again using a simple threshold to evalulate how accurately we identify the calcifications

• In model-based analysis every step we peform, simple or complicated is related to an underlying model of the system we are dealing with
• Occam's Razor is very important here : The simplest solution is usually the right one

• What comes out of our detector / enhancement process

• Identify objects by eye
  • Count, describe qualitatively: “many little cilia on surface”, “long curly flaggelum”, “elongated nuclear structure”
• Morphometrics
  • Trace the outline of the object (or sub-structures)
  • Can calculate the area by using equal-weight-paper
  • Employing the “cut-and-weigh” method

• \( F_{system}(a,b) = a*b \)
• \( I_{stimulus} = \textrm{Beam}_{profile} \)
• \( S_{system} = \alpha(\vec{x}) \)

\( \longrightarrow \alpha(\vec{x})=\frac{I_{measured}(\vec{x})}{\textrm{Beam}_{profile}(\vec{x})} \)

In many setups there is un-even illumination caused by incorrectly adjusted equipment and fluctations in power and setups

Frequently there is a fall-off of the beam away from the center (as is the case of a Gaussian beam which frequently shows up for laser systems). This can make extracting detail away from the center challenging

• For absorption/attenuation imaging \( \rightarrow \) Beer-Lambert Law \[ I_{detector} = \underbrace{I_{source}}_{I_{stimulus}}\underbrace{\exp(-\alpha d)}_{S_{sample}} \]

  • Different components have a different \( \alpha \) based on the strength of the interaction between the light and the chemical / nuclear structure of the material \[ I_{sample}(x,y) = I_{source}\exp(-\alpha(x,y) d) \] \[ \alpha = f(N,Z,\sigma,\cdots) \]

• For segmentation this model is:

  • there are 2 (or more) distinct components that make up the image
  • these components are distinguishable by their values (or vectors, colors, tensors, …)

Start out with a simple image of a cross with added noise \[ I(x,y) = f(x,y) \]

The intensity can be described with a probability density function \[ P_f(x,y) \]

By examining the image and probability distribution function, we can deduce that the underyling model is a whitish phase that makes up the cross and the darkish background

Applying the threshold is a deceptively simple operation

\[ I(x,y) = \begin{cases} 1, & f(x,y)\geq0.5 \\ 0, & f(x,y)<0.5 \end{cases} \]

• We can peform the same sort of analysis with this image of cells
• This time we can derive the model from the basic physics of the system
  • The field is illuminated by white light of nearly uniform brightness
  • Cells absorb light causing darker regions to appear in the image
  • Lighter regions have no cells
  • Darker regions have cells

There are two broad classes of data and scientific studies.

Observational

• Exploring large datasets looking for trends
• Population is random
• Not always hypothesis driven
• Rarely leads to causation

We examined 100 people and the ones with blue eyes were on average 10cm taller

In 100 cake samples, we found a 0.9 correlation between cooking time and bubble size

Controlled

• Most scientific studies fall into this category
• Specifics of the groups are controlled
• Can lead to causation

We examined 50 mice with gene XYZ off and 50 gene XYZ on and as the foot size increased by 10%

We increased the temperature and the number of pores in the metal increased by 10%

Coin Flip

1. Each flip gives us a small piece of information about the flipper and the coin
2. More flips provides more information
   • Random / Stochastic variations in coin and flipper cancel out
   • Systematic variations accumulate

Real Experiment

1. Each measurement tells us about our sample, out instrument, and our analysis
2. More measurements provide more information
   • Random / Stochastic variations in sample, instrument, and analysis cancel out
   • Normally the analysis has very little to no stochastic variation
   • Systematic variations accumulate

Once the reproducibility has been measured, it is possible to compare groups. The idea is to make a test to assess the likelihood that two groups are the same given the data

1. List assumptions
2. Establish a null hypothesis
   • Usually both groups are the same
3. Calculate the probability of the observations given the truth of the null hypothesis
   • Requires knowledge of probability distribution of the data
   • Modeling can be exceptionally complicated

Loaded Coin

We have 1 coin from a magic shop

• our assumptions are
  • we flip and observe flips of coins accurately and independently
  • the coin is invariant and always has the same expected value
• our null hypothesis is the coin is unbiased \( E(\mathcal{X})=0.5 \)
• we can calculate the likelihood of a given observation given the number of flips (p-value)

How good is good enough?

Since we do not usually know our distribution very well or have enough samples to create a sufficient probability model

Student T Distribution

We assume the distribution of our stochastic variable is normal (Gaussian) and the t-distribution provides an estimate for the mean of the underlying distribution based on few observations.

• We estimate the likelihood of our observed values assuming they are coming from random observations of a normal process

Student T-Test

Incorporates this distribution and provides an easy method for assessing the likelihood that the two given set of observations are coming from the same underlying process (null hypothesis)

• Assume unbiased observations
• Assume normal distribution

Back to the magic coin, let's assume we are trying to publish a paper, we heard a p-value of < 0.05 (5%) was good enough. That means if we get 5 heads we are good!

Clearly this is not the case, otherwise we could keep flipping coins or ask all of our friends to flip until we got 5 heads and publish

The p-value is only meaningful when the experiment matches what we did.

• We didn't say the chance of getting 5 heads ever was < 5%
• We said if we have exactly 5 observations and all of them are heads the likelihood that a fair coin produced that result is <5%

Many methods to correct, most just involve scaling \( p \). The likelihood of a sequence of 5 heads in a row if you perform 10 flips is 5x higher.

This is very bad news for us. We have the ability to quantify all sorts of interesting metrics

• cell distance to other cells
• cell oblateness
• cell distribution oblateness

So lets throw them all into a magical statistics algorithm and push the publish button

With our p value of less than 0.05 and a study with 10 samples in each group, how does increasing the number of variables affect our result

Using the simple correction factor (number of tests performed), we can make the significant findings constant again

So no harm done there we just add this correction factor right? Well what if we have exactly one variable with shift of 1.0 standard deviations from the other.

There are two representations for videos

• a list of images
• a single 3- or 4-D image \( (X,Y,Z,t) \)

List of Images

• More processing and memory efficient
• More options for looking through

3-/4-D Image

• Large sizes in memory
• Possibility to combine neighbor images

In-between

• A number of smaller 3-/4D Images
• Chunked together (eg. 10 frames each)