Loading [MathJax]/jax/output/HTML-CSS/jax.js
  • 2. Statistics
    • Built-in support for statistics
    • Distribution functions
    • Example
    • Create a normal distribution curve
    • Simple testing
    • Example analysis
    • Linear Regression
    • Exercise: Exercise 6
    • More words of warning

2. Statistics

Built-in support for statistics

  • R is a statistical programming language:
    • Classical statistical tests are built-in
    • Statistical modeling functions are built-in
    • Regression analysis is fully supported
    • Additional mathematical packages are available (MASS, Waves, sparse matrices, etc)

Distribution functions

  • Most commonly used distributions are built-in, functions have stereotypical names, e.g. for normal distribution:
    • pnorm - cumulative distribution for x
    • qnorm - inverse of pnorm (from probability gives x)
    • dnorm - distribution density
    • rnorm - random number from normal distribution
distributions

distributions

  • Available for variety of distributions: punif (uniform), pbinom (binomial), pnbinom (negative binomial), ppois (poisson), pgeom (geometric), phyper (hyper-geometric), pt (T distribution), pf (F distribution)

  • 10 random values from the Normal distribution with mean 10 and standard deviation 5:

rnorm(10, mean=10, sd=5)
 [1]  5.693620  4.429255  0.442925  9.223783 14.217208 10.117813 10.052955  9.347432
 [9]  7.879201  3.260119
  • The probability of drawing exactly 10 from this distribution:
dnorm(10, mean=10, sd=5)
[1] 0.07978846
dnorm(100, mean=10, sd=5)
[1] 3.517499e-72
  • The probability of drawing a value smaller than 10:
pnorm(10, mean=10, sd=5)
[1] 0.5
  • The inverse of pnorm():
qnorm(0.5, mean=10, sd=5)
[1] 10
  • How many standard deviations for statistical significance?
qnorm(0.95, mean=0, sd=1)
[1] 1.644854

Example

Recall our histogram of Wind Speed from yesterday:

  • The data look to be roughly normally-distributed
  • An assumption we rely on for various statistical tests
weather <- read.csv("ozone.csv")
hist(weather$Wind, col="steelblue", xlab="Wind Speed",
     main="Distribution of Wind Speed",
     breaks = 20, freq=FALSE)

Create a normal distribution curve

  • If our data are normally-distributed, we can calculate the probability of drawing particular values.
    • e.g. a Wind Speed of 10
windMean <- mean(weather$Wind)
windSD <- sd(weather$Wind)
dnorm(10, mean=windMean, sd=windSD)
[1] 0.1132311
  • We can overlay this on the histogram using points as we just saw:
hist(weather$Wind, col="steelblue", xlab="Wind Speed",
     main="Distribution of Wind Speed",
     breaks = 20, freq=FALSE)
points(10, dnorm(10, mean=windMean, sd=windSD),
       col="red", pch=16)

  • We can repeat the calculation for a vector of values
    • remember that functions in R are often vectorized
    • use lines in this case rather than points
xs <- c(0,5,10,15,20)
ys <- dnorm(xs, mean=windMean, sd=windSD)
hist(weather$Wind, col="steelblue", xlab="Wind Speed",
     main="Distribution of Wind Speed",
     breaks = 20, freq=FALSE)
lines(xs, ys, col="red")

  • For a smoother curve, use a longer vector:
    • we can generate x values using the seq() function
  • Inspecting the data in this manner is usually acceptable to assess normality
    • the fit doesn’t have to be exact
    • the shapiro test is also available ?shapiro.test (but not really recommended by statisticians)
hist(weather$Wind,col="steelblue",xlab="Wind Speed",
     main="Distribution of Wind Speed",breaks = 20,freq=FALSE)
xs <- seq(0,20, length.out = 10000)
ys <- dnorm(xs, mean=windMean,sd=windSD)
lines(xs,ys,col="red")

Simple testing

  • If we want to compute the probability of observing a particular Wind Speed, from the same distribution, we can use the standard formula to calculate a t statistic:

t=ˉx−μ0s/√(n)

  • Say a Wind Speed of 2; which from the histogram seems to be unlikely
t <- (windMean - 2) / (windSD/sqrt(length(weather$Wind)))
t
[1] 27.93897
  • …or use the t.test() function to compute the statistic and corresponding p-value
t.test(weather$Wind, mu=2)

    One Sample t-test

data:  weather$Wind
t = 27.939, df = 152, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 2
95 percent confidence interval:
  9.394804 10.520229
sample estimates:
mean of x 
 9.957516
  • A variety of tests are supported the R authors have tried to make them as consistent as possible
?var.test
?t.test
?wilcox.test
?prop.test
?cor.test
?chisq.test
?fisher.test

Example analysis

  • We have already seen that men in our patients dataset tend to be heavier than women
  • We can test this formally in R
    • N.B. the weight of people in a population tends to be normally distributed, so we can probably be safe to use a parametric test

We need to run this if we don’t have the patients data in our R environment

patients <- read.delim("patient-info.txt")

We can test if the variance of the two groups is the same

  • this will influence what flavour of t-test we use
var.test(patients$Weight~patients$Sex)

    F test to compare two variances

data:  patients$Weight by patients$Sex
F = 0.14216, num df = 49, denom df = 44, p-value = 3.59e-10
alternative hypothesis: true ratio of variances is not equal to 1
95 percent confidence interval:
 0.07900337 0.25344664
sample estimates:
ratio of variances 
         0.1421572 
t.test(patients$Weight~patients$Sex, var.equal=FALSE)

    Welch Two Sample t-test

data:  patients$Weight by patients$Sex
t = -11.204, df = 55.168, p-value = 7.786e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -15.62079 -10.88094
sample estimates:
mean in group Female   mean in group Male 
            68.95980             82.21067

If were unwilling to make a assumption of normality, a non-parametric test could be used.

wilcox.test(patients$Weight~patients$Sex)

    Wilcoxon rank sum test with continuity correction

data:  patients$Weight by patients$Sex
W = 59, p-value = 1.993e-15
alternative hypothesis: true location shift is not equal to 0

Linear Regression

  • Linear modeling is supported by the function lm():
    • example(lm)
  • The output assumes you know a fair bit about the subject
  • lm is really useful for plotting lines of best fit to XY data, in order to determine intercept, gradient and Pearson’s correlation coefficient
    • This is very easy in R

  • Three steps to plotting with a best fit line:

    • Plot XY scatter-plot data
    • Fit a linear model
    • Add bestfit line data to plot with abline() function

  • Let’s see a toy example:-

x <- c(1, 2.3, 3.1, 4.8, 5.6, 6.3)
y <- c(2.6, 2.8, 3.1, 4.7, 5.1, 5.3)
plot(x,y, xlim=c(0,10), ylim=c(0,10))

  • The ~ is used to define a formula; i.e. “y is given by x”
    • Take care about the order of x and y in the plot and lm expressions
plot(x,y, xlim=c(0,10), ylim=c(0,10))
myModel <- lm(y~x)
abline(myModel, col="red")

The generic summary function give an overview of the model

  • particular components are accessible if we know their names
summary(myModel)

Call:
lm(formula = y ~ x)

Residuals:
       1        2        3        4        5        6 
 0.33159 -0.22785 -0.39520  0.21169  0.14434 -0.06458 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)   
(Intercept)  1.68422    0.29056   5.796   0.0044 **
x            0.58418    0.06786   8.608   0.0010 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.3114 on 4 degrees of freedom
Multiple R-squared:  0.9488,    Adjusted R-squared:  0.936 
F-statistic:  74.1 on 1 and 4 DF,  p-value: 0.001001
names(myModel)  # Names of the objects within myModel
 [1] "coefficients"  "residuals"     "effects"       "rank"          "fitted.values"
 [6] "assign"        "qr"            "df.residual"   "xlevels"       "call"         
[11] "terms"         "model"
  • alternatively, various helper functions are provided.
coef(myModel)   # Coefficients
(Intercept)           x 
  1.6842239   0.5841843 
resid(myModel)  # Residuals
          1           2           3           4           5           6 
 0.33159186 -0.22784770 -0.39519512  0.21169160  0.14434418 -0.06458482 
fitted(myModel) # Fitted values
       1        2        3        4        5        6 
2.268408 3.027848 3.495195 4.488308 4.955656 5.364585 
residuals(myModel) + fitted(myModel) # what values does this give?
  1   2   3   4   5   6 
2.6 2.8 3.1 4.7 5.1 5.3

You can also get some diagnostic information on the model.

  • Some explanation can be found here and here
par(mfrow=c(2,2)) 
plot(myModel)

  • R has a very powerful formula syntax for describing statistical models
  • Suppose we had two explanatory variables x and z, and one response variable y
  • We can describe a relationship between, say, y and x using a tilde ~, placing the response variable on the left of the tilde and the explanatory variables on the right:
    • y~x
  • It is very easy to extend this syntax to do multiple regressions, ANOVAs, to include interactions, and to do many other common modelling tasks. For example
y~x       #If x is continuous, this is linear regression
y ~ x
y~x       #If x is categorical, ANOVA
y ~ x
y~x+z     #If x and z are continuous, multiple regression
y ~ x + z
y~x+z     #If x and z are categorical, two-way ANOVA
y ~ x + z
y~x+z+x:z # : is the symbol for the interaction term
y ~ x + z + x:z
y~x*z     # * is a shorthand for x+z+x:z
y ~ x * z

Exercise: Exercise 6

  • There are suggestions that Ozone level could be influenced by Temperature:
plot(weather$Temp, weather$Ozone,xlab="Temperature",ylab="Ozone level",pch=16)

  • Perform a linear regression analysis to assess this:
    • Fit the linear model and print a summary of the output
    • Plot the two variables and overlay a best-fit line
    • What is the equation of the best-fit line in the form y=ax+b
  • Calculate the correlation between the two variables using the function cor (?cor)
    • can you add text to the plot to show the correlation coefficient?
    • HINT: The paste can be used to join strings of text together, or variables
paste("Hello","World")
[1] "Hello World"
age <- 35
paste("My age is", age)
[1] "My age is 35"
## Your Answer Here ##

More words of warning

Correlation != Causation

http://tylervigen.com/spurious-correlations

So if I want to win a nobel prize, I should eat even more chocolate?!?!?

But no-one would ever take such trends seriously….would they?

Cutting-down on Ice Cream was recommended as a safeguard against polio!

Cutting-down on Ice Cream was recommended as a safeguard against polio!

LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIFNvbHZpbmcgQmlvbG9naWNhbCBQcm9ibGVtcyBVc2luZyBSIC0gRGF5IDIiCmF1dGhvcjogTWFyayBEdW5uaW5nLCBTdXJhaiBNZW5vbiBhbmQgQWlvcmEgWmFiYWxhLiBPcmlnaW5hbCBtYXRlcmlhbCBieSBSb2JlcnQgU3Rvam5pxIcsCiAgTGF1cmVudCBHYXR0bywgUm9iIEZveSwgSm9obiBEYXZleSwgRMOhdmlkIE1vbG7DoXIgYW5kIElhbiBSb2JlcnRzCmRhdGU6ICdgciBmb3JtYXQoU3lzLnRpbWUoKSwgIkxhc3QgbW9kaWZpZWQ6ICVkICViICVZIilgJwpvdXRwdXQ6CiAgaHRtbF9ub3RlYm9vazoKICAgIHRvYzogeWVzCiAgICB0b2NfZmxvYXQ6IHllcwotLS0KCgojIDIuIFN0YXRpc3RpY3MKIyNCdWlsdC1pbiBzdXBwb3J0IGZvciBzdGF0aXN0aWNzCi0gUiBpcyBhIHN0YXRpc3RpY2FsIHByb2dyYW1taW5nIGxhbmd1YWdlOgogICAgKyBDbGFzc2ljYWwgc3RhdGlzdGljYWwgdGVzdHMgYXJlIGJ1aWx0LWluCiAgICArIFN0YXRpc3RpY2FsIG1vZGVsaW5nIGZ1bmN0aW9ucyBhcmUgYnVpbHQtaW4KICAgICsgUmVncmVzc2lvbiBhbmFseXNpcyBpcyBmdWxseSBzdXBwb3J0ZWQKICAgICsgQWRkaXRpb25hbCBtYXRoZW1hdGljYWwgcGFja2FnZXMgYXJlIGF2YWlsYWJsZSAoYE1BU1NgLCBXYXZlcywgc3BhcnNlIG1hdHJpY2VzLCBldGMpCiAgCiMjRGlzdHJpYnV0aW9uIGZ1bmN0aW9ucyAgCi0gTW9zdCBjb21tb25seSB1c2VkIGRpc3RyaWJ1dGlvbnMgYXJlIGJ1aWx0LWluLCBmdW5jdGlvbnMgaGF2ZSBzdGVyZW90eXBpY2FsIG5hbWVzLCBlLmcuIGZvciBub3JtYWwgZGlzdHJpYnV0aW9uOgogICAgKyAqKmBwbm9ybWAqKiAtIGN1bXVsYXRpdmUgZGlzdHJpYnV0aW9uIGZvciB4CiAgICArICoqYHFub3JtYCoqIC0gaW52ZXJzZSBvZiBwbm9ybSAoZnJvbSBwcm9iYWJpbGl0eSBnaXZlcyB4KQogICAgKyAqKmBkbm9ybWAqKiAtIGRpc3RyaWJ1dGlvbiBkZW5zaXR5CiAgICArICoqYHJub3JtYCoqIC0gcmFuZG9tIG51bWJlciBmcm9tIG5vcm1hbCBkaXN0cmlidXRpb24KICAKIVtkaXN0cmlidXRpb25zXShpbWFnZXMvZGlzdHJpYnV0aW9ucy5wbmcpICAKICAKLSBBdmFpbGFibGUgZm9yIHZhcmlldHkgb2YgZGlzdHJpYnV0aW9uczogYHB1bmlmYCAodW5pZm9ybSksIGBwYmlub21gIChiaW5vbWlhbCksIGBwbmJpbm9tYCAobmVnYXRpdmUgYmlub21pYWwpLCBgcHBvaXNgIChwb2lzc29uKSwgYHBnZW9tYCAoZ2VvbWV0cmljKSwgYHBoeXBlcmAgKGh5cGVyLWdlb21ldHJpYyksIGBwdGAgKFQgZGlzdHJpYnV0aW9uKSwgcGYgKEYgZGlzdHJpYnV0aW9uKSAKCgotIDEwIHJhbmRvbSB2YWx1ZXMgZnJvbSB0aGUgTm9ybWFsIGRpc3RyaWJ1dGlvbiB3aXRoIG1lYW4gMTAgYW5kIHN0YW5kYXJkIGRldmlhdGlvbiA1OgpgYGB7cn0Kcm5vcm0oMTAsIG1lYW49MTAsIHNkPTUpCmBgYAotIFRoZSBwcm9iYWJpbGl0eSBvZiBkcmF3aW5nIGV4YWN0bHkgMTAgZnJvbSB0aGlzIGRpc3RyaWJ1dGlvbjoKYGBge3J9CmRub3JtKDEwLCBtZWFuPTEwLCBzZD01KQpgYGAKCmBgYHtyfQpkbm9ybSgxMDAsIG1lYW49MTAsIHNkPTUpCmBgYAoKCi0gVGhlIHByb2JhYmlsaXR5IG9mIGRyYXdpbmcgYSB2YWx1ZSBzbWFsbGVyIHRoYW4gMTA6CmBgYHtyfQpwbm9ybSgxMCwgbWVhbj0xMCwgc2Q9NSkKYGBgCi0gVGhlIGludmVyc2Ugb2YgYHBub3JtKClgOgpgYGB7cn0KcW5vcm0oMC41LCBtZWFuPTEwLCBzZD01KQpgYGAKLSBIb3cgbWFueSBzdGFuZGFyZCBkZXZpYXRpb25zIGZvciBzdGF0aXN0aWNhbCBzaWduaWZpY2FuY2U/CmBgYHtyfQpxbm9ybSgwLjk1LCBtZWFuPTAsIHNkPTEpCmBgYAoKIyMgRXhhbXBsZQoKUmVjYWxsIG91ciBoaXN0b2dyYW0gb2YgV2luZCBTcGVlZCBmcm9tIHllc3RlcmRheToKCi0gVGhlIGRhdGEgbG9vayB0byBiZSByb3VnaGx5IG5vcm1hbGx5LWRpc3RyaWJ1dGVkCi0gQW4gYXNzdW1wdGlvbiB3ZSByZWx5IG9uIGZvciB2YXJpb3VzIHN0YXRpc3RpY2FsIHRlc3RzCgpgYGB7cn0Kd2VhdGhlciA8LSByZWFkLmNzdigib3pvbmUuY3N2IikKaGlzdCh3ZWF0aGVyJFdpbmQsIGNvbD0ic3RlZWxibHVlIiwgeGxhYj0iV2luZCBTcGVlZCIsCiAgICAgbWFpbj0iRGlzdHJpYnV0aW9uIG9mIFdpbmQgU3BlZWQiLAogICAgIGJyZWFrcyA9IDIwLCBmcmVxPUZBTFNFKQpgYGAKCiMjIENyZWF0ZSBhIG5vcm1hbCBkaXN0cmlidXRpb24gY3VydmUKCi0gSWYgb3VyIGRhdGEgYXJlIG5vcm1hbGx5LWRpc3RyaWJ1dGVkLCB3ZSBjYW4gY2FsY3VsYXRlIHRoZSBwcm9iYWJpbGl0eSBvZiBkcmF3aW5nIHBhcnRpY3VsYXIgdmFsdWVzLgogICAgICArIGUuZy4gYSBXaW5kIFNwZWVkIG9mIDEwCgpgYGB7cn0Kd2luZE1lYW4gPC0gbWVhbih3ZWF0aGVyJFdpbmQpCndpbmRTRCA8LSBzZCh3ZWF0aGVyJFdpbmQpCmRub3JtKDEwLCBtZWFuPXdpbmRNZWFuLCBzZD13aW5kU0QpCmBgYAoKLSBXZSBjYW4gb3ZlcmxheSB0aGlzIG9uIHRoZSBoaXN0b2dyYW0gdXNpbmcgYHBvaW50c2AgYXMgd2UganVzdCBzYXc6CmBgYHtyfQpoaXN0KHdlYXRoZXIkV2luZCwgY29sPSJzdGVlbGJsdWUiLCB4bGFiPSJXaW5kIFNwZWVkIiwKICAgICBtYWluPSJEaXN0cmlidXRpb24gb2YgV2luZCBTcGVlZCIsCiAgICAgYnJlYWtzID0gMjAsIGZyZXE9RkFMU0UpCnBvaW50cygxMCwgZG5vcm0oMTAsIG1lYW49d2luZE1lYW4sIHNkPXdpbmRTRCksCiAgICAgICBjb2w9InJlZCIsIHBjaD0xNikKYGBgCgotIFdlIGNhbiByZXBlYXQgdGhlIGNhbGN1bGF0aW9uIGZvciBhIHZlY3RvciBvZiB2YWx1ZXMKICAgICsgcmVtZW1iZXIgdGhhdCBmdW5jdGlvbnMgaW4gUiBhcmUgb2Z0ZW4gKioqdmVjdG9yaXplZCoqKgogICAgKyB1c2UgYGxpbmVzYCBpbiB0aGlzIGNhc2UgcmF0aGVyIHRoYW4gYHBvaW50c2AKICAgIApgYGB7cn0KeHMgPC0gYygwLDUsMTAsMTUsMjApCnlzIDwtIGRub3JtKHhzLCBtZWFuPXdpbmRNZWFuLCBzZD13aW5kU0QpCmhpc3Qod2VhdGhlciRXaW5kLCBjb2w9InN0ZWVsYmx1ZSIsIHhsYWI9IldpbmQgU3BlZWQiLAogICAgIG1haW49IkRpc3RyaWJ1dGlvbiBvZiBXaW5kIFNwZWVkIiwKICAgICBicmVha3MgPSAyMCwgZnJlcT1GQUxTRSkKbGluZXMoeHMsIHlzLCBjb2w9InJlZCIpCmBgYAoKCgoKLSBGb3IgYSBzbW9vdGhlciBjdXJ2ZSwgdXNlIGEgbG9uZ2VyIHZlY3RvcjoKICAgICsgd2UgY2FuIGdlbmVyYXRlIHggdmFsdWVzIHVzaW5nIHRoZSBgc2VxKClgIGZ1bmN0aW9uCi0gSW5zcGVjdGluZyB0aGUgZGF0YSBpbiB0aGlzIG1hbm5lciBpcyB1c3VhbGx5IGFjY2VwdGFibGUgdG8gYXNzZXNzIG5vcm1hbGl0eQogICAgKyB0aGUgZml0IGRvZXNuJ3QgaGF2ZSB0byBiZSBleGFjdAogICAgKyB0aGUgc2hhcGlybyB0ZXN0IGlzIGFsc28gYXZhaWxhYmxlIGA/c2hhcGlyby50ZXN0YCAoYnV0IG5vdCByZWFsbHkgcmVjb21tZW5kZWQgYnkgc3RhdGlzdGljaWFucykKCgpgYGB7cn0KaGlzdCh3ZWF0aGVyJFdpbmQsY29sPSJzdGVlbGJsdWUiLHhsYWI9IldpbmQgU3BlZWQiLAogICAgIG1haW49IkRpc3RyaWJ1dGlvbiBvZiBXaW5kIFNwZWVkIixicmVha3MgPSAyMCxmcmVxPUZBTFNFKQp4cyA8LSBzZXEoMCwyMCwgbGVuZ3RoLm91dCA9IDEwMDAwKQp5cyA8LSBkbm9ybSh4cywgbWVhbj13aW5kTWVhbixzZD13aW5kU0QpCmxpbmVzKHhzLHlzLGNvbD0icmVkIikKYGBgCgojIyBTaW1wbGUgdGVzdGluZwoKLSBJZiB3ZSB3YW50IHRvIGNvbXB1dGUgdGhlIHByb2JhYmlsaXR5IG9mIG9ic2VydmluZyBhIHBhcnRpY3VsYXIgV2luZCBTcGVlZCwgZnJvbSB0aGUgc2FtZSBkaXN0cmlidXRpb24sIHdlIGNhbiB1c2UgdGhlIHN0YW5kYXJkIGZvcm11bGEgdG8gY2FsY3VsYXRlIGEgdCBzdGF0aXN0aWM6CgokJHQgPSBcZnJhY3tcYmFye3h9IC1cbXVfMH17cyAvIFxzcXJ0KG4pfSQkCgotIFNheSBhIFdpbmQgU3BlZWQgb2YgMjsgd2hpY2ggZnJvbSB0aGUgaGlzdG9ncmFtIHNlZW1zIHRvIGJlIHVubGlrZWx5CgpgYGB7cn0KdCA8LSAod2luZE1lYW4gLSAyKSAvICh3aW5kU0Qvc3FydChsZW5ndGgod2VhdGhlciRXaW5kKSkpCnQKYGBgCgotIC4uLm9yIHVzZSB0aGUgKipgdC50ZXN0KClgKiogZnVuY3Rpb24gdG8gY29tcHV0ZSB0aGUgc3RhdGlzdGljIGFuZCBjb3JyZXNwb25kaW5nIHAtdmFsdWUKCmBgYHtyfQp0LnRlc3Qod2VhdGhlciRXaW5kLCBtdT0yKQpgYGAKCi0gQSB2YXJpZXR5IG9mIHRlc3RzIGFyZSBzdXBwb3J0ZWQgdGhlIFIgYXV0aG9ycyBoYXZlIHRyaWVkIHRvIG1ha2UgdGhlbSBhcyBjb25zaXN0ZW50IGFzIHBvc3NpYmxlCgpgYGB7cn0KP3Zhci50ZXN0Cj90LnRlc3QKP3dpbGNveC50ZXN0Cj9wcm9wLnRlc3QKP2Nvci50ZXN0Cj9jaGlzcS50ZXN0Cj9maXNoZXIudGVzdApgYGAKCiAgICAKLSBCb3R0b20tbGluZTogUHJldHR5IG11Y2ggYW55IHN0YXRpc3RpY2FsIHRlc3QgeW91IGNhcmUgdG8gbmFtZSB3aWxsIHByb2JhYmx5IGJlIGluIFIKICAgICsgVGhpcyBpcyBub3Qgc3VwcG9zZWQgdG8gYmUgYSBzdGF0aXN0aWNzIGNvdXJzZSAoc29ycnkhKQogICAgKyBOb25lIG9mIHRoZW0gYXJlIHBhcnRpY3VsYXIgaGFyZGVyIHRoYW4gb3RoZXJzIHRvIHVzZQogICAgKyBUaGUgZGlmZmljdWx0eSBpcyBkZWNpZGluZyB3aGljaCB0ZXN0IHRvIHVzZToKICAgICAgICArIHdoZXRoZXIgdGhlIGFzc3VtcHRpb25zIG9mIHRoZSB0ZXN0IGFyZSBtZXQsIGV0Yy4KICAgICsgQ29uc3VsdCB5b3VyIGxvY2FsIHN0YXRpc3RpY2lhbiBpZiBub3Qgc3VyZQogICAgKyBBbiB1cGNvbWluZyBjb3Vyc2UgdGhhdCB3aWxsIGhlbHAKICAgICAgICArIFtJbnRyb2R1Y3Rpb24gdG8gU3RhdGlzdGljYWwgQW5hbHlzaXNdKGh0dHA6Ly9iaW9pbmZvcm1hdGljcy1jb3JlLXNoYXJlZC10cmFpbmluZy5naXRodWIuaW8vSW50cm9kdWN0aW9uVG9TdGF0cy8pCiAgICArIFNvbWUgZ29vZCByZWZlcmVuY2VzOgogICAgICAgICsgW1N0YXRpc3RpY2FsIEFuYWx5c2lzIFVzaW5nIFIgKENvdXJzZSBmcm9tIHRoZSBCYWJhcmFoYW0gQmlvaW5mb3JtYXRpY3MgQ29yZSldKGh0dHA6Ly90cmFpbmluZy5jc3guY2FtLmFjLnVrL2Jpb2luZm9ybWF0aWNzL2V2ZW50LzE4Mjc3NzEpCiAgICAgICAgKyBbUXVpY2stUiBndWlkZSB0byBzdGF0c10oaHR0cDovL3d3dy5zdGF0bWV0aG9kcy5uZXQvc3RhdHMvaW5kZXguaHRtbCkKICAgICAgICArIFtTaW1wbGUgUiBlQm9va10oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvZG9jL2NvbnRyaWIvVmVyemFuaS1TaW1wbGVSLnBkZikKICAgICAgICArIFtSIHdpa2ldKGh0dHBzOi8vZW4ud2lraWJvb2tzLm9yZy93aWtpL1JfUHJvZ3JhbW1pbmcvRGVzY3JpcHRpdmVfU3RhdGlzdGljcykKICAgICAgICArIFtSIHR1dG9yXShodHRwOi8vd3d3LnItdHV0b3IuY29tL2VsZW1lbnRhcnktc3RhdGlzdGljcykKICAgICAgICArIFtTdGF0aXN0aWNhbCBDaGVhdHNoZWV0XShodHRwczovL3Jhd2dpdC5jb20vYmlvaW5mb3JtYXRpY3MtY29yZS1zaGFyZWQtdHJhaW5pbmcvaW50ZXJtZWRpYXRlLXN0YXRzL21hc3Rlci9jaGVhdHNoZWV0LnBkZikKICAgLSBSIHdpbGwgZG8gd2hhdGV2ZXIgeW91IGFzayBpdCwgaXQgd2lsbCBuZXZlciBjaGVjayB0aGUgYXNzdW1wdGlvbnMgb3IgaGVscCB5b3UgdG8gaW50ZXJwcmV0IHRoZSByZXN1bHQKCiFbXShpbWFnZXMvY2xpcHB5LmpwZykKCiMjIEV4YW1wbGUgYW5hbHlzaXMKCi0gV2UgaGF2ZSBhbHJlYWR5IHNlZW4gdGhhdCBtZW4gaW4gb3VyIGBwYXRpZW50c2AgZGF0YXNldCB0ZW5kIHRvIGJlIGhlYXZpZXIgdGhhbiB3b21lbgotIFdlIGNhbiAqKnRlc3QgdGhpcyBmb3JtYWxseSoqIGluIFIKICAgICsgTi5CLiB0aGUgd2VpZ2h0IG9mIHBlb3BsZSBpbiBhIHBvcHVsYXRpb24gdGVuZHMgdG8gYmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQsIHNvIHdlIGNhbiBwcm9iYWJseSBiZSBzYWZlIHRvIHVzZSBhIHBhcmFtZXRyaWMgdGVzdAogICAgCiFbXShpbWFnZXMvbWFsZXMtdmVyc3VzLWZlbWFsZXMucG5nKQoKCgoKV2UgbmVlZCB0byBydW4gdGhpcyBpZiB3ZSBkb24ndCBoYXZlIHRoZSBwYXRpZW50cyBkYXRhIGluIG91ciBSIGVudmlyb25tZW50CmBgYHtyfQpwYXRpZW50cyA8LSByZWFkLmRlbGltKCJwYXRpZW50LWluZm8udHh0IikKYGBgCgpXZSBjYW4gdGVzdCBpZiB0aGUgdmFyaWFuY2Ugb2YgdGhlIHR3byBncm91cHMgaXMgdGhlIHNhbWUKCisgdGhpcyB3aWxsIGluZmx1ZW5jZSB3aGF0IGZsYXZvdXIgb2YgdC10ZXN0IHdlIHVzZQoKYGBge3J9CnZhci50ZXN0KHBhdGllbnRzJFdlaWdodH5wYXRpZW50cyRTZXgpCmBgYAoKCmBgYHtyfQp0LnRlc3QocGF0aWVudHMkV2VpZ2h0fnBhdGllbnRzJFNleCwgdmFyLmVxdWFsPUZBTFNFKQpgYGAKCklmIHdlcmUgdW53aWxsaW5nIHRvIG1ha2UgYSBhc3N1bXB0aW9uIG9mIG5vcm1hbGl0eSwgYSBub24tcGFyYW1ldHJpYyB0ZXN0IGNvdWxkIGJlIHVzZWQuCgpgYGB7cn0Kd2lsY294LnRlc3QocGF0aWVudHMkV2VpZ2h0fnBhdGllbnRzJFNleCkKYGBgCgojIyBMaW5lYXIgUmVncmVzc2lvbgoKLSBMaW5lYXIgbW9kZWxpbmcgaXMgc3VwcG9ydGVkIGJ5IHRoZSBmdW5jdGlvbiBgbG0oKWA6CiAgICArIGBleGFtcGxlKGxtKWAKLSBUaGUgb3V0cHV0IGFzc3VtZXMgeW91IGtub3cgYSBmYWlyIGJpdCBhYm91dCB0aGUgc3ViamVjdAotIGBsbWAgaXMgcmVhbGx5IHVzZWZ1bCBmb3IgcGxvdHRpbmcgbGluZXMgb2YgYmVzdCBmaXQgdG8gWFkgZGF0YSwgaW4gb3JkZXIgdG8gZGV0ZXJtaW5lIGludGVyY2VwdCwgZ3JhZGllbnQgYW5kIFBlYXJzb24ncyBjb3JyZWxhdGlvbiBjb2VmZmljaWVudAogICAgKyBUaGlzIGlzIHZlcnkgZWFzeSBpbiBSCi0gVGhyZWUgc3RlcHMgdG8gcGxvdHRpbmcgd2l0aCBhIGJlc3QgZml0IGxpbmU6CgogICAgKyBQbG90IFhZIHNjYXR0ZXItcGxvdCBkYXRhCiAgICArIEZpdCBhIGxpbmVhciBtb2RlbAogICAgKyBBZGQgYmVzdGZpdCBsaW5lIGRhdGEgdG8gcGxvdCB3aXRoIGBhYmxpbmUoKWAgZnVuY3Rpb24KLSBMZXQncyBzZWUgYSB0b3kgZXhhbXBsZTotCgpgYGB7cn0KeCA8LSBjKDEsIDIuMywgMy4xLCA0LjgsIDUuNiwgNi4zKQp5IDwtIGMoMi42LCAyLjgsIDMuMSwgNC43LCA1LjEsIDUuMykKcGxvdCh4LHksIHhsaW09YygwLDEwKSwgeWxpbT1jKDAsMTApKQpgYGAKCi0gVGhlIGB+YCBpcyB1c2VkIHRvIGRlZmluZSBhIGZvcm11bGE7IGkuZS4gInkgaXMgZ2l2ZW4gYnkgeCIKICAgICsgVGFrZSBjYXJlIGFib3V0IHRoZSBvcmRlciBvZiBgeGAgYW5kIGB5YCBpbiB0aGUgYHBsb3RgIGFuZCBgbG1gIGV4cHJlc3Npb25zCgpgYGB7cn0KcGxvdCh4LHksIHhsaW09YygwLDEwKSwgeWxpbT1jKDAsMTApKQpteU1vZGVsIDwtIGxtKHl+eCkKYWJsaW5lKG15TW9kZWwsIGNvbD0icmVkIikKYGBgCgpUaGUgZ2VuZXJpYyBgc3VtbWFyeWAgZnVuY3Rpb24gZ2l2ZSBhbiBvdmVydmlldyBvZiB0aGUgbW9kZWwKCi0gcGFydGljdWxhciBjb21wb25lbnRzIGFyZSBhY2Nlc3NpYmxlIGlmIHdlIGtub3cgdGhlaXIgbmFtZXMKCmBgYHtyfQpzdW1tYXJ5KG15TW9kZWwpCm5hbWVzKG15TW9kZWwpICAjIE5hbWVzIG9mIHRoZSBvYmplY3RzIHdpdGhpbiBteU1vZGVsCmBgYAoKLSBhbHRlcm5hdGl2ZWx5LCB2YXJpb3VzIGhlbHBlciBmdW5jdGlvbnMgYXJlIHByb3ZpZGVkLgoKYGBge3J9CmNvZWYobXlNb2RlbCkgICAjIENvZWZmaWNpZW50cwpyZXNpZChteU1vZGVsKSAgIyBSZXNpZHVhbHMKZml0dGVkKG15TW9kZWwpICMgRml0dGVkIHZhbHVlcwpyZXNpZHVhbHMobXlNb2RlbCkgKyBmaXR0ZWQobXlNb2RlbCkgIyB3aGF0IHZhbHVlcyBkb2VzIHRoaXMgZ2l2ZT8KYGBgCgpZb3UgY2FuIGFsc28gZ2V0IHNvbWUgZGlhZ25vc3RpYyBpbmZvcm1hdGlvbiBvbiB0aGUgbW9kZWwuCgotIFNvbWUgZXhwbGFuYXRpb24gY2FuIGJlIGZvdW5kIFtoZXJlXShodHRwOi8vZGF0YS5saWJyYXJ5LnZpcmdpbmlhLmVkdS9kaWFnbm9zdGljLXBsb3RzLykgYW5kIFtoZXJlXShodHRwOi8vc3RyYXRhLnVnYS5lZHUvNjM3MC9ydGlwcy9yZWdyZXNzaW9uUGxvdHMuaHRtbCkKCmBgYHtyfQpwYXIobWZyb3c9YygyLDIpKSAKcGxvdChteU1vZGVsKQpgYGAKCi0gUiBoYXMgYSB2ZXJ5IHBvd2VyZnVsIGZvcm11bGEgc3ludGF4IGZvciBkZXNjcmliaW5nIHN0YXRpc3RpY2FsIG1vZGVscwotIFN1cHBvc2Ugd2UgaGFkIHR3byBleHBsYW5hdG9yeSB2YXJpYWJsZXMgYHhgIGFuZCBgemAsIGFuZCBvbmUgcmVzcG9uc2UgdmFyaWFibGUgYHlgCi0gV2UgY2FuIGRlc2NyaWJlIGEgcmVsYXRpb25zaGlwIGJldHdlZW4sIHNheSwgYHlgIGFuZCBgeGAgdXNpbmcgYSB0aWxkZSBgfmAsIHBsYWNpbmcgdGhlIHJlc3BvbnNlIHZhcmlhYmxlIG9uIHRoZSBsZWZ0IG9mIHRoZSB0aWxkZSBhbmQgdGhlIGV4cGxhbmF0b3J5IHZhcmlhYmxlcyBvbiB0aGUgcmlnaHQ6CiAgICArIHl+eAotIEl0IGlzIHZlcnkgZWFzeSB0byBleHRlbmQgdGhpcyBzeW50YXggdG8gZG8gbXVsdGlwbGUgcmVncmVzc2lvbnMsIEFOT1ZBcywgdG8gaW5jbHVkZSBpbnRlcmFjdGlvbnMsIGFuZCB0byBkbyBtYW55IG90aGVyIGNvbW1vbiBtb2RlbGxpbmcgdGFza3MuIEZvciBleGFtcGxlCgpgYGB7cn0KeX54ICAgICAgICNJZiB4IGlzIGNvbnRpbnVvdXMsIHRoaXMgaXMgbGluZWFyIHJlZ3Jlc3Npb24KeX54ICAgICAgICNJZiB4IGlzIGNhdGVnb3JpY2FsLCBBTk9WQQp5fngreiAgICAgI0lmIHggYW5kIHogYXJlIGNvbnRpbnVvdXMsIG11bHRpcGxlIHJlZ3Jlc3Npb24KeX54K3ogICAgICNJZiB4IGFuZCB6IGFyZSBjYXRlZ29yaWNhbCwgdHdvLXdheSBBTk9WQQp5fngreit4OnogIyA6IGlzIHRoZSBzeW1ib2wgZm9yIHRoZSBpbnRlcmFjdGlvbiB0ZXJtCnl+eCp6ICAgICAjICogaXMgYSBzaG9ydGhhbmQgZm9yIHgreit4OnoKYGBgCgojIyBFeGVyY2lzZTogRXhlcmNpc2UgNgoKLSBUaGVyZSBhcmUgc3VnZ2VzdGlvbnMgdGhhdCBPem9uZSBsZXZlbCBjb3VsZCBiZSBpbmZsdWVuY2VkIGJ5IFRlbXBlcmF0dXJlOgpgYGB7cn0KcGxvdCh3ZWF0aGVyJFRlbXAsIHdlYXRoZXIkT3pvbmUseGxhYj0iVGVtcGVyYXR1cmUiLHlsYWI9Ik96b25lIGxldmVsIixwY2g9MTYpCmBgYAoKLSBQZXJmb3JtIGEgbGluZWFyIHJlZ3Jlc3Npb24gYW5hbHlzaXMgdG8gYXNzZXNzIHRoaXM6CiAgICArIEZpdCB0aGUgbGluZWFyIG1vZGVsIGFuZCBwcmludCBhIHN1bW1hcnkgb2YgdGhlIG91dHB1dAogICAgKyBQbG90IHRoZSB0d28gdmFyaWFibGVzIGFuZCBvdmVybGF5IGEgYmVzdC1maXQgbGluZQogICAgKyBXaGF0IGlzIHRoZSBlcXVhdGlvbiBvZiB0aGUgYmVzdC1maXQgbGluZSBpbiB0aGUgZm9ybQogICAgICAkJCB5ID0gYXggKyBiJCQKLSBDYWxjdWxhdGUgdGhlIGNvcnJlbGF0aW9uIGJldHdlZW4gdGhlIHR3byB2YXJpYWJsZXMgdXNpbmcgdGhlIGZ1bmN0aW9uIGNvciAoP2NvcikKICAgICsgY2FuIHlvdSBhZGQgdGV4dCB0byB0aGUgcGxvdCB0byBzaG93IHRoZSBjb3JyZWxhdGlvbiBjb2VmZmljaWVudD8KICAgICsgSElOVDogVGhlIGBwYXN0ZWAgY2FuIGJlIHVzZWQgdG8gam9pbiBzdHJpbmdzIG9mIHRleHQgdG9nZXRoZXIsIG9yIHZhcmlhYmxlcwoKYGBge3J9CnBhc3RlKCJIZWxsbyIsIldvcmxkIikKYWdlIDwtIDM1CnBhc3RlKCJNeSBhZ2UgaXMiLCBhZ2UpCmBgYAoKCmBgYHtyfQoKIyMgWW91ciBBbnN3ZXIgSGVyZSAjIwoKYGBgCgohW10oaW1hZ2VzL2V4ZXJjaXNlNi5wbmcpCiFbXShpbWFnZXMvZXhlcmNpc2U2Yi5wbmcpCgoKIyMgTW9yZSB3b3JkcyBvZiB3YXJuaW5nCgoqKipDb3JyZWxhdGlvbiAhPSBDYXVzYXRpb24qKioKCiFbXShpbWFnZXMvc3B1cmlvdXMucG5nKQoKaHR0cDovL3R5bGVydmlnZW4uY29tL3NwdXJpb3VzLWNvcnJlbGF0aW9ucwoKCiFbXShpbWFnZXMvbm9iZWwtcHJpemUuanBlZykKCltTbyBpZiBJIHdhbnQgdG8gd2luIGEgbm9iZWwgcHJpemUsIEkgc2hvdWxkIGVhdCBldmVuIG1vcmUgY2hvY29sYXRlPyE/IT9dKGh0dHA6Ly93d3cuYnVzaW5lc3NpbnNpZGVyLmNvbS9jaG9jb2xhdGUtY29uc3VtcHRpb24tdnMtbm9iZWwtcHJpemVzLTIwMTQtND9JUj1UKQoKQnV0IG5vLW9uZSB3b3VsZCBldmVyIHRha2Ugc3VjaCB0cmVuZHMgc2VyaW91c2x5Li4uLndvdWxkIHRoZXk/CgoKIVtDdXR0aW5nLWRvd24gb24gSWNlIENyZWFtIHdhcyByZWNvbW1lbmRlZCBhcyBhIHNhZmVndWFyZCBhZ2FpbnN0IHBvbGlvIV0oaW1hZ2VzL2ljZWNyZWFtdnNwb2xpby5qcGcpCgo=