# Probability By [Untitled](https://paragraph.com/@0xd65808da91cf1d2594b2754add8e916a8cb67441) · 2022-07-06 --- Probability ----------- Probability theory talks about the construction and implications of probability models. For example, given a probability distribution, what are the mean and variance? What is the distribution of a transformed random variable? In computer simulations, probability theory tells us what will happen to the generated realizations, in particular when the experiments can be repeated as many times as the researcher wishes. This is a real-world analogue of the frequentist’s interpretation of probability. Probability Space ----------------- A _sample space_ \\(\\Omega\\) is a collection of all possible outcomes. It is a set of things. An _event_ \\(A\\) is a subset of \\(\\Omega\\). It is something of interest on the sample space. A \\(\\sigma\\)-_field_ is a complete set of events that include all countable unions, intersections, and differences. It is a well-organized structure built on the sample space. A _probability measure_ satisfies * (positiveness) \\(P\\left(A\\right)\\geq0\\) for all events; * (countable additivity) If \\(A\_{i}\\), \\(i\\in\\mathbb{N}\\), are mutually disjoint, then \\(P\\left(\\bigcup\_{i\\in\\mathbb{N}}A\_{i}\\right)=\\sum\_{i\\in\\mathbb{N}} P \\left(A\_{i}\\right).\\) * \\(P(\\Omega) = 1\\). The above construction gives a mathematically well-defined probability measure, but we have not yet answered “How to assign the probability?” There are two major schools of thinking on probability assignment. One is the _frequentist_, who considers probability as the average chance of occurrence if a large number of experiments are carried out. The other is the _Bayesian_, who deems probability as a subjective brief. The principles of these two schools are largely incompatible, while each school has peculiar pros and cons under different real-world contexts. Random Variable --------------- A _random variable_ maps an event to a real number. If the outcome is multivariate, we call it a _random vector_. Distribution Function --------------------- We go back to some terminologies we learned in an undergraduate probability course. A _(cumulative) distribution function_ \\(F:\\mathbb{R}\\mapsto \[0,1\]\\) is defined as \\\[ F\\left(x\\right)=P\\left(X\\leq x\\right). \\\] It is often abbreviated as CDF, and it has the following properties. * \\(\\lim\_{x\\to-\\infty}F\\left(x\\right)=0\\), * \\(\\lim\_{x\\to\\infty}F\\left(x\\right)=1\\), * non-decreasing, * right-continuous \\(\\lim\_{y\\to x^{+}}F\\left(y\\right)=F\\left(x\\right).\\) The \\(q\\)-th quantile of a random variable is \\(\\min\_{x\\in \\mathbb R} P(X \\leq x) \\geq q\\). For a continuous distribution, if its CDF is differentiable, then \\\[ f(x) = d F\\left(x\\right) / d x \\\] is called the _probability density function_ of \\(X\\), often abbreviated as PDF. It is easy to show that \\(f\\left(x\\right)\\geq0\\), and by the Leibniz integral rule \\(\\int\_{a}^{b}f\\left(x\\right)dx=F\\left(b\\right)-F\\left(a\\right)\\). For a discrete random variable, its CDF is obviously non-differentiable at any jump points. In this case, we define the _probability mass function_ \\(f(x) = F(x) - \\lim\_{y \\to x^{-}} F(y)\\). We have learned many parametric distributions. A parametric distribution can be completely characterized by a few parameters. **Examples**: * Binomial distribution. \\\[f(x = k ;p) = \\binom{n}{k} p^k (1-p)^{n-k}\\\] * Poisson distribution. \\\[f(x = k;\\lambda) = \\frac{\\lambda^k \\exp(-\\lambda)}{k!}\\\] * Uniform distribution. \\\[f(x; a, b) = \\frac{1}{b-a} \\cdot \\mathbf{1}\\{a\\leq x \\leq b\\}\\\] * Normal distribution. \\\[ f(x; \\mu, \\sigma^2) = \\frac{1}{\\sqrt{2\\pi} \\sigma} \\exp\\left( - \\frac{(x-\\mu)^2}{2\\sigma^2}\\right) \\\] Its mean is \\(\\mu\\) and variance \\(\\sigma^2\\). * Log-normal distribution. \\\[ f(x; \\mu, \\sigma^2) = \\frac{1}{\\sqrt{2\\pi} \\sigma} \\exp\\left( - \\frac{(\\log(x)-\\mu)^2}{2\\sigma^2}\\right), \\\] Its means is \\(\\exp(\\mu + 0.5 \\sigma^2)\\) and variance \\(\[\\exp(\\sigma^2 - 1)\] \\exp(2\\mu+ \\sigma^2)\\). * \\(\\chi^{2}\\), \\(t\\), and \\(F\\) distributions. **Example** `R` has a rich collection of distributions implemented in a unified pattern: `d` for density, `p` for probability, `q` for quantile, and `r` for random variable generator. For instance, `dnorm`, `pnorm`, `qnorm`, and `rnorm` are the corresponding functions of the normal distribution, and the parameters \\(\\mu\\) and \\(\\sigma\\) can be specified in the arguments of the functions. Integration ----------- In probability theory, an integral \\(\\int X\\mathrm{d}P\\) is called the _expected value,_ or _expectation,_ of \\(X\\). We often use the notation \\(E\\left\[X\\right\]\\), instead of \\(\\int X\\mathrm{d}P\\), for convenience. The expectation is the average of a random variable, despite that we cannot foresee the realization of a random variable in a particular trial (otherwise there is no uncertainty). In the frequentist’s view, the expectation is the average outcome if we carry out a large number of independent trials. If we know the probability mass function of a discrete random variable, its expectation is calculated as \\(E\\left\[X\\right\]=\\sum\_{x}xP\\left(X=x\\right)\\). If a continuous random variable has a PDF \\(f(x)\\), its expectation can be computed as \\(E\\left\[X\\right\]=\\int xf\\left(x\\right)\\mathrm{d}x\\). Here are some properties of the expectation. * \\(E\\left\[X^{r}\\right\]\\) is call the \\(r\\)-moment of \\(X\\). The _mean_ of a random variable is the first moment \\(\\mu=E\\left\[X\\right\]\\), and the second _centered_ moment is called the _variance_ \\(\\mathrm{var}\\left\[X\\right\]=E \[(X-\\mu)^{2}\]\\). The third centered moment \\(E\\left\[\\left(X-\\mu\\right)^{3}\\right\]\\), called _skewness_, is a measurement of the symmetry of a random variable, and the fourth centered moment \\(E\\left\[\\left(X-\\mu\\right)^{4}\\right\]\\), called _kurtosis_, is a measurement of the tail thickness. * We call \\(E\\left\[\\left(X-\\mu\\right)^{3}\\right\]/\\sigma^{3}\\) the _skewness coefficient_, and \\(E\\left\[\\left(X-\\mu\\right)^{4}\\right\]/\\sigma^{4}-3\\) _degree of excess_. A normal distribution’s skewness and degree of excess are both zero. * Moments do not always exist. For example, the mean of the Cauchy distribution does not exist, and the variance of the \\(t(2)\\) distribution does not exist. * \\(E\[\\cdot\]\\) is a linear operation. \\(E\[a X\_1 + b X\_2\] = a E\[X\_1\] + b E\[X\_2\].\\) * _Jensen’s inequality_ is an important fact. A function \\(\\varphi(\\cdot)\\) is convex if \\(\\varphi( a x\_1 + (1-a) x\_2 ) \\leq a \\varphi(x\_1) + (1-a) \\varphi(x\_2)\\) for all \\(x\_1,x\_2\\) in the domain and \\(a\\in\[0,1\]\\). For instance, \\(x^2\\) is a convex function. Jensen’s inequality says that if \\(\\varphi\\left(\\cdot\\right)\\) is a convex function, then \\\[ \\varphi\\left(E\\left\[X\\right\]\\right)\\leq E\\left\[\\varphi\\left(X\\right)\\right\]. \\\] * _Markov inequality_ is another simple but important fact. If \\(E\\left\[\\left|X\\right|^{r}\\right\]\\) exists, then \\\[ P\\left(\\left|X\\right|>\\epsilon\\right)\\leq E\\left\[\\left|X\\right|^{r}\\right\]/\\epsilon^{r} \\\] for all \\(r\\geq1\\). _Chebyshev inequality_ \\(P\\left(\\left|X\\right|>\\epsilon\\right)\\leq E\\left\[X^{2}\\right\]/\\epsilon^{2}\\) is a special case of the Markov inequality when \\(r=2\\). Multivariate Random Variable# ----------------------------- A bivariate random variable is a vector of two scalar random variables. More generally, a multivariate random variable has the _joint CDF_ as \\\[ F\\left(x\_{1},\\ldots,x\_{n}\\right)=P\\left(X\_{1}\\leq x\_{1},\\ldots,X\_{n}\\leq x\_{n}\\right). \\\] Joint PDF is defined similarly. It is illustrative to introduce the joint distribution, conditional distribution and marginal distribution in the simple bivariate case, and these definitions can be extended to multivariate distributions. Suppose a bivariate random variable \\((X,Y)\\) has a joint density \\(f(\\cdot,\\cdot)\\). The _marginal density_ \\(f\\left(y\\right)=\\int f\\left(x,y\\right)dx\\) integrates out the coordinate that is not interested. The _conditional density_ can be written as \\(f\\left(y|x\\right)=f\\left(x,y\\right)/f\\left(x\\right)\\) for \\(f(x) \\neq 0\\). Independence ------------ For two events \\(A\_1\\) and \\(A\_2\\), the _conditional probability_ is \\\[ P\\left(A\_1|A\_2\\right) = \\frac{P\\left(A\_1 A\_2\\right)}{ P\\left(A\_2\\right) } \\\] if \\(P(A\_2) \\neq 0\\). In this definition of conditional probability, \\(A\_2\\) plays the role of the outcome space so that \\(P(A\_1 A\_2)\\) is standardized by the total mass \\(P(A\_2)\\). Since \\(A\_1\\) and \\(A\_2\\) are symmetric, we have \\(P(A\_1 A\_2) = P(A\_2|A\_1)P(A\_1)\\). It implies \\\[ P(A\_1 | A\_2)=\\frac{P\\left(A\_2| A\_1\\right)P\\left(A\_1\\right)}{P\\left(A\_2\\right)} \\\] This formula is the well-known _Bayes’ Theorem_, which is a cornerstone of _decision theory_. **Example** \\(A\_1\\) is the event “a student can survive CUHK’s MSc program”, and \\(A\_2\\) is his or her application profile. We say two events \\(A\_1\\) and \\(A\_2\\) are _independent_ if \\(P(A\_1A\_2) = P(A\_1)P(A\_2)\\). If \\(P(A\_2) \\neq 0\\), it is equivalent to \\(P(A\_1 | A\_2 ) = P(A\_1)\\). In words, knowing \\(A\_2\\) does not change the probability of \\(A\_1\\). If \\(X\\) and \\(Y\\) are independent, \\(E\[XY\] = E\[X\]E\[Y\]\\). Law of Iterated Expectations# ----------------------------- In the bivariate case, if the conditional density exists, the conditional expectation can be computed as \\(E\\left\[Y|X\\right\]=\\int yf\\left(y|X\\right)dy\\). The law of iterated expectation implies \\(E\\left\[E\\left\[Y|X\\right\]\\right\]=E\\left\[Y\\right\]\\). Below are some properties of conditional expectations 1. \\(E\\left\[E\\left\[Y|X\_{1},X\_{2}\\right\]|X\_{1}\\right\]=E\\left\[Y|X\_{1}\\right\];\\) 2. \\(E\\left\[E\\left\[Y|X\_{1}\\right\]|X\_{1},X\_{2}\\right\]=E\\left\[Y|X\_{1}\\right\];\\) 3. \\(E\\left\[h\\left(X\\right)Y|X\\right\]=h\\left(X\\right)E\\left\[Y|X\\right\].\\) **Application** Regression is a technique that decomposes a random variable \\(Y\\) into two parts, a conditional mean and a residual. Write \\(Y=E\\left\[Y|X\\right\]+\\epsilon\\), where \\(\\epsilon=Y-E\\left\[Y|X\\right\]\\). Show that \\(E\[\\epsilon\] = 0\\) and \\(E\[\\epsilon E\[Y|X\] \] = 0\\). --- *Originally published on [Untitled](https://paragraph.com/@0xd65808da91cf1d2594b2754add8e916a8cb67441/probability)*