Probability Theory

1. Probability
Probability is measure of expressing the belief that an event will occur or has occurred.
Consider a discrete variable X taking values from definition set U and a is value on such set. Probability of that X takes a is expressed as:
\[
P(X = a) = \frac{\text{the number of }a}{\sum_{i \in U }\text{the number of } i}
\]
P(X = a) is real-value in integral [0, 1] and $\sum_{i \in U} P(X = i) = 1$
2. Expectation:
a) Expectation value (mean) of X is weighted average of possible values that X can take. Weight is exactly the probability of that value occurrence:
\[
E[X] = \sum p(X = x) \times x
\]
Expectation is the value that you expect the result of your experiment to be average.
Example:
The studying outcome (in 5-point scale) of a student A is given by :

Grade (X)	Probability
5	3/20
4	9/20
3	4/20
2	3/20
1	1/20

In order to obtain the studying quality of a student A, we consider Expectation of grade X as representation value:
\[
\begin{align}
E[x] & = p(X = 1) \times 1 + p(X = 2) \times 2 + p(X = 3) \times 3 + p(X = 4) \times 4 + p(X=5) \times 5\\
& = 1/20 \times 1 + 3/20 \times 2 + 4/20 \times 3 + 9/20 \times 4 + 3/20 \times 5\\
& = 3.5
\end{align}
\]
b) Expectation value of a function
To find E[f(X)], where f(X) is a function of X, uses this formula:
\[
E[f(X)] = \sum p(X = x) f(x)
\]

Example:
Given a function $f(X) = X^2$ with:
f(X = 1) = 1 ; p(X = 1) = 1/6
f(X = 2) = 4 ; p(X = 2) = 3/6
f(X = 3) = 9 ; p(X = 3) = 2/6
The expectation value are obtained as:
\[
\begin{align}
E[f(X)] & = p(X=1) \times f(X=1) + p(X=2) \times f(X=2) + p(X=3) \times f(X=3)\\
& = 1/6 \times 1 + \3/6 \times 4 + 2/6 \times 9 \\
& = 31/6
\end{align}
\]

c) Expectation property
- Linear operator
\[
E[a X + b] = a E[X] + b
\]
where a, b are constants

3. Variance
a) Variance
Variance of X characterizes the spread of possible values of X and is written as:
\[
Var(X) = E[(X - E(X))^2]
\]
This can be also written as:
\[
Var(X) = E[X^2] - E^2(X)
\]
Proof:
\[
\begin{align}
Var(X) & = E[(X - E[X])^2] \\
& = \sum p(X = x) ( x - E[X])^2\\
& = \sum p(X = x) ( x^2 - 2xE[X]+E^2[X])\\
& = \sum p(X = x) x^2 - \sum p(X = x) 2xE[X]+\sum p(X = x)E^2[X])\\
& = E[X^2] - 2E[X]\sum p(X = x) x + E^2[X]\\
& = E[X^2] - 2E^2[X]+ E^2[X]\\
& = E[X^2] - E^2[X]
\end{align}
\]

b) Variance Property:
+ $Var(aX + b) = a^2 Var(X)$
where a, b are constants
Proof:
\[
\begin{align}
Var(aX +b) & = E[(aX+b)^2] - E^2[(aX+b)]\\
& = E[a^2X^2+2abX+ b^2] - (aE[X]+b)^2\\
& = a^2E[X^2]+2abE[X]+ b^2 - (a^2E^2[X]+2abE[X] + b^2)\\
& = a^2E[X^2] - a^2E^2[X]\\
& = a^2(E[X^2] - E^2[X])\\
& = a^2Var(X)
\end{align}
\]

4. Standard deviation
Standard deviation is square root of variance.

5. Probability function
Probability function covers both probability mass function (discrete probability function) and probability density function (continuous probability function)

$\circ$ Discrete probability function (probability mass function)
Suppose that U is a discrete set of possible values. Then $X: U \rightarrow R$ is a discrete variable. Probability mass function $f(x): R \rightarrow [0, 1]$ is defined as:
\[
f(X = x) = P(X = x) = P({s \in U| X(s) = x})
\]
Example:
Suppose the U is sample of simple toss of fair coin and X is random variaele on U by assigning 1 to "head" and 0 to "tail". Since coin is fair, so probability mass function of coin toss is:
\[
f(x) = \left\{ \begin{matrix} \frac{1}{2} \text{ , } x \in \{0, 1\}\\ 0 \text{ , } x\not\in \{0, 1\} &\end{matrix} \right
\]

$\circ$ Continuous probability function (probability density funcition - pdf, probability distribution fucntion or just density)
A function f(X) is considered as density function of continuous random variable X if it satisfies :
\[
P(a \leq X \leq b) = \int_{a}^{b} f(x) dx
\]
This definition leads to:
+ $0 \leq f(x) \leq 1$ with any x
+ $\int f(x) dx = 1$
Because continuous probability function is considered for the infinite number of points over continuous intervals, probability at a single point is always zero. That is why continuous probability function is defined over an interval rather than at one point.

6. Cumulative distribution function (Cumulative frequency function)
Cumulative distribution function is sum of probability values of probability function f(x) found at values less than or equal to x:
\[
F(x) = \int_{-\inf}^x f(x) dx
\]
Therefor, F(x) is an area under probability distribution curve from negative infinite to x.
For discrete probability function:
\[
F(x) = \int_{-\inf}^x f(x) dx = P(X \leq x)
\]

Note:
- The term "probability distribution function" is usually used to denote the "probability density function", but there isn't the standard among statisticians and probabilists. That leads to that the term "probability distribution function" are considered as "accumulative density function " or even "probability mass function" in some special cases.

7. Correlation Coefficient
http://www.aiaccess.net/English/Glossaries/GlosMod/e_gm_correlation_coefficient.htm
http://www.itl.nist.gov/div898/handbook/pmc/section5/pmc542.htm

Reference:
[1] Grading
[2] Expectation & Variance on Mathsrevision
[3] Distribution function on Wolfram Mathworld
[4] Probability funciton on Wolfram Mathworld
[5] Probability Distribution on Engineering Statistics handbook
[6] Probability density function on Wikipedia
[7] Cumulative distribution function on Wikipedia
[8] Probability mass function on Wikipedia