Random processes, Markov chains, and probabilistic modeling
← Back to MathematicsReview essential probability theory needed for stochastic processes.
Learn the basic concepts and classification of stochastic processes.
Master the theory and applications of discrete-time Markov chains.
Study Markov chains in continuous time and their generator matrices.
Learn renewal processes and their applications to reliability and maintenance.
Study Brownian motion and diffusion processes in continuous time.
Master martingale theory and its applications to stochastic processes.
Apply stochastic processes to model waiting lines and service systems.
Study stationary processes and their spectral analysis.
Explore advanced stochastic processes and modern applications.
Review essential probability theory and establish the mathematical foundations for stochastic processes.
Review the measure-theoretic foundation: (Ω, ℱ, P) where Ω is sample space, ℱ is sigma-algebra, and P is probability measure. Understand sigma-algebras and measurability.
Master random variables as measurable functions X: Ω → ℝ. Review discrete and continuous distributions, CDFs, PDFs, and transformations of random variables.
Study expectation E[X] = ∫X dP, variance, higher moments, and their properties. Learn Fubini's theorem and change of variables for expectations.
Review conditional probability P(A|B) and conditional expectation E[X|Y]. Understand the law of total expectation and law of total variance.
Master independence of events and random variables. Study correlation, covariance, and understand that independence implies uncorrelatedness but not vice versa.
Learn φ_X(t) = E[e^(itX)] and their properties. Understand uniqueness theorem, continuity theorem, and applications to sums of independent random variables.
Review Law of Large Numbers (weak and strong) and Central Limit Theorem. Understand their importance for asymptotic behavior of stochastic processes.
Study convergence in probability, almost surely, in distribution, and in L^p. Learn relationships between different modes of convergence.
Learn the fundamental concepts and classification of stochastic processes.
Understand {X(t), t ∈ T} as a family of random variables indexed by time parameter T. Learn finite-dimensional distributions and consistency conditions.
Study sample paths ω ↦ X(·,ω) as realizations of the process. Understand continuity, differentiability, and other path properties.
Classify processes by state space: discrete (countable states) vs. continuous (uncountable states). Learn examples of each type.
Distinguish discrete-time {X_n, n = 0,1,2,...} from continuous-time {X(t), t ≥ 0} processes. Understand modeling implications of each choice.
Learn strict stationarity (distribution invariant under time shifts) vs. weak stationarity (first two moments invariant). Understand practical importance.
Study processes where increments over disjoint intervals are independent. Learn that this leads to Lévy processes and includes Brownian motion and Poisson processes.
Study fundamental examples: random walk, Poisson process, Brownian motion, birth-death processes, and autoregressive processes.
Learn filtration {ℱ_t} as increasing family of sigma-algebras representing information available up to time t. Understand adapted processes.
Master the theory and applications of discrete-time Markov chains.
Learn the memoryless property: P(X_{n+1} = j | X_0,...,X_n) = P(X_{n+1} = j | X_n). Understand transition probabilities and matrices.
Study P^{(n+m)} = P^{(n)} · P^{(m)} relating n-step and m-step transition probabilities. Understand matrix exponentiation in discrete time.
Learn recurrent vs. transient states, periodic vs. aperiodic states. Understand positive recurrent, null recurrent, and absorbing states.
Study irreducible chains where all states communicate. Learn period of a state and aperiodic chains. Understand decomposition into communicating classes.
Find stationary distributions π satisfying π = πP. Learn existence and uniqueness conditions, and connection to eigenvectors of transition matrix.
Study convergence to stationary distribution: lim P^{(n)} = π for irreducible, aperiodic, positive recurrent chains. Learn ergodic theorem for Markov chains.
Calculate probability of eventual absorption into absorbing states. Learn to solve linear systems arising from first-step analysis.
Study hitting times T_j = inf{n ≥ 1: X_n = j} and their distributions. Learn mean return times and their relationship to stationary probabilities.
Study Markov chains in continuous time and their infinitesimal generators.
Master the fundamental continuous-time process: homogeneous Poisson process with rate λ. Learn counting process properties and exponential inter-arrival times.
Understand that Markov chains spend exponentially distributed time in each state. Learn the memoryless property of exponential distribution.
Study infinitesimal generator Q where Q_{ij} is rate of transition from i to j. Learn that row sums equal zero and P(t) = e^{Qt}.
Learn forward equation P'(t) = P(t)Q and backward equation P'(t) = QP(t). Understand their probabilistic interpretations and solutions.
Study processes where transitions are only to neighboring states. Learn birth rates λ_i and death rates μ_i, and detailed balance equations.
Find limiting distributions π satisfying πQ = 0. Learn conditions for existence and uniqueness, and connection to discrete-time embedded chain.
Transform continuous-time chain to discrete-time by uniformizing with rate ν ≥ max_i |Q_{ii}|. Learn how this enables computational methods.
Apply continuous-time Markov chains to model queueing systems. Study M/M/1 queue as birth-death process with arrival rate λ and service rate μ.