Discrete Dynamic Programming

Pablo Winant

Markov Chains

A worker’s employment dynamics obey the stochastic matrix

\[P = \begin{bmatrix} 1-\alpha & \alpha \\ \beta & 1-\beta \end{bmatrix}\]

with \(\alpha\in(0,1)\) and \(\beta\in (0,1)\). First line corresponds to employment, second line to unemployment.

Which is the stationary equilibrium? (choose any value for \(\alpha\) and \(\beta\))

α = 0.2
β = 0.5
P = [(1-α)  α; β (1-β)]
μ0 = [0.5, 0.5]
y = [0.0, 0.0]

2-element Vector{Float64}:
 0.0
 0.0

# quick and dirty
(μ0'*P^100)'

2-element Vector{Float64}:
 0.7142857142857181
 0.28571428571428714

# clean iteration 
# define f(m) and copmute iterations $m_n = f(m_{n-1})$

# identity matrix
 # matlab: eye(2)

using LinearAlgebra: I


I *2.0

LinearAlgebra.UniformScaling{Float64}
2.0*I

# solve a linear system
# P' x = y  and |x| = 1

# solve M x = y and |x| = 1 with

M = P' - I

2×2 Matrix{Float64}:
 -0.2   0.5
  0.2  -0.5

# option 1 : replace last row with indpeendent condition

M[end,:] .= 1.0
M

rhs = copy(y)
rhs[end] = 1.0

M \ rhs

2-element Vector{Float64}:
 0.7142857142857144
 0.28571428571428564

# option 2 : use nonsquare system
M = [ 
    P' - I ;
    ones(2)'
]
M

R = [
    y;
    1
]

M \ R

2-element Vector{Float64}:
 0.7142857142857144
 0.28571428571428564

In the long run, what will the the fraction \(p\) of time spent unemployed? (Denote by \(X_m\) the fraction of dates were one is unemployed)

# your code here

Illustrate this convergence by generating a simulated series of length 10000 starting at \(X_0=1\). Plot \(X_m-p\) against \(m\). (Take \(\alpha=\beta=0.1\)).

Basic Asset Pricing model

A financial asset yields dividend \((x_t)\), which follows an AR1. It is evaluated using the stochastic discount factor: \(\rho_{0,t} = \beta^t \exp(y_t)\) where \(\beta<1\) and \(y_t\) is an \(AR1\). The price of the asset is given by \(p_0 = \sum_{t\geq 0} \rho_{0,t} U(x_t)\) where \(U(u)=\exp(u)^{0.5}/{0.5}\). Our goal is to find the pricing function \(p(x,y)\), which yields the price of the asset in any state.

Write down the recursive equation which must be satisfied by \(p\).

# your code here

Compute the ergodic distribution of \(x\) and \(y\).

# your code here

Discretize processes \((x_t)\) and \((y_t)\) using 2 states each. How would you represent the unknown \(p()\)?

# your code here

Solve for \(p()\) using successive approximations

# your code here

Solve for \(p()\) by solving a linear system (homework)

# your code here

Asset replacement (from Compecon)

At the beginning of each year, a manufacturer must decide whether to continue to operate an aging physical asset or replace it with a new one.

An asset that is \(a\) years old yields a profit contribution \(p(a)\) up to \(n\) years, at which point, the asset becomes unsafe and must be replaced by law.

The cost of a new asset is \(c\). What replacement policy maximizes profits?

Calibration: profit \(p(a)=50-2.5a-2.5a^2\). Maximum asset age: 5 years. Asset replacement cost: 75, annual discount factor \(\delta=0.9\).

Define kind of problem, the state space, the actions, the reward function, and the Bellman updating equation

# your code here

Solve the problem using Value Function Iteration

# your code here

Solve the problem using Policy Iteration. Compare with VFI.

# your code here