Discrete Dynamic Programming

Pablo Winant

Markov Chains

A worker’s employment dynamics obey the stochastic matrix

\[P = \begin{bmatrix} 1-\alpha & \alpha \\ \beta & 1-\beta \end{bmatrix}\]

with \(\alpha\in(0,1)\) and \(\beta\in (0,1)\). First line corresponds to employment, second line to unemployment.

Which is the stationary equilibrium? (choose any value for \(\alpha\) and \(\beta\))

α, β = 0.2, 0.9;
P = [α 1-α; 1-β β];
μ0 = [0.5, 0.5];

(μ0' * P^100)' # quick and dirty

2-element Vector{Float64}:
 0.11111111111111122
 0.8888888888888898

# we want to solve M x = y
# it's almost like x = P' x or (I-P') x = 0
# but it needs an additional constraint |x|=1

# we import the identity operator
using LinearAlgebra: I
I

2-element Vector{Float64}:
 1.0
 1.0

# option 1
M = I - P'
M[end,:] .= 1 # change last row (. for vectorized assigment)

y = zeros(2)
y[end] = 1

# solve for x in  Mx=y   solve(M,y)
M\y

2-element Vector{Float64}:
 0.11111111111111105
 0.888888888888889

ones(2)'

1×2 adjoint(::Vector{Float64}) with eltype Float64:
 1.0  1.0

#option 2
M =[
    I-P' ; # concatenate along first dimension
    ones(2)' #
]
# Mx = y
y = [0, 0, 1]
M


M\y      # lstq(M,y)  minimizes |Mx - y| in x

2-element Vector{Float64}:
 0.11111111111111112
 0.8888888888888891

In the long run, what will the the fraction \(p\) of time spent unemployed? (Denote by \(X_m\) the fraction of dates were one is unemployed)

# your code here

Illustrate this convergence by generating a simulated series of length 10000 starting at \(X_0=1\). Plot \(X_m-p\) against \(m\). (Take \(\alpha=\beta=0.1\)).

Basic Asset Pricing model

A financial asset yields dividend \((x_t)\), which follows an AR1. It is evaluated using the stochastic discount factor: \(\rho_{0,t} = \beta^t \exp(y_t)\) where \(\beta<1\) and \(y_t\) is an \(AR1\). The price of the asset is given by \(p_0 = \sum_{t\geq 0} \rho_{0,t} U(x_t)\) where \(U(u)=\exp(u)^{0.5}/{0.5}\). Our goal is to find the pricing function \(p(x,y)\), which yields the price of the asset in any state.

Write down the recursive equation which must be satisfied by \(p\).

# your code here

Compute the ergodic distribution of \(x\) and \(y\).

# your code here

Discretize processes \((x_t)\) and \((y_t)\) using 2 states each. How would you represent the unknown \(p()\)?

# your code here

Solve for \(p()\) using successive approximations

# your code here

Solve for \(p()\) by solving a linear system (homework)

# your code here

Asset replacement (from Compecon)

At the beginning of each year, a manufacturer must decide whether to continue to operate an aging physical asset or replace it with a new one.

An asset that is \(a\) years old yields a profit contribution \(p(a)\) up to \(n\) years, at which point, the asset becomes unsafe and must be replaced by law.

The cost of a new asset is \(c\). What replacement policy maximizes profits?

Calibration: profit \(p(a)=50-2.5a-2.5a^2\). Maximum asset age: 5 years. Asset replacement cost: 75, annual discount factor \(\delta=0.9\).

Define kind of problem, the state space, the actions, the reward function, and the Bellman updating equation

# your code here

Solve the problem using Value Function Iteration

# your code here

Solve the problem using Policy Iteration. Compare with VFI.

# your code here