Discrete Dynamic Programming

Pablo Winant

Markov Chains

A worker’s employment dynamics obey the stochastic matrix

\[P = \begin{bmatrix} 1-\alpha & \alpha \\ \beta & 1-\beta \end{bmatrix}\]

with \(\alpha\in(0,1)\) and \(\beta\in (0,1)\). First line corresponds to employment, second line to unemployment.

Which is the stationary equilibrium? (choose any value for \(\alpha\) and \(\beta\))

α = 0.2
β = 0.3
P = [
    1-α    α ;
    β     (1-β)
]

μ0 = [0.4, 0.6]
μ0'*P - μ0' # this should be zero

1×2 adjoint(::Vector{Float64}) with eltype Float64:
 0.1  -0.1

μ0'*P^50 # really bad idea

1×2 adjoint(::Vector{Float64}) with eltype Float64:
 0.6  0.4

μ0 # a vector

2-element Vector{Float64}:
 0.4
 0.6

μ0' # a 1x2 matrix

1×2 adjoint(::Vector{Float64}) with eltype Float64:
 0.4  0.6

μ0''

2-element Vector{Float64}:
 0.4
 0.6

maximum(abs,μ0)

0.6000000000000003

# loop
μ0 = [0.4, 0.6]
for t=1:10000
    μ1 = (μ0'*P)'
    if maximum(abs,μ1 -μ0) < 1e-8
        print("Converged after $t iterations")
        break
    end
    μ0 = μ1
end
μ0

Converged after 25 iterations

2-element Vector{Float64}:
 0.5999999880790714
 0.4000000119209292

# identity operator:
using LinearAlgebra: I
P' - I

2×2 Matrix{Float64}:
 -0.2   0.3
  0.2  -0.3

# we want to solve mu' P = mu' in mu'
# or:  P' mu = mu    (sound slike a linear algebra problem)

# or: (P'-I) mu = 0

# BUT, because of mass preservation, columns of P sum to 1
# rows of P' sum to 1
# rows of P' - I sum to 1

# need to :
# remove one of the colinear rows
# add another condition: sum(mu) = 1 -> add one row 1*mu[1] + 1*mu[2] == 1

using LinearAlgebra: I
M = P' - I
y = [0.0, 0.0]

# Change last row:
M[end,:] .= 1.0
# chante last condition (`==1`)
y[end] = 1.0

#solve M x= y in x
# x= (M)^(-1) y   # never!!! invert a matrix
x= M \ y # equivalent to solve(M,y)

2-element Vector{Float64}:
 0.6000000000000001
 0.39999999999999997

# we can also construct as system with more equations

M = [
    P' - I;
    1.0 1.0
]
y = [0.0, 0.0, 1.0]

# solve??? for M x =y deosn't make sense 
# but min_x |Mx -y| does makes sense (leastsq)

x = M \ y  #also works

2-element Vector{Float64}:
 0.6
 0.3999999999999999

In the long run, what will the the fraction \(p\) of time spent unemployed? (Denote by \(X_m\) the fraction of dates were one is unemployed)

# your code here

Illustrate this convergence by generating a simulated series of length 10000 starting at \(X_0=1\). Plot \(X_m-p\) against \(m\). (Take \(\alpha=\beta=0.1\)).

Basic Asset Pricing model

A financial asset yields dividend \((x_t)\), which follows an AR1. It is evaluated using the stochastic discount factor: \(\rho_{0,t} = \beta^t \exp(y_t)\) where \(\beta<1\) and \(y_t\) is an \(AR1\). The price of the asset is given by \(p_0 = \sum_{t\geq 0} \rho_{0,t} U(x_t)\) where \(U(u)=\exp(u)^{0.5}/{0.5}\). Our goal is to find the pricing function \(p(x,y)\), which yields the price of the asset in any state.

Write down the recursive equation which must be satisfied by \(p\).

# your code here

Compute the ergodic distribution of \(x\) and \(y\).

# your code here

Discretize processes \((x_t)\) and \((y_t)\) using 2 states each. How would you represent the unknown \(p()\)?

# your code here

Solve for \(p()\) using successive approximations

# your code here

Solve for \(p()\) by solving a linear system (homework)

# your code here

Asset replacement (from Compecon)

At the beginning of each year, a manufacturer must decide whether to continue to operate an aging physical asset or replace it with a new one.

An asset that is \(a\) years old yields a profit contribution \(p(a)\) up to \(n\) years, at which point, the asset becomes unsafe and must be replaced by law.

The cost of a new asset is \(c\). What replacement policy maximizes profits?

Calibration: profit \(p(a)=50-2.5a-2.5a^2\). Maximum asset age: 5 years. Asset replacement cost: 75, annual discount factor \(\delta=0.9\).

Define kind of problem, the state space, the actions, the reward function, and the Bellman updating equation

# your code here

Solve the problem using Value Function Iteration

# your code here

Solve the problem using Policy Iteration. Compare with VFI.

# your code here