Teaching a cache to think beyond LRU
I wanted to see whether a Deep Q-Network could make better cache eviction decisions than least recently used (LRU). I built an L2 cache simulator in Python and PyTorch: 1024 bytes, eight-way set associative, with 64-byte blocks and a small 128-neuron MLP.
The 42% wall
The hit rate stalled around 42%. The network was learning the absolute positions of blocks, but blocks changed slots between episodes. I sorted the eight candidates by frequency and recency before presenting them to the model. With a consistent order, the reported hit rate rose to 44.87%.
Making the reward reflect the hardware
A dirty writeback and a cache miss each cost 100 DRAM cycles in my simulator, so I gave them equal penalties. The dirty writeback rate dropped from 35.4% to 31.8%. Once the model stopped making the worst evictions on its own, I removed the hard-coded safety overrides.
What the result means
On the tested Zipfian workload, the DQN reached 98.9 cycles average memory access time against 101.5 for LRU. LRU still had the higher hit rate; the advantage came from accounting for expensive dirty evictions. The policy sees normalized recency, frequency, age and dirty-bit metadata, never the actual memory address. This is a result for this simulator and workload, rather than a claim about production hardware.