From 1984 to 2023, the median cost of a new home rose from $79,900 to $417,700, more than double the rate of overall price inflation.