The average interest rate on 30-year fixed mortgages hit about 4.5 percent this week. Still a pretty good rate, but up from 3.6 percent in early May. The market’s gotten a bit spooked by fears the Federal Reserve might taper off its purchases of mortgage-backed bonds.

But those higher mortgage rates aren’t putting much of a dent in the housing recovery.

“Although the increase in mortgage rates may be a bit of a drag on the housing market, we still anticipate the recovery is going to continue,” says Mike Fratantoni, vice president of research and economics for the Mortgage Bankers Association.

Fratantoni adds that’s because home prices are still below the housing bubble’s peak. Plus the job market is better, which means incomes are up and so are mortgage applications.

But Robert Denk, senior economist at the National Association of Home Builders, says what really counts is how many of those mortgage applications end up in the reject pile.

“If there is very limited access to credit, it doesn’t really matter whether the rates are historically low or whether they’re historically high,” he says. “Nobody’s getting the loans.”

As for those still thinking about refinancing? Denk says hurry up and do it before interest rates go up even more.