Five years ago this month, the Troubled Asset Relief Program pumped hundreds of billions of dollars into banks. The big ones have largely paid the government back. But many smaller community banks still owe the feds, and life is getting tougher for those who do.

TARP was designed so that banks wouldn’t want to stay in it long. One of the ways the government gave banks money was by buying preferred stock in the bank. In exchange, banks had to pay the government a 5 percent dividend. The deal was that after five years, banks that hadn’t paid off TARP would have to pay a higher dividend: 9 percent. Community banks that have been able to pay off TARP have done so. Those that haven’t now worry about the drag on business of having to pay the government more.

Community banks did gain new customers during the financial crisis, as people rethought whether they wanted to have their money in giant banks that got much of the blame for the meltdown. But the smaller banks are still in a challenging position.

“Many of them are in a more difficult situation than our large banks, which have recovered well from the financial crisis,” says Texas Tech finance professor Scott Hein. “That’s an irony that we still need to address.”

A key complaint community banks have is that new regulations put in place after the financial crisis are hurting them far more than big banks. JPMorgan Chase, Citi, and other large banks certainly aren’t fans of the new rules, but at least they have armies of lawyers and staff to wade through them. Small banks say they can’t afford that kind of help and they’re struggling. Higher TARP expenses will make the life of a community banker even tougher.