Neighbors fear the affordable housing project will further concentrate poverty in a distressed neighborhood.