Critics say at the current cost of building affordable housing, $250 million in pandemic funds didn't put a dent in the home shortage.