Affordability issues, high inflation, and management of public finances are among the reasons they say Mr. Trudeau should step down.