The ACT Party says its confidence in the Speaker of the House is “falling by the day”, accusing Gerry Brownlee of failing to address racial harassment in Parliament.