You cannot criticise people for going to the match - everyone has their reasons for going, some taking their kids, some it's a form of escape (good or bad) from daily life, others (like the guy I sit next to) have been going since the 70s - it's in his DNA to go.
I honestly don't believe he would care if attendance figure dropped as the TV is that much these days.
What I think he would/does care about is bad publicity - he even dragged himself in from of a House of Commons committee to defend himself. We need coordinated protests - in numbers - like Liverpool fans did. It needs to make the national press - not the likes of the Chronicle for him to care. And it needs to be constant until he's gone. And stop shopping at SD in Northumberland street.