If you buy a football club, you've got to know that you're buying an asset that isn't really "yours"--not in the same way most other assets are. People are emotionally invested in it in ways they aren't to most other business enterprises. As an owner you've got to realize that those people are stakeholders, and if you're smart you realize that you owe some level of duty to treat them with respect. Even if you don't share the same enthusiasm for the club, those people are still your customers after all.
Ashley treats the fans--his customers--with nothing but sheer contempt, at least to the extent he even acknowledges them as anything at all other than a line on his accounts.
In my opinion we owe the club and ourselves the duty to let Ashley know, in no uncertain terms, that treating the club and the supporters with contempt is not fucking acceptable and that it's our club as much as it is his. If people still go to games and put money in his pocket, but vociferously protest, I've got no real issue with that, because they're off-setting the money they're putting in with their protesting. People that are willing to just sit back and get fucked by him, though . . I suppose they're getting the fucking they want.
Edit: Or, more succinctly, what Dave said.