did exactly the same, and exactly the same sort of thing happened! We just sat and loudly over-analysed the game, much to the confusion of a group of locals sat in front who seemed to think we were taking it far too seriously.
That's possibly the dumbest thing about going to American sports. There have been a few times I'd be at are local NFL stadium for college bowl or championship games, and with 5 or so minutes left we'd try to move maybe 5 rows up because everyone had left. But still, they wouldn't let us go, not sure we're hurting in that situation, but oh well.
It's worse if it happens at a supporters section too, that's supposed to be for anyone.