There are two problems here. One is the "walking four people abreast" problem. This takes up the whole sidewalk. Since you're probably chatting as you go, or you'd have broken up the line, you're not moving very fast. No one can pass you, no one can do anything. You have failed.
The other is the "I want to have a conversation right here" problem. In my experience, this is generally caused by teenagers. When I was trying to get to the train from the fireworks on the Parkway (along with the rest of the Philadelphia metropolitan area), there was a group of teenagers just hanging out on the corner, shooting the breeze. To drive this home: it was raining, it was after 11 pm, and everyone in the world was trying to get through that intersection. Step off the sidewalk! Save it! Get out of my way!
Or today, when I was cycling through a college campus. Now, I own a road bike. It likes pavement. As far as I know, the surface on which I stand has very little effect on my ability to converse. It would be my feeling, therefore, that if you have thirty kids loitering outside a building, that it is their prerogative to get the hell off the sidewalk. But no. I said "excuse me," but even the ones who were facing me simply failed to react. Someone has to beat some sense into these people.