In all honesty, it isn't the gun that would cause me issues. Some guys just have the look of someone that isn't completely stable and I avoid these people if at all possible but would do so even more if they had a shotgun over their shoulder. Any normal person, I wouldn't care at all. I would figure you were hunting birds and just didn't have a place to drop it off or using it to keep bears away if we were in an area where that was a problem.
Actually, I guess it depends on the location. If you are going to one of the meat markets where fishermen are everywhere and there is no obvious hunting nearby, I might put you more into that "probably not completely stable" part and get out of there before some guy casting over your line causes you to lose your best fish of the day and things get ugly. Out in the less populated areas, not a big deal.