I wore a vest for years and always found when fully loaded it gave me a sore back after a few hours. I imagine this is due to all the weight being in the front. The convenience of having everything in immediately accessible pockets is hard to beat though. Another thing I didn't like about the vest is the compartments are not waterproof and my fly boxes, tackle in ziplocks, etc got covered in water droplets and had to be aired/dryed out when I got home.
I guess the bottom line is each way of carrying your tackle has it's costs and benifets. For me all the tackle I need immediate access to can fit in my wading jacket. Everything else goes in the pack. Having the weight in the back feels a lot more comfortable and doesn't give me a sore back.