I can explain because I felt the same strain and even posted here about it (first post on this forum). I went with a Swiss broker at first before consulting the forum (UBS) and then switched to another Swiss broker (PF) after consulting the forum, as they have substantially lower - not AUM based - custody fees. The transaction fees at PF hurt me, and even hinder me (is this good or bad?!) making transactions on the fly, or on a whim. 500CHF is not a life-changing or even significant amount, but it’s still not an amount I’d piss away that easily, and reshuffling my portfolio would cost me that, or more.
It’s mentally comforting to know I can literally walk down to a post office in my town and talk to a human - say because I screwed up my TFA or locked my account. Edit: That’s a human mistake which can happen, and I won’t relinquish the ability to do that, for almost any fee, and if someone calls me an idiot I’ll just call them a child who has much to learn about life 
Also, given I’ve been building this portfolio with a bit of cash at a time, and get paid in a Swiss bank account anyway, I know it’d cause me a lot of mental grief and stress to transfer all my savings with a few clicks to somewhere on the net - even if I am already doing that. I’ve never transferred big amounts in my life and I don’t find it obviously simple, emotionally, to simply sell and transfer 100s of k or more to IBKR or anywhere else. It’s the familiarity that drives being with PF. Also if I want to sell I can do that and the money is instantly in the same bank account, no transfer needed, no questions asked.
Also, regardless of whether this applies or not, I feel some sense of emotional security taxwise because given I declare these accounts on my tax forms I tell myself “The Swiss tax authority knows about it for years already and can track how they’re going, so there won’t be any questions if I want to sell - the money is in a Swiss account already for years now”. I know this is bullshit, but to me it makes some sense.
I think once someone becomes a citizen they can maintain their Swiss account wherever they are (?), so that’s my plan anyway.