I would recommend instead of creating a decision tree (“Swiss or not”, then “cheap or not”; two-step-decision), to decide it in one step. This makes the decision sharper: How much are you willing to pay more for swissness? I would also include fees to move your securities out, in addition to the usual suspects (currency fees, commissions, depot fee, Swiss stamp duty, higher spreads, TER of the funds available, withholding tax, etc.).
I will give you now my feedback to your reasons. Feel free to ignore it.
- Location of assets
This is not very defined. What exactly do you fear?
IB keeps the assets for european clients with IB UK. The regulatory agency FCA does not have a worse reputation than the Swiss Finma.
If you buy US funds, then you are possibly subject to US inheritance tax (and a duty to file a declaration after your demise) no matter if the broker is a Swiss one or not. Swiss ones, when not qualified intermediaries, might rat you out or not. However …
… is for US funds much harder without a qualified intermediary. You either lose 15% withholding tax on dividends (also with irish etf) or you file a partial US return.
For Swiss taxes, it’s not much more complicated. You fill it all in Wertschriftenverzeichnis or for US assets in DA-1 (a different sheet).
Legal point of view
Believe me, it’s not easier in Switzerland. Will give reasons if requested.
There is a whole thread about this. IB does have several interfaces.