This is correct, but: for long term investment recurring fees (TER + taxes on dividends + custody fees) are more important than fixed costs incurred when buying. So even 1% currency exchange fee would be acceptable, but there are also ways to push it down to 0.5% and lower.
Furthermore, the most tax-effective geographically-balanced investment strategy for Swiss investors seems to be: keep only US exposure in taxable account and gain exposure to the rest of the world market via 3a index funds.
You can invest in something like VUSA or any MSCI USA ETF (Irish or US) with a Swiss broker. There are brokers with reasonable fees. You can also go to a European broker, get lower fees and a wider choice of ETFs. You would have to pre-exchange your CHF to EUR. Or you can go to IB, but you probably know this story.