If they don’t try to collect by pursuing legal means, you can contact the bank after one year and they’ll have to free the caution then. If the landlord tries to collect (for example, by requesting it from a collection agency (no idea how to name it in English, Office des poursuites et faillites, in French)), it becomes more complex. You can search litigation for free (the authority responsible for it varies from Canton to Canton) but you’d end up with an entry on the collection registry for 5 years if you loose.
None of this is informed legal advice and I’m no specialist of the field. I’d consult with an attorney before making my decision (you can get a short free consultation by asking the lawyers’ association in some Cantons, it’s worth checking on their website).
I’ve lived a pretty similar situation recently. My analysis was that it was worth paying to avoid the hassle (though I was in a state of general tiredness at the time). I ended up paying and getting the hassle because my landlord was a real dumb dumb. Live and learn, I guess.