Nah, I worked for a service provider that had a couple of public sector clients. In full detail, I technically worked at a freelance agency, who lend me to their service provider client, who then had me work on their clients.
Long story short: There are freelancing agencies that do all the admin work for you so you don’t need to found your own freelance business. Technically you are their employee, practically you are a freelancer without all the hassle bits.
Some agencies I know are: GULP, Adecco, Hays, Michael Page. You can check their project offers. Or, what I prefer: Have freelancing prominent on your LinkedIn profile and every now and then you get offers from them.