New Zealand FSPs are a huge headache to run if you are non-resident. The government is clamping down on non-compliant FSPs, having performed over 600 deregistration so far this year. Deregistration notice - 11 October 2012 | Companies Office FSP
FSPs can be great vehicles but you should think carefully and seek legal counsel before spending time and money on setting it up.
If you want a legitimate financial services company, I find that Gibraltar, Malta, Hong Kong, Singapore, and UK have some of the most favourable legislation and easy-going FIUs.
No one is going to take a financial services company in Costa Rica or Seychelles seriously, unless you are a known name in the industry and have all the right connections or unless you are into some very shady activities. If you engage in surreptitious activities, it may benefit you to be in such jurisdictions.