That's a big down side of the medal. Anyway, to buy Gold for bitcoins is possible as I read, it's not big of an issue. AdvCash offer a debit card and it's possible to load your card with btc to fiat and fiat to card, it's what you describe.but of course it won't work to buy gold bars with debit cards offered by crypto exchanges.
second that, it's so easy to do with most of the EMI's today. AdvCash is such an example where you can exchange BTC ->FIAT ->LOAD CARD in seconds.why not load a card and buying as you said @boomy ?