It's much easier than you think. Nowadays, I store my 24 word recovery key phrase in clear text on a piece of paper in three different locations, some of them pretty insecure, open to see for anyone that wants to take a look.
Sounds idiotic, right?
Well, not really. Losing the full recovery phrase is of much bigger risk for me, and I can't even memorize my phone number (my shirts have a longer lifespan than my SIM cards), and 24 random words in the right order are pretty much impossible to keep in mind for the long run (for me at least).
There is a very elegant option that makes all the worries go away, takes all risks away regarding storing the 24 words without other people having access to it, and you just need to remember ONE word/phrase/combination chosen by yourself. It also fixes the problem someone holding a gun to your head.
https://www.ledger.com/academy/passphrase-an-advanced-security-feature