Would you find it rude if someone declined to answer the question..?

It's rude to ask unless you have opened a repoire with someone already, and feel the question is not rude, and can accept a lack of answering as not rude if the person refuses to answer. Just like any other subject. That's why small talk is always about the weather.