(Old fashioned) Not bothered. Uninterested in the outcome. A response to an act or words that are not of importance. Since skinny your nose would be uncomfortable, this says it’s not that bad, matter of fact. (Compare to No big deal or La-dee-dah.)

“You want to drive your car down that old dirt road? Well, it’s no skin off my nose if you do.”