Living in the poor side of town. Often the railroad tracks would divide the wealthy part of town from the poorer area. So to live on the wrong side of the tracks is to be in the slums.

“Dad would never let me marry him, because he says my boyfriend comes from the wrong side of the tracks.”