Deterioration. Ruination. Suggests a better time that has since been depleted.  ‘Pot’ is used in the sense of worthless, perhaps from an old plant going bad in it’s planter pot. Also used as ‘gone to the dogs’. (Compare to Go downhill.)

 “Since his girlfriend left him, he’s just gone to pot, doesn’t shower, sits around the house watching TV all day.”