Old Hippie reference to being almost out of marijuana because the smokable leaves are gone and only stems and seeds remain. The part of a marijuana plant that provides a ‘high’ are the dried leaves, but they often came with the seeds and stems of the plant, hippies had to carefully leave the stems and seeds in the bag and not smoke them as they cause headaches. So the phrase implies that one is either too broke to buy pot or lost your contact for procuring and all you have is the bag of stems and seeds. (Compare to Down and out.)
Down to seeds and stems
DEFINITION