The theme of rejected love is more implied than stated in the poem. What the poem presents, essentially, is a scene where the persona attempts to woo his mistress into having a sexual relationship with him.
In stanza after stanza, the persona conjures up different types of images, first to ease his mistress's fears by suggesting that time and distance are not of the essence in their love relationship and he can afford to sit by "the tide/ Of Humber and complain while she refuses" "Till" the conversion of the Jews" and he spends countless number of years adoring the physical features of his mistress.
When his mistress does not appear to have been moved, the persona invokes the spectre of death in stanza 2, with "Time's winged chariot hurrying near". Other images associated with the grave and decay are made use of, and the mistress is reminded of her own mortality and the work that worms will do to her virginity if she does not give in to his sexual overtures.
Since this may appear to have frightened his mistress enough, the persona argues in the 3rd stanza that now is time for them to consummate their love, for the "youthful hue/sits on thy skin like morning dew" and her body and senses seem aroused.
Throughout the poem, though, it is only the persona who speaks, pleads, coaxes and threatens. Not a word is heard from his mistress; yet it is the persona's turns of language used that suggest a rejection of his love by this mistress.