Snape was in love with Lily Potter since he was a boy; they were best friends from childhood and through much of their time at Hogwarts. Eventually, however, Snape began hanging out with the soon-to-be Death Eaters and adopting their beliefs about mudbloods and the superiority of pure-blooded wizards (probably because of his awful experience with his Muggle father). He and Lily grew apart in their later years at Hogwarts because of this. At the same time, Lily began dating Snape's greatest enemy at school, James Potter. James and Sirius Black enjoyed teasing and harassing Snape, partly because he was weird, but also because James had a crush on Lily.
For Snape, Harry is the physical embodiment of Lily and James' love — he reminds Snape that Lily never loved him that way, and that she instead loved his rival. Additionally, Harry looks and acts a lot like James (Snape also says several times that he is "as arrogant as his father").
Finally, I think that Snape is probably also bitter/sad/angry that Lily sacrificed herself for Harry. If it weren't for Harry, Lily could still be alive.