I used to think it was just rumor as well - Snape obviously loves (consider his opening speech) and is skilled at (he brews the complicated Wolfsbane) potions - but then came OotP:

'You applied first for the Defence Against the Dark Arts post, I believe?' Professor Umbridge asked Snape.
'Yes,' said Snape quietly.
'But you were unsuccessful?'
Snape's lip curled.
'Obviously.'
Professor Umbridge scribbled on her clipboard.
'And you have applied regularly for the Defence Against the Dark Arts post since you first joined the school, I believe?'
'Yes,' said Snape quietly, barely moving his lips. He looked very angry.