This is going to kill me, literally...
While I Was Trying to Explain This To You, My Head Literally Exploded
Before we get into the details of how you've been using the word “literally” wrong, or how annoying it is when you do, or in what specific way it makes you sound silly, a couple of caveats:
I am not judging you as a person; nor should you judge yourself. We tend to take the “love the sinner, hate the sin” attitude in this matter.
You're in good company; even intelligent, professional people, from NPR commentators to government officials, get it wrong all the time. It's no excuse, but there's no sense feeling lonely while you blunder.
Now you may already be feeling defensive, because you're an educated person and you think the way you've been saying “literally” is probably right because most of the time you use words correctly, and that if you had been using it wrong an English teacher would have told you, and anyway it's just a word and words are open to interpretation, right? Well, yes, sometimes they are, but in the same way that “1” can't really be interpreted as “0,” or “Republican” as “Democrat,” you can't really use “literally” in certain ways.
Here's a list. It is not exhaustive, but it's pretty comprehensive for a non-academic blog.
It does not mean figuratively, as in the case of the woman I heard once say to a colleague, “I was literally dead on my feet yesterday.” Presumably this was not the case, as she didn't appear to be dead when she told this story, but the important point here is that she actually meant figuratively, which is the opposite of literally. Being “dead on your feet” is a figure of speech that emphasizes exhaustion by exaggerating your case (unless you happen to be both dead and standing up, which I don't think anybody can manage for very long). Now there might be a time that you do literally take the money and run, but I'd rather maintain plausible deniability, so if that happens, don't tell me.
It does not mean strongly, vehemently, or any other word that imparts emphasis. Many people use it that way, and some dictionaries even list that as one of its meanings, but the people who are in charge of dictionaries these days are so craven that they'll include any made-up, misused word if enough people use it, because who wants to be like, y'know, correct? Totes ridic.
It does not mean seriously, as in not in jest. This is why you think you can, but in fact cannot, say, “Dude, I mean it, stay away from my sister or literally I will kill you” without committing assault.
And it doesn't really work in matters of opinion; no one is “literally the best” unless specific criteria are under consideration. So while you could be literally the tallest person in a room, you could not be literally the most annoying, because there's no objective measurement of annoyance (although if there was, correcting the grammar of your friends, neighbors and co-workers would post huge numbers).
Part of what makes this so difficult is that English words are often rich with multiple definitions. For instance, “I had to cut Bob today” could mean you fired him or that you're a violent felon. So it's not that we are without sympathy – just that I think you, and other well-spoken, well-educated people, should hold yourself to a higher standard and use complex words correctly. So please do so from now on, or I will literally lose my mind.