There are a few characters I often had a hard time watching and especially listening to.

-Michael. He was continually upset, often suspicious and accusing others unjustly, and who can forget him yelling WALT. He was resentful and distrustful. He didn't really bond with anyone for a long time. Eventually he formed a kind of friendship with Jin.
Mostly, his personality rubbed me the wrong way. And eventually he betrayed the others. Even though it was part of a deal to get Walt back and go home, I disliked him for it.

Charlie Pace. I know he eventually gave his life for others, but that was at the end. He had a fatuous affectation, stupidly feigning ignorance and a lack of thought in order to be an irritant, as though he felt others didn't really like him so he would act the fool. I have known people like him who, afraid of being rejected if people knew them for who they were, they passive aggressively irritated them to force the rejection they believed was coming. Kind of "Oh, people think I am stupid, so I will act stupid on purpose, and pretend I don't know I am irritating them." And this behavior becomes a habit for them. Charlie knew Jin could not understand him but badgered him on whether he like the band The Kinks. He continued on in his quite louder than normal, fake friendly voice. Charlie was a continual fuckup. He was incompetent at most tasks, yet insisted on being involved in things in spite of this.
He was prone to becoming hyper excitable to the point of near hysteria. When Boone was injured Charlie began shouting at Jack "Where is Shannon?" Jack answered "I don't know." Charlie was nearly hysterical and continued yelling "Where is Shannon?" while Jack told him he didn't know, until Jack finally shouted back "I DON'T KNOW".
Charlie lacked impulse control, he could be demanding, and often urgently insisted that others share his hysteria, though he had no idea what to do. It was as though he felt others needed to feel as out of control as he felt. Like many addicts, Charlie is a manipulative and chronic liar. And I hate liars. I know he had his good points, but his personality was so damned irritating to me that I found it hard to like him for his good points.

Claire. Her nasal whining voice really got to me, especially when she was crying.

Ben Linus. I basically hated this guy because of his dictatorial, criminal, and cruel actions. He directed his people to kidnap and murder the survivors of the crash, as well as to hold them prisoners under harsh conditions. I disliked him, but he was not as irritating as the other three.

Many people don't like Ana Lucia. She was a psychologically damaged cop. She shot Shannon. But I didn't find her annoying or irritating. I understood how she made her errors, and I understood that she was mentally damaged, still suffering PTSD. So I didn't dislike her, even though I hated what she did. (I really liked Shannon. I like Maggie Grace.) Ana Lucia's voice didn't grate my nerves like Charlie's, Claire's, and Micheal's voices so often did.