The soreness you feel after a workout essentially comes about because you damaged your muscle fibers. Exercise physiologists refer to this as delayed-onset muscle soreness (DOMS), and it typically becomes most noticeable a day or two after a workout. What happens is that microtears in your muscle fibers generate calcium leakage and an accumulation of histamincs, potassium, prostaglandins and local edema (accumulation of fluid) in the affected muscle. The edema stimulates the nerve endings, causing the sensation of pain.
The damage to your muscles and subsequent soreness occur to a greater degree as a result of the negative (eccentric) component of an exercise. That may be why you don't always experience the same level of soreness; not all exercises place the same type of eccentric stress on your muscles.
Now, you've probably noticed that the amount of soreness decreases as you become more accustomed to a particular workout and/or exercise. You're basically adapting to the stress of training, and you won't experience the same type of pain unless you increase the stress to an even greater degree. But I have to ask you: How much harder can you train? At some point, it would seem that you'd have to train till you fall over in each set. That's hardly reasonable.
A certain amount of damage to muscles seems to be required for increases in strength and size to occur. But the damage need not be so extensive that you constantly experience DOMS. Moreover, you absolutely do not need to train to failure in every set or even every workout. If you do, you're on the right track to becoming overtrained, in which case you can kiss your gains goodbye and say hello to an increased chance of injury.
The best way to go about your training is to manipulate how intensely you train over time, and do the same thing with your training volume and frequency. Termed penriodization, this has been found to be the most effective training system. The basic premise is that you'll observe times of highly intense training followed by less-intense training.
A little stiffness is to be expected after almost all workouts, but if you end up being sore for more than one day, you probably went too far. My advice is to make sure this doesn't happen too frequently.