What can be said against Utilitarianism? Are all actions indeed an act towards happiness?

Im finding it hard to find fault in this theory, unless by being an empiricist Mill is ignoring other factors (though I cannot seem to find any). He seems to have explained all human discourse in a simpe framework. Can his theory interlock and marry with others that seem to defy his?