The essential idea of How to Do Things with Words is that understanding a considerable part of our communication depends on regarding it not as description-like (constative) but as a a practical action (performative). There are statements that require this kind of treatment. For instance, if I say, "I hereby undertake to pay all your bills," I'm not just describing something, I am using these words to do something. To explore how this works, Austin looks at ways in which you can fail to do something with performatives. If you can fail, that means that some condition for success was not met, and by collecting the conditions for success you get a profile of what these performatives are. These conditions for success are called 'felicity conditions'. Austin proposes six of them; he regards them as in some sense exhaustive, but in working them out recognizes that they don't all work the same way.
(A.1) There must exist an accepted conventional procedure having a certain conventional effect, that procedure to include the uttering of certain words by certain persons in certain circumstances, and further,[ J. L. Austin, How to Do Things with Words, 2nd edition, Urmson and Sbisà, eds., Harvard UP (Cambridge MA: 1975) pp. 14-15.]
(A.2) the particular persons and circumstances in a given case must be appropriate for the invocation of the particular procedure invoked.
(B.1) The procedure must be executed by all participants both correctly and
(B.2) completely.
(Γ.1) Where, as often, the procedure is designed for use by persons having certain thoughts or feelings, or for the inauguration of certain consequential conduct on the part of any participant, then a person participating in and so invoking the procedure must in fact have those thoughts or feelings, and the participants must intend so to conduct themselves, and further
(Γ.2) must actually so conduct themselves subsequently.
In a sense A and B rules have to do with the esse of the performative and Γ rules with the bene esse, so I'll set aside Γ, at least mostly, for now. (There's a reason Austin gives it a Greek letter to mark it off from the others.)
So let's take an example. I smash a bottle on a ship and say, "I name this ship such-and-such." We have a conventional procedure here -- ship christening -- and a conventional effect -- ship gets a name -- which requires people saying something in a certain role in certain circumstances. That's (A.1). Now, for a ship christening actually to succeed, I have to be the right kind of person doing this in the right circumstances -- I can't just be a random person going around smashing bottles on ships, and if for some reason we were just rehearsing, obviously I didn't actually christen the ship, any more than you get married at the wedding rehearsal. That's (A.2). The ship christening could also fail, and need to be redone, if I somehow did it wrong (B.1) or if I were interrupted (B.2).
Austin therefore gives us a taxonomy of failure (and thus negatively of success), and proposes, tentatively, some names for each kind. One possible way of doing it that mixes and matches from his various suggestions is this:
A & B: Misfires (act purported but void)
A: Misinvocations
A.1: Non-plays
A.2: Misapplications or Misplays
B: Misexecutions or Miscarriages
B.1 Flaws
B.2 Hitches or Non-executions
Γ: Abuses (acts professed but hollow)
Γ.1: Dissimulations
Γ.2: Breaches
But it probably doesn't hugely matter what you call them, as long as you know what condition is violated.
Austin is not just exploring an idea in language, though; these six, with perhaps minor modification at times, are found for all sorts of ceremonies and rites. (Austin himself notes this.) And indeed, one way to understand Austin's argument is to understand it as an argument for the importance of the ritual aspect of language. He will even sometimes talk about rites instead of 'conventions' in discussing the issues with regard to (A.1). And any kind of communicative ritual will be describable in Austin's terms. Thus you can apply this kind of thinking to rituals like sacraments. Nor is this surprising; if you look at his examples, it is pretty clear that a lot of his thought on this subject is based on reflecting on (Anglican) rituals and sacraments in the first place -- baptism and marriage being the primary ones. And Catholic sacraments properly done fulfill all six felicity conditions.
And by the same token, one can fail at them in ways analogous to failing at promising, or betting, or ceremonial naming, or any of the many other performatives at which one can fail. Austin's A-type failures are in the cases of sacraments causes of invalidity. So for instance, one can purport to be ordaining someone but fail because you make up your own right (violating A.1) or because you do not have the authority of a bishop (violating A.2). Someone who baptizes "In the name of the Creator, the Redeemer, and the Sanctifier" has failed to baptize; they violated A.1. Someone trying to ordain a woman as a Catholic priest has failed; they are violating A.2. Someone trying to marry two people of the same sex in a Catholic marriage have violated A.2. And so on and so forth.
B.1 failures in the case of sacraments can also be causes of invalidity, although perhaps not -- it depends on how the error effects what's essential to the sacrament. This is something that has had extensive discussion over the centuries. One that comes to mind is the case of the priest who's ignorant of Latin trying to baptize "In nomine patris et filiae et spiritus sanctus". The 'filae' makes it invalid (because 'filiae' can only mean 'of the Daughter'), whereas there's a very good argument that the 'spiritus sanctus' on its own does not (because there's nothing else it could mean in context except 'of the Holy Spirit'). Both are wrong, but the former error changes something essential, wrecking the sacrament, whereas the latter is arguably just bad grammar. If the priest saying the Mass in English says, without realizing it, "This is body my", he did it wrong, but no one need be distressed about whether it is valid. Priests (unfortunately) often don't say the words of absolution correctly in the sacrament of reconciliation; while this is not a great thing, it doesn't automatically make it sacramentally invalid.
B.2 failures can also be causes of invalidity. They certainly make it so that the sacrament is not valid, but sometimes (we might say) they only make it 'nonvalid' rather than 'invalid'. If a priest said the Mass, and having raised the host begins the words of institution, "Th--" but at that moment collapses or is shot and never finishes, there is no valid eucharist even though he didn't actually do anything invalid or wrong -- it just never got finished. On the other hand, if he just skips the words of institution, this is very definitely wrong and invalid.
Γ failures, of course, do not affect validity; they are failures in doing the sacraments worthily. (It is a common error to think that sacraments can automatically be invalidated by secretly redirecting your intent; this confuses 'intent' or 'intention' in the modern sense with 'intention' in the scholastic sense, which means the disposition of the act to an end. If the conditions for the act are as they should be and the act is done so as to communicate doing what the Church does, there is nothing wrong with the act itself, although you may be doing it in a bad spirit.)
A complication is that in the case of sacraments we are always evaluating at three different levels -- what's essential to the sacrament, what's in accordance with the rules and laws meant to protect the sacrament, and what's in accordance with morals given the sacrament. One has the same kind of errors, it's just we sometimes want to make a distinction between an A.2 violation that makes the sacrament invalid and an A.2 violation that may or may not make it invalid but is still out of bounds -- or illicit, as we often say.
An implication of all this is that discussions of validity and invalidity, licitness and illicitness, in the sacraments are not arcane legalisms; they are discussions of facts about actions -- whether you actually performed the action, correctly and completely, as it should be done. There's a real fact of the matter, even if there are cases that are difficult to evaluate or need further clarification. Likewise, they aren't weird. You get exactly the same issues, adapting to the differences in actions, whenever you deal with any kind of ritual or communicative action. And what is more, it is not trivial. Take, for instance, an apology: you can fail in an apology if you don't actually do anything that people could take as an apology (A.1), or if you don't apologize to the person to whom you should actually be apologizing (A.2), or if you mangle it too badly (B.1) or break off (B.2), or if your apology is a lie (Γ.1 and Γ.2). And it matters, because apology matters. So too with every other kind of ritual.