(1) Obviously, a key concept here is 'wishing'. Weinberg says:
The first thing to note is that a wish is a kind of desire. As we’ll see later, wishes are different from other kinds of desires, but for now what matters is what they have in common with other desires: they are not chosen.Here is an obvious potential problem: this is not how people in general talk about wishing. We usually talk about wishing something as if it were a deliberate act, not an unchosen desire. If I wish you a happy birthday, it (one hopes) is true that I am wanting you to have a happy birthday, but that is not what is meant by wishing someone a happy birthday, or joy in life, or anything like this.
The natural way to explain this is that 'wishing' in fact covers multiple things that are different but related. We use 'wish' to cover a certain kind of desire, deliberate internal expressions springing from desires, and external expressions that are signs of desires. We use it to cover velleities, longings, and fantasies.
The real question, then, is not, "What kind of thing is wishing in general?" but "In talking specifically about wishing someone harm, what kind of thing is meant?" That so many people think it wrong is in and of itself evidence that, in this context, Weinberg's analysis of it as an unchosen desire is incorrect; it's certainly a sign that this should be seen as a highly controvertible claim.
(2) Are desires unchosen? Some immediate desires are unchosen. But is it really true that all desires are unchosen? Weinberg himself pulls back, later saying that "most desires are unchosen". I'm not sure this is true, either, but the 'most' raises the question of whether wishing (in the context of wishing someone harm) is a kind of desire that is, in fact, chosen, a desire you cultivate in yourself by deliberate actions. Well-wishing and ill-wishing are, again, things we talk about as if they were cultivated, or at least directed.
(3) This relates to another point. What we do, and the desires we have, are, when those desires can be cultivated or directed, expressions of character, and even involuntary desires (in the sense of desires we have spontaneously without choosing to have them) can be expressions of bad character. If an adult goes around having spontaneous feelings of wanting babies to come to harm, for instance, there is something wrong with them; and if they could reduce them or redirect them in less troubling directions or at least struggle against them, but instead deliberately indulge them, their feeling these things is an expression of bad character. There is something wrong with their ill-wishing, and it is a moral wrongness, even if it is not so in the same sense of 'wrong' that we use when we apply to actions.
(4) Weinberg's second reason for thinking that wishing someone harm is not morally wrong is that wishing is disconnected from reasons for action. I confess I am completely unclear why Weinberg thinks this is always the case. Mere wishes are wishes that don't issue into any kind of action, but wishing in fact regularly leads to action. If I wish you good luck as you go off to college, I might buy you a greeting card to that effect. I am not even sure what it would mean to say that wishing someone luck never gives a reason for expressing to them that you wish them luck. Perhaps the reason-relation here is oblique -- that is, expressing that you wish someone luck has a different relation to a desire for luck than somehow giving them luck -- but it's a reason-relation, nonetheless.
And it does seem that people do sometimes act out of wishing for harm for someone; at least, it seems reasonable to suggest that if A wishes harm for B, A would be more likely actually to harm B if the opportunity ever arises than he would be if A wishes B well. Again, perhaps wishing as a more generic desire deals with more generic reasons, and thus doesn't give us a specific reason to do specific things -- but there is no reason to deny that we can have generic reasons that, while not specific reasons themselves, combine with other reasons in particular situations to become specific reasons. We must be wary of any overly simplistic account of how desires interact to issue in action.
If it is the case that wishing is not disconnected from reasons for action, but only more obliquely or indirectly connected to them (which seems to fit our way of describing it better), then Weinberg's second reason doesn't seem to work.
(5) And likewise, although it may seem strange, it causes a problem for Weinberg's third reason. The third reason is that wishes aren't magic, but what this means in context is that wishing someone harm doesn't make it more likely that they will in fact be harmed. But as previously noted, this is not true: wishing someone harm can increase the likelihood that they will be harmed. If you don't come into contact with the person to whom you wish harm, and if all indirect influence is swamped out by other things, it is true that harm is unlikely to result; there is no causal channel available for it. But who is more likely to try to assassinate someone, the person who wishes them harm or the person who does not wish them harm? If you wish someone harm, you may harm them if you have the opportunity, and even if not, you may fail to do something that would keep them from harm, and even if not, you may influence others in a way that will make it more likely for those people to harm them or to fail to prevent harm to them. Wishes may not wash the dishes, but if you keep wishing that the dishes were washed, you might end up doing something about it, like buying a washing machine, or even just expressing your wishes for clean dishes -- which, as we know from birthdays and Christmas, sometimes increases the chance of someone else acting on your wish. Even if you don't express the wish, having the wish repeatedly, dwelling on it, etc., may lead you to act in such a way that it changes other people's behaviors in ways relating to dishes.
Wishes play such an obvious role in our social relations with other people, it's just too obvious that they influence results somehow. If by 'magic' you mean a direct influence, it's clear that this is often not the case; but wishing feeds into our social interactions in ways that clearly shift the results of those actions. This is very obviously true with wishing someone well, so why wouldn't it be true with wishing someone harm?
(6) There is plenty of reason to think that wishing someone well or wishing them harm is not usually a spontaneous desire. You can imagine a situation in which you might spontaneously wish someone harm for entirely understandable reasons -- for instance, if someone is torturing you right now, or if you have a vivid memory of being tortured by them in the past. But certainly wishing someone harm when you don't know them personally is not usually like this; it usually happens because you dwell on thinking about them in a bad light. This dwelling-on is mostly under your control. And even if the wishing itself didn't originally arise in this way, we can consent to and indulge ourselves in the wishing, and this is under our control. One of Weinberg's is to think that if we can't control our desires, that ends the story; but this is not true. We not only desire, we consent to or struggle against our desires. And this can very definitely be a kind of moral action, much more closely intermingled with the desire itself than any outward expression. Indulging yourself in wishing someone harm is a good sign of a character that is in some way malicious, and is itself often malicious, often selfish, often arrogant.
It is generally wrong, outside of extraordinary cases, to wish someone harm; it is wrong to indulge at length in wishing someone harm; it is wrong simply to let yourself be the sort of person who easily wishes another harm rather than work to be a better sort of person.