Seneca continues to battle the interlocutor accusing philosophers of hypocrisy for not practicing all that they teach. Perhaps philosophers sometimes fail to meet their own standards; but this does not somehow make the good they accomplish any less good, and, moreover, it is because their standards are themselves very good. People should be praised for a pursuit that seeks the best and does good, even if it fails to accomplish everything. In reality, philosophers are being attacked precisely for the greatness of their ideas and the excellence of their work. Moreover, those who make this attack show the cramped nature of their minds: a generous human spirit is always aiming for higher than it can certainly accomplish. If it fails -- it fails in a glorious pursuit. The critics are creatures of night complaining about the splendor of the light.
The Stoic diatribe continues with the objector still pressing the issue, complaining that Stoics say wealth, health, and the like should be despised but spend an immense part of their effort on them anyway. But, Seneca notes, the way one despises lesser things is not by eliminating them but by not grasping after them, or holding them in fear. Fleeing the thing can be as much an enslavement as pursuing it. Having wealth or good health is as much a forum for virtue as not having them.
What wealth and health do is not a matter of a virtue itself, but they do increase the opportunity for exercise of virtue. To this extent the Stoic sage will be pleased when fortune smiles on him so as to give him these things, because the whole point of the Stoic moral philosophy is that virtue is the one and only intrinsic good, so things have their value insofar as they are suitable for the use of virtue. The difference will be precisely in taking virtue as supreme: "In fine, my riches belong to me, you belong to your riches."
Nothing about wisdom requires that it be poor, but only that it strive for justice. The sage will neither be boastful of the gifts of Fortune nor ashamed of them. Instead he will use them carefully for good.
It is a mistake to think that virtue involves giving things away profligately. On the contrary, it requires giving things rationally, and this is very difficult. Benefits should be given so as to give others the possibility of giving benefits in return. Moreover, the opportunity to benefit can be found even in one's own household, because nature requires that we benefit human beings, and it does not matter whether they are one's own slaves or freeborn men; what matters is only that they are worthy of the benefit. (A full discussion of this short argument would no doubt require serious consideration of Seneca's treatise on benefits.) Moreover, the critic errs in conflating students of wisdom with the wise; the former will necessarily be imperfect. The important thing is that they value virtue above wealth.
If Seneca is placed in a house brimming with gold and silver, he thinks no more of himself than if he were in a hovel, because they are not part of who he is. Socrates would be the same way. The critics think that philosophers fail to meet their own standards because they have not actually understood what the standards are.
If the objector is motivated by the notion that this makes the wise man no different from the fool, he is simply wrong. The fool clings to wealth, so he thinks the wise man should cling to poverty -- but in reality, precisely the mark of the wise man is that he doesn't care enough about wealth to cling to either, choosing to regard it simply as a tool for virtue. The fool is devastated by loss of his wealth, but the sage is not harmed, merely limited. What is more, just like Socrates, the sage is not going to be bothered by this sort of objection; it is little more than useless whining.
Seneca imagines Socrates addressing the critic, admonishing them to praise good men if they can, and if they can't, to be silent; they are simply wasting their time. And harming themselves as well, because while they go about rebuking others for their failings, they spend little time rebuking themselves for their own. There is more than enough for them to do simply in improving themselves to be wasting their time attacking people better than they are.
And Seneca ends the book still addressing the critic in the person of Socrates, contrasting the smugness of their faces with the insecurity of their position.
Quotations are from the Aubrey Stewart translation, or (occasionally) directly from the Latin.