David Erlich seeks a writ of prohibition to restrain respondent court from issuing its writ of execution on a judgment for costs incurred on a successful appeal by Chaim I. Etner, the real party in interest, from a judgment for Erlich in a trade libel action.

Erlich also seeks to compel respondent court to quash any writ of execution heretofore issued;' to reset the trade libel action for trial at the earliest convenient date; and, with respect to an independent proceeding in respondent court commenced by Erlich, to vacate orders denying his application for a preliminary injunction, sustaining Etner’s demurrer without leave to amend, and granting Etner’s motion for summary judgment.

In his action for trade libel Erlich recovered a judgment against Etner for $30,000 general damages and $5,000 punitive damages. Subsequently, on the appeal, the judgment was reversed on the ground of insufficiency of the evidence in support of damages as awarded, and the cause was remanded for retrial on that sole issue. (Erlich v. Etner, 224 Cal. App.2d 69 [ 36 Cal.Rptr. 256 ].) After the filing of the remittitur, Etner obtained a judgment for costs on appeal of $1,802.03.

Etner is insolvent and unable to respond to or satisfy the judgment in whatever amount which may be rendered against him in the trade libel action. Erlich twice made motions to forestall enforcement of the cost judgment until final determination on retrial. In this connection he offered to post a stay bond in an amount to be designated by respondent court or to deposit $2,000 in a savings account and leave the passbook with Etner’s attorney pending final determination on retrial.

In response to the first motion, respondent court entered an order as follows: “Motion to recall and quash writ of execution is denied. The Court notes that no writ of execution on the judgment for costs has been issued. ...”

The court denied the second motion, citing First National Bank v. Stansbury, 214 Cal. 190 [ 5 P.2d 11 , 78 A.L.R. 358 ], and without otherwise indicating the basis of its decision.

The court subsequently granted a motion by Etner that the trade libel action should go off calendar to be reset upon payment of the appeal costs.

Erlich thereupon commenced an independent action in equity to enjoin Etner from collecting his judgment for costs, again offering to post security to guarantee Etner his costs in the event the judgment in the trade libel action was not sufficient to offset those costs. The court denied the application for a preliminary injunction, vacated a stay order previously entered, sustained a demurrer by Etner without leave to amend, granted a motion by him for summary judgment and, after the filing of the present petition, entered a summary judgment. The injunction and the trade libel actions have been consolidated.

In its order the court stated that it had no discretion in the circumstances. Although it is not entirely clear, the court apparently was of the opinion that the claim asserted by Erlich had been adjudged against him in earlier proceedings.

In the return to the alternative writ in the instant proceedings Etner states that he is without funds to pay his attorney or for the reporter’s transcript of the trial in the trade libel action; that the reporter refuses to give him a copy for use in the retrial until paid; and that he wishes to use the funds due on the cost judgment to pay such debts.

Section 440 of the Code of Civil Procedure provides: “When cross-demands have existed between persons under such circumstances that, if one had brought an action against the other, a counterclaim could have been set up, the two demands shall be deemed compensated, so far as they equal each other. ...” It is thus clear that coexisting cross-demands must be “compensated so far as they equal each other” which necessarily means that each of the claimants is deemed paid to the extent that their claims may be balanced in amount.

A principle akin to that enunciated by section 440 has been applied where the claim against which a setoff is sought is a judgment. Thus, a judgment debtor who by assignment or otherwise becomes the owner of a claim against his judgment creditor in equity may have his claim set off against the judgment, and the fact that the claim has not itself been reduced to judgment is not an obstacle. Nor must the claim of the judgment debtor be liquidated (Hauger v. Gates, 42 Cal.2d 752, 755 [ 269 P.2d 609 ]) ; it being sufficient that the claim though unliquidated has matured. (Coonan v. Loewenthal, 147 Cal. 218 , 223 et seq. [ 81 P. 527 , 109 Am.St.Rep. 128 ].) And there appears to be no reason to treat a judgment for costs on appeal differently from other judgments. Finally, the insolvency of the party against whom the relief is sought affords sufficient ground for invoking this equitable principle. (Harrison v. Adams, 20 Cal.2d 646, 648-649 [ 128 P.2d 9 ]; Machado v. Borges, 170 Cal. 501, 502-503 [ 150 P. 351 ] ; Coonan v. Loewenthal, supra, 147 Cal. 218 , 223 et seq.; cf. 11 U.S.C. §108 .)

It is apparent in such a case that the judgment debtor should not be required to pay the judgment prior to the determination of the validity of his own claim, as should such claim subsequently prove valid he would be deprived of his right to setoff and, in all likelihood, even the recovery of his payment to the extent of his judgment. Such a result is obviously unjust and contrary to the provisions of section 440 of the Code of Civil Procedure and the cases establishing the equitable right of setoff. The power to stay execution of a judgment on the ground that there is an action pending on a disputed claim by the judgment debtor against the judgment creditor was thus recognized in California Cotton Credit Corp. v. Superior Court, 127 Cal.App. 472, 475 [ 15 P.2d 1108 ].

The mere fact that a judgment debtor asserts a claim against the creditor, however, does not of itself mean that he is entitled to enjoin collection of the judgment, and the trial court has broad discretion in determining whether to grant the stay or enjoin collection of the judgment. (California Cotton Credit Corp. v. Superior Court, supra, 127 Cal.App. 472, 475 .) If the rule were otherwise, the judgment debtor could delay and harass his creditor by the filing of merely frivolous claims.

In determining whether to enjoin collection of the judgment pending decision of the validity of the disputed claim, the court should consider the likelihood that the judgment debtor will recover upon his claim, the probability and comparative amount of recovery, and the ability of the judgment creditor to respond should a judgment be rendered against him in the action on the disputed claim. The fact that the judgment creditor may have an immediate need for the funds due to him upon his judgment is not a basis for denial of relief where the foregoing considerations otherwise compel relief.

In the instant case, while the record shows that liability exists upon Erlich’s claim and that Etner will be unable to pay any judgment which might be rendered against him on the claim, it nevertheless fails to establish the amount which Erlich is likely to recover and, accordingly, we cannot say that as a matter of law Erlich was entitled to have collection of the judgment enjoined. It does not follow, however, that for such reason Erlich is not entitled to issuance of the peremptory writ. The availability of mandate is not limited to those situations where there hag been an abuse of discretion but also extends to cases where a trial court refuses to exercise its discretion because of a mistaken belief that the court had no discretion in the premises. (See In re Brumback, 46 Cal.2d 810, 813 [ 299 P.2d 217 ]; cf. Weber v. Superior Court, 53 Cal.2d 403, 406 [ 2 Cal.Rptr. 9 , 348 P.2d 572 ].) It thus follows that the instant petition for mandate is not foreclosed because the trial court, in sustaining Etner’s demurrer and granting his motion for summary judgment in the injunction proceeding, expressly did so upon the erroneous basis that it had no discretion..

Nor does it appear, as urged by Etner, that Erlich is barred by the 'doctrine of res jiidicata. The burden of estab lishing the defense of re's judicata is upon the party asserting it. (Dillard v. McKnight, 34 Cal.2d 209, 218 [ 209 P.2d 387 , 11 A.L.R.2d 835 ].) The order denying the first motion to recall or quash the writ of execution, which recites that no writ had issued, appears to be a determination that the motion was premature or that the form of the motion was improper until issuance of the writ, and such an order which is not based on the merits is not a bar to a subsequent proceeding in proper form. (Mercer Casualty Co. v. Lewis, 41 Cal. App.2d 918, 923-924 [ 108 P.2d 65 ] ; see Rest., Judgments (1942) §54; 3 Witkin, Cal. Procedure (1954) 1939-1940; cf. Keidatz v. Albany, 39 Cal.2d 826, 828 [ 249 P.2d 264 ]; Goddard v. Security Title Ins. & Guar. Co., 14 Cal.2d 47, 52-53 [ 92 P.2d 804 ].) The only evidence as to the basis of the order denying the second motion is the citation by the court of First National Bank v. Stansbury, supra, 214 Cal. 190 . That case, on the merits, is clearly distinguishable because the claim urged by the judgment debtor there had not matured at the time he attempted to use it as a setoff. Moreover, it does not appear whether there was a hearing on the merits of Erlich’s second motion or what showing was made in support of it, and in this state of the record the order denying the second motion may not be treated as a bar to relief.

Etner’s contention that an appeal from the judgment in the equitable proceeding is an adequate remedy is without merit. The absence of another adequate remedy was determined by this court when we granted the alternative writ. (City & County of San Francisco v. Superior Court, 53 Cal. 2d 236, 242 [ 1 Cal.Rptr. 158 , 347 P.2d 294 ] ; City of Los Angeles v. Superior Court, 51 Cal.2d 423, 429 [ 333 P.2d 745 ].)

The power of a trial court to stay the retrial of an action until costs on appeal are paid was recognized in Weile v. Sturtevant, 176 Cal. 767 , 768 et seq. [ 169 P. 685 ], The court there pointed out that this power is analogous to the power of a court to prevent oppression resulting from the multiplicity of actions by staying successive actions where costs of prior actions have not been paid; that the power to stay a retrial, which existed at common law, is a most useful one to prevent the hardship" to which the unlimited right of suit sometimes leads; and that the power should not be exercised in every ease where the costs of appeal have not been paid but only where the facts show oppression and hardship. As we have seen, Erlich has offered to post security for the cost judgment, and it is established as a matter of law that liability exists for the trade libel. Under the circumstances this is not a case where justice is accomplished by a stay of retrial.

We have examined the other contentions made by Etner and find that they are without merit or that they are not determinative in the present proceeding.

The alternative writ heretofore issued is discharged. Let a peremptory writ of mandate issue directing respondent court to set the action for trade libel for retrial and to consider, in accordance with the views expressed above, whether Erlich is entitled to have the collection of the judgment for costs enjoined pending the determination of the trade libel action. In all other respects the petition is denied.

Traynor, C. J., McComb, J., Peters, J., Tobriner, J., Mosk, J., and Burke, J., concurred.

The petition of the real party in interest for a rehearing was denied December 15, 1965.