OPINION

The Inlandboatmens’ Union of the Pacific (“IBU”) brought this action against the Dutra Group, alleging that Dutra violated the terms of a settlement agreement between the two parties. Dutra contends that because the IBU failed to seek arbitration of this dispute as required by the collective bargaining agreement between the parties, the federal courts lack jurisdiction over the case. The district court granted Dutra’s motion to dismiss, and the IBU appeals.

I. BACKGROUND

Dutra is a marine construction, towing, and dredging company based in Northern California. The IBU represents deckhands employed by Dutra on its barges and tugboats that operate in the San Francisco Bay and its tributaries. In November, 1997, the IBU filed a grievance with Dutra regarding a subcontracting arrangement which, according to the IBU, violated the parties’ collective bargaining agreement (“CBA”). The IBU contended that Dutra leased one of its barges, the “Harry B,” to another company, Master’s Tug & Tow, and subcontracted with Master’s to complete work for Dutra. The CBA provides that Dutra will use only IBU-represented personnel to perform Dutra’s work; however, Master’s did not employ IBU members. The IBU contended that three of its members were laid off when the nonunion employer took over the operation of the Harry B in Dutra’s stead.

The CBA between Dutra and the IBU contains a reasonably broad arbitration provision; it governs “[a]ny dispute concerning ... wages, working conditions, or any other matters referred to in this [CBA].” After the IBU filed the “Harry B” grievance, the parties arranged for an arbitration of the dispute, as required by the CBA. Prior to that formal proceeding, however, the parties engaged in less formal mediation. The mediation succeeded in resolving the dispute, and resulted in a settlement agreement between the parties that was concluded on August 23, 1999,.

The settlement agreement had five elements to it. First, Dutra agreed to compensate IBU workers who would have worked on the Harry B jobs had the work not been subcontracted and performed by non-union workers. Second, the parties agreed to renew the CBA and agreed upon its general terms. Third, Dutra agreed that it would subcontract work to Master’s only if the subcontractor agreed to employ IBU members for labor to be performed on behalf of Dutra. Fourth, Dutra agreed to support the IBU in a jurisdictional dispute with another union. Finally, the parties agreed that the settlement agreement would resolve “all outstanding disputes and grievances” between them, and that any payments that Dutra owed under the settlement agreement would be made by September 15,1999.

The IBU contends that shortly after the settlement agreement was concluded, Du-tra breached its third provision by once again subcontracting with Master’s, even though Master’s continued to use the services of workers who did not belong to the IBU to perform the sub-contracted work. Soon thereafter, the IBU filed this action under § 301 of the Labor Management Relations Act (“LMRA”), 29 U.S.C. § 185 (a), seeking enforcement of the set tlement agreement’s subcontracting provision, as well as damages for its breach.

Dutra asserts that because the dispute is governed by the arbitration clause of the CBA, and because the IBU failed to exhaust its non-judicial remedies, the district court lacked jurisdiction over the action. The district court granted Dutra’s motion to dismiss pursuant to Federal Rule of Civil Procedure 12(b)(1). The IBU appeals that decision. We review the grant of a motion to dismiss de novo. Mundy v. United States, 983 F.2d 950, 952 (9th Cir.1993).

II. DISCUSSION

This case presents the question of when disputes regarding the terms or provisions of side agreements not included in the parties’ basic written collective bargaining-agreement are covered by the arbitration clause contained in the CBA. The IBU contends that when a side agreement, such as the settlement agreement at issue here, does not expressly include a requirement for arbitration within its terms, the union should be able to pursue its rights under that agreement in federal court pursuant to § 301 of the LMRA. Dutra responds that in this case the dispute over the terms of the third provision of the settlement agreement is governed by the arbitration clause of the CBA, and that the union must accordingly exhaust its non-judicial remedies before proceeding to court.

A. ARBITRABILITY OF SIDE AGREEMENTS BY PARTIES TO A COLLECTIVE BARGAINING AGREEMENT

We start by recognizing two general principles of labor law. The first is that federal labor policy as declared by the Supreme Court provides a strong preference for the arbitration of labor-management disputes:

AT & T Technologies, Inc. v. Communications Workers of America, 475 U.S. 643, 650 , 106 S.Ct. 1415 , 89 L.Ed.2d 648 (1986) (quoting Steelworkers v. Warrior & Gulf Navigation Co., 363 U.S. 574, 582-83 , 80 S.Ct. 1347 , 4 L.Ed.2d 1409 (1960)). Where, as here, the arbitration provision of a CBA is reasonably broad in scope, the presumption of arbitration is “particularly potent.” Dennis L. Christensen Gen'l Bldg. Contractor v. Southern Calif. Conf. of Carpenters, 952 F.2d 1073, 1077 (9th Cir.1991). The burden thus falls upon the party contesting arbitrability to show how the language of the arbitration clause excludes a dispute from the clause’s purview. Phoenix Newspapers, Inc. v. Phoenix Mailers Union Local 752, 989 F.2d 1077, 1080 (9th Cir.1993).

The second general principle is that a collective bargaining agreement is not a narrow document limited to its specific terms and provisions; rather, it is a means of ordering more generally the labor-management relationship between the parties to it. As the Supreme Court has explained:

Warrior & Gulf Navigation Co., 363 U.S. at 578-80 , 80 S.Ct. 1347 (citations omitted). Thus, a collective bargaining agreement is not limited solely to the specific provisions of the basic labor contract formally executed by the parties, but it may also include, among other things, written side agreements and oral understandings entered into by the parties to the collective bargaining relationship, including settlement agreements.

In this case, the settlement agreement resolved a dispute that the parties agree was in the first instance arbitrable under the CBA. The question presented here, then, is whether the present dispute over the alleged breach of the settlement agreement falls within the CBA’s arbitration clause as well. We have not previously addressed the arbitrability of disputes regarding side agreements, such as settlement agreements, that are not set forth in the parties’ basic written collective bargaining agreement. Several other circuits, however, have spoken to the issue.

In determining whether a dispute arising under a side agreement must be arbitrated pursuant to an arbitration clause of a CBA, the Second and Fourth Circuits consider the similarity of the subject matter of the side agreement to that of the CBA. Where the subject matter is dissimilar, these circuits deem the side agreement “collateral” to the CBA and do not apply the CBA’s arbitration clause. Where the side agreement is “integral” to the CBA, these courts have held that a dispute over its provisions is arbitrable. See Cornell Univ. v. UAW Local 2300, 942 F.2d 138 , 140 (2d Cir.1991) (holding that a “letter of understanding” was “collateral” to the CBA and therefore not governed by the CBA arbitration clause); Adkins v. Times-World Corp., 771 F.2d 829, 830-31 (4th Cir.1985) (holding that an “addendum” to the CBA was integral to it and therefore subject to the arbitration clause).

Two other circuits, the Third and Seventh, have adopted a different approach. These circuits have held that unless the parties specify otherwise, disputes involving a side agreement are arbitrable if the subject matter of the side agreement is within the scope of the arbitration clause of the CBA. See Niro v. Fearn Int’l., Inc., 827 F.2d 173, 175 (7th Cir.1987) (holding that “a settlement agreement is an arbi-trable subject when the underlying dispute is arbitrable”); L.O. Koven & Bro., Inc. v. Local Union No. 5767, United Steelworkers of America, 381 F.2d 196, 204-05 (3rd Cir.1967) (concluding that a dispute over a side agreement that was silent as to arbitrability was governed by the CBA arbitration clause because the underlying subject was one “ordinarily a matter for consideration by an arbitrator” under the CBA.).

We essentially agree with the approach followed by the Third and Seventh Circuits. Whether a dispute arising under a side agreement is arbitrable depends on the scope of the arbitration clause in the CBA. We hold that disputes arising under a side agreement must be arbitrated if the dispute relates to a subject that is within the scope of the CBA’s arbitration clause. For example, if the arbitration clause in a CBA were even broader than the one at issue here, and covered “all disputes that may arise” between the parties, then any dispute over any matter, whether or not it relates to a side agreement, would unquestionably be arbitrable. In contrast, if the arbitration clause were far narrower and covered only, for example, disputes over discipline and discharge, then a dispute arising under a side agreement concerning the assignment of vacation days would not be arbitrable.

Here, the arbitration clause of the CBA is reasonably broad, and covers “[a]ny dispute concerning the interpretation or any other dispute between the parties hereto regarding wages, working conditions, or any other matters referred to in this Agreement....” The subject matter of the side agreement that is in dispute here^ — Dutra’s subcontracting practices — is a matter explicitly referred to in the written collective bargaining agreement, and thus clearly falls within the purview of the arbitration clause. The clause contains no exclusion for disputes arising under side agreements. Thus, the settlement agreement at issue in this case resolved a grievance that, under the provisions of the CBA, was required to be arbitrated under its arbitration clause.

The approach that we adopt is not only required by precedent, but establishes greater certainty in the bargaining process and thereby promotes the important goal of industrial peace. See Auciello Iron Works, Inc. v. NLRB, 517 U.S. 781, 785 , 116 S.Ct. 1754 , 135 L.Ed.2d 64 (1996) (“The object of the National Labor Relations Act is industrial peace and stability, fostered by collective-bargaining agreements providing for the orderly resolution of labor disputes between workers and employees.”). By avoiding the uncertainty inherent in the “collateral contract” analysis, we enable parties entering into side agreements to have a clearer understanding of whether disputes arising under those side agreements will be arbitrable. If the parties do not reach an explicit understanding regarding the arbitrability of disputes arising under a proposed side agreement, then they merely need examine the arbitration clause of the CBA which they have previously executed and determine whether it applies to the subject matter of the proposed agreement. They need do so, of course, in light of the presumption of arbitrability that we have discussed earlier, and that the Supreme Court has repeatedly emphasized. The general arbitration clause will apply to a dispute over a side agreement to the same extent that it would govern any other disagreement between the parties.

The IBU contends that our decision in Alpha Beta Co. v. Retail Store Employees Union, Local 428, 671 F.2d 1247 (9th Cir.1982), controls the outcome of this case. In Alpha Beta, the union and Alpha Beta, which were parties to a CBA, settled, by means of a written agreement, a dispute regarding the wrongful discharge of certain union members. When individual employees subsequently filed charges with the NLRB regarding those discharges, the employer requested the union to arbitrate in order to obtain a definitive interpretation of the CBA provision at issue in the settled dispute. The union rejected the request on the ground that it had no dispute with the employer over the meaning of the provision. We first ruled that no dispute existed between the union and the employer, and then, on that basis, held that arbitration was not required. See id., at 1250 . We suggested in dicta, however, that if a dispute actually had existed, then “Alpha Beta’s remedy would be to bring an action for breach of the settlement agreement, not to seek arbitration of the underlying dispute.” Id. at 1249 . That speculative observation does not control here; in Alpha Beta we did not consider and resolve the question of law cited by the IBU. See United States v. Johnson, 256 F.3d 895, 914-15 (9th Cir.2001) (en banc). The comment in Alpha Beta that the IBU asks us to treat as establishing the controlling law here was merely tangential to our holding that no actual dispute existed between the union and the employer; therefore, the remark does not serve as precedent or govern our analysis here. See Ruff v. Sullivan, 907 F.2d 915, 918 (9th Cir.1990) (“This panel is not bound by dicta from prior cases.”). Moreover, to the extent that the dicta in Alpha Beta suggests that as a general rule CBAs’ arbitration clauses do not apply to disputes arising under settlement agreements, we reject that suggestion.

The IBU also suggests that the result we reach will lead to a circular process in which a party will initiate arbitration, the dispute will be settled, one party will breach the settlement, and the other party will then be forced back into arbitration to resolve the dispute about the settlement. That may happen occasionally, as it did here. However, one could just as easily make the same argument about litigation and the settlement of lawsuits. Moreover, no party is compelled to settle an arbitra-ble dispute without obtaining an enforceable order. Here, for example, the union could have insisted that the settlement be issued in the form of an arbitration award. It simply failed to do so.

The IBU finally contends that even if the arbitration clause of the CBA would generally govern disputes arising under side agreements, the time elements relating to this case require that the union be allowed to seek enforcement of the agreement in a judicial forum directly, without first arbitrating the dispute. The expiration date of the 1994-99 CBA was June 30, 1999, and the succeeding CBA was not signed until September 17, 1999. Because the settlement agreement was concluded in August, 1999, in the interval between the expiration date of the 1994 CBA and the execution date of the 1999 agreement, the IBU argues that the arbitration clause of the 1999 CBA should not be construed to govern disputes arising under the side agreement. In support of this contention, the IBU points to the Second Circuit’s decision in Peerless Imps., Inc. v. Wine, Liquor & Distillery Workers Union, Local One, 908 F.2d 924 (2d Cir.1990).

The union’s argument is contradicted by the terms of the 1999 CBA. That collective bargaining agreement included a provision that set its effective date as July 1 of that year. The effective date clause thus unambiguously removed any real time gap that may have existed and brought disputes that arose during the interval between the expiration date of the 1994 CBA and the execution date of the succeeding CBA under the purview of the arbitration clause of the 1999 CBA. The IBU was well aware when it agreed to an effective date of July 1, 1999, that the settlement agreement had been concluded at a time subsequent to that date and thus at a time when the arbitration clause of the succeeding agreement would unquestionably be deemed to have been effective. In addition, the Peerless case actually supports the conclusion that the dispute before us is subject to the 1999 CBA’s arbitration clause, rather than the IBU’s contrary arguments. In Peerless, the court held that the arbitration clause there could “not be used to reach back to cover disputes arising before the agreement was executed, unless such preexisting disputes are brought within the scope of the clause.” Id. at 928 (emphasis added). Here, the effective date provision of the succeeding CBA served directly to bring disputes arising prior to the execution date within the scope of that clause.

B. SCOPE OF EYIDENCE IN A MOTION TO DISMISS

Dutra moved to dismiss prior to answering the IBU’s complaint, pursuant to Federal Rule of Civil Procedure 12. It is generally true that a trial court may not consider evidence outside the pleadings in connection with a motion to dismiss. See Hal Roach Studios, Inc. v. Richard Feiner & Co., 896 F.2d 1542 , 1555 n. 19 (9th Cir.1990). When a district court does consider such extraneous evidence, the general rule is that the motion is converted into a motion for summary judgment, and the non-moving party must be allowed to conduct discovery in order to oppose that motion. See Fed.R.Civ.P. 12(b), Branch v. Tunnell, 14 F.3d 449, 453-54 (9th Cir.1994).

Here, the district court stated that in deciding Dutra’s motion it considered the two documents cited in the complaint — the CBA and the settlement agreement — as well as the pleading itself. The IBU contends that because the court looked to materials beyond the complaint, it erred in not converting the motion to one for summary judgment, and in not affording the IBU a “reasonable opportunity to present all material made pertinent to such a motion by Rule 56.” Fed.R.Civ.P. 12(b).

The district court correctly ruled on Dutra’s motion as a rule 12 motion to dismiss. We have held that a failure to exhaust non-judicial remedies must be raised in a motion to dismiss, and should be treated as such even if raised as part of a motion for summary judgment. Ritza, 837 F.2d at 368. The district court also correctly looked to the settlement agreement and the CBA in ruling on the motion. This is so for two reasons. First, a district judge may generally consider a document outside the complaint when deciding a motion to dismiss if the complaint specifically refers to the document and if its authenticity is not questioned. Townsend v. Columbia Operations, 667 F.2d 844, 848-49 (9th Cir.1982). The complaint referred to both the settlement agreement and the CBA, and the IBU does not contest the authenticity of either document. Second, the motion raised a jurisdictional issue. We have held that when deciding a motion to dismiss for lack of jurisdiction, a district court “is free to hear evidence regarding jurisdiction and to resolve factual disputes in determining whether it has jurisdiction over a claim.” Steen v. John Hancock Mutual Life Ins. Co., 106 F.3d 904, 910 (9th Cir.1997); See also Ritza, 837 F.2d at 369 (stating that “the [district] court has a broad discretion as to the method to be used in resolving [a] factual dispute” over a matter in abatement.) (quotation omitted).

The IBU further contends that extrinsic evidence not taken into account by the district court supported the conclusion that the parties did not intend that the settlement agreement be covered by the arbitration clause. Specifically, the IBU argues that an earlier threat made by Dutra to sue the IBU for breach of the settlement agreement should have been considered by the district judge in determining whether the dispute over the settlement agreement is arbitrable. Because, as set forth above, the district court may consider evidence when resolving a question of jurisdiction on a motion to dismiss, the district court should have considered the IBU’s contention. However, the error was harmless. Dutra’s earlier threat of a lawsuit is, in itself, insufficient to overcome the clear language of the arbitration clause, which unquestionably covers the dispute at issue in this case. The statement by Dutra does not constitute either a waiver of its rights to arbitration under the CBA, or a repudiation of the CBA’s arbitration clause. See Bailey v. Bicknell Minerals, Inc. 819 F.2d 690, 692-93 (7th Cir.1987).

The IBU’s additional contention that it should be allowed discovery to demonstrate that it did not intend that disputes arising under the settlement be arbitrable is also without merit, as the IBU’s intent is irrelevant in light of the clear language of the arbitration clause, and the absence of any provision in the settlement agreement excluding its terms from the reach of that arbitration clause.

CONCLUSION

Because the dispute over the settlement agreement between Dutra and the IBU was arbitrable under the terms of the CBA between the parties, we AFFIRM the district court’s dismissal of the complaint on the ground that the IBU failed to exhaust its non-judicial remedies.

AFFIRMED.

. The expiration date of the CBA under which the initial dispute arose was June 30, 1999. The succeeding CBA was executed on September 17, 1999, but the effective date was made July 1, 1999. The grievance and arbitration procedures remained unchanged. . Although it does not affect the outcome of the case, we note that the district court incorrectly dismissed the case pursuant to Rule 12(b)(1), for lack of subject matter jurisdiction. We have held that a failure to exhaust non-judicial remedies is a "matter in abatement” that is "related” to our jurisdiction, but that such a failure does not fall within any of the categories of reasons for dismissal specified in Rule 12(b). Ritza v. Int’l Longshoremen’s and Warehousemen's Union, 837 F.2d 365, 368-69 (9th Cir.1988) (per curiam). Thus, we have concluded, a motion to dismiss for a failure to exhaust non-judicial remedies is properly considered a "non-enumerated” Rule 12(b) motion. Id. at 369 . . Section 301 provides that: [sjuits for violations of contracts between an employer and a labor organization representing employees in an industry affecting commerce as defined in this chapter, or between any such labor organizations, may be brought in any district court of the United States having jurisdiction of the parties, without respect to the amount in controversy or without regard to the citizenship of the parties. 29 U.S.C. § 185 (a). There is no dispute that if the arbitration clause does not govern this dispute, the federal courts have jurisdiction to hear the IBU's claims. Section 301 has been broadly construed to include not only CBAs, but also other "agreement[s] between employers and labor organizations significant to the maintenance of labor peace between them.” Retail Clerics Int’l. Ass’n, Local Unions Nos. 128 & 633 v. Lion Dry Goods, Inc., 369 U.S. 17, 28 , 82 S.Ct. 541 , 7 L.Ed.2d 503 (1962). . For the sake of simplicity, we will hereafter use the all-inclusive terms "settlement agreement" or "side agreement" when referring to the third provision of the settlement agreement, rather than identifying the third provision itself on each occasion. In doing so, however, we do not intend to express any view as to the arbitrability of disputes regarding other provisions of the agreement, which are not at issue in this litigation, although there appears to be little doubt as to their arbitrability, with the possible exception of disputes arising under the second provision. See n. 5, infra. . We exempt from our analysis a dispute over the terms to be included in a new CBA or over whether to extend an existing CBA, upon either the same or different terms. The arbitration of such disputes is known as "interest arbitration." Phoenix Newspapers, Inc. v. Phoenix Mailers Union, Local 752, 989 F.2d 1077, 1082 (9th Cir.1993). There.must be a clear agreement between the parties to impose interest arbitration. It is a comparatively rare practice in most industries, although it is more prevalent in the public sector, at least in some states. Michael Goldberg, "Law, Labor and the Mainstream Press,” 15 Labor Lawyer 93, 149 n. 196 (1999). Our decision in this case applies only to disputes subject to the "grievance arbitration” clause of a CBA, which typically covers "disputes regarding the interpretation of the parties' agreement,” Phoenix Newspapers, 989 F.2d at 1082 , or arising under it. . It is possible, of course, for parties to a CBA to exclude disputes arising under a side agreement from the arbitration clause of the CBA not only by a clear statement to that effect in the arbitration clause itself, but by including a specific provision in the side agreement. Niro, 827 F.2d at 175 ("[i]f parties desire that a settlement agreement should not be arbitrable they may so prescribe.”). As we stated above, the arbitration clause of the CBA contains no such exclusion. Nor, in the case before us, does the side agreement. Thus, the IBU's allegation that Dutra has breached the settlement agreement is arbitra-ble. . Because a principal goal of the NLRA is to resolve labor disputes efficiently, the establishment of clear rules that decrease uncertainty in the resolution of disagreements is particularly appropriate. See Auciello, 517 U.S. at 785-86 , 116 S.Ct. 1754 (observing that the NLRB has adopted "various presumptions” to facilitate the "orderly resolution of labor disputes.”). . The union termed the process "Kafka-es-que” in its argument to the district court. The fallacy in the union's position may be seen by positing a settlement of a grievance at an earlier stage of the grievance process. In such a case, where the formal arbitration proceedings had not yet been commenced, a dispute over an alleged violation of the settlement agreement would clearly be arbitrable. The invocation of the formal arbitration mechanisms prior to the settlement does not change this result. We also note that even if an arbitration award had been issued following a full arbitration proceeding, it is possible that a dispute over the meaning and application of that award would be arbitrable. While we need not decide that question here, we observe that an outright refusal by a party to comply with an award warrants the filing of an action to compel. International Union of Petroleum & Indus. Workers v. Western Indus. Maint., 707 F.2d 425, 428 (9th Cir.1983). In contrast, if several years after an award is issued a disagreement arises between the parties as to, for example, whether a particular type of work is covered by the terms of an award governing sub-contracting, generally that dispute, like other disputes over the meaning of the CBA, will be subject to arbitration. For, arbitration awards themselves become part of the "collective bargaining agreement," in the broadest sense of that term, Drummond Coal Co. v. United Mine Workers, 748 F.2d 1495, 1497 & n. 3 (11th Cir.1984), and thus are an important element of the "common law of the shop” which governs the continuous collective bargaining relationship. Hawaii Teamsters and Allied Workers Union, Local 996 v. United Parcel Serv., 241 F.3d 1177, 1183-84 (9th Cir.2001). . Although we use the term "succeeding" agreement here, we could as easily refer to the 1999 CBA as a "new,” "amended,” "revised,” or "successor” CBA. The terminology is unimportant, as all such terms would have the same meaning and legal consequence. . We need not decide more generally when a dispute over events that occur during an interval between the expiration date of an old contract and the execution date of its successor are subject to arbitration. The answer may depend on a number of circumstances that it is unnecessary for us to consider here. In any event, by referring to Peerless as we do, we do not intend to adopt its implication that where such a gap exists, some specific action must be taken in order to overcome a "reach-back” problem.