CONVERTIBLE PROMISSORY NOTE

EXHIBIT 2.3

CONVERTIBLE PROMISSORY NOTE

FOR VALUE RECEIVED, PFF BANCORP, INC., a Delaware corporation located at 9337 Milliken Avenue, Rancho Cucamonga, California 91729 (“Maker”) hereby promises to pay to the order of FBOP Corporation, an Illinois corporation (“Payee”), located at Oak Park, Illinois, the principal sum of Seven Million and no/100 Dollars ($7,000,000.00) together with interest as provided herein on all amounts outstanding hereunder, in lawful money of the United States of America.

The principal amount outstanding under this Promissory Note shall bear interest at a rate (the “Interest Rate”) equal to One Month LIBOR (as defined below) in effect from time to time plus three and one-half percent per annum. As used herein, One Month LIBOR shall mean the independent index which is the British Bankers Association (BBA) LIBOR for a one-month period as reported by a major news service selected by Payee (such as Reuters, Bloomberg or Moneyline Telerate), as in effect as of the date of this Promissory Note and as of the first day of each calendar month within the term of this Promissory Note thereafter. The Interest Rate is subject to change from time to time based on changes in One Month LIBOR. If One Month LIBOR for a particular period is not provided or reported on the first day of a month because, for example, it is a weekend or holiday or for another reason, the One Month LIBOR shall be established as of the preceding day on which One Month LIBOR is provided for such period and reported by the selected news service. All interest rate changes will become effective without notice to Maker. If One Month LIBOR becomes unavailable during the term of this Promissory Note, Payee may substitute a comparable index in its sole discretion. The interest payments for this Note shall be computed on a 365/360 basis; that is, by applying the ratio of the annual interest rate over a year of 360 days, multiplied by the outstanding principal balance, multiplied by the actual number of days the principal balance is outstanding.

Accrued interest shall be payable by Maker to Payee on the same day of each month within the term hereof commencing on September 30, 2008 and continuing on the last day of each month thereafter and at maturity. Unless earlier paid, all outstanding principal, accrued and unpaid interest and other sums payable under the terms of this Promissory Note shall be due and payable without demand on the Maturity Date. As used herein, the “Maturity Date” shall mean June 12, 2009, subject to the acceleration provisions hereof. All payments of principal and interest hereunder shall be made to Payee at its address set forth above, or such other place as the legal holder hereof may from time to time in writing appoint by notification to Maker.

Without limiting the other provisions of this Promissory Note, Payee shall have the right, at its sole discretion, at any time upon written notice to Maker, and subject to applicable regulatory approvals, to convert all or a

portion of this Promissory Note into shares of Preferred Stock of Maker representing 19.9% (or, if less than all of this Promissory Note is converted, the appropriate proportionate share) of the aggregate voting power of Maker’s outstanding capital stock, and containing other customary terms and provisions for securities of this nature. Maker covenants and agrees to use its commercially reasonable best efforts to issue such Preferred Stock within 7 days of receipt of a conversion notice from FBOP.

Maker represents and warrants to the Payee that the indebtedness represented by this Promissory Note is a business loan and that the proceeds of this Promissory Note will be used solely for the business purposes described herein. Maker covenants and agrees that the proceeds of the indebtedness represented by this Promissory Note will be used solely as follows: to make on the date of the funding of this Promissory Note a capital contribution in the amount of $7,000,000 to the capital of PFF Bank & Trust, a federal savings institution and a wholly owned subsidiary of Maker (the “Bank”).

This Promissory Note is secured, inter alia, pursuant to the terms of a certain Pledge Agreement dated as of even date herewith by Maker in favor of Payee (such Pledge Agreement, as the same may be amended or modified from time to time, the “Pledge Agreement”).

Maker shall have the right upon two (2) business days’ advance written notice to prepay without penalty any sum due under this Promissory Note, and any such partial prepayment shall be applied first to accrued but unpaid interest and charges, and then to principal.

Notwithstanding anything to the contrary herein, in the event that the Agreement and Plan of Merger of even date herewith among Maker, Payee and California Madison Holdings, Inc. (the “Merger Agreement”) is terminated due to a “Superior Proposal” (as defined in the Merger Agreement), (i) all outstanding principal, accrued and unpaid interest and other sums outstanding under the terms of this Promissory Note shall, at the option of Payee, become immediately due and payable without demand, presentment or notice of any kind, and (ii) in addition to the amounts otherwise payable under the terms of this Promissory Note, Maker shall also immediately pay to Payee the sum of Two Hundred Fifty Thousand Dollars ($250,000).

The occurrence of any of the following events shall constitute an “Event of Default” hereunder: (i) if Maker fails to pay any of the indebtedness evidenced by this Promissory Note when due by maturity, acceleration or otherwise, or fails in any other respect to comply with any other of its obligations set forth in this Promissory Note, (ii) if Maker fails in any respect to comply with the terms of the Pledge Agreement, or if the Pledge Agreement ceases or fails to be in full force or effect, (iii) if any warranty or representation made by Maker in the Pledge Agreement shall be discovered to be untrue in any material respect, (iv) if Maker makes an assignment for the benefit of creditors, files a petition in bankruptcy, is unable generally to pay its debts as they come due, is adjudicated insolvent or bankrupt or there is entered any order or decree granting relief in any involuntary case commenced against Maker under any applicable bankruptcy, insolvency or other similar law now or hereafter in effect, (v) if Maker petitions or applies to any tribunal for any receiver, trustee, liquidator, assignee,

2

custodian, sequestrator or other similar official for Maker or of any substantial part of the assets and properties thereof, or commences any proceeding in a court of law for a reorganization, readjustment of debt, dissolution, liquidation or other similar procedure under the law or statutes of any jurisdiction, whether now or hereafter in effect, (vi) if there is commenced against Maker any involuntary case under any applicable bankruptcy, insolvency or other similar law now or hereafter in effect or any proceeding in a court of law for a reorganization, readjustment of debt, dissolution, liquidation or other similar procedure, which remains undismissed or not be discharged, vacated or stayed, within thirty (30) days after commencement, or Maker by any act, indicates its consent to, approval of, or acquiescence in any such proceeding in a court of law, (vii) if there is filed or issued a levy or writ of attachment or garnishment or other like judicial process upon Maker or any of the collateral securing this indebtedness, or (viii) if Maker ceases doing business as a going concern.

Upon the occurrence of any of any Event of Default, Payee may at its option and without prior notice to Maker or any other person or entity, declare any or all of the indebtedness evidenced hereby to be immediately due and payable, set off against the indebtedness any amounts owing by Payee or any of its affiliates to Maker or any of its affiliates, and exercise any one or more of the rights and remedies granted to Payee by any agreement with Maker or given to it under applicable law. The remedies of Payee, as provided herein, shall be cumulative and concurrent, and may be pursued singularly, successively or together, at the sole discretion of Payee, and may be exercised as often as occasion therefore shall arise.

After the Maturity Date, or the earlier acceleration of the indebtedness evidenced by this Promissory Note, Maker shall pay interest on the balance of principal remaining unpaid during any such period at an annual rate equal to two percent (2%) plus the Interest Rate then in effect under this Promissory Note (the “Default Rate”). The interest accruing under this paragraph shall be immediately due and payable by Maker to the holder of this Note and shall be additional indebtedness evidenced by this Promissory Note.

If this Promissory Note is past due and is placed in the hands of an attorney for collection or enforcement or is collected or enforced through any legal proceedings, then Maker shall pay to Payee all attorneys’ fees, costs and expenses incurred in connection therewith in addition to all other amounts due hereunder.

Whenever under the terms of this Promissory Note the time for any payment due falls on a Saturday, Sunday or holiday, the time for such payment shall be extended to the next business day.

Notwithstanding any provisions of this Promissory Note or any instrument securing payment of the indebtedness evidenced by this Promissory Note to the contrary, it is the intent of Payee and Maker that Payee shall never be entitled to receive, collect or apply, as interest on principal of the indebtedness, any amount in excess of the maximum rate of interest permitted to be charged by applicable law; and if under any circumstance whatsoever, fulfillment of any provision of this Promissory Note, at the time performance of such provision shall be due, shall involve transcending the limit of validity prescribed by applicable law, then, ipso facto, the obligation to be fulfilled shall be reduced to the limit of such validity; and in the event Payee

3

ever receives, collects or applies as interest any such excess, such amount which would be excess interest shall be deemed a permitted partial prepayment of principal and treated hereunder as such; and if the principal of the indebtedness secured hereby is paid in full, any remaining excess funds shall forthwith be paid to Maker.

Maker and any and all others who are now or may become liable for all or any part of the obligations of Maker under this Promissory Note, including any guarantor hereof, agree to (i) waive presentment and demand for payment, notices of nonpayment and dishonor, protest of dishonor and notice of protest; (ii) waive all notices in connection with the delivery and acceptance hereof and all other notices in connection with the performance, default or enforcement of the payment hereof or hereunder; (iii) waive any and all lack of diligence and delays in enforcement of the payment hereof; (iv) agree that the liability of each such person or entity shall be unconditional and without regard to the liability of any other person or entity for the payment hereof and shall not in any manner be affected by any indulgence or forbearance granted or consented to by the Payee to any of them with respect hereto; (v) consent to any and all extensions of time, renewals, waivers or modifications which may be granted by Payee with respect to the payment or other provisions hereof, and to the release of any security at any time given for the payment hereof or any part hereof, with or without substitution, and to the release of any person or entity liable for the payment hereof; and (vi) consent to the addition of any and all other makers, endorsers, guarantors and other obligors for the payment hereof, and to the acceptance of any and all security for the payment hereof.

This Promissory Note shall be binding upon Maker and its successors and permitted assigns, and shall inure to the benefit of Payee and its successors and assigns. Maker may not assign its rights hereunder without the prior written consent of Payee, in its sole discretion. Payee may assign all or a part of its interest in this Promissory Note and its rights hereunder; provided, that so long as no Event of Default has occurred and the Merger Agreement remains in effect, Payee shall not assign all or a portion of this Promissory Note without the consent of the Maker, not to be unreasonably withheld.

If any provision of this Promissory Note or any payments pursuant to the terms hereof shall be invalid or unenforceable to any extent, the remainder of this Promissory Note and any other payments hereunder shall not be affected thereby and shall be enforceable to the greatest extent permitted by law.

This Promissory Note may not be amended, modified or changed, nor shall any waiver of any of the provisions hereof be effective, except only by an instrument in writing and signed by the party against whom enforcement of any waiver, amendment, change, modification or discharge is sought. Maker acknowledges and agrees that this Promissory Note is the entire agreement with respect to the subject matter hereof (other than the Pledge Agreement) and that there are no contrary agreements, oral or written, establishing a term of this Promissory Note

Maker agrees that this Promissory Note and the rights and obligations of all parties hereunder shall be governed by and construed under the substantive laws of the State of Illinois, without reference to the conflict of laws principles of such state.

4

ANY LITIGATION BASED HEREON OR ARISING OUT OF OR IN CONNECTION WITH THIS PROMISSORY NOTE OR ANY OTHER DOCUMENTS OR AGREEMENTS ENTERED INTO IN CONNECTION HEREWITH SHALL BE BROUGHT AND MAINTAINED EXCLUSIVELY IN THE COURTS OF THE STATE OF ILLINOIS OR IN THE UNITED STATES DISTRICT COURT FOR THE NORTHERN DISTRICT OF ILLINOIS. MAKER HEREBY EXPRESSLY AND IRREVOCABLY SUBMITS TO THE JURISDICTION OF THE COURTS OF THE STATE OF ILLINOIS AND OF THE UNITED STATES DISTRICT COURT FOR THE NORTHERN DISTRICT OF ILLINOIS FOR THE PURPOSE OF ANY SUCH LITIGATION. MAKER FURTHER IRREVOCABLY CONSENTS TO THE SERVICE OF PROCESS BY REGISTERED MAIL AT THE ADDRESS OF MAKER SET FORTH BELOW AND SERVICE SO MADE SHALL BE DEEMED TO BE COMPLETED THREE (3) BUSINESS DAYS AFTER DEPOSIT IN THE U.S. MAILS, POSTAGE PREPAID. MAKER HEREBY EXPRESSLY AND IRREVOCABLY WAIVES, TO THE FULLEST EXTENT PERMITTED BY LAW, ANY OBJECTION TO THE LAYING OF VENUE OF ANY SUCH LITIGATION BROUGHT IN ANY SUCH COURT AND ANY CLAIM THAT ANY SUCH LITIGATION HAS BEEN BROUGHT IN AN INCONVENIENT FORUM.

EACH OF MAKER AND PAYEE HEREBY KNOWINGLY, VOLUNTARILY AND INTENTIONALLY WAIVE ANY RIGHT TO A TRIAL BY JURY IN RESPECT OF ANY LITIGATION BASED HEREON, OR ARISING OUT OF, UNDER OR IN CONNECTION WITH, THIS PROMISSORY NOTE

IN WITNESS WHEREOF, Maker has duly executed and delivered this Note as of the date and year first above written.

5