PLEDGE AGREEMENT

EXHIBIT 2.4

PLEDGE AGREEMENT

THIS PLEDGE AGREEMENT (this “Agreement”), dated as of June 13, 2008, is made by PFF Bancorp, Inc., having an office at 9337 Milliken Avenue, Rancho Cucamonga, California 91729 (“Pledgor”) in favor of and to the benefit of FBOP Corporation, an Illinois corporation (“Pledgee”).

R E C I T A L S

WHEREAS, Pledgor is the record and beneficial owner of the shares of stock described in Schedule I hereto (the “Pledged Shares”) issued by Diversified Builder Services, Inc., a California corporation and Glencrest Investment Advisors, Inc., a Delaware corporation (collectively, the “Companies”); and

WHEREAS, Pledgor has requested that the Pledgee provide a loan to Pledgor in the principal amount of Seven Million and no/100 Dollars ($7,000,000) to pursuant to the terms of a Promissory Note dated as of even date herewith by Pledgor to the order of Pledgee (such Promissory Note, as the same may be modified, amended, replaced or supplemented from time to time, the “Promissory Note”); and

WHEREAS, in connection with the making of the loan evidenced by the Promissory Note and as a condition thereto, the Lender is requiring that Pledgor shall have executed and delivered this Pledge Agreement and granted the security interest contemplated hereby.

NOW, THEREFORE, in consideration of the premises and the covenants hereinafter contained and to induce the Lender to make the loan evidenced by the Promissory Note, Pledgor hereby agrees as follows:

1. Pledge. Pledgor hereby pledges and grants to the Lender a first priority security interest in the Pledged Shares and the certificates representing the Pledged Shares (the “Pledged Collateral”) as security for the prompt payment in full when due, whether at stated maturity, by acceleration or otherwise, of all indebtedness and obligations of Pledgor now or hereafter existing under the Promissory Note and all obligations of Pledgor now or hereafter existing under this Pledge Agreement (collectively, the “Secured Obligations”).

2. Delivery of Pledged Collateral. Upon the execution and delivery of this Agreement and as a condition to the making of the loan evidenced by the Promissory Note,

Pledgor shall deliver to Pledgee all certificates representing or evidencing the Pledged Collateral as of the date of this Pledge Agreement to and held by or on behalf of Pledgee pursuant hereto and accompanied by duly executed instruments of transfer or assignment in blank, all in form and substance satisfactory to Pledgee. Upon the issuance of any additional certificates or instruments representing or evidencing all or any additional part of the Pledged Collateral after the date hereof, Pledgor shall promptly but in any event within one (1) business day of the issuance thereof deliver the same to Pledgee together with duly executed instruments of transfer or assignment in blank, all in form and substance satisfactory to Pledgee.

3. Further Documentation. At any time and from time to time and at the sole expense of Pledgor, Pledgor will promptly and duly execute and deliver any and all further instruments and documents and take such further action as Pledgee may reasonably request to obtain the full benefits of this Pledge Agreement.

4. Representations and Warranties. Pledgor hereby represents and warrants to Pledgee that:

(a) Pledgor is, and at all times during the terms of this Agreement, will be, the sole holder of record and the sole beneficial owner of its Pledged Collateral, free and clear of any lien, encumbrance or claim of third party thereon or thereto or affecting the title thereto except for the lien created by this Agreement.

(b) Pledgor has the right and requisite capacity or authority to pledge, assign, transfer, deliver, deposit and set over the Pledged Collateral to Pledgee as provided herein.

(c) None of the Pledged Collateral has been transferred in violation of the securities registration, securities disclosure or similar laws of any jurisdiction to which such transfer may be subject.

(d) No consent, approval, authorization or other order of any person or entity and no consent, authorization, approval or other action by, and no notice to or filing with, any governmental authority or regulatory body is required to be made or obtained by any Pledgor either (i) for the pledge of the Pledged Collateral pursuant to this Agreement or for the execution, delivery or performance of this Agreement by Pledgor or (ii) for the exercise by Pledgee of the voting or other rights provided for in this Agreement or the remedies in respect of the Pledged Collateral pursuant to this Agreement, except as may be required in connection with such disposition by laws affecting the offering and sale of securities generally.

(e) The Pledged Collateral constitutes one hundred percent (100%) of the issued and outstanding capital stock of each of the Companies, and there are no outstanding options, warrants or other undertakings or commitments therefor, issued or outstanding.

(f) The pledge, assignment and delivery of the Pledged Collateral pursuant to this Agreement will create a valid first priority lien and encumbrance on and a first priority perfected security interest in the Pledged Collateral, and the proceeds thereof, securing the payment of the Secured Obligations.

2

(g) This Agreement has been duly executed and delivered by the Pledgor and constitutes a legal, valid and binding obligation of the Pledgor enforceable in accordance with its terms.

The representations and warranties set forth in this Section 4 shall survive the execution and delivery of this Agreement.

5. Covenants. Pledgor covenants and agrees that:

(a) Without the prior written consent of Pledgee, Pledgor will not (i) sell, assign, transfer, pledge or otherwise encumber any of its rights in or to the Pledged Collateral or any unpaid dividends or other distributions or payments with respect thereto or grant a lien or security interest in any thereof, or (ii) amend, supplement, modify or terminate the charter, bylaws or other constituent document of either of the Companies, or cause or suffer to permit the issuance of any additional shares of the capital stock of either of the Companies, or any options, warrants or commitments therefore, or any additional certificates or instruments of any kind representing or evidencing the Pledged Collateral or any portion thereof.

(b) Pledgor has and will defend the title to the Pledged Collateral and the liens of Pledgee thereon against the claim of any person or entity and will maintain and preserve such liens until the Secured Obligations are repaid in full.

(c) All distributions in respect of any of the Pledged Collateral shall, if received by any Pledgor, be received in trust for the benefit of Pledgee, be segregated from the other property or funds of such Pledgor, and be forthwith delivered to Pledgee as Pledged Collateral in the same form as so received (with any necessary indorsement).

6. Pledgor’s Right to Vote. As long as no Event of Default shall have occurred, Pledgor shall have the right, from time to time, to vote and give consents with respect to the Pledged Collateral or any part thereof for all purposes not inconsistent with the provisions of this Agreement; provided, however, that no vote shall be cast, and no consent shall be given or action taken, which would have the effect of impairing the position or interest of Pledgee in respect of the Pledged Collateral or which would authorize or effect (i) the dissolution or liquidation, in whole or in part, of any of the Companies, (ii) the consolidation or merger of the Companies with any other person or entity, (iii) the sale, disposition or encumbrance of all or substantially all of the assets of the Companies, (iv) any change in the authorized number of shares, the stated capital or the authorized capital of the Companies or the issuance of any additional capital stock or other ownership or debt instruments of the Companies, or (v) the alteration of the voting rights with respect to the capital stock of the Companies.

7. Defaults and Remedies.

(a) Upon the occurrence of an Event of Default (as such term is defined in the Promissory Note), Pledgee (personally or through an agent) is hereby authorized and empowered to transfer and register in its name or in the name of its nominee the whole or any part of the Pledged Collateral, to exchange certificates or instruments representing or evidencing Pledged

3

Collateral for certificates or instruments of smaller or larger denominations, to exercise the voting rights with respect thereto, to collect and receive all cash dividends and other distributions made thereon, to sell in one or more sales after ten (10) days’ notice of the time and place of any public sale or of the time after which a private sale is to take place (which notice Pledgor agree is commercially reasonable), but without any previous notice or advertisement, the whole or any part of the Pledged Collateral and to otherwise act with respect to the Pledged Collateral as though Pledgee was the outright owner thereof, Pledgor hereby irrevocably constituting and appointing Pledgee as the proxy and attorney-in-fact of Pledgor, with full power of substitution to do so; provided, however, Pledgee shall not have any duty to exercise any such right or to preserve the same and shall not be liable for any failure to do so or for any delay in doing so. Any sale shall be made at a public or private sale at Pledgee’s place of business, or at any public building in the City of Chicago or elsewhere to be named in the notice of sale, either for cash or upon credit or for future delivery at such price as Pledgee may deem fair, and Pledgee may be the purchaser of the whole or any part of the Pledged Collateral so sold and hold the same thereafter in its own right free from any claim of Pledgor or any right of redemption. Each sale shall be made to the highest bidder, but Pledgee reserves the right to reject any and all bids at such sale which, in its discretion, it shall deem inadequate. Demands of performance, except as otherwise herein specifically provided for, notices of sale, advertisements and the presence of property at sale are hereby waived and any sale hereunder may be conducted by an auctioneer or any officer or agent of Pledgee.

(b) If, at the original time or times appointed for the sale of the whole or any part of the Pledged Collateral, the highest bid, if there be but one sale, shall be inadequate to discharge in full all the Secured Obligations, or if the Pledged Collateral be offered for sale in lots, if at any of such sales, the highest bid for the lot offered for sale would indicate to Pledgee, in its discretion, the unlikelihood of the proceeds of the sales of the whole of the Pledged Collateral being sufficient to discharge all the Secured Obligations, Pledgee may, on one or more occasions and in its discretion, postpone any of said sales by public announcement at the time of sale or the time of previous postponement of sale, and no other notice of such postponement or postponements of sale need be given, any other notice being hereby waived; provided, however, that any sale or sales made after such postponement shall be after ten (10) days’ notice to Pledgor.

(c) In the event of any sales hereunder, Pledgee shall, after deducting all costs or expenses of every kind (including reasonable attorneys’ fees and disbursements) for care, safekeeping, collection, sale, delivery or otherwise, apply the residue of the proceeds of the sales to the payment or reduction, either in whole or in part, of the Secured Obligations in accordance with the agreements and instruments governing and evidencing the Secured Obligations, returning the surplus, if any, to Pledgor.

(d) Pledgor agrees that monetary damages would not be adequate compensation for any loss incurred by reason of a breach by it of the provisions of this Agreement and hereby agree to waive the defense in any action for specific performance that a remedy at law would be adequate.

4

(e) If, at any time when Pledgee shall determine to exercise its right to sell the whole or any part of the Pledged Collateral hereunder, such Pledged Collateral or the part thereof to be sold shall not, for any reason whatsoever, be effectively registered under the Securities Act of 1933, as amended (the “Act”), Pledgee may, in its discretion (subject only to applicable requirements of law), sell such Pledged Collateral or part thereof by private sale in such manner and under such circumstances as Pledgee may deem necessary or advisable, but subject to the other requirements of this Section 7, and shall not be required to effect such registration or to cause the same to be effected. Without limiting the generality of the foregoing, in any such event Pledgee in its discretion (a) may, in accordance with applicable securities laws, proceed to make such private sale notwithstanding that a registration statement for the purpose of registering such Pledged Collateral or part thereof could be or shall have been filed under the Act (or similar statute), (b) may approach and negotiate with a single possible purchaser to effect such sale, and (c) may restrict such sale to a purchaser who will represent and agree that such purchaser is purchasing for its own account, for investment and not with a view to the distribution or sale of such Pledged Collateral or part thereof. In addition to a private sale as provided above in this Section 7, if any of the Pledged Collateral shall not be freely distributable to the public without registration under the Act (or similar statute) at the time of any proposed sale pursuant to this Section 7, then Pledgee shall not be required to effect such registration or cause the same to be effected but, in its discretion (subject only to applicable requirements of law), may require that any sale hereunder (including a sale at auction) be conducted subject to restrictions (i) as to the financial sophistication and ability of any person or entity permitted to bid or purchase at any such sale, (ii) as to the content of legends to be placed upon any certificates representing the Pledged Collateral sold in such sale, including restrictions on future transfer thereof, (iii) as to the representations required to be made by each person or entity bidding or purchasing at such sale relating to that person’s or entity’s access to financial information about the Companies and such person’s or entity’s intentions as to the holding of the Pledged Collateral so sold for investment, for its own account, and not with a view to the distribution thereof, and (iv) as to such other matters as Pledgee may, in its discretion, deem necessary or appropriate in order that such sale (notwithstanding any failure so to register) may be effected in compliance with the laws affecting the enforcement of creditors’ rights and the Act and all applicable state securities laws.

(f) Pledgor acknowledges that notwithstanding the legal availability of a private sale or a sale subject to the restrictions described above in paragraph (e), Pledgee may, in its discretion, elect to register any or all the Pledged Collateral under the Act (or any applicable state securities law) in accordance with its rights hereunder. Pledgor, however, recognizes that Pledgee may be unable to effect a public sale of any or all the Pledged Collateral and may be compelled to resort to one or more private sales thereof. Pledgor also acknowledges that any such private sale (conducted in a commercially reasonable manner for private sales) may result in prices and other terms less favorable to the seller than if such sale were a public sale and, notwithstanding such circumstances, agrees that any such private sale shall be deemed to have been made in a commercially reasonable manner. Pledgee shall be under no obligation to delay a sale of any of the Pledged Collateral for the period of time necessary to permit the registrant to register such securities for public sale under the Act, or under applicable state securities laws, even if the issuer of the Pledged Collateral or Pledgor would agree to do so.

5

(g) Pledgor agrees that following the occurrence of an Event of Default, Pledgor will not at any time plead, claim or take the benefit of any appraisal, valuation, stay, extension, moratorium or redemption law now or hereafter in force in order to prevent or delay the enforcement of this Agreement, or the absolute sale of the whole or any part of the Pledged Collateral or the possession thereof by any purchaser at any sale hereunder, and each Pledgor waives the benefit of all such laws to the extent it lawfully may do so. Pledgor agrees that it will not interfere with any right, power and remedy of Pledgee provided for in this Agreement or now or hereafter existing at law or in equity or by statute or otherwise, or the exercise or beginning of the exercise by Pledgee of any one or more of such rights, powers or remedies. No failure or delay on the part of Pledgee to exercise any such right, power or remedy and no notice or demand which may be given to or made upon Pledgor by Pledgee with respect to any such remedies shall operate as a waiver thereof, or limit or impair Pledgee’s right to take any action or to exercise any power or remedy hereunder, without notice or demand, or prejudice its rights as against Pledgor in any respect.

(h) Pledgor further agrees that a breach of any of the covenants contained in this Section 7 will cause irreparable injury to Pledgee, that Pledgee has no adequate remedy at law in respect of such breach and, as a consequence, agrees that each and every covenant contained in this Section 7 shall be specifically enforceable against Pledgor, and Pledgor hereby waives and agrees not to assert any defenses against an action for specific performance of such covenants except for a defense that the Secured Obligations are not then due and payable in accordance with the agreements and instruments governing and evidencing such obligations.

8. Application of Proceeds. Any cash held by Pledgee as Pledged Collateral and all cash proceeds received by Pledgee in respect of any sale of, liquidation of, or other realization upon all or any part of the Pledged Collateral shall be applied by Pledgee as follows:

(a) First, to the payment of the costs and expenses of such sale, including reasonable compensation to Pledgee and its agent and counsel, and all expenses, liabilities and advances made or incurred by Pledgee in connection therewith;

(b) Next, to the payment of the Secured Obligations; and

(c) Finally, after payment in full of all Secured Obligations, to the payment to Pledgor, or its successors or assigns, or to whomsoever may be lawfully entitled to receive the same or as a court of competent jurisdiction may direct, of any surplus then remaining from such proceeds.

9. No Waiver. No delay on Pledgee’s part in exercising any power of sale, lien, option or other right hereunder, and no notice or demand which may be given to or made upon Pledgor by Pledgee with respect to any power of sale, lien, option or other right hereunder, shall constitute a waiver thereof, or limit or impair Pledgee’s right to take any action or to exercise any power of sale, lien, option, or any other right hereunder, without notice or demand, or prejudice Pledgee’s rights as against Pledgor in any respect.

6

10. Assignment. Pledgee may assign, indorse or transfer any instrument evidencing all or any part of the Secured Obligations as provided in the Promissory Note, and the holder of such instrument shall be entitled to the benefits of this Agreement.

11. Lien Absolute. All rights of Pledgee hereunder, and all obligations of Pledgor hereunder, shall be absolute and unconditional irrespective of:

(a) any amendment, modification, supplement or restatement of the Promissory Note or any document, instrument or agreement executed or delivered in connection therewith;

(b) any change in the time, manner or place of payment of, or in any other term of, all or any part of the Secured Obligations, or any other amendment or waiver of or any consent to any departure from the Promissory Note or any other agreement or instrument governing or evidencing any Secured Obligations; or

(c) any exchange, release or non-perfection of any other collateral, or any release or amendment or waiver of or consent to departure from any guaranty, for all or any of the Secured Obligations.

12. No Impairment. Pledgor consents and agrees that Pledgee may at any time, or from time to time, in its discretion (a) renew, extend or change the time of payment, and/or the manner, place or terms of payment of all or any part of the Secured Obligations and (b) exchange, release and/or surrender all or any of the Pledged Collateral, or any part(s) thereof, by whomsoever deposited, which is now or may hereafter be held by Pledgee in connection with all or any of the Secured Obligations; all in such manner and upon such terms as Pledgee may deem proper, and without notice to or further assent from such Pledgor, it being hereby agreed that such Pledgor shall be and remain bound upon this Agreement, irrespective of the existence, value or condition of any of the Pledged Collateral, and notwithstanding any such change, exchange, settlement, compromise, surrender, release, renewal or extension, and notwithstanding also that the Secured Obligations may, at any time exceed the aggregate principal amount thereof set forth in the Promissory Note or any other agreement governing any Secured Obligations. Pledgor hereby waives notice of acceptance of this Agreement, and also presentment, demand, protest and notice of dishonor of any and all of the Secured Obligations, and promptness in commencing suit against any party hereto or liable hereon, and in giving any notice to or of making any claim or demand hereunder upon Pledgor. No act or omission of any kind on Pledgee’s part shall in any event affect or impair this Agreement.

13. Reinstatement. This Agreement shall remain in full force and effect and continue to be effective should any petition be filed by or against Pledgor for liquidation or reorganization, should Pledgor become insolvent or make an assignment for the benefit of creditors or should a receiver or trustee be appointed for all or any significant part of any Pledgor’s assets, and shall continue to be effective or be reinstated, as the case may be, if at any time payment and performance of the Secured Obligations, or any part thereof, is, pursuant to applicable law, rescinded or reduced in amount, or must otherwise be restored or returned by any obligee of the Secured Obligations, whether as a “voidable preference,” “fraudulent

7

conveyance,” or otherwise, all as though such payment or performance had not been made. In the event that any payment, or any part thereof, is rescinded, reduced, restored or returned, the Secured Obligations shall be reinstated and deemed reduced only by such amount paid and not so rescinded, reduced, restored or returned.

14. Miscellaneous.

(a) Pledgee may execute any of its duties hereunder by or through agents or employees and shall be entitled to advice of counsel concerning all matters pertaining to its duties hereunder.

(b) Neither Pledgee nor any of its officers, directors, employees, agents or counsel shall be liable for any action lawfully taken or omitted to be taken by it or them hereunder or in connection herewith, except for its or their own gross negligence or willful misconduct.

(c) This Agreement shall be binding upon Pledgor and its successors and assigns, and shall inure to the benefit of, and be enforceable by, Pledgee and its successors and assigns, and shall be governed by, and construed and enforced in accordance with, the internal laws in effect in the State of Illinois without giving effect to principles of choice of law, and none of the terms or provisions of this Agreement may be waived, altered, modified or amended except in writing duly signed for and on behalf of Pledgee and Pledgor.

15. Severability. If for any reason any provision or provisions hereof are determined to be invalid and contrary to any existing or future law, such invalidity shall not impair the operation of or effect those portions of this Agreement which are valid.

16. Notices. Except as otherwise provided herein, whenever it is provided herein that any notice, demand, request, consent, approval, declaration or other communication shall or may be given to or served upon any of the parties by another, or whenever any of the parties desires to give or serve upon another any communication with respect to this Agreement, each such notice, demand, request, consent, approval, declaration or other communication shall be in writing and shall be delivered in person (by personal delivery, delivery service or overnight courier service) with receipt acknowledged, or telecopied and confirmed immediately in writing by a copy mailed by registered or certified mail, return receipt requested, postage prepaid, addressed as hereinafter set forth, or sent by registered or certified mail, return receipt requested, postage prepaid, addressed as follows:

PFF Bancorp, Inc.

9337 Milliken Avenue

Rancho Cucamonga, CA 91730

Facsimile: (409)  ###-###-####

Attn: Robert L. Golish

8

FBOP Corporation

11 West Madison Street

Oak Park, IL 60302

Facsimile: 708 ###-###-####

Attn: Edward C. Fitzpatrick

The giving of any notice required hereunder may be waived in writing by the party entitled to receive such notice. Every notice, demand, request, consent, approval, declaration or other communication hereunder shall be deemed to have been duly given or served on the date on which personally delivered, in person, by delivery service or by overnight courier service, with receipt acknowledged, or the date of the telecopy transmission, or three (3) business days after the same shall have been deposited in the United States mail. Failure or delay in delivering copies of any notice, demand, request, consent, approval, declaration or other communication to the persons designated above to receive copies shall in no way adversely affect the effectiveness of such notice, demand, request, consent, approval, declaration or other communication.

17. CONSENT TO JURISDICTION. ANY LITIGATION BASED HEREON, OR ARISING OUT OF, UNDER, OR IN CONNECTION WITH, THIS AGREEMENT OR ANY COURSE OF CONDUCT, COURSE OF DEALING, STATEMENTS (WHETHER VERBAL OR WRITTEN) IN CONNECTION HEREWITH SHALL BE BROUGHT AND MAINTAINED EXCLUSIVELY IN THE COURTS OF THE STATE OF ILLINOIS OR IN THE UNITED STATES DISTRICT COURT FOR THE NORTHERN DISTRICT OF ILLINOIS. PLEDGOR HEREBY EXPRESSLY AND IRREVOCABLY SUBMITS TO THE JURISDICTION OF THE COURTS OF THE STATE OF ILLINOIS AND OF THE UNITED STATES DISTRICT COURT FOR THE NORTHERN DISTRICT OF ILLINOIS FOR THE PURPOSE OF ANY SUCH LITIGATION AS SET FORTH ABOVE. PLEDGOR FURTHER IRREVOCABLY CONSENTS TO THE SERVICE OF PROCESS BY REGISTERED MAIL AT PLEDGOR’S ADDRESS SET FORTH HEREIN, POSTAGE PREPAID. PLEDGOR HEREBY EXPRESSLY AND IRREVOCABLY WAIVES, TO THE FULLEST EXTENT PERMITTED BY LAW, ANY OBJECTION WHICH IT MAY HAVE OR HEREAFTER MAY HAVE TO THE LAYING OF VENUE OF ANY SUCH LITIGATION BROUGHT IN ANY SUCH COURT REFERRED TO ABOVE AND ANY CLAIM THAT ANY SUCH LITIGATION HAS BEEN BROUGHT IN AN INCONVENIENT FORUM.

18. WAIVER OF JURY TRIAL. EACH OF PLEDGOR AND PLEDGEE HEREBY EXPRESSLY WAIVES ANY RIGHT TO TRAIL BY JURY OF ANY CLAIM, DEMAND, ACTION OR CAUSE OF ACTION ARISING HEREUNDER OR IN CONNECTION HEREWITH.

19. Section Titles. The Section titles contained in this Agreement are and shall be without substantive meaning or content of any kind whatsoever and are not a part of the agreement between the parties hereto.

9

20. Counterparts. This Agreement may be executed in any number of counterparts, which shall, collectively and separately, constitute one agreement.

[Signature Page Follows]

10

IN WITNESS WHEREOF, Pledgor has caused this Pledge Agreement to be duly executed as of the date first written above.

11

Schedule I

Pledged Shares