FIFTH AMENDMENT AGREEMENT To Pledge and Security Agreement and Irrevocable Proxy Dated as of June 30, 2009 by and among RFC ASSETHOLDINGS II, LLC, PASSIVE ASSET TRANSACTIONS, LLC AND CERTAIN AFFILIATES THEREOF FROM TIME TO TIME PARTY HERETO, as Grantors,

Exhibit 10.36

EXECUTION COPY

FIFTH AMENDMENT AGREEMENT

To Pledge and Security Agreement and Irrevocable Proxy

Dated as of June 30, 2009

by and among

RFC ASSET HOLDINGS II, LLC,

PASSIVE ASSET TRANSACTIONS, LLC

AND CERTAIN

AFFILIATES THEREOF FROM

TIME TO TIME PARTY HERETO,

as Grantors,

and

GMAC LLC,

as Lender Agent

This FIFTH AMENDMENT AGREEMENT (this “Agreement”), dated as of June 30, 2009 (the “Amendment Effective Date”), is by and among RFC Asset Holdings II, LLC, a Delaware limited liability company (“RAHI”), and Passive Asset Transactions, LLC, a Delaware limited liability company (“PATI”; and together with RAHI, each a “Borrower” and collectively, the “Borrowers”); Residential Capital, LLC, a Delaware limited liability company (“ResCap”), Residential Funding Company, LLC, a Delaware limited liability company (“RFC”), and GMAC Mortgage, LLC, a Delaware limited liability company (“GMAC Mortgage”; and together with ResCap and RFC, each herein a “Guarantor” and collectively, the “Guarantors”), and the other parties hereto as Grantors (each, together with each Borrower and each Guarantor, a “Grantor” and collectively, the “Grantors”); and GMAC LLC, a Delaware limited liability company, as Lender Agent for the Lender Parties.

Reference is hereby made to the Pledge and Security Agreement and Irrevocable Proxy dated as of November 20, 2008 among the Grantors and the Lender Agent (as amended and modified through the date hereof, the “Security Agreement”).

RECITALS

1. Each of the parties hereto is a party to the Security Agreement.

2. The parties hereto desire to make certain amendments to the Security Agreement.

3. Each of the parties hereto, by its signature hereto, hereby acknowledges, consents and agrees to the changes set forth herein.

4. In consideration of the promises and mutual agreements herein contained and for good and valuable consideration, the receipt and sufficiency of which is hereby acknowledged, the parties hereto agree as follows:

ARTICLE I

DEFINED TERMS

SECTION 1.1 Capitalized terms used herein and not otherwise defined shall have the meaning set forth in the Security Agreement.

ARTICLE II

AMENDMENTS TO THE SECURITY AGREEMENT

SECTION 2.1 Amendments to Section 1.

(a) The following definitions in Section 1 to the Security Agreement are hereby amended and restated in their entirety to read as follows:

First Savings Warehouse Note means that Promissory Note, dated as of April 1, 2005 and that Promissory Note dated as of July 1, 2009 and issued by First Savings Mortgage Corporation in favor of RFC, as the same may be amended, supplemented, restated or otherwise modified from time to time, and including any notes given in substitution or replacement therefor.

(b) The following definitions are hereby added to Section 1 to the Security Agreement:

MSR Documents means (a) the MSR Loan Agreement, (b) the “Notes, “ the “Guarantee,” the “Account Control Agreements,” and the “Servicing Contracts” (each as defined in the MSR Loan Agreement) and (c) all notices, certificates, financing statements and other documents to be executed and delivered by RFC, GMAC Mortgage or ResCap in connection with the transactions contemplated by the MSR Loan Agreement.

November Documents means (a) the Loan Agreement, the Notes, this Agreement, the Account Control Agreements, the GSAP Indenture Transaction Documents and the Flume No. 8 Security Documents, (b) after the Custodial Transfer Date, the Master Custody Agreement, (c) any document designated as a “Security Document” in any Collateral Addition Designation Notice, (d) all of the security agreements, pledges, collateral assignments, mortgages, deeds of trust, trust deeds or other instruments evidencing or creating or purporting to create any security interests in favor of the Lender Agent for its benefit and for the benefit of the Lender Parties, (e) the Contribution Agreements and (f) all notices, certificates, financing statements, agreements and other documents to be executed and delivered by RAHI, PATI, ResCap, RFC or GMAC Mortgage pursuant to the foregoing or otherwise in connection with the Loan Agreement or the extension of financing by the Lenders contemplated thereunder.

Warehouse Loans means, as the context may require, loans made by RFC pursuant to a Warehouse Agreement to the borrower under such agreement and/or any mortgage loans or other loans or assets purchased by RFC pursuant to the First Savings Warehouse Agreement.

(c) The definitions of “Permitted Liens”, “Secured Parties” and “Credit Agent” are hereby deleted in their entirety from Section 1 to the Security Agreement.

SECTION 2.2 Amendments to Schedule IV.

(a) Exhibit A to Schedule IV of the Fourth Security Agreement is hereby amended by inserting the following:

(b) Exhibit C to Schedule IV of the Security Agreement is hereby amended by amending and restating the third Pledged Note listed therein as follows:

(c) Exhibit D to Schedule IV of the Security Agreement is hereby amended by inserting the following:

ARTICLE III

CONDITIONS TO EFFECTIVENESS

SECTION 3.1 Amendment Effective Date. This Agreement and the provisions contained herein shall become effective as of the Amendment Effective Date.

SECTION 3.2 Opinions and Other Deliverables. The Obligors covenant and agree to deliver or cause to be delivered (a) opinions of counsel to the Obligors with respect to the transactions contemplated hereby, which opinions shall be in form and substance satisfactory to the Lender Agent, on or before July 8, 2009 and (b) such other documents, including but not limited to the Obligors’ board resolutions approving this Agreement, as the Lender Agent may reasonably request, which documents will be in form and substance satisfactory to the Lender Agent, on or before July 15, 2009. The Obligors acknowledge and agree that the Lender Agent may, upon the failure to deliver any of the items set forth in clauses (a) and (b) above in the timeframes set forth therein, declare an Event of Default.

ARTICLE IV

ACKNOWLEDGEMENTS, CONSENTS, NOTICE, CONFIRMATION AND

REPRESENTATIONS AND WARRANTIES

SECTION 4.1 Notice. Each party hereto hereby acknowledges timely notice of the execution of this Agreement and of the transactions and amendments contemplated hereby. Each party hereto hereby waives any notice requirement contained in the Security Agreement or the Facility Documents with respect to the execution of this Agreement.

SECTION 4.2 Confirmation of the Security Agreement. The Grantors each hereby acknowledge and agree that the Security Agreement and each other Facility Document (each as amended as of the date hereof) are each ratified and confirmed in all respects and shall remain in full force and effect in accordance with their respective terms. Without limiting the foregoing, each Grantor reaffirms its grant of a security interest in all the Collateral pledged by it, and agrees that such security interest secures all Obligations as defined in the Loan Agreement. As of the Amendment Effective Date, each reference in the Security Agreement to “this Agreement” or in any other Facility Document to the “Security Agreement” shall mean the Security Agreement as amended by this Agreement, and as hereinafter amended or restated.

SECTION 4.3 Representations and Warranties. By its signature hereto, each Grantor hereby represents and warrants that, before and after giving effect to this Agreement, as follows:

(a) Its representations and warranties set forth in the Facility Documents (each as amended as of the date hereof) are true and correct as if made on the date hereof, except to the extent they expressly relate to an earlier date; and

(b) No Default has occurred and is continuing.

ARTICLE V

MISCELLANEOUS

SECTION 5.1 GOVERNING LAW. THIS AGREEMENT SHALL BE GOVERNED BY AND CONSTRUED IN ACCORDANCE WITH THE LAWS OF THE STATE OF NEW YORK WITHOUT REGARD TO CONFLICTS OF LAWS PRINCIPLES (BUT WITH REFERENCE TO SECTION 5-1401 OF THE NEW YORK GENERAL OBLIGATIONS LAW, WHICH BY ITS TERMS APPLIES TO THIS AGREEMENT).

SECTION 5.2 Execution in Counterparts. This Agreement may be executed in any number of counterparts and by different parties hereto in separate counterparts, each of which when so executed shall be deemed to be an original (whether such counterpart is originally executed or an electronic copy of an original and each party hereto expressly waives its rights to receive originally executed documents) and all of which when taken together shall constitute one and the same agreement.

SECTION 5.3 WAIVER OF JURY TRIAL. EACH OF THE PARTIES HEREBY IRREVOCABLY WAIVES ANY AND ALL RIGHT TO A TRIAL BY JURY WITH RESPECT TO ANY LEGAL PROCEEDING ARISING OUT OF OR RELATING TO THIS AGREEMENT.

SECTION 5.4 Entire Agreement. This Agreement, the Security Agreement and the other Facility Documents embody the entire agreement and understanding of the parties hereto and supersede any and all prior agreements, arrangements and understanding relating to the matters provided for herein.

SECTION 5.5 Captions. The various captions in this Agreement are included for convenience only and shall not affect the meaning or interpretation of any provision of this Agreement.

SECTION 5.6 Severability. If any provision of this Agreement, or the application thereof to any party or any circumstance, is held to be unenforceable, invalid or illegal (in whole or in part) for any reason (in any jurisdiction), the remaining terms of this Agreement, modified by the deletion of the unenforceable invalid or illegal portion (in any relevant jurisdiction), will continue in full force and effect, and such unenforceability, invalidity or illegality will not otherwise affect the enforceability, validity or legality of the remaining terms of this Agreement so long as this Agreement, as so modified, continues to express, without material change, the original intentions of the parties as to the subject matter hereof and the deletion of such portion of this Agreement will not substantially impair the respective expectations of the parties or the practical realization of the benefits that would otherwise be conferred upon the parties.

SECTION 5.7 SUBMISSION TO JURISDICTION. EACH PARTY HERETO HEREBY IRREVOCABLY SUBMITS TO THE NONEXCLUSIVE JURISDICTION OF THE COURTS OF THE STATE OF NEW YORK LOCATED IN THE CITY OF MANHATTAN OR IN THE UNITED STATES DISTRICT COURT FOR THE SOUTHERN DISTRICT OF NEW YORK FOR PURPOSES OF ALL LEGAL PROCEEDINGS ARISING OUT OF OR RELATING TO THIS AGREEMENT OR THE TRANSACTIONS CONTEMPLATED HEREBY AND IRREVOCABLY AGREES TO BE BOUND BY ANY JUDGMENT RENDERED THEREBY IN CONNECTION WITH SUCH LITIGATION. EACH PARTY HERETO IRREVOCABLY WAIVES, TO THE FULLEST EXTENT PERMITTED BY LAW, ANY OBJECTION WHICH IT MAY NOW OR HEREAFTER HAVE TO THE LAYING OF THE VENUE OF ANY SUCH PROCEEDING BROUGHT IN SUCH A COURT AND ANY CLAIM THAT ANY SUCH PROCEEDING BROUGHT IN SUCH A COURT HAS BEEN BROUGHT IN AN INCONVENIENT FORUM. EACH PARTY HERETO HEREBY CONSENTS TO PROCESS BEING SERVED IN ANY SUIT, ACTION OR PROCEEDING WITH RESPECT TO THIS AGREEMENT, OR ANY DOCUMENT DELIVERED PURSUANT HERETO BY THE MAILING OF A COPY THEREOF BY REGISTERED OR CERTIFIED MAIL, POSTAGE PREPAID, RETURN RECEIPT REQUESTED, TO ITS RESPECTIVE ADDRESS SPECIFIED AT THE TIME FOR NOTICES UNDER THIS AGREEMENT OR TO ANY OTHER ADDRESS OF WHICH IT SHALL HAVE GIVEN WRITTEN OR ELECTRONIC NOTICE TO THE OTHER PARTIES. THE FOREGOING SHALL NOT LIMIT THE ABILITY OF ANY PARTY HERETO TO BRING SUIT IN THE COURTS OF ANY JURISDICTION.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

IN WITNESS WHEREOF, the parties have caused this Agreement to be executed by their respective officers thereunto duly authorized, as of the date first above written.