AMENDMENTNUMBER FOUR to the MASTERREPURCHASE AGREEMENT Dated as of November 20, 2012, among PENNYMAC CORP., MORGAN STANLEY BANK. N.A. and MORGAN STANLEY MORTGAGE CAPITAL HOLDINGS LLC

Exhibit 10.96

EXECUTION COPY

AMENDMENT NUMBER FOUR

to the

MASTER REPURCHASE AGREEMENT

Dated as of November 20, 2012,

among

PENNYMAC CORP.,

MORGAN STANLEY BANK. N.A.

and

MORGAN STANLEY MORTGAGE CAPITAL HOLDINGS LLC

This AMENDMENT NUMBER FOUR (this “Amendment Number Four”) is made this 19^th day of December, 2013, among PENNYMAC CORP., a Delaware corporation, as seller (“Seller”), MORGAN STANLEY BANK, N.A., a national banking association, as buyer (“Buyer”) and MORGAN STANLEY MORTGAGE CAPITAL HOLDINGS LLC, a New York limited liability company, as agent for Buyer (“Agent”), to the Master Repurchase Agreement, dated as of November 20, 2012, between Seller and Buyer, as such agreement may be amended from time to time (the “Agreement”). Capitalized terms used but not otherwise defined herein shall have the meanings assigned to such terms in the Agreement.

RECITALS

WHEREAS, Seller, Buyer and Agent have agreed to amend the Agreement to extend the Termination Date thereunder for a period of 364 days and, in connection with such extension, make certain additional modifications thereto, each as more specifically set forth herein; and

WHEREAS, as of the date hereof, Seller represents to Buyer and Agent that Seller is in full compliance with all of the terms and conditions of the Agreement and each other Program Document and no Default or Event of Default has occurred and is continuing under the Agreement or any other Program Document.

NOW THEREFORE, for good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, and for the mutual covenants herein contained, the parties hereto hereby agree as follows:

SECTION 1. Amendment. Effective as of December 19, 2013 (the “Amendment Effective Date”),

(a) the Recital to the Agreement is hereby amended to read in its entirety as follows:

Buyer shall, with respect to the Committed Amount and may, with respect to the Uncommitted Amount, from time to time, upon the terms and conditions set forth herein, agree to enter into transactions in which Seller transfers to Buyer Eligible Mortgage Loans against the transfer of funds by Buyer, with a simultaneous agreement by Buyer to transfer to Seller such Purchased Loans at a date certain, against the transfer of funds by Seller. Each such transaction shall be referred to herein as a “Transaction,” and, unless otherwise agreed in writing, shall be governed by this Agreement.

(b) Section 1.01 of the Agreement is hereby amended by adding the following new defined term immediately following the defined term “Commitment Fee Percentage:”

“Committed Amount” shall have the meaning assigned thereto in the Pricing Side Letter.

(c) Section 1.01 of the Agreement is hereby amended by adding the following new defined term immediately following the defined term “Confirmation:”

“Contractual Obligation” shall mean as to any Person, any provision of any agreement, instrument or other undertaking to which such Person is a party or by which it or any of its property is bound or any provision of any security issued by such Person.

(d) the defined term “Maximum Amount” in Section 1.01 of the Agreement is hereby amended to read in its entirety as follows:

“Maximum Amount” shall mean the sum of (i) the Committed Amount and (ii) in Buyer’s sole discretion, the Uncommitted Amount.

(e) the defined term “Termination Date in Section 1.01 of the Agreement is hereby amended to read in its entirety as follows:

“Termination Date” shall mean December 18, 2014 or such earlier date on which this Repurchase Agreement shall terminate in accordance with the provisions hereof or by operation of law.

(f) Section 1.01 of the Agreement is hereby amended by adding the following new defined term immediately following the defined term “Type”:

“Uncommitted Amount” shall have the meaning assigned thereto in the Pricing Side Letter.

(g) Section 2.01(a) of the Agreement is hereby amended to read in its entirety as follows:

(a) Subject to the terms and conditions of the Repurchase Documents and provided that no Default or Event of Default shall have occurred and be continuing hereunder, the Buyer shall, with respect to the Committed Amount and may, with respect to the Uncommitted Amount, from time to time as requested by the Seller, enter into Transactions with an aggregate Purchase Price for all Purchased Loans acquired by the Buyer not to exceed the lesser of (i) the Maximum Amount and (ii) the Margin Base as in effect from time to time (after giving effect to the purchase of such Eligible Mortgage Loans). The Buyer shall have the obligation, subject to the terms and conditions of the Repurchase Documents, to enter into Transactions up to the Committed Amount and shall have no obligation to enter into Transactions with respect to the Uncommitted Amount, which Transactions shall be entered into in the sole discretion of the Buyer. All purchases of Loans hereunder shall be first deemed committed up to the Committed Amount and then the remainder, if any, shall be deemed uncommitted up to the Uncommitted Amount.

(h) Section 2.02(b) of the Agreement is hereby amended to read in its entirety as follows:

(b) Upon receipt from the Seller of a Transaction Request pursuant to Section 2.02(a), the Buyer shall, with respect to the Committed Amount, and may, with respect to the Uncommitted Amount, upon satisfaction of all applicable conditions precedent set forth in Sections 5.01 and 5.02 hereof and provided that no Default or

2

Event of Default shall have occurred and be continuing, enter into such Transaction with the Seller. In the event that the Buyer (or the Agent on behalf of the Buyer) determines to enter into a Transaction, the Buyer (or the Agent) shall specify the terms for such proposed Transaction, including the Purchase Price for the applicable Eligible Mortgage Loans, the Pricing Rate for the Transaction, the Market Value for the applicable Eligible Mortgage Loans, the Repurchase Date in respect of such Transaction and any additional terms or conditions of the Transaction, in a Confirmation to be delivered to the Seller on or prior to the applicable Purchase Date. The Buyer shall have no obligation to enter into any Transactions with respect to the Uncommitted Amount, which Transactions shall be entered into in the sole discretion of the Buyer.

(i) Section 2.05 of the Agreement is hereby amended by adding the following new Sub-section (e) to the end thereof:

(e) The Buyer may, in its sole discretion, at any time, terminate any Transactions with respect to the Uncommitted Amount by providing written notice to the Seller. Not later than the earlier of thirty calendar days following receipt of such notice or the next Repurchase Date, the Seller agrees to repurchase all such Mortgage Loans at the Repurchase Price and to satisfy all of its Repurchase Obligations with respect to such Mortgage Loans.

(j) Section 3.04 of the Agreement is hereby amended to read in its entirety as follows:

The Seller agrees to pay to the Buyer a commitment fee equal to the product of (a) the Commitment Fee Percentage and (b) the Committed Amount (the “Commitment Fee”), such payment to be made in Dollars in immediately available funds, without deduction, set off or counterclaim, to the Buyer as provided in the Pricing Side Letter. The Seller also hereby agrees to pay to the Buyer any other fees provided in the Pricing Side Letter as specified therein. The Buyer may, in its sole discretion, net any of the Commitment Fee or any other fee from the proceeds of any Purchase Price paid to the Seller. Each of such fees shall be deemed to be fully earned and non-refundable when paid.

(k) Section 4.12 of the Agreement is hereby amended to read in its entirety as follows:

(a) All payments made by the Seller under this Repurchase Agreement shall be made free and clear of, and without deduction or withholding for or on account of, any present or future taxes, levies, imposts, deductions, charges or withholdings, and all liabilities (including penalties, interest and additions to tax) with respect thereto imposed by any Governmental Authority (collectively, “Taxes”), all of which shall be paid by the Seller for its own account not later than the date when due. Taxes as defined in this Section 4.12(a) shall not include (i) taxes imposed on the net income of the Buyer, (ii) taxes imposed under Section 1471 through Section 1474 of the Code, any current or future regulations or official interpretations thereof, any agreement entered into pursuant to Section 1471(b) of the Code, or any fiscal or regulatory legislation, rules or practices adopted pursuant to any intergovernmental agreement entered into in connection with the implementation of such Sections of the Code, (iii) branch profits taxes, or (iv) franchise taxes. If the Seller is required by law or regulation to deduct or withhold any Taxes from or in respect of any amount payable hereunder, it shall: (a) make such deduction or

3

withholding; (b) pay the amount so deducted or withheld to the appropriate Governmental Authority not later than the date when due; (c) deliver to the Buyer, promptly, original tax receipts and other evidence satisfactory to the Buyer of the payment when due of the full amount of such Taxes; and (d) pay to the Buyer such additional amounts as may be necessary so that the Buyer receives, free and clear of all Taxes, a net amount equal to the amount it would have received under this Repurchase Agreement, as if no such deduction or withholding had been made. In addition, the Seller agrees to pay to the relevant Governmental Authority in accordance with applicable law any current or future stamp or documentary taxes or any other excise or property taxes, charges or similar levies (including, without limitation, mortgage recording taxes, transfer taxes and similar fees) imposed by any Governmental Authority that arise from any payment made hereunder or from the execution, delivery or registration of, or otherwise with respect to, this Repurchase Agreement (“Other Taxes”).

(b) The Seller agrees to indemnify and hold the Buyer harmless for the full amount of Taxes (including additional amounts with respect thereto) and Other Taxes, and the full amount of Taxes of any kind imposed by any jurisdiction on amounts payable under this Section 4.12, and any liability (in each case, including penalties, interest, additions thereto and expenses) arising therefrom or with respect thereto, provided that the Buyer shall have provided the Seller with evidence, reasonably satisfactory to the Seller, of payment of Taxes or Other Taxes, as the case may be.

(c) Without prejudice to the survival or any other agreement of the Seller hereunder, the agreements, covenants and obligations of the Seller contained in this Section 4.12 shall survive the termination of this Repurchase Agreement. Nothing contained in this Section 4.12 shall require the Buyer to make available any of its tax returns or other information that it deems to be confidential or proprietary or to incur additional costs or regulatory burdens that the Buyer considers in its good faith reasonable judgment to be material.

(d) Each party to this Repurchase Agreement acknowledges that it is its intent for purposes of U.S. federal, state and local income and franchise taxes to treat each Transaction as indebtedness of the Seller that is secured by the Purchased Loans and that the Purchased Loans are owned by the Seller in the absence of an Event of Default or any event under Section 13.21 of this Repurchase Agreement which the Buyer determines to be inconsistent with such treatment. All parties to this Repurchase Agreement agree to such treatment and agree to take no action inconsistent with this treatment, unless required by law.

(l) Sections 7.14, 7.15 and 7.16 of the Agreement are hereby amended to read in their entirety as follows:

7.14 Maintenance of Tangible Net Worth. The Seller shall not permit its Tangible Net Worth at any time to be less than $150,000,000 and the Seller shall not permit the Tangible Net Worth of Guarantor at any time to be less than $860,000,000.

7.15 Maintenance of Ratio of Total Indebtedness to Tangible Net Worth. The Seller shall not permit the ratio of Total Indebtedness to Tangible Net Worth at any time to be greater than 10:1. The Seller shall not permit the Guarantor’s ratio of Total Indebtedness to Tangible Net Worth at any time to be greater than 5:1.

4

7.16 Maintenance of Profitability. The Seller shall not permit the Guarantor’s Net Income (before income taxes), generated over a consecutive six month period, measured on the last day of each fiscal quarter, to be less than $1.00.

(m) Section 7.18 of the Agreement is hereby amended to read in its entirety as follows:

7.18 Maintenance of Liquidity. The Seller shall ensure that, as of the end of each calendar month, it has unencumbered Cash Equivalents in an amount of not less than $10,000,000 and that the Guarantor has unencumbered Cash Equivalents in an amount of not less than $40,000,000.

(n) Section 13.20(b) of the Agreement is hereby amended to read in its entirety as follows:

(b) the Buyer and the Agent have no fiduciary relationship to the Seller, and the relationship between the Seller, on the one hand, and the Buyer (and the Agent of the Buyer, as applicable), on the other hand, is solely that of the Seller and the Buyer (and the Agent of the Buyer, as applicable); and

SECTION 2. Defined Terms. Any terms capitalized but not otherwise defined herein shall have the respective meanings set forth in the Agreement.

SECTION 3. Effectiveness. This Amendment Number Four shall become effective as of the date that the Agent shall have received counterparts hereof duly executed by each of the parties hereto.

SECTION 4. Fees and Expenses. Seller agrees to pay to Buyer and Agent all reasonable out of pocket costs and expenses incurred by Buyer or Agent in connection with this Amendment Number Four (including all reasonable fees and out of pocket costs and expenses of Buyer’s or Agent’s legal counsel) in accordance with Section 13.04 and 13.06 of the Agreement.

SECTION 5. Representations. Seller hereby represents to Buyer and Agent that as of the date hereof and taking into account the terms of this Amendment Number Four, Seller is in full compliance with all of the terms and conditions of the Agreement and each other Program Document and no Default or Event of Default has occurred and is continuing under the Agreement or any other Program Document.

SECTION 6. Binding Effect; Governing Law. THIS AMENDMENT NUMBER FOUR SHALL BE BINDING AND INURE TO THE BENEFIT OF THE PARTIES HERETO AND THEIR RESPECTIVE SUCCESSORS AND PERMITTED ASSIGNS. THIS AMENDMENT NUMBER FOUR SHALL BE CONSTRUED IN ACCORDANCE WITH, AND GOVERNED BY, THE LAWS OF THE STATE OF NEW YORK, WITHOUT GIVING EFFECT TO THE CONFLICT OF LAWS PRINCIPLES THEREOF (EXCEPT FOR SECTION 5-1401 OF THE NEW YORK GENERAL OBLIGATIONS LAW WHICH SHALL GOVERN).

SECTION 7. Counterparts. This Amendment Number Four may be executed by each of the parties hereto on any number of separate counterparts, each of which shall be an original and all of which taken together shall constitute one and the same instrument.

SECTION 8. Limited Effect. Except as amended hereby, the Agreement shall continue in full force and effect in accordance with its terms. Reference to this Amendment Number Four

5

need not be made in the Agreement or any other instrument or document executed in connection therewith, or in any certificate, letter or communication issued or made pursuant to, or with respect to, the Agreement, any reference in any of such items to the Agreement being sufficient to refer to the Agreement as amended hereby.

[Signature Page Follows]

6

IN WITNESS WHEREOF, Seller, Buyer and Agent have caused this Amendment Number Four to be executed and delivered by their duly authorized officers as of the Amendment Effective Date.

Amendment Number Four to Master Repurchase Agreement