MODIFICATION TO CONSENT AND AMENDMENT NO. 4 TO TERM LOAN AGREEMENT

Exhibit 10.36

EXECUTION COPY

MODIFICATION TO CONSENT AND AMENDMENT NO. 4

TO TERM LOAN AGREEMENT

THIS MODIFICATION TO CONSENT AND AMENDMENT NO. 4 TO TERM LOAN AGREEMENT (this “Modification to Consent”) is entered into as of March     , 2013, by and among the Lenders identified on the signature pages hereof (such Lenders, together with their respective successors and permitted assigns, are referred to hereinafter each individually as a “Lender” and collectively as the “Lenders”), CERBERUS BUSINESS FINANCE LLC, as agent for the Lenders (in such capacity, “Agent”), KELLWOOD COMPANY, a Delaware corporation (“Parent”), the Domestic Subsidiaries of Parent party hereto as Borrowers (together with Parent, “Borrowers”) and the other Obligors party hereto (together with the Borrowers, the “Loan Parties”).

WHEREAS, the Loan Parties, Agent, and Lenders are parties to that certain Term Loan Agreement dated as of October 19, 2011 (as amended, modified or supplemented from time to time, the “Term Loan Agreement”);

WHEREAS, the Loan Parties, Agent and Lenders are also parties to that certain Consent and Amendment No. 3 to Term Loan Agreement dated as of December 31, 2012 (the “Phat Fashions Consent”);

WHEREAS, pursuant to the Phat Fashions Consent, Agent and Lenders consented to the sale by Phat Fashions of the PF Transferred Assets to PF Purchaser pursuant to the Phat Fashions Purchase Agreement, subject to certain terms and conditions;

WHERAS, the Loan Parties have requested that certain of such terms and conditions to such consent be modified and Agent and Lenders are willing to provide their consent to such modification on the terms provided herein;

NOW THEREFORE, in consideration of the premises and mutual agreements herein contained, the parties hereto agree as follows:

1. Defined Terms. Unless otherwise defined herein, capitalized terms used herein shall have the meanings ascribed to such terms in the Phat Fashions Consent.

2. Amendments to Phat Fashions Consent. In reliance upon the representations and warranties of the Borrower set forth in Section 6 below and subject to the conditions to effectiveness set forth in Section 5 below, the Phat Fashions Consent is hereby amended as follows:

(a) Section 2(ii) is amended and restated in its entirety as follows:

(ii) the sale by Phat Fashions of the PF Transferred Assets to PF Purchaser pursuant to the Phat Fashions Purchase Agreement and the release of Agent’s Lien on the PF Transferred Assets, in each case, so long as (A) such sale is consummated on or before May 31, 2013, (or such later date as Agent shall agree to), (B) the amount of the gross proceeds arising from the cash purchase price paid on the Closing (as defined in the Phat Fashions Purchase Agreement) for the PF Transferred Assets (such amount, the “Initial PF Consideration”) is at least $4,800,000, and (C) the Net Cash Proceeds of the Initial PF Consideration is remitted directly to Agent for application to the outstanding Revolving Loans,

(b) The Phat Fashions Purchase Agreement attached as Exhibit C to the Phat Fashions Consent is replaced with Exhibit C hereto. All references to the Phat Fashions Purchase Agreement shall mean the agreement attached hereto as Exhibit C.

3. Continuing Effect. Except as expressly set forth in Section 2 of this Modification to Consent, nothing in this Modification to Consent shall constitute a modification or alteration of the terms, conditions or covenants of the Term Loan Agreement or any other Loan Document (including the Phat Fashions Consent), or a waiver of any other terms or provisions thereof, and the Term Loan Agreement and the other Loan Documents shall remain unchanged and shall continue in full force and effect, in each case as amended hereby.

4. Reaffirmation and Confirmation; Covenant.

(a) Each Loan Party hereby ratifies, affirms, acknowledges and agrees that the Term Loan Agreement and the other Loan Documents represent the valid, enforceable (except as enforceability may be limited by bankruptcy, insolvency or similar laws affecting the enforcement of creditors’ rights generally) and collectible obligations of such Loan Party, and further acknowledges that there are no existing claims, defenses, personal or otherwise, or rights of setoff whatsoever with respect to the Term Loan Agreement or any other Loan Document. Each Loan Party hereby agrees that this Modification to Consent in no way acts as a release or relinquishment of the Liens and rights securing payments of the Obligations. The Liens and rights securing payment of the Obligations are hereby ratified and confirmed by each Loan Party in all respects.

(b) Each of (i) SCSF Kellwood Finance, LLC (“SCSF Finance”) and Sun Kellwood Finance, LLC (“Sun Finance”, and together with SCSF Finance, the “Sun Guarantors”) hereby ratifies, affirms, acknowledges and agrees that its obligations under that certain Guaranty, dated as of October 19, 2011, made by the Sun Guarantors in favor of the Agent remain in full force and effect and (ii) SK Financial Services Corp. hereby ratifies, affirms, acknowledges and agrees that its obligations under that certain Limited Guaranty dated as of March 23, 2012, made by SK Financial Services Corp. in favor of the Agent remains in full force and effect.

5. Conditions to Effectiveness. This Modification to Consent shall become effective upon the satisfaction of each of the following conditions precedent, each in form and substance acceptable to Agent:

(a) Agent shall have received a fully executed copy of this Modification to Consent;

(b) Agent shall have received a fully executed copy of an amendment to the Revolver Loan Documents, with respect to the matters addressed in this Modification to Consent;

-2-

(c) Agent shall have received a fully executed copy of an amendment to the Sun Loan Documents, with respect to the matters addressed in this Modification to Consent;

(d) No Default or Event of Default shall have occurred and be continuing on the date hereof or as of the date of the effectiveness of this Modification to Consent.

6. Representations and Warranties. In order to induce Agent and Lenders to enter into this Modification to Consent, each Loan Party hereby represents and warrants to Agent and Lenders that, after giving effect to this Modification to Consent:

(a) All representations and warranties contained in the Term Loan Agreement and the other Loan Documents are true and correct in all material respects on and as of the date of this Modification to Consent, in each case as if then made, other than representations and warranties that expressly relate solely to an earlier date (in which case such representations and warranties were true and correct in all material respects on and as of such earlier date);

(b) No Default or Event of Default has occurred and is continuing; and

(c) This Modification to Consent and the Term Loan Agreement, as modified hereby, constitute legal, valid and binding obligations of each Loan Party and are enforceable against each Loan Party in accordance with their respective terms, except as enforceability may be limited by bankruptcy, insolvency or similar laws affecting the enforcement of creditors’ rights generally.

7. Miscellaneous.

(a) Expenses. The Loan Parties jointly and severally agree to pay on demand all expenses of Agent (including, without limitation, the fees and expenses of outside counsel for Agent) in connection with the preparation, negotiation, execution, delivery and administration of this Modification to Consent and all other instruments or documents provided for herein or delivered or to be delivered hereunder or in connection herewith, all in accordance with Section 3.4 of the Term Loan Agreement. All obligations provided herein shall survive any termination of this Modification to Consent and the Term Loan Agreement as modified hereby.

(b) Governing Law. This Modification to Consent shall be a contract made under and governed by the internal laws of the State of New York.

(c) Counterparts. This Modification to Consent may be executed in any number of counterparts, and by the parties hereto on the same or separate counterparts, and each such counterpart, when executed and delivered, shall be deemed to be an original, but all such counterparts shall together constitute but one and the same Amendment. Delivery of an executed counterpart of this Modification to Consent by facsimile or electronic mail shall be equally effective as delivery of an original executed counterpart of this Modification to Consent.

-3-

8. Release.

(a) In consideration of the agreements of Agent and Lenders contained herein and for other good and valuable consideration, the receipt and sufficiency of which is hereby acknowledged, each Loan Party, on behalf of itself and its successors, assigns, and other legal representatives, hereby absolutely, unconditionally and irrevocably releases, remises and forever discharges Agent and Lenders, and their successors and assigns, and their present and former shareholders, affiliates, subsidiaries, divisions, predecessors, directors, officers, attorneys, employees, agents and other representatives (Agent, each Lender and all such other Persons being hereinafter referred to collectively as the “Releasees” and individually as a “Releasee”), of and from all demands, actions, causes of action, suits, controversies, damages and any and all other claims, counterclaims, defenses, rights of set-off, demands and liabilities whatsoever (individually, a “Claim” and collectively, “Claims”) of every name and nature, known or unknown, suspected or unsuspected, both at law and in equity, which any Loan Party or any of their respective successors, assigns, or other legal representatives may now or hereafter own, hold, have or claim to have against the Releasees or any of them for, upon, or by reason of any circumstance, action, cause or thing whatsoever in relation to, or in any way in connection with any of the Term Loan Agreement, or any of the other Loan Documents or transactions thereunder or related thereto which arises at any time on or prior to the day and date of this Modification to Consent.

(b) Each Loan Party understands, acknowledges and agrees that the release set forth above may be pleaded as a full and complete defense and may be used as a basis for an injunction against any action, suit or other proceeding which may be instituted, prosecuted or attempted in breach of the provisions of such release.

(c) Each Loan Party agrees that no fact, event, circumstance, evidence or transaction which could now be asserted or which may hereafter be discovered shall affect in any manner the final, absolute and unconditional nature of the release set forth above.

[signature pages follow]

-4-

IN WITNESS WHEREOF, the parties hereto have caused this Modification to Consent to be executed by their respective officers thereunto duly authorized and delivered as of the date first above written.

Modification to Consent and Amendment No. 4 to Term Loan Agreement

Modification to Consent and Amendment No. 4 to Term Loan Agreement

Modification to Consent and Amendment No. 4 to Term Loan Agreement

Agreed and acknowledged with respect to Section 4(b):

Modification to Consent and Amendment No. 4 to Term Loan Agreement