AMENDMENT NO. 14 TO LOAN AND SECURITY AGREEMENT

Exhibit 10.1

AMENDMENT NO. 14 TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 14 TO LOAN AND SECURITY AGREEMENT, dated as of October 27, 2009 (this “Amendment No. 14”), entered into by and among Wachovia Bank, National Association, successor by merger to Congress Financial Corporation (Florida), in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, LLC, a Delaware limited liability company formerly known as Supreme International, Inc. (“Supreme”), Jantzen, LLC, a Delaware limited liability company formerly known as Jantzen, Inc. (“Jantzen”), Perry Ellis Menswear, LLC, a Delaware limited liability company formerly known as Perry Ellis Menswear, Inc. (“Perry Ellis Menswear”), Perry Ellis Europe Limited, a private limited company incorporated in England and Wales formerly known as Farah Manufacturing (U.K.) Limited (“Perry Europe”), Salant Holding, LLC, a Delaware limited liability company formerly known as Salant Holding Corporation (“Salant Holding” and together with Supreme, Jantzen, Perry Europe and Perry Ellis Menswear, each individually “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel, LLC, a Delaware limited liability company formerly known as Jantzen Apparel Corp. (“Jantzen Apparel”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, a Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), Perry Ellis Shared Services Corporation, a Delaware corporation (“PE Shared Services”), Winnsboro DC, LLC, a Delaware limited liability company (“Winnsboro”), Tampa DC, LLC, a Delaware limited liability company (“Tampa DC”), Perry Ellis International Group Holdings Limited, a private company incorporated under the laws of Ireland having its principal place of business in the Bahamas (“Group Holdings”) and Perry Ellis Real Estate, LLC, a Delaware limited liability company formerly known as Perry Ellis Real Estate Corporation (“PE Real Estate” and, together, with Parent, PEI Licensing, Jantzen Apparel, Supreme I, Supreme II, Supreme Realty, Group Holdings, PE Shared Services, Winnsboro, Tampa DC, and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

W I T N E S S E T H :

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1, 2002, by and among Agent, Lenders, Borrowers and Guarantors, as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19, 2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003, Amendment No. 3 to Loan and Security Agreement, dated December 1, 2003, Amendment No. 4 to Loan and Security Agreement, dated February 25, 2004, Amendment No. 5 to Loan and Security Agreement, dated July 1, 2004, Amendment No. 6 to Loan and Security Agreement, dated as of September 30, 2004, Amendment No. 7 to Loan and Security Agreement, dated as of February 26, 2005, Amendment No. 8 to Loan and Security Agreement, dated as of

September 30, 2005, Amendment No. 9 to Loan and Security Agreement, dated as of February 24, 2006, Amendment No. 10 to Loan and Security Agreement, dated as of August 28, 2006, Amendment No 11 to Loan and Security Agreement, dated as of November 29, 2006, Amendment No. 12 and Consent to Loan and Security Agreement, dated as of December 6, 2006 and Amendment No. 13 to Loan and Security Agreement, dated as of October 30, 2008 (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders agree to (a) release Agent’s security interest and lien upon the assets of Perry Europe, and (b) make certain amendments to the Loan Agreement, and Agent and Lenders are willing to agree to such release and such amendments, subject to the terms and conditions set forth in this Amendment No. 14; and

WHEREAS, by this Amendment No. 14, Agent, Lenders, Borrowers and Guarantors desire and intend to evidence such release and amendments;

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions.

1.1 Additional Definitions.

(a) “Amendment No. 14” shall mean Amendment No. 14 to Loan and Security Agreement by and among Agent, Lenders, Borrowers and Guarantors, as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

(b) “Amendment No. 14 Effective Date” shall mean the first date on which all of the conditions precedent to the effectiveness of Amendment No. 14 shall have been satisfied and/or waived.

2. Transfers of Assets. Section 9.7 (b) of the Loan Agreement is hereby amended to add a new Section 9.7(b)(xii) at the end thereof as follows:

“(xii) the sale of any Intellectual Property; provided, that, (A) Agent shall have received not less than ten (10) days prior notice together with true, correct and complete copies of all agreements, documents and instruments to be executed and/or delivered relating to such sale and such other information with respect thereto as Agent may reasonably request; (B) At Agent’s option, any Inventory bearing the trademark(s) sold in accordance with this Section 9.7(b)(xii) shall cease to constitute Eligible Inventory to the extent that such Inventory may no longer be sold using or incorporating such trademark

or, in the reasonable determination of Agent, such Inventory is not anticipated to be sold during any remaining period during which such Borrower continues to be authorized to use such trademark prior to the termination of its rights to do so under the applicable agreement; (C) the aggregate value (as last reported by Borrowers and Guarantors to Agent on the financial statements most recently delivered to Agent pursuant to Section 9.6) of the Intellectual Property to be sold shall not exceed $30,000,000 in any twelve (12) consecutive calendar month period; and (D) as of the date of such sale and after giving effect thereto, no Default or Event of Default shall exist or have occurred and be continuing.”

3. Encumbrances. Section 9.8 of the Loan Agreement is hereby amended by (a) deleting the period at the end of subsection (r) of such Section and replacing it with “; and”, and (b) adding at the end of such Section a new subsection as follows:

“(s) the security interests in and liens on assets of Perry Europe (but not any other Borrower or Guarantor) to secure the Indebtedness permitted under Section 9.9(s) hereof.”

4. Indebtedness. Section 9.9 of the Loan Agreement is hereby amended by deleting subsection (s) of such Section entirely and replacing it with the following:

“(s) Indebtedness of Perry Europe to Barclays Bank PLC or another financial institution acceptable to Agent (which Indebtedness may be guaranteed by Parent), provided, that, (i) in no event shall the aggregate outstanding principal amount of such Indebtedness or such guarantee exceed £4,500,000, (ii) such Indebtedness of Parent shall be unsecured, (iii) Agent shall have received true, correct and complete copies of all agreements, documents and instruments evidencing or otherwise relating to such Indebtedness, in form and substance satisfactory to Agent, as duly executed and delivered by the parties thereto, and (iii) Perry Europe and Parent shall furnish to Agent all notices or demands in connection with such Indebtedness either received by Perry Europe or Parent or on its behalf promptly after the receipt thereof, or concurrently with the sending thereof, as the case may be.”

5. Release of Liens on Assets of Perry Europe. Effective upon the Amendment No. 14 Effective Date, Agent hereby releases all liens and security interests held by Agent on the assets of Perry Europe. Following the Amendment No. 14 Effective Date and upon the request and at the expense of Borrowers, Agent shall execute and deliver such further documents and instruments as may be necessary to release the lien and security interest held by Agent on the assets of Perry Europe.

6. Amendment Fee. In consideration of this Amendment No. 14 and, in addition to, and not in limitation of, any other fee paid or payable to Agent under any of the Financing Agreements at any time, Borrowers shall, on the Amendment No. 14 Effective Date, pay to Agent, or Agent, at its option, may charge the loan account of Borrowers maintained by Agent, an amendment fee in the amount of $25,000, which fee shall be fully earned and payable as of the Amendment No. 14 Effective Date and shall constitute part of the Obligations

7. Representations, Warranties and Covenants. Borrowers and Guarantors, jointly and severally, represent, warrant and covenant with and to Agent and Lenders as follows, which representations, warranties and covenants shall survive the execution and delivery hereof:

(a) this Amendment No. 14 and all other documents, agreements and instruments executed by any Borrower or Guarantor in connection herewith (together with this Amendment No. 14, the “Amendment Documents”) have been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and are in full force and effect as of the date hereof, and the agreements and obligations of Borrowers and Guarantors contained herein and therein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (i) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (ii) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law);

(b) none of the Amendment Documents nor the transactions contemplated thereby are in contravention of any applicable law, or the terms of any agreement to which any Borrower or Guarantor is a party or by which any property of any Borrower or Guarantor is bound; and

(c) as of the date hereof, no Default or Event of Default exists or has occurred and is continuing.

8. Conditions Precedent. The terms and provisions of this Amendment No. 14 shall only be effective upon the satisfaction of each of the following conditions precedent in a manner satisfactory to Agent:

(a) Agent shall have received executed counterparts of this Amendment No. 14, duly authorized, executed and delivered by Borrowers and Guarantors and such Lenders as are required under the Loan Agreement to approve the transactions contemplated by this Amendment No. 14;

(b) Agent shall have received copies of all agreements, documents and instruments evidencing or otherwise relating to the Indebtedness and liens permitted under Sections 9.8(s) and 9.9(s) of the Loan Agreement as amended hereby, in form and substance satisfactory to Agent, duly executed and delivered by the parties thereto; and

(c) No Default or Event of Default shall exist or have occurred and be continuing.

9. Effect of this Amendment. This Amendment No. 14 and the other Amendment Documents constitute the entire agreement of the parties with respect to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly provided herein, no other changes or modifications to the Financing Agreements are intended or implied, and in all other respects the Financing Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 14, the provisions of this Amendment No. 14 shall control.

10. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent to effectuate the provisions and purposes of this Amendment No. 14.

11. Release of Claims. No Borrower or Guarantor has any actual or potential claim or cause of action against Agent or any Lender with respect to any matters relating to the Financing Agreements and related transactions through the date hereof, and hereby waives and releases any right to assert same.

12. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

13. Binding Effect. This Amendment No. 14 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

14. Counterparts. This Amendment No. 14 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 14, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 14 by telecopier or other method of electronic transmission shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 14. Any party delivering an executed counterpart of this Amendment No. 14 by telecopier or other method of electronic transmission also shall deliver an original executed counterpart of this Amendment No. 14, but the failure to deliver an original executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 14 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 14 to be duly executed and delivered by their authorized officers as of the day and year first above written.

Signature Page to Amendment No. 14

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

Signature Page to Amendment No. 14

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

Signature Page to Amendment No. 14

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

Signature Page to Amendment No. 14

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

AGREED:

WACHOVIA BANK, NATIONAL ASSOCIATION,

successor by merger to Congress Financial Corporation (Florida),

as Agent and a Lender

BANK OF AMERICA, N.A., as Syndication Agent and a Lender

THE CIT GROUP/COMMERCIAL SERVICES, INC.

ISRAEL DISCOUNT BANK OF NEW YORK

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

Signature Page to Amendment No. 14

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

HSBC BANK USA, NATIONAL ASSOCIATION

Signature Page to Amendment No. 14