AMENDMENT NO. 10 TO LOAN AND SECURITY AGREEMENT

Exhibit 10.84

[Execution]

AMENDMENT NO. 10 TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 10 TO LOAN AND SECURITY AGREEMENT, dated as of August 28, 2006, entered into by and among Wachovia Bank, National Association, successor by merger to Congress Financial Corporation (Florida), in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, LLC, a Delaware limited liability company formerly known as Supreme International, Inc. (“Supreme”), Jantzen, LLC, a Delaware limited liability company formerly known as Jantzen, Inc. (“Jantzen”), Perry Ellis Menswear, LLC, a Delaware limited liability company formerly known as Perry Ellis Menswear, Inc. (“Perry Ellis Menswear”), Perry Ellis Europe Limited, formerly known as Farah Manufacturing (U.K.) Limited, a private limited company incorporated in England and Wales (“Perry Europe”), Salant Holding, LLC, a Delaware limited liability company formerly known as Salant Holding Corporation (“Salant Holding” and together with Supreme, Jantzen, Perry Europe and Perry Ellis Menswear, each individually “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel, LLC, a Delaware limited liability company formerly known as Jantzen Apparel Corp. (“Jantzen Apparel”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, a Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), Perry Ellis Shared Services Corporation, a Delaware corporation (“PE Shared Services”), Winnsboro DC, LLC, a Delaware limited liability company (“Winnsboro”), Tampa DC, LLC, a Delaware limited liability company (“Tampa DC”), Perry Ellis International Group Holdings Limited, a private company incorporated under the laws of Ireland having its principal place of business in the Bahamas (“Group Holdings”) and Perry Ellis Real Estate, LLC, a Delaware limited liability company formerly known as Perry Ellis Real Estate Corporation (“PE Real Estate” and, together, with Parent, PEI Licensing, Jantzen Apparel, Supreme I, Supreme II, Supreme Realty, Group Holdings, PE Shared Services, Winnsboro, Tampa DC, and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

W I T N E S S E T H :

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1, 2002, by and among Agent, Lenders, Borrowers and Guarantors, as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19, 2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003, Amendment No. 3 to Loan and Security Agreement, dated December 1, 2003, Amendment No. 4 to Loan and Security Agreement, dated February 25, 2004, Amendment No. 5 to Loan and

1

Security Agreement, dated July 1,2004, Amendment No. 6 to Loan and Security Agreement, dated as of September 30, 2004, Amendment No. 7 to Loan and Security Agreement, dated as of February 26, 2005, Amendment No. 8 to Loan and Security Agreement, dated as of September 30, 2005 and Amendment No. 9 to Loan and Security Agreement, dated as of February 24, 2006 (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders extend the term of the Loan Agreement and agree to make certain other amendments to the Loan Agreement, and Agent and Lenders are willing to do so, subject to the terms and conditions set forth in this Amendment No. 10; and

WHEREAS, by this Amendment No. 10, Agent, Lenders, Borrowers and Guarantors desire and intend to evidence such extension and amendments;

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions.

1.1 Additional Definition. As used herein, the term “Amendment No. 10” shall mean Amendment No. 10 to Loan and Security Agreement by and among Agent, Lenders, Borrowers and Guarantors, as the same, now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced, and the Loan Agreement and the other Financing Agreements are hereby amended to include, in addition and not in limitation, the following definitions:

1.2 Interpretation. For purposes of this Amendment No. 10, unless otherwise defined herein, all capitalized terms used herein which are defined in the Loan Agreement shall have the meanings given to such terms in the Loan Agreement.

2. Term. The first sentence of Section 13.1 (a) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(a) This Agreement and the other Financing Agreements shall become effective as of the date set forth on the first page hereof and shall continue in full force and effect for a term ending on January 1, 2008 (the “Renewal Date”) and from year to year thereafter, unless sooner terminated pursuant to the terms hereof; provided, that, except as Agent and Borrower Agent may otherwise agree, in the event that the term of this Agreement shall continue for any additional year after January 1, 2008, Borrowers and Guarantors shall pay to Agent an extension fee in the amount of one-sixth (1/6%) percent of the Maximum Credit which fee shall be earned and payable in full on January 1, 2008 in respect of the extension for each year after the Renewal Date.”

2

3. Representations, Warranties and Covenants. Borrowers and Guarantors, jointly and severally, represent, warrant and covenant with and to Agent and Lenders as follows, which representations, warranties and covenants shall survive the execution and delivery hereof:

3.1 this Amendment No. 10 has been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and is in full force and effect as of the date hereof, and the agreements and obligations of Borrowers and Guarantors contained herein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (i) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (ii) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law);

3.2 neither this Amendment No. 10 nor the transactions contemplated hereby are in contravention of any applicable law, or the terms of any agreement to which any Borrower or Guarantor is a party or by which any property of any Borrower or Guarantor is bound; and

3.3 as of the date hereof, no Default or Event of Default exists or has occurred and is continuing.

4. Conditions Precedent. The effectiveness of the amendments contained herein shall only be effective upon the satisfaction of each of the following conditions precedent in a manner satisfactory to Agent:

4.1 Agent shall have received executed counterparts of this Amendment No. 10, duly authorized, executed and delivered by Borrowers, Guarantors and the Lenders; and

4.2 No Default or Event of Default shall exist or have occurred and be continuing.

5. Effect of this Amendment. This Amendment No. 10 and the instruments and agreements delivered pursuant hereto (if any) constitute the entire agreement of the parties with respect to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly amended pursuant hereto, no other changes or modifications to the Financing Agreements are intended or implied, and in all other respects the Financing Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 10, the provisions of this Amendment No. 10 shall control.

3

6. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent to effectuate the provisions and purposes of this Amendment No. 10.

7. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

8. Binding Effect. This Amendment No. 10 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

9. Counterparts. This Amendment No. 10 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 10, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 10 by telecopier or other method of electronic transmission shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 10. Any party delivering an executed counterpart of this Amendment No. 10 by telecopier or other method of electronic transmission also shall deliver an original executed counterpart of this Amendment No. 10, but the failure to deliver an original executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 10 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

4

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 10 to be duly executed and delivered by their authorized officers as of the day and year first above written.

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

AGREED:

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

AGREED:

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

AGREED:

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]