AMENDMENT NO. 3 TO LOAN AND SECURITY AGREEMENT

Exhibit 10.59

AMENDMENT NO. 3 TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 3 TO LOAN AND SECURITY AGREEMENT, dated as of December 1, 2003, entered into by and among Congress Financial Corporation (Florida), a Florida corporation, in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, Inc., a Delaware corporation (“Supreme”), Jantzen, Inc., a Delaware corporation (“Jantzen”), Salant Corporation, a Delaware corporation (“Salant”), Salant Holding Corporation, a Delaware corporation (“Salant Holding”, and together with Supreme, Jantzen and Salant, each individually a “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PET Licensing”), Jantzen Apparel Corp., a Delaware corporation (“Jantzen Apparel”), BBI Retail, L.L.C., a Florida limited liability company (“BBI”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), and Perry Ellis Real Estate Corporation, a Delaware corporation (“PE Real Estate”, and together with Parent, PEI Licensing, Jantzen Apparel, BBI, Supreme I, Supreme II, Supreme Realty and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

WITNESSETH:

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1,2002, by and among Agent, Lenders, Borrowers and Guarantors as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19, 2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003 and as amended hereby (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Borrowers and Guarantors have entered into new Interest Rate Protection Agreements with a party that is not an affiliate of Agent;

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders agree to amend the Loan Agreement to reflect such new Interest Rate Protection Agreements; and

WHEREAS, by this Amendment No. 3, Agent, Lenders, Borrowers and Guarantors desire and intend to evidence such amendments.

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions

(a) Additional Definitions. As used herein, the term “Amendment No. 3” shall mean this Amendment No. 3 to Loan and Security Agreement as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced, and the Loan Agreement and the other Financing Agreements are hereby amended to include, in addition and not in limitation, such definition.

(b) Interpretation. For purposes of this Amendment No. 3, unless otherwise defined herein, all capitalized terms used herein which are defined in the Loan Agreement shall have the meanings given to such terms in the Loan Agreement.

4. Payments. Section 6.4(a) of the Loan Agreement is hereby amended to delete the clause “fourth” contained therein and to substitute the following therefor:

“fourth, to pay principal in respect of the Loans and to pay or prepay Obligations arising under or pursuant to any Interest Rate Protection Agreements of a Borrower or Guarantors with an Affiliate of Agent or another financial institution acceptable to Agent in accordance with the terms of the arrangements with such Affiliate or other financial institution (up to the amount of any then effective Reserve established in respect of such Obligations arising under or pursuant to any Interest Rate Protection Agreements);”

5. Representations, Warranties and Covenants. Borrowers and Guarantors jointly and severally represent, warrant and covenant with and to Agent and Lenders that this Amendment No. 3 has been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and is in full force and effect as of the date hereof, and the agreements and obligations of Borrowers and Guarantors contained herein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (a) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (b) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law).

2

6. Conditions Precedent. The effectiveness of the consent and amendments contained herein shall only be effective upon the satisfaction of each of the following conditions precedent in a manner satisfactory to Agent:

6.1 Agent shall have received an executed original or executed original counterparts of this Amendment No. 3, duly authorized, executed and delivered by the respective party or parties hereto;

6.2 Agent shall have received, in form and substance satisfactory to Agent, the written consent of Required Lenders consenting to this Amendment No. 3, duly authorized, executed and delivered by such Lenders.

7. Effect of this Amendment. This Amendment No.3 and the instruments and agreements delivered pursuant hereto constitute the entire agreement of the parties with respect to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly amended pursuant hereto and except for the amendments expressly contained herein, no other changes or modifications or waivers to the Financing Agreements are intended or implied, and in all other respects the Financing Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 3, the provisions of this Amendment No. 3 shall control.

8. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent or Lenders to effectuate the provisions and purposes of this Amendment No. 3.

9. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

10. Binding Effect. This Amendment No. 3 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

11. Counterparts. This Amendment No. 3 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 3, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 3 by telecopier shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 3. Any party delivering an executed counterpart of this Amendment No. 3 by telecopier also shall deliver an original executed counterpart of this Amendment No.3, but the failure to deliver an original

3

executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 3 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

4

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 3 to be duly executed and delivered by their authorized officers as of the day and year first above written.

5

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

6

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

7