AMENDMENT NO. 2 TO LOAN AND SECURITY AGREEMENT

Exhibit 10.58

AMENDMENT NO. 2 TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 2 TO LOAN AND SECURITY AGREEMENT, dated September 22, 2003, entered into by and among Congress Financial Corporation (Florida), a Florida corporation, in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, Inc., a Delaware corporation (“Supreme”), Jantzen, Inc., a Delaware corporation (“Jantzen”), Salant Corporation, a Delaware corporation (“Salant”), Salant Holding Corporation, a Delaware corporation (“Salant Holding”, and together with Supreme, Jantzen and Salant, each individually a “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel Corp., a Delaware corporation (“Jantzen Apparel”), BBI Retail, L.L.C., a Florida limited liability company (“BBI”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), and Perry Ellis Real Estate Corporation, a Delaware corporation (“PE Real Estate”, and together with Parent, PEI Licensing, Jantzen Apparel, BBI, Supreme I, Supreme II, Supreme Realty and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

W I T N E S S E T H :

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1, 2002, by and among Agent, Lenders, Borrowers and Guarantors as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19, 2003, and as amended hereby (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Parent intends to issue new 8 ⁷/8% Senior Subordinated Notes due 2013, the proceeds of which are to be used to, among other things, repay all of the obligations of Parent evidenced by or arising under the Existing Subordinated Notes (as hereinafter defined);

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders agree to amend the Loan Agreement to permit the issuance of such new notes and the repayment of the Existing Subordinated Notes and make certain other amendments to the Loan Agreement; and

WHEREAS, by this Amendment No. 2, Agent, Lenders, Borrowers and Guarantors desire and intend to evidence such amendments.

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions

1.1 Additional Definitions. As used herein, the following terms shall have the meanings given to them below, and the Loan Agreement and the other Financing Agreements are hereby amended to include, in addition and not in limitation, the following definitions:

(a) “Amendment No. 2” shall mean this Amendment No. 2 to Loan and Security Agreement as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

(b) “Existing Note Payoff Account” shall mean account number                          maintained at Wachovia Bank, National Association in the name of Parent, which account shall only hold proceeds of the issuance of the Subordinated Notes payable to Parent and shall have been established solely for such purpose.

(c) “Existing Subordinated Note Agreements” shall mean, collectively, (i) the Existing Subordinated Notes, (ii) the Existing Subordinated Note Indenture and (iii) all other agreements, documents and instruments related thereto.

(d) “Existing Subordinated Note Guarantors” shall mean, collectively, the following (together with their respective successors and assigns) to the extent that each has guaranteed the Indebtedness of Parent under the Existing Subordinated Notes: (i) Supreme Canada, (ii) Supreme International Corporation de Mexico, S.A. de C.V., a Mexico corporation, (iii) BBI, (iv) Jantzen Apparel, (v) PEI Licensing, (vi) Supreme, (vii) Jantzen, (vii) Supreme I, (viii) Supreme II, (ix) Supreme Realty, (x) PE Real Estate, (xi) Salant and (xii) Salant Holding.

(e) “Existing Subordinated Note Indenture” shall mean the Indenture, dated April 6, 1999, by and between Parent, as issuer, the Existing Subordinated Note Guarantors, as subsidiary guarantors, and the Existing Subordinated Note Trustee, as trustee, as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

(f) “Existing Subordinated Notes” shall mean, collectively, the 12¼% Series B Senior Subordinated Notes due 2006 in the original principal amount of $100,000,000, as the same now exist or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

(g) “Existing Subordinated Note Trustee” shall mean US Bank National Association, formerly known as State Street Bank and Trust Company, in its capacity as trustee pursuant to the Existing Subordinated Note Indenture.

2

(h) “Subordinated Note Agreements” shall mean, collectively, (i) the Subordinated Notes, (ii) the Subordinated Note Indenture and (iii) all other agreements, documents and instruments related thereto.

1.2 Amendments to Definitions.

(a) All references to the term “Subordinated Note Guarantors” in the Loan Agreement or any of the other Financing Agreements and each such reference is hereby amended to mean, collectively, the following (together with their respective successors and assigns): (i) Supreme, (ii) Jantzen, (iii) Salant, (iv) Salant Holding, (v) PEI Licensing, (vi) Jantzen Apparel, (vii) BBI, (viii) Supreme I, (ix) Supreme II, (x) Supreme Realty and (xi) PE Real Estate.

(b) All references to the term “Subordinated Note Indenture” in the Loan Agreement or any of the other Financing Agreements and each such reference is hereby amended to mean the Indenture, dated September 22, 2003, by and between Parent, as issuer, the Subordinated Note Guarantors, as Subsidiary Guarantors, and US Bank National Association, Inc., as trustee, as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

(c) All references to the term “Subordinated Notes” in the Loan Agreement or any of the other Financing Agreements and each such reference is hereby amended to mean, collectively, the 8 ⁷/8% Senior Subordinated Notes due 2013 in the original principal amount of $150,000,000, as the same now exist or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

(d) All references to the term “Financing Agreements” in the Loan Agreement or any of the other Financing Agreements and each such reference is hereby amended to include, in addition and not in limitation, collectively, this Amendment No. 2.

1.3 Interpretation. For purposes of this Amendment No. 2, unless otherwise defined herein, all capitalized terms used herein which are defined in the Loan Agreement shall have the meanings given to such terms in the Loan Agreement.

2. Indebtedness. Section 9.9(g)(i) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(i) the aggregate amount of such Indebtedness shall not exceed $150,000,000, less the aggregate amount of all repayments or redemptions, whether optional or mandatory, in respect thereof, plus interest thereon at the rate provided for in the Subordinated Notes as in effect on the date hereof,”

3. Intentionally deleted.

3

4. Consent. Subject to the terms and conditions contained herein, to the extent such consent is or may be required under the Loan Agreement, Agent and Lenders hereby consent to the issuance by Parent of the Subordinated Notes and the repayment in full of all of the Indebtedness and other obligations of Parent under the Existing Subordinated Notes in accordance with the terms contained herein.

5. Representations, Warranties and Covenants. Borrowers and Guarantors jointly and severally represent, warrant and covenant with and to Agent and Lenders as follows, which representations, warranties and covenants shall survive the execution and delivery hereof:

5.1 This Amendment No. 2 has been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and is in full force and effect as of the date hereof, and the agreements and obligations of Borrowers and Guarantors contained herein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (i) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (ii) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law).

5.2 No action of, or filing with, or consent of any Governmental Authority and no approval or consent of any other party, is required to authorize, or is otherwise required in connection with, the execution, delivery and performance of this Amendment No. 2.

5.3 None of the transactions contemplated by this Amendment No. 2 are in contravention of any applicable law, or the terms of any agreement to which any Borrower or Guarantor is a party or by which any property of any Borrower or Guarantor is bound.

5.4 The Subordinated Note Agreements and the transactions contemplated thereby have been duly executed, delivered and performed in accordance with their terms, including the fulfillment (not the waiver, except as disclosed and consented to by Agent) of all conditions precedent set forth therein.

5.5 On or before September 16, 2003, Borrowers and Guarantors shall have issued or caused to be issued a redemption notice with respect to the repayment in full of all of the Indebtedness and other obligations of Borrowers and Guarantors evidenced by or arising under the Existing Subordinated Notes and Borrowers and Guarantors shall provide to Agent on or before such date evidence, in form and substance satisfactory to Agent, of same. Such redemption notice shall provide for all of the Existing Subordinated Notes to be repaid in full on or before October 22, 2003 (the “Existing Subordinated Note Payoff Date”). A true, correct and complete copy of such notice shall have been delivered to Agent on or before the date of the issuance of such redemption notice. On or before October 22, 2003, Borrowers and Guarantors shall cause all of such Indebtedness and other obligations to be repaid in full with the portion of the proceeds payable to Parent from the issuance of the Subordinated Notes that have been deposited into the Existing Note Payoff Account and Borrowers and Guarantors shall provide to Agent on or before such date evidence, in form and substance satisfactory to Agent, that (a) all Indebtedness and other obligations of Borrowers and Guarantors evidenced by or arising under

4

the Existing Subordinated Notes and the other Existing Subordinated Note Agreements have been duly and validly repaid and satisfied in full in accordance with the terms of the Existing Subordinated Note Indenture and the other Existing Subordinated Note Agreements using proceeds of the issuance of the Subordinated Notes that have been deposited in the Existing Note Payoff Account and (b) Parent and the Existing Subordinated Note Guarantors have been released from all liability and obligations in connection with the Existing Subordinated Note Indenture and the other Existing Subordinated Note Agreements therewith.

5.6 On or before September 19, 2003, Parent shall cause all amounts payable to it with the proceeds from the issuance of the Subordinated Notes to be remitted to the Existing Note Payoff Account, other than those proceeds paid to Agent for application to the Obligations, and Borrowers and Guarantors shall provide to Agent on or before such date evidence, in form and substance satisfactory to Agent, of same. No other funds shall be held in the Existing Note Payoff Account and all of the funds held therein shall only be used to repay the Indebtedness and other obligations of Borrowers and Guarantors evidenced by or arising under the Existing Subordinated Note Indenture and the other Existing Subordinated Note Agreements on or before October 22, 2003. The amounts so paid to the Existing Note Payoff Account shall be sufficient to repay all of the Indebtedness and other obligations of Borrowers and Guarantors evidenced by or arising under the Existing Subordinated Note Indenture and the other Existing Subordinated Note Agreements and Borrowers and Guarantors shall provide to Agent evidence, in form and substance satisfactory to Agent, of same. All funds so deposited into the Existing Note Payoff Account shall at all times be held in such account free and clear of any security interests, liens, pledge, encumbrances or claims of any kind or nature.

5.7 The Subordinated Notes have been duly authorized, issued and delivered by Parent pursuant to the Subordinated Note Indenture and the other Subordinated Note Agreements, and the transactions contemplated thereunder have been performed in accordance with their terms by the respective parties thereto in all respects, including the fulfillment (not merely the waiver) of all conditions precedent set forth therein.

5.8 All actions and proceedings required by the Subordinated Notes and the other Subordinated Note Agreements, applicable law or regulations, have been taken, and the transactions required thereunder have been (or will be when required to under the Subordinated Note Agreements or applicable law or regulations) duly and validly taken and consummated.

5.9 The execution, delivery and performance by Borrowers and Guarantors of the Subordinated Note Agreements and the consummation of the transactions therein contemplated, and compliance with the provisions thereof (a) are all within each Borrower’s and Guarantor’s corporate or limited liability company powers, (b) have been duly authorized by each Borrower and Guarantor, (c) do not and will not violate any law or regulation or any order or decree of any court or other Governmental Authority in any respect, (d) do not or shall not conflict with or result in the breach of, or constitute a default in any respect under, any indenture, mortgage, deed of trust, security agreement, agreement or instrument to which any Borrower or Guarantor is a party or may be bound, (e) will not result in the creation or imposition of, or require or give rise to any obligation to grant, any lien, security interest, charge or other encumbrance upon any property of any Borrower or Guarantor or (f) violate any provision of the certificate of

5

incorporation, certificate of formation, by-laws or other organizational documents of any Borrower or Guarantor.

5.10 All payments that are required or anticipated to be made on the Existing Subordinated Note Payoff Date and the names of the payees to be paid using all of the proceeds of the Subordinated Notes contemplated by the Subordinated Note Agreements are set forth on Schedule 5.9 hereto.

5.11 No court of competent jurisdiction has issued any injunction, restraining order or other order which prohibits consummation of the issuance of the Subordinated Notes and the transactions related to the other Subordinated Note Agreements and no governmental or other action or proceeding has been threatened or commenced, seeking any injunction, restraining order or other order which seeks to avoid or otherwise modify the issuance of the Subordinated Notes or any of the other Subordinated Note Agreements and the transactions related thereto.

5.12 Agent has, on or before the date hereof, received from Borrowers, true, complete and correct copies of the Subordinated Note Agreements, and all notices, instruments, documents and agreements related thereto, including all exhibits and schedules thereto.

6. Conditions Precedent. The effectiveness of the consent and amendments contained herein shall only be effective upon the satisfaction of each of the following conditions precedent in a manner satisfactory to Agent:

6.1 Agent shall have received an executed original or executed original counterparts of this Amendment No. 2 (including all schedules and exhibits hereto), duly authorized, executed and delivered by the respective party or parties hereto;

6.2 Agent shall have received, in form and substance satisfactory to Agent, true, correct and complete copies of all of the Subordinated Note Agreements, duly executed, authorized and delivered by each of the parties thereto, which shall in all respects be in form and substance satisfactory to Agent;

6.3 Agent shall have received not less than $41,000,000 from the proceeds of the issuance of the Subordinated Notes for application to the Obligations in accordance with the terms of the Loan Agreement;

6.4 No Default or Event of Default shall exist or have occurred and be continuing;

6.5 Agent shall have received, in form and substance satisfactory to Agent, the written consent of Required Lenders consenting to this Amendment No. 2, duly authorized, executed and delivered by such Lenders; and

6.6 Each of the foregoing conditions set forth in this Section 6 shall have been satisfied by no later than September 21, 2003, as determined by Agent.

7. Effect of this Amendment. This Amendment No.2 and the instruments and agreements delivered pursuant hereto constitute the entire agreement of the parties with respect

6

to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly amended pursuant hereto and except for the amendments expressly contained herein, no other changes or modifications or waivers to the Financing Agreements are intended or implied, and in all other respects the Financing Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 2, the provisions of this Amendment No. 2 shall control.

8. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent or Lenders to effectuate the provisions and purposes of this Amendment No. 2.

9. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

10. Binding Effect. This Amendment No. 2 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

11. Counterparts. This Amendment No. 2 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 2, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 2 by telecopier shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 2. Any party delivering an executed counterpart of this Amendment No. 2 by telecopier also shall deliver an original executed counterpart of this Amendment No. 2, but the failure to deliver an original executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 2 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

7

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 2 to be duly executed and delivered by their authorized officers as of the day and year first above written.

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

8

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

9