WITNESSETH:

Exhibit 10.60

[EXECUTION]

AMENDMENT NO. 4 TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 4 TO LOAN AND SECURITY AGREEMENT, dated February 25, 2004, entered into by and among Congress Financial Corporation (Florida), a Florida corporation, in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, Inc., a Delaware corporation (“Supreme”), International, Inc., a Delaware corporation (“Jantzen”), Salant Corporation, a Delaware corporation (“Salant”), Salant Holding Corporation, a Delaware corporation (“Salant Holding”, and together with Supreme, Jantzen and Salant, each individually a “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel Corp., a Delaware corporation (“Jantzen Apparel”), BBI Retail, L.L.C., a Florida limited liability company (“BBI”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), and Perry Ellis Real Estate Corporation, a Delaware corporation (“PE Real Estate”, and together with Parent, PEI Licensing, Jantzen Apparel, BBI, Supreme I, Supreme II, Supreme Realty and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

WITNESSETH:

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1,2002, by and among Agent, Lenders, Borrowers and Guarantors as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19,2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003, Amendment No. 3 to Loan and Security Agreement dated December 1,2003 and as amended hereby (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Borrowers and Guarantors have requested that Lenders agree to increase the maximum amount available under the Loan Agreement and Lenders are willing to so agree, subject to the terms and conditions contained herein;

WHEREAS, by this Amendment No. 4, Agent, Lenders, Borrowers and Guarantors desire and intend to evidence such amendment;

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions.

(a) Additional Definitions. As used herein, the following terms shall have the meanings given to them below, and the Loan Agreement and the other Financing Agreements are hereby amended to include, in addition and not in limitation, the following definitions:

(i) “Amendment No. 4” shall mean this Amendment No. 4 to Loan and Security Agreement by and among Agent, Lenders, Borrowers and Guarantors, as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

(ii) “International Trademark Interests” shall mean the equitable and beneficial interests of a Borrower or Guarantor in certain trademarks that are registered in jurisdictions outside the United States of America and certain related rights as described on Exhibit A hereto.

(b) Amendments to Definitions.

(i) Effective February 23, 2004, all references to the term “Maximum Credit” in the Loan Agreement shall be deemed and each such reference is hereby amended to mean $130,000,000, provided, that, the term “Maximum Credit” shall mean $110,000,000 effective on and after the earlier of (A) June 7, 2004, and (B) the date of: (1) the receipt by Agent of the written request of Borrowers to reduce the Maximum Credit to such amount and (2) the payment of any amounts required to be repaid pursuant to Section 2 below in connection with such reduction.

(ii) All references to the term “Commitment” in the Loan Agreement or any of the other Financing Agreements shall mean, at any time, as to each Lender the principal amount set forth on Exhibit B hereto as to such Lender for the periods specified therein or on Schedule 1 to the Assignment and Acceptance Agreement entered into after the date hereof pursuant to which such Lender becomes a Lender hereunder or otherwise increases the amount of its Commitment after the date hereof, in each case as the same may be adjusted from time to time in accordance with the terms hereof; sometimes being collectively referred to in the Loan Agreement and the other Financing Agreements as “Commitments”.

(c) Interpretation. For purposes of this Amendment No. 4, unless otherwise defined herein, all capitalized terms used herein which are defined in the Loan Agreement shall have the meanings given to such terms in the Loan Agreement.

2. Mandatory Prepayment. Unless sooner demanded by Agent in accordance with the terms of the Loan Agreement or the other Financing Agreements, on the date of any reduction in the Maximum Credit, Borrowers shall absolutely and unconditionally, automatically and without notice or demand, make a payment to Agent in respect of the Loans outstanding in an amount equal to the excess, if any, of the aggregate unpaid principal amount of the Loans over the

Maximum Credit as so reduced in immediately available funds, together with all interest in respect thereof.

3. Collection of Accounts. Section 6.3(a)(ii) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor: “(ii) Excess Availability is less than $35,000,000, provided, that, for purposes of this clause (ii), so long as the Maximum Credit is $130,000,000, Excess Availability shall be calculated solely on the basis of the Borrowing Base of each Borrower without regard to the Loan Limit for such Borrower and at all times thereafter shall be calculated as provided in the definition of such term contained herein.”

4. Transfers of International Trademark Interests. Section 9.7(b)(x) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(x) the transfer of International Trademark Interests owned by any Borrower or Guarantor (including Salant or Salant Holdings, or any entity that subsequently becomes a Borrower or Guarantor), to Perry Ellis International Group Holdings Limited, a corporation organized under the laws of Ireland and having its principal place of business in The Commonwealth of the Bahamas, or any other wholly-owned direct or indirect Subsidiary of Parent that is not organized under the laws of the United States of America or Canada, provided, that, (A) as to any such transfer: (1) Agent shall have received true, correct and complete copies of all agreements relating to such transfer and such other information with respect thereto as Agent may reasonably request from time to time, (2) in no event shall such arrangements limit or impair the rights of any Borrower or Guarantor (or Agent) to use the trademarks subject to such transfer (except in jurisdictions outside the United States of America or Canada) or create any material liabilities of any Borrower or Guarantor in connection with such transfer, (3) as of the date of any such transfer and after giving effect thereto, no Default or Event of Default shall exist or have occurred and be continuing, and (4) such transfer shall be permitted or not prohibited under the Senior Note Indenture and Borrowers and Guarantors shall have complied with all requirements, if any, with respect to such transfer provided for therein and such transfer shall not result in the breach of, or constitute a default under, any indenture, agreement or instrument to which any Borrower or Guarantor is a party or by which it or its assets may be bound, and (B) upon the satisfaction of each of the conditions set forth in the clause (A) of this Section 9.7(b)(x) above as to any such transfer, upon the written request of Borrower Agent, Agent shall, at Borrower’s expense, execute and deliver to Borrower Agent a release instrument with respect to the International Trademark Interests to be so transferred in form and substance reasonably satisfactory to Agent, provided, that, (1) the release pursuant to such release instrument shall only be effective so long as any other party with a security interest, lien or encumbrance with respect to such International Trademark Interests to be so transferred has released and terminated such interests, (2) Agent shall have received evidence that Senior Note Trustee, if the Senior Note Trustee has a security interest therein, and any other party with a security interest therein has on or before the effectiveness of the release by Agent of such security interest unconditionally executed and delivered to Borrower Agent a release instrument with respect thereto, (3) such release by Agent shall only be effective upon the Business Day immediately prior to, and to the extent of, the valid and enforceable transfer of the applicable International Trademark Interests by such Borrower

or Guarantor to Perry Ellis International Group Holdings Limited and (4) if any such transfer is not effective, and if Agent shall not have received evidence of the effectiveness of any such transfer, within ten (10) Business Days after the delivery be Agent of such release instrument to Borrower Agent, then Borrower Agent shall promptly return such release instrument to Agent and the release provided for therein shall be null and void ab initio and shall not be nor have ever been of any force and effect;”

5. Amendment Fee. Borrowers shall pay to Agent, for the account of Lenders (in accordance with the arrangements between Agent and Lenders), an amendment fee: (a) on the date hereof in the amount of $20,000 which shall be fully earned as of the date hereof, and (b) commencing on March 23, 2004, and every thirty (30) days thereafter an additional $20,000 for each such thirty (30) day period (or any part thereof) that the Maximum Credit is greater than $110,000,000 or there are any Loans outstanding in excess of the Maximum Credit (after giving effect to any reduction thereto), without limiting any other rights of Agent and Lenders.

6. Representations, Warranties and Covenants. Borrowers and Guarantors jointly and severally represent, warrant and covenant with and to Agent and Lenders that this Amendment No. 4 has been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and is in full force and effect as of the date hereof, and the agreements and obligations of Borrowers and Guarantors contained herein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (a) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (b) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law).

7. Conditions Precedent. The effectiveness of the consent and amendments contained herein shall only be effective upon the satisfaction of each of the following conditions precedent in a manner satisfactory to Agent:

(a) Agent shall have received an executed original or executed original counterparts of this Amendment No. 4, duly authorized, executed and delivered by the respective party or parties hereto;

(b) Agent shall have received, in form and substance satisfactory to Agent, the written consent of Lenders to this Amendment No. 4, duly authorized, executed and delivered by such Lenders.

8. Effect of this Amendment. This Amendment No. 4 and the instruments and agreements delivered pursuant hereto constitute the entire agreement of the parties with respect to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly amended pursuant hereto and except for the amendments expressly contained herein, no other changes or modifications or waivers to the Financing Agreements are intended or implied, and in all other respects the Financing

Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 4, the provisions of this Amendment No. 4 shall control.

9. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent or Lenders to effectuate the provisions and purposes of this Amendment No. 4.

10. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

11. Binding Effect. This Amendment No. 4 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

12. Counterparts. This Amendment No. 4 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 4, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 4 by telecopier shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 4. Any party delivering an executed counterpart of this Amendment No. 4 by telecopier also shall deliver an original executed counterpart of this Amendment No. 4, but the failure to deliver an original executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 4 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 4 to be duly executed and delivered by their authorized officers as of the day and year first above written.

EXHIBIT A

TO

AMENDMENT NO. 4 TO LOAN AND SECURITY AGREEMENT

The term “International Trademark Interests” as used in Amendment No. 4 shall mean, collectively, the following:

(a) All of the equitable and beneficial interests of a Borrower or Guarantor in and to the trademarks owned by such Borrower or Guarantor identified by such Borrower or Guarantor to Agent in the information provided by such Borrower or Guarantor to Agent in accordance with Section 9.7(b)(x) of the Loan Agreement in connection with the transfer of such interests permitted thereunder to the extent such trademarks are registered and used outside the United States of America, its territories and possessions (collectively, the “International Trademarks”), provided, that, the term International Trademarks does not include any trademarks to the extent registered or used in the United States of America, its territories or possessions or the legal title and ownership of any Borrower or Guarantor of such trademarks;

(b) all trade dress, logos, slogans, designs, domain names, trade names, business names, corporate names and other source identifiers, whether or not registered, whether currently in use or not, incorporated in or part of the International Trademarks, including, without limitation, all common law rights and all rights therein provided by international treaties or conventions, together in each case with the goodwill of the business connected with and symbolized by any of the foregoing, and all rights, including the right to grant additional licenses thereof, corresponding thereto throughout the world, excluding the United States of America, its territories and possessions, and all other rights of any kind whatsoever accruing thereunder or pertaining thereto;

(c) all agreements, permits, consents, orders and franchises, in each case relating exclusively and directly to the International Trademarks and all proceeds, income, royalties and other payments now or hereafter due and/or payable with respect thereto, subject, in each case, to the terms of such agreements, permits, authorizations and franchises; and

(d) all proceeds of the International Trademarks.

Provided, that, in no event shall the term “International Trademark Interests” be construed to include or be deemed to mean (a) any of the trademarks of any Borrower or Guarantor to the extent registered or used in the United States of America, its territories or possessions or (b) the legal title and ownership of any Borrower or Guarantor of such trademarks.

EXHIBIT B

TO

AMENDMENT NO. 4 TO LOAN AND SECURITY AGREEMENT

The term “Reduction Date” shall mean the earlier of (A) June 7, 2004, and (B) the date of: (1) the receipt by Agent of the written request of Borrowers to reduce the Maximum Credit to $110,000,000 and (2) the payment of any amounts required to be repaid pursuant to Section 2 below in connection with such reduction.