AMENDMENT NO. 5 TO THE SENIOR CREDIT FACILITY

Exhibit 10.78

[Execution]

AMENDMENT NO. 5 TO THE SENIOR CREDIT FACILITY

AMENDMENT NO. 5 TO LOAN AND SECURITY AGREEMENT, dated as of July 1, 2004, entered into by and among Congress Financial Corporation (Florida), a Florida corporation, in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, Inc., a Delaware corporation (“Supreme”), Jantzen, Inc., a Delaware corporation (“Jantzen”), Salant Corporation, a Delaware corporation (“Salant”), Salant Holding Corporation, a Delaware corporation (“Salant Holding”, and together with Supreme, Jantzen and Salant, each individually a “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel Corp., a Delaware corporation (“Jantzen Apparel”), BBI Retail, L.L.C., a Florida limited liability company (“BBI”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), and Perry Ellis Real Estate Corporation, a Delaware corporation (“PE Real Estate”, and together with Parent, PEI Licensing, Jantzen Apparel, BBI, Supreme I, Supreme II, Supreme Realty and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

W I T N E S S E T H :

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1, 2002, by and among Agent, Lenders, Borrowers and Guarantors as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19, 2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003, Amendment No. 3 to Loan and Security Agreement dated December 1, 2003, Amendment No. 4 to Loan and Security Agreement dated February 25, 2004 and as amended hereby (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Borrowers and Guarantors have advised Agent and Lenders that (i) Salant will be changing its name to Perry Ellis Menswear, Inc.; (ii) Borrowers and Guarantors intend to amend the amount of their lines of credit under the Letter of Credit Facility Agreements with the Letter of Credit Issuers (although the aggregate amount of such lines of credit will continue to be limited to $60,000,000 as is currently permitted under the Loan Agreement); and (iii) Borrowers

and Guarantors intend to merge Salant Holding with and into Supreme, with Supreme as the surviving corporation;

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders consent to each of such actions contemplated by them and Agent and Lenders are willing to so consent, subject to the terms and conditions set forth in this Amendment No. 5;

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions.

(a) Additional Definitions. As used herein, the term “Amendment No. 5” shall mean this Amendment No. 5 to Loan and Security Agreement by and among Agent, Lenders, Borrowers and Guarantors, as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced and the Loan Agreement and the other Financing Agreements are hereby amended to include, in addition and not in limitation, such definition.

(b) Interpretation. For purposes of this Amendment No. 5, unless otherwise defined herein, all capitalized terms used herein which are defined in the Loan Agreement shall have the meanings given to such terms in the Loan Agreement.

2. Consent to Name Change. Agent and Lenders hereby consent to the change of the name of Salant Corporation to Perry Ellis Menswear, Inc. and waive the requirement under Section 9.1(b)(i) of the Loan Agreement that Agent receive not less than thirty (30) days prior written notice of such change, provided, that, Agent shall receive a copy of the amendment to the certificate of incorporation of Salant, as certified by the Secretary of State of the jurisdiction of incorporation of Salant, evidencing that the name change has been filed with such Secretary of State, as soon as it is available.

3. Consent to Amendments to Letter of Credit Facility Agreements. Subject to the terms and conditions set forth herein, Agent and Lenders hereby consent to (a) the amendment of the Letter of Credit Facility Agreements of Salant, Salant Holding, Supreme and Jantzen with HSBC Bank USA to increase the maximum amount of the obligations of all of such Borrowers to HSBC Bank USA and its affiliates under such agreements to $30,000,000; (b) the amendment of the Letter of Credit Facility Agreements of Salant, Salant Holding, Supreme and Jantzen with CommerceBank, N.A. to decrease the maximum amount of the obligations of all of such Borrowers to CommerceBank, N.A. under such agreements to $22,000,000; and (c) the amendment of the Letter of Credit Facility Agreements of Salant, Salant Holding, Supreme and Jantzen with Israel Discount Bank of New York to decrease the maximum amount of the obligations of all of such Borrowers to Israel Discount Bank of New York under such agreements to $8,000,000. The effectiveness of the consent of Agent and Lenders set forth in this Section 3 is subject to the receipt by Agent of true, correct and complete copies of all of the agreements related to such amendments as executed by each of the parties thereto, which agreements shall be in form and substance reasonably satisfactory to Agent.

2

4. Merger of Salant Holding with and into Supreme. Section 9.7(a) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(a) merge into or with or consolidate with any other Person or permit any other Person to merge into or with or consolidate with it; except that Salant Holding may merge with and into Supreme; provided, that, each of the following conditions is satisfied: (i) Agent shall have received not less than ten (10) Business Days’ prior written notice of the effective date of such merger, (ii) Agent shall have received, true, correct and complete copies of all agreements, documents and instruments relating to such merger or consolidation, including, when available, the certificate or certificates of merger to be filed with each appropriate Secretary of State or similar Governmental Authority, foreign or domestic (with a copy as filed promptly after such filing), (iii) Supreme shall be the surviving corporation and as the surviving corporation shall expressly confirm, ratify and assume the Obligations and the Financing Agreements to which it is a party in writing, in form and substance satisfactory to Agent, and Borrowers and Guarantors shall execute and deliver such other agreements, documents and instruments as Agent may request in connection therewith, (iv) after giving effect to such merger and as a result thereof, no Default or Event of Default shall exist or have occurred, (v) in no event shall Supreme or any other Borrower or Guarantor become liable for any Indebtedness or other obligations (contingent or otherwise) as a result of such merger that it is not otherwise permitted to have hereunder, and (vi) such merger shall have occurred by not later than February 28, 2005,”

5. Representations, Warranties and Covenants. Borrowers and Guarantors jointly and severally represent, warrant and covenant with and to Agent and Lenders that this Amendment No. 5 has been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and is in full force and effect as of the date hereof, and the agreements and obligations of Borrowers and Guarantors contained herein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (a) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (b) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law);

6. Conditions Precedent. The effectiveness of the consents and amendments contained herein shall only be effective upon the receipt by Agent of an executed original or executed original counterparts of this Amendment No. 5, duly authorized, executed and delivered by Borrowers, Guarantors and the Required Lenders.

7. Effect of this Amendment. This Amendment No. 5 and the instruments and agreements delivered pursuant hereto constitute the entire agreement of the parties with respect to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly amended pursuant hereto and except

3

for the amendments expressly contained herein, no other changes or modifications to the Financing Agreements are intended or implied, and in all other respects the Financing Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. Any acknowledgment or consent contained herein shall not be construed to constitute a consent to any other or further action by any Borrower or Guarantor or to entitle any Borrower or Guarantor to any other or further consent. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 5, the provisions of this Amendment No. 5 shall control.

8. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent or Lenders to effectuate the provisions and purposes of this Amendment No. 5.

9. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

10. Binding Effect. This Amendment No. 5 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

11. Counterparts. This Amendment No. 5 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 5, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 5 by telecopier shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 5. Any party delivering an executed counterpart of this Amendment No. 5 by telecopier also shall deliver an original executed counterpart of this Amendment No. 5, but the failure to deliver an original executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 5 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

4

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 5 to be duly executed and delivered by their authorized officers as of the day and year first above written.

5

6

I, HARRIET SMITH WINDSOR, SECRETARY OF STATE OF THE STATE OF DELAWARE, DO HEREBY CERTIFY THE ATTACHED IS A TRUE AND CORRECT COPY OF THE CERTIFICATE OF AMENDMENT OF “SALANT CORPORATION”, CHANGING ITS NAME FROM “SALANT CORPORATION” TO “PERRY ELLIS MENSWEAR, INC.”, FILED IN THIS OFFICE ON THE FIRST DAY OF JULY, A.D. 2004, AT 4:17 O’CLOCK P.M.

A FILED COPY OF THIS CERTIFICATE HAS BEEN FORWARDED TO THE NEW CASTLE COUNTY RECORDER OF DEEDS.

CERTIFICATE OF AMENDMENT

TO

RESTATED CERTIFICATE OF INCORPORATION

OF

SALANT CORPORATION

Salant Corporation, a Delaware corporation (the “Corporation”), pursuant to Section 242 of the Delaware General Corporation Law, hereby files this Certificate of Amendment to its Restated Certificate of Incorporation:

1. The name of the corporation (hereinafter called the “Corporation”) is SALANT CORPORATION.

2. The Restated Certificate of Incorporation of the Corporation is hereby amending Article First and restating it as follows:

FIRST: The name of the corporation (hereinafter called the “Corporation”) is PERRY ELLIS MENSWEAR. INC.

3. The Restated Certificate of Incorporation of the Corporation is hereby amending Article Eleventh and restating it as follows:

ELEVENTH: The Corporation shall have the power to, and may, indemnify its officers, directors, employees and such other persons whom it has the power to indemnify under Delaware law, to the fullest extent permitted by Delaware law.

4. The foregoing amendment to the Restated Certificate of Incorporation has been duly adopted by joint unanimous written consent of the board of directors and sole stockholder and in accordance with the provisions of Sections 228 and 242 of the Delaware General Corporation Law.

IN WITNESS WHEREOF, the undersigned has executed this Certificate of Amendment as of the 30th day of June, 2004.