AMENDMENT NO. 6 TO LOAN AND SECURITY AGREEMENT

Exhibit 10.63

[Execution]

AMENDMENT NO. 6 TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 6 TO LOAN AND SECURITY AGREEMENT, dated as of September 30, 2004, entered into by and among Congress Financial Corporation (Florida), a Florida corporation, in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, Inc., a Delaware corporation (“Supreme”), Jantzen, Inc., a Delaware corporation (“Jantzen”), Perry Ellis Menswear, Inc., formerly known as Salant Corporation, a Delaware corporation (“Perry Ellis Menswear”), Salant Holding Corporation, a Delaware corporation (“Salant Holding”, and together with Supreme, Jantzen and Perry Ellis Menswear, each individually a “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel Corp., a Delaware corporation (“Jantzen Apparel”), BBI Retail, L.L.C., a Florida limited liability company (“BBI”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, a Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), and Perry Ellis Real Estate Corporation, a Delaware corporation (“PE Real Estate”, and together with Parent, PEI Licensing, Jantzen Apparel, BBI, Supreme I, Supreme II, Supreme Realty and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

W I T N E S S E T H :

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1, 2002, by and among Agent, Lenders, Borrowers and Guarantors as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19, 2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003, Amendment No. 3 to Loan and Security Agreement dated December 1, 2003, Amendment No. 4 to Loan and Security Agreement dated February 25, 2004, Amendment No. 5 to Loan and Security Agreement dated July 1, 2004, and as amended hereby (as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”); and

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders extend the term of the Loan Agreement and make certain other amendments to the Loan Agreement and Agent and Lenders are willing to so consent, subject to the terms and conditions set forth in this Amendment No. 6.

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions.

(a) Additional Definitions. As used herein, the term “Amendment No. 6” shall mean this Amendment No. 6 to Loan and Security Agreement by and among Agent, Lenders, Borrowers and Guarantors, as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced and the Loan Agreement and the other Financing Agreements are hereby amended to include, in addition and not in limitation, such definition.

(b) Amendments to Definitions.

(i) All references to the term “Applicable Margin” in the Loan Agreement or any other Financing Agreements and each such reference is hereby amended to mean, at any time, as to the Interest Rate for Prime Rate Loans and the Interest Rate for Eurodollar Rate Loans the applicable percentage (on a per annum basis) set forth below if either (A) the sum of: (1) the Quarterly Average Excess Availability for the immediately preceding fiscal quarter plus (2) the Excess Cash as of the last day of the immediately preceding fiscal quarter is at or within the amounts indicated for such percentage or (B) the Leverage Ratio as of the last day of the immediately preceding fiscal quarter (which ratio for this purpose shall be calculated based on the four (4) immediately preceding fiscal quarters) is at or within the levels indicated for such percentage:

provided, that, (A) the Applicable Margin shall be calculated and established once each fiscal quarter (commencing with the fiscal quarter ending on October 31, 2004, but with the Applicable

2

Margin based on the above to be effective as of October 1, 2004) and shall remain in effect until adjusted thereafter at the end of the next quarter and (B) the Applicable Margin shall be the lower percentage set forth above based on (1) the sum of the Quarterly Average Excess Availability and the Excess Cash as provided above or (2) the Leverage Ratio.

(ii) The definition of the term “Consolidated Net Income” is hereby amended to delete the portion of the definition prior to clause (a) thereof and substitute the following therefor:

“Consolidated Net Income” shall mean, with respect to any Person for any period, the aggregate of the net income (loss) of such Person and its Subsidiaries, on a consolidated basis, for such period (excluding to the extent included therein any extraordinary or non-recurring gains and extraordinary non-cash charges, including impairment charges to property, plant and equipment, Intellectual Property or goodwill and non-cash charges related to employee benefits or other management compensation plans and any non-cash compensation charges arising from any grants, issuance or repricing of stock, stock options or other equity based awards) after deducting all charges which should be deducted before arriving at the net income (loss) for such period and after deducting the Provision for Taxes for such period, all as determined in accordance with GAAP; provided, that,”

(iii) The reference to the amount of “$20,000,000” in the definition of the term “Eligible Factor Receivables” in the Loan Agreement is hereby deleted and the amount of “$30,000,000” substituted therefor.

(iv) All references to the term “Inventory Loan Limit” in the Loan Agreement or any of the other Financing Agreements and each such reference is hereby amended to mean $65,000,000.

(c) Interpretation. For purposes of this Amendment No. 6, unless otherwise defined herein, all capitalized terms used herein which are defined in the Loan Agreement shall have the meanings given to such terms in the Loan Agreement.

2. Letter of Credit Accommodations. Section 2.2(b) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(b) In addition to any charges, fees or expenses charged by any bank or issuer in connection with the Letter of Credit Accommodations, Borrowers shall pay to Agent, for the benefit of Lenders, (i) a letter of credit fee at a rate equal to one quarter (1/4 %) percent per annum on the daily outstanding balance of the Letter of Credit Accommodations consisting of Commercial Letters of Credit for the immediately preceding month (or part thereof), payable in arrears as of the first day of each succeeding month and (ii) a letter of credit fee at a rate equal to one (1%) percent per annum on the daily outstanding balance of all other Letter of Credit Accommodations for the immediately preceding month (or part thereof), payable in arrears as of the first day of each succeeding month, except that Agent may, and upon the written direction of Required Lenders shall, require Borrowers to pay to Agent for the benefit of Lenders

3

such letter of credit fee at a rate equal to two and one quarter (2 1/4%) percent per annum on the daily outstanding balance of the Letter of Credit Accommodations consisting of Commercial Letters of Credit and at a rate equal to three (3%) percent per annum on the daily outstanding balance of all other Letter of Credit Accommodations for: (A) the period from and after the date of termination hereof until Agent and Lenders have received full and final payment of all Obligations (notwithstanding entry of a judgment against any Borrower) and (B) the period from and after the date of the occurrence of an Event of Default for so long as such Event of Default is continuing as determined by Agent. Such letter of credit fee shall be calculated on the basis of a three hundred sixty (360) day year and actual days elapsed and the obligation of Borrowers to pay such fee shall survive the termination of this Agreement.”

3. Unused Line Fee. Section 3.2(a) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(a) Borrowers shall pay to Agent for the ratable benefit of Lenders monthly an unused line fee at a rate equal to the percentage (on a per annum basis) set forth below calculated upon the amount by which $70,000,000 for the period commencing October 1 of each year and ending on March 31 of the immediately following year, or $50,000,000 for the period commencing on April 1 of each year and ending on September 30 of such year, exceeds the average daily principal balance of the outstanding Loans and Letter of Credit Accommodations during the immediately preceding month (or part thereof) while this Agreement is in effect and for so long thereafter as any Obligations are outstanding. Such fee shall be payable on the first day of each month in arrears. The percentage used for determining the unused line fee shall be as set forth below if either (i) the sum of the Quarterly Average Excess Availability for the immediately preceding fiscal quarter plus the Excess Cash as of the last day of the immediately preceding fiscal quarter is at or within the amounts indicated for such percentage or (ii) the Leverage Ratio as of the last day of the immediately preceding fiscal quarter (which ratio for this purpose shall be calculated based on the four (4) immediately preceding fiscal quarters) is at or within the levels indicated for such percentage:

4

provided, that, (A) the unused line fee percentage shall be calculated and established once each fiscal quarter (commencing with the fiscal quarter ending on October 31, 2004, but with the unused line fee percentage based on the above to be effective as of October 1, 2004) and (B) the unused line fee percentage shall be the lower percentage set forth above based on (1) the sum of the Quarterly Average Excess Availability plus the Excess Cash as provided above or (2) the Leverage Ratio.”

4. Mergers. Section 9.7(a) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(a) merge into or with or consolidate with any other Person or permit any other Person to merge into or with or consolidate with it; except that any Borrower may merge with any other Borrower and any Guarantor may merge with any Guarantor or Borrower; provided, that, each of the following conditions is satisfied: (i) Agent shall have received not less than ten (10) Business Days’ prior written notice of the effective date of such merger, (ii) Agent shall have received, true, correct and complete copies of all agreements, documents and instruments relating to such merger or consolidation, including, when available, the certificate or certificates of merger to be filed with each appropriate Secretary of State or similar Governmental Authority, foreign or domestic (with a copy as filed promptly after such filing), (iii) the surviving corporation shall expressly confirm, ratify and assume the Obligations and the Financing Agreements to which it is a party in writing, in form and substance satisfactory to Agent, and Borrowers and Guarantors shall execute and deliver such other agreements, documents and instruments as Agent may request in connection therewith, (iv) as to any merger of a Guarantor with a Borrower, the Borrower shall be the surviving corporation, (v) any assets acquired by a Borrower pursuant to a merger of a Guarantor with such Borrower of a type or category that might be included in the calculation of the Borrowing Base of such Borrower shall only be so included subject to such additional conditions, limitations or other terms as Agent may determine, (vi) the Borrower or Guarantor acquiring any assets as a result of such merger shall acquire such assets subject to the security interest, lien, pledge and any other interest of Agent in the property so acquired and permitting such merger hereunder or otherwise shall not be deemed to authorize the acquisition of such assets pursuant to such merger free and clear of the security interest, lien, pledge and any other interest of Agent therein, (vii) the consideration payable by any Borrower in connection with such merger shall be in amounts and form and otherwise on terms satisfactory to Agent, (viii) after giving effect to such merger and as a result thereof, no Default or Event of Default shall

5

exist or have occurred, and (ix) in no event shall any Borrower or Guarantor become liable for any Indebtedness or other obligations (contingent or otherwise) as a result of such merger that it is not otherwise permitted to have hereunder;”

5. Transfers of Assets. Section 9.7(b) of the Loan Agreement is hereby amended to add a new Section 9.7(b)(xi) at the end thereof as follows:

“(xi) the sale or assignment of any assets (other than Accounts, Inventory or any other assets at any time considered in the calculation of the Borrowing Base of any Borrower) by a Borrower to another Borrower or the sale or assignment of any assets by a Guarantor or Borrower to another Guarantor or Borrower, provided, that, as to any such sale or assignment each of the following conditions is satisfied: (A) Agent shall have received not less than ten (10) Business Days’ prior written notice of the effective date of such sale or assignment, with such information with respect thereto as Agent may require, including a description of the assets to be sold or assigned, the seller or assignor and buyer or assignee, and the consideration for such transfer, its form and manner and timing of payment, (B) Agent shall have received true, correct and complete copies of all agreements relating to such sale or assignment and such other information with respect thereto as Agent may reasonably request from time to time, (C) in no event shall such sale or assignment result in any limit or other impairment of the rights of any Borrower or Guarantor (or Agent) to use any Intellectual Property or other assets or create or give rise to any material liabilities of any Borrower or Guarantor in connection with such sale or assignment, (D) as of the date of any such sale or assignment and after giving effect thereto, no Default or Event of Default shall exist or have occurred and be continuing, (E) the Borrower or Guarantor to whom such assets are sold or assigned shall acquire such assets subject to the security interest, lien, pledge and any other interest of Agent in the property so sold or assigned and permitting such sale or assignment hereunder or otherwise shall not be deemed to authorize such sale or assignment free and clear of the security interest, lien, pledge and any other interest of Agent therein, (F) the consideration payable by any Borrower shall be in amounts and form and otherwise on terms satisfactory to Agent, (G) the Borrower or Guarantor acquiring the assets shall expressly confirm and ratify that the assets sold or assigned are subject to the security interest, lien, pledge and any other interest of Agent in writing, in form and substance satisfactory to Agent, and Borrowers and Guarantors shall execute and deliver such other agreements, documents and instruments as Agent may request in connection therewith, (H) any assets acquired by a Borrower from a Guarantor pursuant such sale or assignment of a type or category that might be included in the calculation of the Borrowing Base of such Borrower shall only be so included subject to such additional conditions, limitations or other terms as Agent may determine, and (I) such sale or assignment shall not result in the breach of, or constitute a default under, any indenture, agreement or instrument to which any Borrower or Guarantor is a party or by which it or its assets may be bound,”

6

6. Indebtedness. Section 9.9(j) of the Loan Agreement is hereby amended to delete Section 9.9(j)(ii) thereof in its entirety and the following substituted therefor: “intentionally omitted;”.

7. End of Fiscal Years; Fiscal Quarters. Section 9.14 of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“9.14 End of Fiscal Years; Fiscal Quarters. Each Borrower and Guarantor shall, for financial reporting purposes, cause its, and each of its Subsidiaries’ (a) fiscal years to end on January 31 of each year except that the fiscal year of Supreme Canada shall end on May 31 of each year (provided that upon sixty (60) days’ prior written notice to Agent, the end of the fiscal year of each Borrower and Guarantor may be changed to December 31) and (b) fiscal quarters to end on April 30, July 31, October 31 and January 31 of each year except that the fiscal quarters of Supreme Canada shall end on August 30, November 30, February 28 and May 31 of each year (provided that upon sixty (60) days’ prior written notice to Agent, the fiscal quarters of each Borrower and Guarantor may be changed to end on March 31, June 30, September 30 and December 31 in connection with the change of the end of its fiscal year to December 31).”

8. Minimum EBITDA. Section 9.17 of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“9.17 Minimum EBITDA. At any time that the aggregate amount of the Excess Availability is less than $25,000,000, (a) the EBITDA of Parent and its Subsidiaries (on a consolidated basis) for the preceding twelve (12) consecutive months (treated as a single accounting period) as of the end of the most recent fiscal month for which Agent or any Lender has received financial statements of Borrowers or Guarantors, shall be not less than $45,000,000 (after giving effect to the EBITDA of Perry Ellis Menswear and its Subsidiaries on a pro forma basis in a manner satisfactory to Agent) and (b) the EBITDA of Parent and its Subsidiaries (on a consolidated basis) as of the end of the most recent two (2) fiscal months, on a combined basis, for which Agent or any Lender has received financial statements of Borrowers or Guarantors shall be positive.”

9. Term. The first sentence of Section 13.1(a) of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“(a) This Agreement and the other Financing Agreements shall become effective as of the date set forth on the first page hereof and shall continue in full force and effect for a term ending on September 30, 2007 (the “Renewal Date”) and from year to year thereafter, unless sooner terminated pursuant to the terms hereof; provided, that, except as Agent and Borrower Agent may otherwise agree, in the event that the term of this Agreement shall continue for any additional year after September 30, 2007, Borrowers and Guarantors shall pay to Agent an extension fee in the amount of one-sixth (1/6th) percent of the Maximum Credit

7

which fee shall be earned and payable in full on September 30, 2007 in respect of the extension for each year after the Renewal Date.”

10. Field Examinations. Without limiting any other rights of Agent to receive information or otherwise with respect to any Borrower or Guarantor, Agent shall conduct three (3) field examinations with respect to the business of Borrowers in any twelve (12) consecutive month period, so long as no Default or Event of Default shall exist or have occurred and Excess Availability is at all times during such period not less than $25,000,000. If at any time there is a Default or Event of Default or Excess Availability is less than such amount, Agent may conduct such field examinations as it determines are necessary or desirable and such field examinations shall not be included in the limitation provided for herein.

11. Amendment Fee.

(a) Borrowers shall pay to Agent, for the account of Lenders (in accordance with the arrangements between Agent and Lenders), an amendment fee on the date hereof in the amount of $250,000 which shall be fully earned as of the date hereof, without limiting any other rights of Agent and Lenders.

(b) In the event that Borrowers and Lenders agree to an amendment and restatement of the financing arrangements provided for herein after the date hereof in connection with an acquisition by Borrowers of the assets of a Person that is not an Affiliate in a bona fide arm’s length transaction, subject to the terms hereof, as a one time accommodation, the closing fee that the parties may agree to as part of such amendment and restatement that would otherwise be payable to a Lender that is a party to this Amendment No. 6 shall be reduced as to each such Lender by the amount of the amendment fee provided for in Section 11(a) above received by such Lender on or about the date hereof; provided, that, (i) such closing fee shall only be reduced as to those Lenders that are party to this Amendment No. 6, (ii) as part of any such amendment and restatement, the Maximum Credit shall be increased by no less than $50,000,000, (iii) a commitment letter issued by Agent setting forth the terms of such amendment and restatement shall be executed and delivered by Borrowers and Guarantors to Agent, and effective by, no later than April 29, 2005, (iv) nothing contained herein shall be construed as a consent to any such acquisition or a waiver of any term or other provision of the Loan Agreement or any of the other Financing Agreements that might be applicable thereto or otherwise, (v) nothing contained herein shall be construed to limit or affect the rights of Agent and Lenders to establish any terms or conditions to be included in any such amendment and restatement, (vi) the reduction in such closing fee shall not apply to any other fees at any time provided for in the Loan Agreement or any amendment and restatement thereof and (vii) the reduction in the closing fee payable to a Lender that is party hereto in connection with the amendment and restatement of the financing arrangements will only apply to the extent that such Lender continues as a Lender in the financing arrangements as so amended and restated.

12. Representations, Warranties and Covenants. Borrowers and Guarantors jointly and severally represent, warrant and covenant with and to Agent and Lenders that: (a) this Amendment No. 6 has been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and is in full force and effect as of the date hereof, and the agreements

8

and obligations of Borrowers and Guarantors contained herein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (i) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (ii) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law) and (b) as of the date hereof and after giving effect to the amendments provided for herein, no Default or Event of Default exists or has occurred and is continuing.

13. Conditions Precedent. The effectiveness of the consents and amendments contained herein shall only be effective upon the receipt by Agent of an executed original or executed original counterparts of this Amendment No. 6, duly authorized, executed and delivered by Borrowers, Guarantors and the Required Lenders by no later than October 4, 2004.

14. Effect of this Amendment. This Amendment No. 6 and the instruments and agreements delivered pursuant hereto constitute the entire agreement of the parties with respect to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly amended pursuant hereto and except for the amendments expressly contained herein, no other changes or modifications to the Financing Agreements are intended or implied, and in all other respects the Financing Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. Any acknowledgment or consent contained herein shall not be construed to constitute a consent to any other or further action by any Borrower or Guarantor or to entitle any Borrower or Guarantor to any other or further consent. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 6, the provisions of this Amendment No. 6 shall control.

15. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent or Lenders to effectuate the provisions and purposes of this Amendment No. 6.

16. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

17. Binding Effect. This Amendment No. 6 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

18. Counterparts. This Amendment No. 6 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 6, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 6 by telecopier shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 6. Any party delivering an executed counterpart of this Amendment No. 6 by telecopier also shall deliver an

9

original executed counterpart of this Amendment No. 6, but the failure to deliver an original executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 6 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

10

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 6 to be duly executed and delivered by their authorized officers as of the day and year first above written.