AMENDMENT NO. 12 AND CONSENT TO LOAN AND SECURITY AGREEMENT

Exhibit 10.86

AMENDMENT NO. 12 AND CONSENT TO LOAN AND SECURITY AGREEMENT

AMENDMENT NO. 12 TO LOAN AND SECURITY AGREEMENT, dated as of                     , 2006 (this “Amendment No. 12”), entered into by and among Wachovia Bank, National Association, successor by merger to Congress Financial Corporation (Florida), in its capacity as agent acting for and on behalf of the parties to the Loan Agreement (as hereinafter defined) as lenders (in such capacity, “Agent”), the parties to the Loan Agreement as lenders (individually a “Lender” and collectively, “Lenders”), Supreme International, LLC, a Delaware limited liability company formerly known as Supreme International, Inc. (“Supreme”), Jantzen, LLC, a Delaware limited liability company formerly known as Jantzen, Inc. (“Jantzen”), Perry Ellis Menswear, LLC, a Delaware limited liability company formerly known as Perry Ellis Menswear, Inc. (“Perry Ellis Menswear”), Perry Ellis Europe Limited, formerly known as Farah Manufacturing (U.K.) Limited, a private limited company incorporated in England and Wales (“Perry Europe”), Salant Holding, LLC, a Delaware limited liability company formerly known as Salant Holding Corporation (“Salant Holding” and together with Supreme, Jantzen, Perry Europe and Perry Ellis Menswear, each individually “Borrower” and collectively, “Borrowers”), Perry Ellis International, Inc., a Florida corporation (“Parent”), PEI Licensing, Inc., a Delaware corporation (“PEI Licensing”), Jantzen Apparel, LLC, a Delaware limited liability company formerly known as Jantzen Apparel Corp. (“Jantzen Apparel”), Supreme Real Estate I, LLC, a Florida limited liability company (“Supreme I”), Supreme Real Estate II, LLC, a Florida limited liability company (“Supreme II”), Supreme Realty, LLC, a Florida limited liability company (“Supreme Realty”), Supreme Munsingwear Canada Inc., a Canada corporation (“Supreme Canada”), Perry Ellis Shared Services Corporation, a Delaware corporation (“PE Shared Services”), Winnsboro DC, LLC, a Delaware limited liability company (“Winnsboro”), Tampa DC, LLC, a Delaware limited liability company (“Tampa DC”), Perry Ellis International Group Holdings Limited, a private company incorporated under the laws of Ireland having its principal place of business in the Bahamas (“Group Holdings”) and Perry Ellis Real Estate, LLC, a Delaware limited liability company formerly known as Perry Ellis Real Estate Corporation (“PE Real Estate” and, together, with Parent, PEI Licensing, Jantzen Apparel, Supreme I, Supreme II, Supreme Realty, Group Holdings, PE Shared Services, Winnsboro, Tampa DC, and Supreme Canada, each individually a “Guarantor” and collectively, “Guarantors”).

W I T N E S S E T H :

WHEREAS, Agent, Lenders, Borrowers and Guarantors have entered into financing arrangements pursuant to which Lenders (or Agent on behalf of Lenders) have made and may make loans and advances and provide other financial accommodations to Borrowers as set forth in the Loan and Security Agreement, dated October 1, 2002, by and among Agent, Lenders, Borrowers and Guarantors, as amended by Amendment No. 1 to Loan and Security Agreement, dated June 19, 2003, Amendment No. 2 to Loan and Security Agreement, dated September 22, 2003, Amendment No. 3 to Loan and Security Agreement, dated December 1, 2003, Amendment No. 4 to Loan and Security Agreement, dated February 25, 2004, Amendment No. 5 to Loan and Security Agreement, dated

July 1, 2004, Amendment No. 6 to Loan and Security Agreement, dated as of September 30, 2004, Amendment No. 7 to Loan and Security Agreement, dated as of February 26, 2005, Amendment No. 8 to Loan and Security Agreement, dated as of September 30, 2005 Amendment No. 9 to Loan and Security Agreement, dated as of February 24, 2006, Amendment No. 10 to Loan and Security Agreement, dated as of August 28, 2006 and Amendment No 11 to Loan and Security Agreement, dated as of November     , 2006(as the same may hereafter be further amended, modified, supplemented, extended, renewed, restated or replaced, the “Loan Agreement”, and together with all agreements, documents and instruments at any time executed and/or delivered in connection therewith or related thereto, as from time to time amended, modified, supplemented, extended, renewed, restated, or replaced, collectively, the “Financing Agreements”);

WHEREAS, Parent, Supreme and Parlux Fragrances, Inc., a Delaware corporation (“Seller”) have entered into or are about to enter into that certain Agreement, dated as of November     , 2006 (the “Parlux Agreement”), pursuant to which (a) Parent or one of its designated subsidiaries has agreed to acquire all of Seller’s right, title and interest in, to and under (i) the License Agreement, dated March 14, 1983 (the “Parlux License Agreement”) between Seller and Parent as assigned to Seller effective December 21, 1994 and (ii) the Licensed Marks (as defined in the Parlux License Agreement as in effect on the date hereof) and (b) Supreme has agreed to acquire all of Seller’s right, title and interest in, to and under all inventory of the Licensed Products (as defined in the Parlux License Agreement as in effect on the date hereof) and certain assets related thereto (but excluding Accounts), in each case in accordance with the terms of the Parlux Agreement as in effect on the date hereof (collectively, the “Parlux Transaction”);

WHEREAS, Borrowers and Guarantors have requested that Agent and Lenders consent to the Parlux Transaction and agree to make certain other amendments to the Loan Agreement, and Agent and Lenders are willing to do so, subject to the terms and conditions set forth in this Amendment No. 12; and

WHEREAS, by this Amendment No. 12, Agent, Lenders, Borrowers and Guarantors desire and intend to evidence such consent and amendments;

NOW, THEREFORE, in consideration of the foregoing, the mutual agreements and covenants contained herein, and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto agree as follows:

1. Definitions.

1.1 Additional Definition.

(a) “Amendment No. 12” shall mean Amendment No. 12 to Loan and Security Agreement by and among Agent, Lenders, Borrowers and Guarantors, as the same now exists or may hereafter be amended, modified, supplemented, extended, renewed, restated or replaced.

2

(b) “Amendment No. 12 Effective Date” shall mean the first date on which all of the conditions precedent to the effectiveness of Amendment No. 12 shall have been satisfied and/or waived.

(c) “Supplemental Amount” shall mean, $25,000,000; provided, that, the Supplemental Amount shall be reduced to zero (0) on the earlier of (i) the date that is six (6) months after the Execution Date (as defined therein) under the Parlux Agreement, (ii) June 30, 2007, (iii) the day on which Borrower Agent shall have notified Agent that it irrevocably elects to reduce the Supplemental Amount to zero (0) or (iv) the day that is 60 days following day on which any Borrower or Guarantor enters into any agreement pursuant to which such Borrower or Guarantor grants a license to a third party to use the “Perry Ellis” trade name in connection with cologne, perfume or any other fragrance product. Notwithstanding any other provision of the Loan Agreement to the contrary, at all times prior to the Parlux Transaction and the satisfaction of the conditions set forth in Section 2(a)-(f) of Amendment No. 12 in accordance with the terms thereof, the Supplemental Amount shall be zero (0).

(d) “Supplemental Availability Amount” shall mean, as to each Borrower, at any time, the amount equal to the Supplemental Amount minus the then outstanding principal amount of Loans and Letter of Credit Accommodations to the other Borrowers based on the Supplemental Amount.

1.2 Amendments to Definitions.

(a) The definition of “Borrowing Base” in Section 1.10 of the Loan Agreement is hereby amended by deleting “, or” appearing at the end of clause (a)(i) of such Section and replacing it with the following: “, plus (D) the Supplemental Availability Amount for such Borrower, or”

(b) Clause (g) of the definition of “Eligible Inventory” in Section 1.30 of the Loan Agreement is hereby amended by deleting such clause in its entirety and replacing it with the following:

“(g) Inventory at premises other than those owned and controlled by a Borrower, except any Inventory which would otherwise be deemed Eligible Inventory that is not located at premises owned and operated by such Borrower may nevertheless be considered Eligible Inventory: (1) as to locations which are leased by such Borrower, if Agent shall have received a Collateral Access Agreement from the owner and lessor of such location, duly authorized, executed and delivered by such owner and lessor, or if Agent shall not have received such Collateral Access Agreement (or Agent shall determine to accept a Collateral Access Agreement that does not include all required provisions or provisions in the form otherwise required by Agent), Agent may, at its option, nevertheless consider Inventory at such location to be Eligible Inventory to the extent Agent shall have established such Reserves in respect of amounts at any time payable by such Borrower to the

3

owner and lessor thereof as Agent shall determine, (2) as to locations owned and operated by a third person, if Agent shall have received a Collateral Access Agreement from such owner and operator with respect to such location, duly authorized, executed and delivered by such owner and operator or if Agent shall not have received such Collateral Access Agreement (or Agent shall determine to accept a Collateral Access Agreement that does not include all required provisions or provisions in the form otherwise required by Agent), Agent may, at its option, nevertheless consider Inventory at such location to be Eligible Inventory to the extent Agent shall have established such Reserves in respect of amounts at any time payable by such Borrower to the owner and operator thereof as Agent shall determine; provided, that, with respect to any Inventory at premises which are leased or utilized by or contracted for by Parlux Fragrances, Inc. or any of its affiliates (collectively, “Parlux”), Parlux and/or its secured lender shall, at the option of Agent, also be deemed to be the owner and operator of such premises for purposes of this clause (g);”

(c) The definition of “Excess Availability” in Section 1.40 of the Loan Agreement is hereby amended by adding the words “plus the Supplemental Availability Amount” after “Accommodations)” and before “, minus” in clause (a)(ii) thereof.

(d) Each reference to the term “Financing Agreements” in the Loan Agreement and any of the other Financing Agreements is hereby amended to include, in addition and not in limitation, collectively, this Amendment No. 12 and the documents, agreements and instruments executed and/or delivered by any Borrower or Guarantor in connection with this Amendment No. 12.

(e) The definition of “Maximum Credit” in Section 1.80 of the Loan Agreement is hereby deleted in its entirety and the following substituted therefor:

“1.80 “Maximum Credit” shall mean (a) from December     , 2006 through and including May 1, 2007, the amount of $210,000,000 and (b) at all other times, the amount of $175,000,000.”

1.3 Interpretation. For purposes of this Amendment No. 12, unless otherwise defined herein, all capitalized terms used herein which are defined in the Loan Agreement shall have the meanings given to such terms in the Loan Agreement.

2. Consent. Subject to the terms and conditions contained herein, to the extent such consent is or may be required under the Loan Agreement, Agent and Lenders hereby consent to the Parlux Transaction in accordance with the terms of the Parlux Agreement as in effect on the date hereof; provided, that, each of the following conditions shall have been satisfied as determined by Agent in good faith:

4

(a) as of the date of the Parlux Transaction and after giving effect thereto, no Default or Event of Default shall exist or have occurred and be continuing;

(b) Agent shall have received the Parlux Agreement, dated not later than December 8, 2006, and all material agreements, documents and instruments executed by any Borrower or Guarantor in connection therewith or related thereto (together with the Parlux Agreement, the “Parlux Documents”), which shall be in form and substance satisfactory to Agent, duly executed and delivered by the parties thereto;

(c) each of the Parlux Documents and the transactions contemplated thereby shall have been consummated in compliance with all applicable laws;

(d) as of the date of the closing of the Parlux Agreement, the aggregate amount of Excess Availability of Borrowers shall have been not less than $35,000,000 for each of the immediately preceding consecutive thirty (30) days and the aggregate amount of Excess Availability of Borrowers shall not be less than $40,000,000 immediately after giving effect to the closing of the Parlux Agreement;

(e) the assets acquired pursuant to the Parlux Agreement shall be free and clear of all liens, security interests or other encumbrances, other than those permitted under the Loan Agreement;

(f) the Parlux License Agreement shall have been terminated; and

(g) in no event shall any Inventory so acquired pursuant to the Parlux Agreement be deemed Eligible Inventory unless and until Agent shall have conducted a field examination with respect thereto (and at Agent’s option, at Borrowers’ expense, obtained an appraisal of such Inventory by an appraiser reasonably acceptable to Agent and in form, scope and methodology reasonably acceptable to Agent and addressed to Agent and upon which Agent is expressly permitted to rely, which appraisal shall be in addition to any appraisals which Agent may obtain pursuant to its rights under Section 7.3 hereof) and then only to the extent the criteria for Eligible Inventory set forth herein are satisfied with respect thereto in accordance with this Agreement (or such other or additional criteria as Agent may, at its option, establish with respect thereto in accordance with this Agreement and subject to such Reserves as Agent may establish in accordance with this Agreement), and upon the request of Agent, Inventory acquired pursuant to the Parlux Agreement and all Accounts arising from the sale of fragrance products shall at all times after such acquisition be separately identified and reported to Agent in a manner satisfactory to Agent.

3. Loans.

(a) Section 2.1(b) of the Loan Agreement is hereby amended by inserting the following immediately before the period at the end of such Section: “, and (iv) the aggregate principal amount of Loans and Letter of Credit Accommodations outstanding at any time to Borrowers based on the Eligible Inventory of Borrowers consisting of cologne, perfume and other fragrances shall not exceed $15,000,000.”

5

(b) Section 2.1(c) of the Loan Agreement is hereby amended by adding after the phrase “based on the Eligible Inventory of all Borrowers exceeds the sublimit set forth above” the phrase “or the aggregate amount of Loans and Letter of Credit Accommodations based on the Eligible Inventory of all Borrowers consisting of cologne, perfume and other fragrances exceeds the applicable sublimit set forth above.”

4. Notice of Fragrance License. If any Borrower or Guarantor enters into any agreement pursuant to which such Borrower or Guarantor grants a license to a third party to use the “Perry Ellis” trade name in connection with cologne, perfume or any other fragrance product, such Borrower or Guarantor shall give Agent prompt written notice thereof.

5. Security Interest in Trademark Collateral. Without limitation upon any Obligations or covenants of Borrowers and Guarantors under the Financing Agreements, at any time that the Supplemental Amount is greater than zero, Borrowers and Guarantors shall not grant to any person nor consent to or otherwise permit or suffer to exist, any lien, claim, encumbrance or security interest in or with respect to any trade names or trademarks or rights in respect thereof owned by any of them, it being acknowledged that termination by the Senior Note Trustee of its security interest in the Released Trademarks has not been recorded at the U.S. Patent and Trademark Office as of the date hereof. If either (a) the aggregate principal amount of Loans and Letter of Credit Accommodations outstanding to Borrowers exceeds the aggregate amount of the Borrowing Bases of Borrowers immediately after the Supplemental Amount shall have been reduced to zero or (b) the definition of Supplemental Amount shall be amended to, directly or indirectly, extend the date by which the Supplemental Amount shall be reduced to zero, then in either case Borrowers and Guarantors shall promptly grant to Agent a perfected first priority security interest in trademarks and trade names owned by Borrowers and Guarantors having an appraised value of not less than $50,000,000 as determined by Agent (or an appraiser selected by Agent) and Borrowers and Guarantors shall promptly execute and deliver such documents, instruments and agreements and take all such other actions as Agent may request in connection therewith, all of which shall be in form and substance satisfactory to Agent.

6. Commitments. In furtherance of the amendment of the Maximum Credit pursuant to this Amendment No. 12, the respective Commitments of the Lenders are hereby amended and restated as set forth on Schedule 1 to Amendment No. 12 which is hereby made a part hereof.

7. Amendment Fee. Borrowers shall pay to Agent, for the account of Lenders (in accordance with the arrangements between Agent and Lenders), the following fees: (a) an amendment fee on the Amendment No. 12 Effective Date in the amount of $90,000, which shall be fully earned as of the date hereof; (b) a Supplemental Availability fee on the Amendment No. 12 Effective Date in the amount of $37,500, which shall be fully earned as of the date hereof; (c) an additional fee in the amount of $37,500, earned as of the date hereof and payable on the date which is ninety (90) days after the first date on which the Supplemental Amount is greater than zero (0), unless on or prior to the date which is ninety (90) days after the first date on which the Supplemental Amount is greater than zero (0), the Supplemental Amount shall have been permanently reduced to zero (0); and (d) an additional fee in the amount of $75,000, payable on the earlier of (i) at any time prior to the permanent reduction of the Supplemental Amount to zero (0), the first

6

date on which the aggregate Excess Availability of Borrowers shall be less than $25,000,000 or (ii) the first date an Event of Default occurs prior to or as a result of the permanent reduction of the Supplemental Amount to zero (0).

8. Representations, Warranties and Covenants. Borrowers and Guarantors, jointly and severally, represent, warrant and covenant with and to Agent and Lenders as follows, which representations, warranties and covenants shall survive the execution and delivery hereof:

(a) this Amendment No. 12 and all other documents, agreements and instruments executed by any Borrower or Guarantor in connection herewith (together with this Amendment No. 12, the “Amendment Documents”) have been duly authorized, executed and delivered by all necessary action on the part of each Borrower and Guarantor which is a party hereto and, if necessary, their respective stockholders, and are in full force and effect as of the date hereof, and the agreements and obligations of Borrowers and Guarantors contained herein and therein constitute legal, valid and binding obligations of Borrowers and Guarantors enforceable against them in accordance with their terms except as such enforceability may be limited by (i) bankruptcy, insolvency, reorganization, moratorium or similar laws of general applicability affecting the enforcement of creditors’ rights and (ii) the application of general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law);

(b) none of the Amendment Documents nor the transactions contemplated thereby are in contravention of any applicable law, or the terms of any agreement to which any Borrower or Guarantor is a party or by which any property of any Borrower or Guarantor is bound; and

(c) as of the date hereof, no Default or Event of Default exists or has occurred and is continuing.

9. Conditions Precedent. The terms and provisions of this Amendment No. 12 shall only be effective upon the satisfaction of each of the following conditions precedent in a manner satisfactory to Agent:

(a) Agent shall have received executed counterparts of this Amendment No. 12, duly authorized, executed and delivered by Borrowers, Guarantors and the Lenders; and

(b) No Default or Event of Default shall exist or have occurred and be continuing.

7

10. Effect of this Amendment. This Amendment No. 12 and the other Amendment Documents constitute the entire agreement of the parties with respect to the subject matter hereof and thereof, and supersede all prior oral or written communications, memoranda, proposals, negotiations, discussions, term sheets and commitments with respect to the subject matter hereof and thereof. Except as expressly provided herein, no other changes or modifications to the Financing Agreements are intended or implied, and in all other respects the Financing Agreements are hereby specifically ratified, restated and confirmed by all parties hereto as of the effective date hereof. To the extent that any provision of the Loan Agreement or any of the other Financing Agreements are inconsistent with the provisions of this Amendment No. 12, the provisions of this Amendment No. 12 shall control.

11. Further Assurances. Each Borrower and Guarantor shall execute and deliver such additional documents and take such additional action as may be reasonably requested by Agent to effectuate the provisions and purposes of this Amendment No. 12.

12. Governing Law. The rights and obligations hereunder of each of the parties hereto shall be governed by and interpreted and determined in accordance with the internal laws of the State of Florida (but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of Florida).

13. Binding Effect. This Amendment No. 12 shall be binding upon and inure to the benefit of each of the parties hereto and their respective successors and assigns.

14. Counterparts. This Amendment No. 12 may be executed in any number of counterparts, but all of such counterparts shall together constitute but one and the same agreement. In making proof of this Amendment No. 12, it shall not be necessary to produce or account for more than one counterpart thereof signed by each of the parties hereto. Delivery of an executed counterpart of this Amendment No. 12 by telecopier or other method of electronic transmission shall have the same force and effect as delivery of an original executed counterpart of this Amendment No. 12. Any party delivering an executed counterpart of this Amendment No. 12 by telecopier or other method of electronic transmission also shall deliver an original executed counterpart of this Amendment No. 12, but the failure to deliver an original executed counterpart shall not affect the validity, enforceability, and binding effect of this Amendment No. 12 as to such party or any other party.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

8

IN WITNESS WHEREOF, the parties hereto have caused this Amendment No. 12 to be duly executed and delivered by their authorized officers as of the day and year first above written.

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

AGREED:

[SIGNATURES CONTINUE ON FOLLOWING PAGE]

[SIGNATURES CONTINUED FROM PRECEDING PAGE]

SCHEDULE 1

TO

AMENDMENT NO. 12 TO LOAN AND SECURITY AGREEMENT

Commitments