TRANSFER AGREEMENT

Exhibit 10.4

TRANSFER AGREEMENT

THIS TRANSFER AGREEMENT (this “Agreement”) is made and entered into as of [        ] [    ], 2013, by and between Kellwood Company, LLC a Delaware limited liability company (“Transferor”), Vince Intermediate Holding, LLC, a Delaware limited liability company (“Transferee”) and solely with respect to Sections 2.04 and 6.01(a) hereof, Apparel Holding Corp., a Delaware corporation (to be renamed Vince Holding Corp., “Apparel Holding”). Transferor and Transferee are referred to collectively herein as the “Parties” and each as a “Party”. Capitalized terms used and not otherwise defined herein have the meanings assigned to such terms in ARTICLE I below.

WHEREAS, Transferor is the owner of 100% of the issued and outstanding equity interests (the “Equity Interests”) of Vince, LLC, a Delaware limited liability company (“Vince”);

WHEREAS, subject to the terms and conditions set forth in this Agreement, Transferor desires to transfer and convey to Transferee, and Transferee desires to receive from Transferor, all of the Equity Interests in exchange for Transferee’s issuance of the promissory note to Transferor attached hereto as Exhibit A (the “Promissory Note”);

NOW, THEREFORE, in consideration of the mutual covenants contained herein and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the Parties agree as follows:

ARTICLE I

DEFINITIONS

1.01 Definitions. The following terms have the meaning set forth below:

“7.625% Notes” is defined in Section 6.01(b).

“12.875% Notes” is defined in Section 6.01(b).

“Additional Sun Capital Contribution” is defined in Section 6.01(b).

“Affiliate” of any particular Person means any other Person controlling, controlled by or under common control with such Person. For purposes of this definition, “control” (including the terms “controlling,” “controlled by” and “under common control with”) means the possession, direct or indirect, of the power to direct or cause the direction of the management and policies of a Person, whether through the ownership of voting securities, by contract or otherwise, and such “control” will be presumed if any Person owns 10% or more of the voting capital stock or other ownership interests, directly or indirectly, of any other Person.

“Agreement” is defined in the Preamble.

“Apparel Holding” is defined in the Preamble.

“Closing” is defined in Section 3.01.

1

“Electronic Delivery” is defined in Section 7.09.

“Encumbrance” means a mortgage, charge, pledge, lien, claim, option, encumbrance, charge, agreement, voting trust, proxy, right of preemption, right of first refusal or any other security interest or transfer restriction of any kind.

“Escrow Agreement” is defined in Section 6.01(b).

“Equity Interest” is defined in the Recitals.

“Initial Public Offering” is defined in Section 6.01(a).

“IPO Proceeds” is defined in Section 2.04.

“Kellwood Business” means the business of the Transferor immediately prior to the Closing, other than the Vince Business.

“Losses” means any loss, liability, demand, claim, action, cause of action, cost, damage, diminution in value, deficiency, tax, penalty, fine or expense, whether or not arising out of third party claims (including interest, penalties, reasonable attorneys’ fees and expenses and all amounts paid in investigation, defense or settlement of any of the foregoing and the enforcement of any rights hereunder).

“Management Services Agreement” is defined in Section 6.01(b).

“Party” is defined in the Preamble.

“Person” means an individual, a corporation, a limited liability company, an association, a joint-stock company, a business trust or other similar organization, a partnership, a joint venture, a trust, an unincorporated organization or a government or any agency, instrumentality or political subdivision thereof.

“Promissory Note” is defined in the Recitals.

“Specific Assets” means any tangible or intangible assets, such as, inter alia, customer and supplier contracts, equipment, drawings or intellectual property.

“Sun Capital” is defined in Section 6.01(b).

“Sun Capital Management” is defined in Section 6.01(b).

“Sun Term Loan A Agreements” is defined in Section 6.01(b).

“Sun Term Loan Agreements” is defined in Section 6.01(b).

“Sun Term B/C/D/E/F/G Loan Agreement” is defined in Section 6.01(b).

“Transferee” is defined in the Preamble.

“Transferor” is defined in the Preamble.

“Vince” is defined in the Recitals.

2

“Vince Business” means the design, development, manufacturing, sourcing, marketing, licensing, distribution and sale of Vince branded apparel and related accessories through wholesale, retail stores, e-commerce and any other distribution channels.

ARTICLE II

THE TRANSACTION

2.01 Transfer of Equity Interests. On and subject to the terms and conditions of this Agreement, in exchange for the issuance of the Promissory Note by Transferee for the benefit of Transferor, (i) Transferor hereby assigns, conveys, sells, transfers and delivers to Transferee, all rights, title and interest in and to the Equity Interests, free and clear of any Encumbrances and (ii) Transferee hereby accepts the Equity Interests from Transferor, free and clear of any Encumbrances.

2.02 Deliveries by Transferee. At the Closing, Transferee shall deliver (a) the Promissory Note duly executed by Transferee and (b) such documents relating to the transactions contemplated by this Agreement as Transferor may reasonably request.

2.03 Deliveries by Transferor. At the Closing, Transferor shall deliver (a) the Equity Interests, and all certificates reflecting ownership thereof, if any, free and clear of any Encumbrances, duly endorsed in blank or accompanied by duly executed unit powers, with appropriate transfer stamps, if any, affixed thereto and (b) such other documents relating to the transactions contemplated by this Agreement as Transferee or its counsel may reasonably request.

2.04 Deliveries by Apparel Holding. Immediately after the closing of the Initial Public Offering (as defined below), Apparel Holding shall deliver the net proceeds from the Initial Public Offering (after giving effect to fees and expenses related thereto, including any underwriting discount, the “IPO Proceeds”) to Transferee.

ARTICLE III

CLOSING OF THE TRANSACTION

3.01 The Closing. The closing of the transactions contemplated hereby (the “Closing”) shall take place at the offices of Kirkland & Ellis LLP in Chicago, Illinois at 8 a.m. on the date hereof, or at such other place or on such other date and time as may be mutually agreeable to the Parties.

ARTICLE IV

REPRESENTATIONS AND WARRANTIES OF TRANSFEROR

Transferor hereby represents to Transferee that:

4.01 Ownership. The Equity Interests are owned of record and beneficially by Transferor and Transferor has good and marketable title to the Equity Interests, free and clear of all Encumbrances, other than pursuant to applicable securities laws. Other than the Equity Interests, there are no issued or outstanding equity interests or other securities of Vince, nor any rights or options to subscribe for or to purchase equity interests or other securities of Vince.

3

4.02 Authorization; No Breach. The execution, delivery, and performance of this Agreement and all other agreements contemplated hereby to which Transferor is a party have been duly authorized by Transferor. This Agreement and all other agreements contemplated hereby each constitutes a valid and binding obligation of Transferor, enforceable in accordance with its terms, except as enforceability may be limited by bankruptcy, insolvency, reorganization, moratorium, or other laws affecting creditors’ rights generally and limitations on the availability of equitable remedies. The execution and delivery by Transferor of this Agreement and all other agreements contemplated hereby to which Transferor is a party, and the fulfillment of and compliance with the respective terms hereof and thereof by Transferor, do not and shall not (i) conflict with or result in a breach of the terms, conditions or provisions of, (ii) constitute a default under, (iii) result in the creation of any Encumbrance upon Transferor’s or Vince’s equity interests or assets pursuant to, (iv) give any third party the right to modify, terminate, or accelerate any obligation under, (v) result in a violation of, or (vi) require any authorization, consent, approval, exemption, or other action by or notice to any court or administrative or governmental body pursuant to, its or Vince’s operating agreement or any material law, statute, rule, or regulation to which Transferor or Vince is subject, or any material agreement, instrument, order, judgment, or decree to which Transferor or Vince is subject, except (A) where the failure to obtain such authorization, consent, approval, exemption, or other action by or notice to any third party under any agreement or instrument would not reasonably be expected to have a material adverse effect on the Vince Business and (B) (ii) for any such occurrence arising in connection with indebtedness that is to be repaid, refinanced, repurchased or discharged in connection with the consummation of the Initial Public Offering (as defined below).

ARTICLE V

REPRESENTATIONS AND WARRANTIES OF TRANSFEREE

Transferee hereby represents to Transferor that:

5.01 Authorization; No Breach. The execution, delivery, and performance of this Agreement and all other agreements contemplated hereby to which Transferee is a party have been duly authorized by Transferee. This Agreement and all other agreements contemplated hereby each constitutes a valid and binding obligation of Transferee, enforceable in accordance with its terms, except as enforceability may be limited by bankruptcy, insolvency, reorganization, moratorium, or other laws affecting creditors’ rights generally and limitations on the availability of equitable remedies. The execution and delivery by Transferee of this Agreement and all other agreements contemplated hereby to which Transferee is a party, and the fulfillment of and compliance with the respective terms hereof and thereof by Transferee, do not and shall not (i) conflict with or result in a breach of the terms, conditions or provisions of, (ii) constitute a default under, (iii) result in the creation of any Encumbrance upon Transferee’s equity interests or assets pursuant to, (iv) give any third party the right to modify, terminate, or accelerate any obligation under, (v) result in a violation of, or (vi) require any authorization, consent, approval, exemption, or other action by or notice to any court or administrative or

4

governmental body pursuant to, its operating agreement or any material law, statute, rule, or regulation to which Transferee is subject, or any material agreement, instrument, order, judgment, or decree to which Transferee is subject.

ARTICLE VI

ADDITIONAL AGREEMENTS

6.01 Post-Closing Transaction.

(a) Apparel Holding hereby covenants and agrees that immediately following the consummation of the initial public offering of its common stock pursuant to an effective registration statement under the Securities Act of 1933, as amended (the “Initial Public Offering”), it shall contribute the IPO Proceeds to Transferee. Transferee hereby covenants and agrees to pay to Transferor, using (i) all but $5.0 million of the IPO Proceeds and (ii) $[        ] million of net borrowings under its new term loan facility, immediately upon receipt of the IPO Proceeds, the principal amount outstanding under the Promissory Note, in full satisfaction thereof by wire transfer of immediately available funds in accordance with instructions provided by Transferor to Transferee in writing not less than two (2) business days prior to the Closing.

(b) Transferor hereby covenants and agrees that immediately following the receipt of the principal amount outstanding under the Promissory Note from Transferee, Transferor shall use such amounts to (i) immediately repay all amounts outstanding under the following instruments (including accrued and unpaid interest thereon and all fees, expenses related thereto), after giving effect to the contribution of certain indebtedness under the Sun Term Loan Agreements (as defined below) to be made by Sun Capital Partners, Inc. (“Sun Capital”) or its affiliates immediately prior to the consummation of the transactions contemplated by that certain Contribution Agreement by and between Apparel Holding and Transferee (the “Additional Sun Capital Contribution”): (A) Transferor’s $55 million term loan facility with Cerberus Business Finance, LLC, dated as of October 19, 2011 (as amended or supplemented); (B) that certain Fifth Amended and Restated Sun Term B/C/D/E/F/G Loan Agreement between Transferor, Sun Kellwood Finance, LLC, SCSF Kellwood Finance, LLC and the other Borrowers party thereto, dated as of June 28, 2013 (as amended or supplemented, the “Sun Term Loan B/C/D/E/F/G Agreement”); and (C) that certain Sun Term A Loan Agreement between Transferor, Sun Kellwood Finance, LLC, SCSF Kellwood Finance, LLC and the other Borrowers party thereto, dated as of October 19, 2011 (as amended or supplemented, the “Sun Term Loan A Agreement” and collectively with the Sun Term Loan B/C/D/E/F/G Agreement, the “Sun Term Loan Agreements”); (ii) redeem at par all of the Transferor’s Second-Priority Senior Secured Payment-In-Kind Notes due 2014 (the “12.875% Notes”) by (A) immediately issuing an unconditional redemption notice with respect to all issued and outstanding 12.875% Notes and (B) immediately depositing with the trustee of the indenture governing the 12.875% Notes an amount sufficient to redeem all issued and outstanding 12.875% Notes (including all accrued and unpaid interest thereon) in accordance with the terms of such indenture and the redemption notice; and (iii) immediately pay (A) that certain debt recovery bonus to Jill Granoff, as contemplated by the terms of her employment agreement, dated as of May 4, 2012 (as amended or supplemented) and (B) the restructuring fee payable to Sun Capital Management V, LLC (“Sun Capital Management”) pursuant to that Management Services Agreement, dated as of May 29, 2008 (as amended or supplemented, the “Management Services Agreement”), by and between Sun Capital Management and Transferor in connection with the consummation of the transactions related to the Initial Public Offering and in accordance with the terms of the Management Services Agreement.

5

Additionally, Transferor may, but is not obligated to, use such amounts to repurchase its 7.625% Debentures due October 5, 2017 (the “7.625% Notes”) and Transferor’s 3.5% Convertible Senior Debentures due June 15, 2034 on or after the closing of the Initial Public Offering. In the event the tender offer for the 7.625% Notes is to be consummated after the closing of the Initial Public Offering, Transferor shall escrow an amount necessary to purchase the 7.625% Notes which may be tendered in the tender offer at the closing of the tender offer with U.S. Bank National Association, as escrow agent, in accordance with the terms of the escrow agreement (substantially in the form attached hereto as Exhibit B, the “Escrow Agreement”) until the earlier of (i) the closing of the tender offer; and (ii) the three-month anniversary of the closing of the Initial Public Offering and Transferee agrees to issue a Joint Written Direction (as defined in the Escrow Agreement) to the extent necessary to affect any payments, whether in whole or in part, with respect to the foregoing clauses (i) through (ii), as reasonably requested by Transferor. Any amounts remaining in the escrow account after the termination of the Escrow Agreement shall be returned to the Transferor.

(c) Transferor hereby further covenants and agrees that immediately following the receipt of the principal amount outstanding under the Promissory Note from Transferee and in no event later than the business day on which the Initial Public Offering is consummated, Transferor shall refinance its $155 million revolving credit facility with Wells Fargo Bank National Association, dated as of October 19, 2011 (as amended or supplemented, the “Wells Fargo Facility”) to remove Vince, LLC as a guarantor or obligor thereunder.

6.02 Further Assurances.

(a) Upon the request of either Party, the other Party shall, without further consideration, execute and deliver, or cause to be executed and delivered, such other instruments of conveyance, transfer, assignment and confirmation, and shall take, or cause to be taken, such further or other actions as the other Party may deem necessary or desirable to carry out the intent and purposes of this Agreement and to consummate and give effect to the transactions contemplated hereby.

6.03 Indemnity.

(a) Transferee shall indemnify Transferor and hold it harmless against any Losses which Transferor may suffer, sustain or become subject to the extent relating to the Vince Business (including Losses of Transferor relating to Section 9 of the Escrow Agreement to the extent such Losses relate to acts or omissions of the Transferee).

(b) Transferor shall indemnify Transferee and hold it harmless against (i) any Losses which Transferee or Vince may suffer, sustain or become subject to the extent relating to (A) the Kellwood Business (including Losses of Transferee relating to Section 9 of the Escrow Agreement to the extent such Losses relate to acts or omissions of the Transferor) or (B) any failure by Transferor to fully perform its obligations under Section 4 of that certain Contribution and Acceptance Agreement by and between Transferor and Vince dated September 1, 2012 and (ii) any Losses which Transferee or Vince may suffer, sustain or become subject to relating to any failure by Transferor to perform its obligations under Section 6.01(b) or Transferor’s inability to remove Vince as an obligor or guarantor under the Wells Fargo Facility.

6

ARTICLE VII

MISCELLANEOUS

7.01 Assignment.

(a) This Agreement and all of the provisions hereof shall be binding upon and inure to the benefit of the Parties and their respective successors and permitted assigns, except that neither this Agreement nor any of the rights, interests or obligations hereunder may be assigned or delegated by a Party without the prior written consent of the other Party.

(b) Either Party may assign in whole or in part its rights and obligations pursuant to this Agreement and all other agreements, documents and instruments executed and/or delivered in connection herewith to one or more of its affiliates. Either Party may assign this Agreement, all other agreements, documents and instruments executed and/or delivered in connection herewith, and its rights and obligations hereunder and thereunder in connection with a merger or consolidation involving such Party or in connection with a sale of stock (or other ownership interests) or assets of such Party. Either Party may assign any or all of its rights pursuant to this Agreement, including its rights to indemnification, and all other agreements, documents and instruments executed and/or delivered in connection herewith, to any of its lender(s) as collateral security.

7.02 Survival. All representations and warranties and covenants contained herein or made in writing by any Party in connection herewith shall survive the execution and delivery of this Agreement and the consummation of the transactions contemplated hereby, regardless of any investigation made by a Party or on its behalf.

7.03 Severability. Whenever possible, each provision of this Agreement shall be interpreted in such manner as to be effective and valid under applicable law, but if any provision of this Agreement or the application of any such provision to any person or circumstance shall be held to be prohibited by or invalid, illegal or unenforceable under applicable law in any respect by a court of competent jurisdiction, such provision shall be ineffective only to the extent of such prohibition or invalidity, illegality or unenforceability, without invalidating the remainder of such provision or the remaining provisions of this Agreement. Furthermore, in lieu of such illegal, invalid or unenforceable provision, there shall be added automatically as a part of this Agreement a legal, valid and enforceable provision as similar in terms to such illegal, invalid, or unenforceable provision as may be possible.

7.04 Interpretation. The headings and captions used in this Agreement and the table of contents to this Agreement are for reference purposes only and shall not affect in any way the meaning or interpretation of this Agreement. Any capitalized term used in any Schedule attached hereto and not otherwise defined therein shall have the meaning set forth in this Agreement. The use of the word “including” herein shall mean “including without limitation.” The word “or” is used in the inclusive sense of “and/or”.

7

7.05 Construction. The language used in this Agreement shall be deemed to be the language chosen by the Parties to express their mutual intent, and no rule of strict construction shall be applied against any Party.

7.06 Amendment and Waiver. Except as provided herein, any provision of this Agreement may be amended or waived only in a writing signed by the Parties. No waiver of any provision hereunder or any breach or default thereof shall extend to or affect in any way any other provision or prior or subsequent breach or default.

7.07 Complete Agreement. This Agreement and the other agreements, instruments, and documents contemplated hereby or executed in connection herewith contain the complete agreement between the Parties and supersede any prior understandings, agreements or representations by or between the Parties, written or oral, which may have related to the subject matter hereof in any way (including any letter of intent or non-binding terms of sale).

7.08 No Third-Party Beneficiaries. This Agreement is for the sole benefit of the Parties hereto and their permitted assigns and nothing herein expressed or implied, shall give or be construed to give any Person, other than the parties hereto and such permitted assigns, any legal or equitable rights hereunder.

7.09 Electronic Delivery; Counterparts. This Agreement and any signed agreement or instrument entered into in connection with this Agreement, and any amendments hereto or thereto, may be executed in one or more counterparts, all of which shall constitute one and the same instrument. Any such counterpart, to the extent delivered by means of a facsimile machine or by .pdf, .tif, ..gif, .peg or similar attachment to electronic mail (any such delivery, an “Electronic Delivery”) shall be treated in all manner and respects as an original executed counterpart and shall be considered to have the same binding legal effect as if it were the original signed version thereof delivered in person. At the request of any Party, each other Party shall re-execute the original form of this Agreement and deliver such form to all other parties. No Party shall raise the use of Electronic Delivery to deliver a signature or the fact that any signature or agreement or instrument was transmitted or communicated through the use of Electronic Delivery as a defense to the formation of a contract, and each such party forever waives any such defense, except to the extent such defense relates to lack of authenticity.

7.10 Governing Law. All questions concerning the construction, validity, enforcement and interpretation of this Agreement shall be governed by the internal Law of the State of Delaware without giving effect to any choice of law or conflict of law provision or rule (whether of the State of Delaware or any other jurisdictions) that would cause the application of the laws of any jurisdictions other than the State of Delaware.

7.11 WAIVER OF TRIAL BY JURY. EACH PARTY HERETO HEREBY IRREVOCABLY WAIVES ALL RIGHT TO TRIAL BY JURY IN ANY LITIGATION, ACTION, PROCEEDING, CROSS-CLAIM, OR COUNTERCLAIM IN ANY COURT (WHETHER BASED ON CONTRACT, TORT, OR OTHERWISE) ARISING OUT OF, RELATING TO OR IN CONNECTION WITH (I) THIS AGREEMENT OR THE VALIDITY, PERFORMANCE, INTERPRETATION, COLLECTION OR ENFORCEMENT HEREOF OR

8

(II) THE ACTIONS OF THE PARTIES IN THE NEGOTIATION, AUTHORIZATION, EXECUTION, DELIVERY, ADMINISTRATION, PERFORMANCE OR ENFORCEMENT HEREOF.

*        *        *

9

IN WITNESS WHEREOF, the Parties hereto have executed this Transfer Agreement as of the date first above written.

Acknowledged and Agreed with respect to Sections 2.04 and 6.01(a) only:

Signature Page to Transfer Agreement

Exhibit A

Form of Promissory Note

See attached. (Exhibit 10.5)

Exhibit B

Form of Escrow Agreement

See attached. (Exhibit 10.65)