Letter Agreement for Exchange of Loans into Common Stock between TPG-MD Investment, LLC and J. Crew Group, Inc. (August 16, 2005)

Exhibit 10.4

EXECUTION VERSION

TPG-MD Investment, LLC

August 16, 2005

J. Crew Group, Inc.

770 Broadway

New York, New York 10003

Ladies and Gentlemen:

We refer to the Credit Agreement, dated as of February 4, 2003, by and among TPG-MD Investment, LLC, J. Crew Operating Corp., J. Crew Group, Inc. and the guarantors named therein (as amended by Amendment No. 1 to the Credit Agreement, dated as of November 21, 2004, the “Credit Agreement”). This letter agreement (the “Letter Agreement”) confirms our understanding with you regarding our agreement to exchange the Loans made under the Credit Agreement into the shares of common stock of J. Crew Group, Inc. (together with its successors and assigns, “J. Crew Group”), par value $0.01 per share (“J. Crew Group Common Stock”). Any capitalized terms used but not defined in this Letter Agreement shall have the meanings assigned to them in the Credit Agreement.

1. In order to facilitate the proposed IPO of J. Crew Group and subject to the terms and conditions set forth in this Letter Agreement, we hereby agree to exchange the Principal Amount of the Loans and the accrued and unpaid interest thereon into J. Crew Group Common Stock in accordance with Article VII of the Credit Agreement.

2. Our agreement set forth in paragraph 1 above shall be subject to the satisfaction (or the waiver) of the conditions to closing set forth in an underwriting agreement to be entered into by and among J. Crew Group and the underwriters to be named therein in connection with the IPO of J. Crew Group, other than any condition requiring the delivery of notice described in paragraph 3 below.

3. Pursuant to Section 7.01 of the Credit Agreement, upon the satisfaction (or the waiver) of such conditions, we shall deliver a written notice to J. Crew Group setting forth the Principal Amount of the Loans and the accrued and unpaid interest thereon as of the Exercise Date, the Exercise Price (subject to adjustment pursuant to Section 7.02 of the Credit Agreement) and the amount of J. Crew Group Common Stock to be issued by J. Crew Group, together with the certificates evidencing the Tranche A Note and the Tranche B Note (or, in lieu thereof, a lost Note affidavit together with an indemnity against third party claims reasonably satisfactory to J. Crew Group). For purposes of the Credit Agreement, the date we deliver such written notice shall be deemed to be the Exercise Date. Upon receipt of such written notice, you agree to deliver to us the share certificates evidencing the amount of J. Crew Group Common Stock to be issued by J. Crew Group in exchange for the Loans.

4. This Letter Agreement shall be governed by and construed in accordance with the laws of the State of New York without giving effect to principles of conflicts of law.

5. This Letter Agreement may be executed in one or more counterparts and by facsimile, each of which shall be deemed an original and all of which shall constitute one and the same Letter Agreement.

2

3

Agreed and acknowledged as of the date first above written:

4