TRADEMARK SECURITY AGREEMENT

Exhibit 10.4

[EXECUTION COPY]

TRADEMARK SECURITY AGREEMENT

This TRADEMARK SECURITY AGREEMENT, dated as of April 23, 2008 (this “Agreement”), is made by ANNCO, INC., a Delaware corporation (“Grantor”), in favor of BANK OF AMERICA, N.A., in its capacity as administrative agent for each of the Lenders now or hereafter party to the Credit Agreement (the “Agent”).

WITNESSETH:

WHEREAS, ANNTAYLOR, INC., a Delaware corporation (“ATI”), Grantor, ANNTAYLOR DISTRIBUTION SERVICES, INC., a Delaware corporation (“ATDS”), ANNTAYLOR RETAIL, INC., a Delaware corporation (“ATR”, and together with ATI, Grantor and ATDS, the “Borrowers” and each, individually, a “Borrower”), the Agent, the lenders referred to therein, the syndication agents named therein, and the issuing banks named therein have entered into that certain $175,000,000 Second Amended and Restated Credit Agreement dated November 14, 2003 (the “Original Credit Agreement”); and

WHEREAS, at the Borrowers’ request, Lenders, the Agent, and Banc of America Securities LLC, as Lead Arranger (in such capacity, the “Arranger”) have agreed to amend and restate the Original Credit Agreement in its entirety pursuant to that certain Third Amended and Restated Credit Agreement dated as of the date hereof (as so amended and restated and as such agreement may be further amended, supplemented, modified, or amended and restated from time to time, the “Credit Agreement”); and

WHEREAS, each of Grantor, ATDS and ATR is, directly or indirectly, a wholly owned Subsidiary of ATI and has and will materially benefit from the Loans and Advances made and to be made, and the Letters of Credit issued and to be issued, under the Credit Agreement; and

WHEREAS, the Borrowers, AnnTaylor Stores Corporation, a Delaware corporation, and the Agent have entered into that certain Third Amended and Restated Pledge and Security Agreement, dated as of the date hereof (as so amended and restated and as such agreement may be further amended, supplemented, modified, or amended and restated from time to time, the “Security Agreement”); and

WHEREAS, as collateral security for payment and performance by the Grantor of its Obligations, the Grantor is willing to continue, amend and grant to the Agent, for the benefit of the Agent and the Lenders, as the case may be, a security interest in certain of its personal property and assets pursuant to the terms of this Agreement, including, without limitation, the Collateral referred to in Section 2 below;

NOW, THEREFORE, in consideration of the above premises and in order to induce the Lenders and the Letter of Credit Issuer to amend and restate the Original Credit Agreement and continue to, respectively, make Loans and issue Letters of Credit under the Credit Agreement, Grantor hereby agrees with the Agent for its benefit, and for the benefit of the Lenders and the Letter of Credit Issuer, by acceptance hereof, as follows:

1. DEFINED TERMS. All capitalized terms used but not otherwise defined in this Agreement have the meanings given to them in the Security Agreement. All other undefined terms contained in this Agreement, unless the context indicates otherwise, have the meanings provided for by the UCC to the extent the same are used or defined therein.

Trademark Security Agreement

2. GRANT OF SECURITY. As security for all Obligations, Grantor hereby pledges, assigns, charges, mortgages, delivers, transfers and grants to the Agent, for the benefit of the Agent and the Lenders, a continuing security interest in, lien on, and right of set-off against, all of the following property and assets of such Grantor, whether now owned or existing or hereafter acquired or arising, regardless of where located (the “Collateral”):

(a) all of each Grantor’s right, title and interest, whether now owned or hereafter acquired, in and to all United States and foreign trademarks, trade names, trade dress, service marks, trademark and service mark registrations, and applications for trademark or service mark registration and any renewals thereof (including without limitation each trademark and service mark registration and application for registration identified in Schedule A attached hereto and incorporated herein by reference) and including all income, royalties, damages and payments now and hereafter due and/or payable with respect thereto (including without limitation damages for past or future infringements thereof), the right to recover for all past, present and future infringements thereof, all rights corresponding thereto throughout the world (but only such rights as now exist or may come to exist under applicable local law) and all other rights of any kind whatsoever of each Grantor accruing thereunder or pertaining thereto, together in each case with the goodwill of the business connected with the use of, and symbolized by, each trademark and service mark (collectively, the “Trademarks”); and

(b) any and all Proceeds of the foregoing.

3. RECORDATION. Grantor authorizes and requests that the Commissioner of Patents and Trademarks and any other applicable government officer record this Agreement.

4. EXECUTION IN COUNTERPARTS. This Agreement may be executed in any number of counterparts, each of which when so executed shall be deemed to be an original and all of which taken together shall constitute one and the same agreement.

5. GRANTS, RIGHTS AND REMEDIES. This Agreement has been entered into in conjunction with the provisions of the Security Agreement. Grantor does hereby acknowledge and confirm that the grant of the security interest hereunder to, and the rights and remedies of, the Agent with respect to the Collateral are more fully set forth in the Security Agreement, the terms and provisions of which are incorporated herein by reference as if fully set forth herein.

6. GOVERNING LAW. This Agreement shall be governed by, and construed in accordance with, the laws of the State of New York.

Trademark Security Agreement

- 2 -

IN WITNESS WHEREOF, the parties hereto have executed this Agreement by their duly authorized officers as of the day and year first above written.

Signature Page

Trademark Security Agreement

STATE OF                                                              

COUNTY OF                                                              

I,                                                              , a Notary Public for said County and State, do hereby certify that                                                               personally came before me this day and acknowledged that (s)he is                                          of                             , a                              , and acknowledged, on behalf of                                                              , the due execution of the foregoing instrument.

Witness my hand and official seal, this the              day of                     ,             .

(Official Seal)

My Commission expires                                                              

Signature Page

Trademark Security Agreement