CASUAL MALE RETAIL GROUP, INC. 2006 INCENTIVE COMPENSATION PLAN

Exhibit 10.4

CASUAL MALE RETAIL GROUP, INC.

2006 INCENTIVE COMPENSATION PLAN

NON-QUALIFIED STOCK OPTION AGREEMENT

(For Non-Employee Directors)

Agreement

1. Grant of Option. CASUAL MALE RETAIL GROUP, INC., a Delaware corporation (the “Company”), hereby grants, as of                      (“Date of Grant”), to                      (the “Optionee”) an option (the “Option”) to purchase up to              shares of the Company’s common stock, $.01 par value per share (the “Shares”), at an exercise price per share equal to $             (the “Exercise Price”). The Option shall be subject to the terms and conditions set forth herein. The Option was issued pursuant to the Company’s 2006 Incentive Compensation Plan (the “Plan”), which is incorporated herein for all purposes. The Option is a Non-Qualified Stock Option, and not an Incentive Stock Option. The Optionee hereby acknowledges receipt of a copy of the Plan and agrees to be bound by all of the terms and conditions hereof and thereof and all applicable laws and regulations.

2. Definitions. Unless otherwise provided herein, terms used herein that are defined in the Plan and not defined herein shall have the meanings attributed thereto in the Plan.

3. Exercise Schedule. Except as otherwise provided in Sections 6 or 9 of this Agreement, or in the Plan, the Option is exercisable in installments as provided below, which shall be cumulative. To the extent that the Option has become exercisable with respect to a percentage of Shares as provided below, the Option may thereafter be exercised by the Optionee, in whole or in part, at any time or from time to time prior to the expiration of the Option as provided herein. The following table indicates each date (the “Vesting Date”) upon which the Optionee shall be entitled to exercise the Option with respect to the percentage of Shares granted as indicated beside the date, provided that the Continuous Service of the Optionee continues through and on the applicable Vesting Date:

Except as otherwise specifically provided herein, there shall be no proportionate or partial vesting in the periods prior to each Vesting Date, and all vesting shall occur only on the appropriate Vesting Date. Upon the termination of the Optionee’s Continuous Service with the Company and its Related Entities, any unvested portion of the Option shall terminate and be null and void, except as may otherwise be determined by the Committee in writing in its sole discretion.

4. Method of Exercise. The vested portion of this Option shall be exercisable in whole or in part in accordance with the exercise schedule set forth in Section 3 hereof by written notice which shall state the election to exercise the Option, the number of Shares in respect of which the Option is being exercised, and such other representations and agreements as to the holder’s investment intent with respect to such Shares as may be required by the Company pursuant to the provisions of the Plan. Such written notice shall be signed by the Optionee and shall be delivered in person or by certified mail to the Secretary of the Company. The written notice shall be accompanied by payment of the Exercise Price. This Option shall be deemed to be exercised after both (a) receipt by the Company of such written notice accompanied by the Exercise Price and (b) arrangements that are satisfactory to the Committee in its sole discretion have been made for Optionee’s payment to the Company of the amount, if any, that is necessary to be withheld in accordance with applicable Federal or state withholding requirements. No Shares shall be issued pursuant to the Option unless and until such issuance and such exercise shall comply with all relevant provisions of applicable law, including the requirements of any stock exchange upon which the Shares then may be traded.

5. Method of Payment. Payment of the Exercise Price shall be by any of the following, or a combination thereof, at the election of the Optionee: (a) cash; (b) check; (c) to the extent permitted by the Committee, with Shares owned by the Optionee, or the withholding of Shares that otherwise would be delivered to the Optionee as a result of the exercise of the Option, (d) pursuant to a “cashless exercise” procedure, by delivery of a properly executed exercise notice together with such other documentation, and subject to such guidelines, as the Committee shall require to effect an exercise of the Option and delivery to the Company by a licensed broker acceptable to the Company of proceeds from the sale of Shares, or (e) such other consideration or in such other manner as may be determined by the Committee in its absolute discretion.

6. Termination of Option.

(a) Any unexercised portion of the Option shall automatically and without notice terminate and become null and void at the time of the earliest to occur of the following:

(i) unless the Committee otherwise determines in writing in its sole discretion, three months after the date on which the Optionee’s Continuous Service is terminated other than by reason of (A) by the Company or a Related Entity for Cause, (B) a Disability of the Optionee as determined by a medical doctor satisfactory to the Committee, or (C) the death of the Optionee;

(ii) immediately upon the termination of the Optionee’s Continuous Service by the Company or a Related Entity for Cause;

(iii) twelve months after the date on which the Optionee’s Continuous Service is terminated by reason of a Disability as determined by a medical doctor satisfactory to the Committee;

(iv) (A) twelve months after the date of termination of the Optionee’s Continuous Service by reason of the death of the Optionee, or, if later, (B) three months after the date on which the Optionee shall die if such death shall occur during the one year period specified in Section 6(a)(iii) hereof; or

2

(v) the tenth anniversary of the date as of which the Option is granted.

(b) To the extent not previously exercised, (i) the Option shall terminate immediately in the event of (A) the liquidation or dissolution of the Company, or (B) any reorganization, merger, consolidation or other form of corporate transaction in which the Company does not survive or the Shares are exchanged for or converted into securities issued by another entity, or an affiliate of such successor or acquiring entity, unless the successor or acquiring entity, or an affiliate thereof, assumes the Option or substitutes an equivalent option or right pursuant to Section 10(c) of the Plan, and (ii) the Committee in its sole discretion may by written notice (“cancellation notice”) cancel, effective upon the consummation of any corporate transaction described in Section 9(b)(i) of the Plan, the Option (or portion thereof) that remains unexercised on such date. The Committee shall give written notice of any proposed transaction referred to in this Section 6(b) a reasonable period of time prior to the closing date for such transaction (which notice may be given either before or after approval of such transaction), in order that the Optionee may have a reasonable period of time prior to the closing date of such transaction within which to exercise the Option if and to the extent that it then is exercisable (including any portion of the Option that may become exercisable upon the closing date of such transaction). The Optionee may condition his exercise of the Option upon the consummation of a transaction referred to in this Section 6(b).

7. Transferability. Unless otherwise determined by the Committee, the Option granted hereby is not transferable otherwise than by will or under the applicable laws of descent and distribution, and during the lifetime of the Optionee the Option shall be exercisable only by the Optionee, or the Optionee’s guardian or legal representative. In addition, the Option shall not be assigned, negotiated, pledged or hypothecated in any way (whether by operation of law or otherwise), and the Option shall not be subject to execution, attachment or similar process. Upon any attempt to transfer, assign, negotiate, pledge or hypothecate the Option, or in the event of any levy upon the Option by reason of any execution, attachment or similar process contrary to the provisions hereof, the Option shall immediately become null and void. The terms of this Option shall be binding upon the executors, administrators, heirs, successors and assigns of the Optionee.

8. No Rights of Stockholders. Neither the Optionee nor any personal representative (or beneficiary) shall be, or shall have any of the rights and privileges of, a stockholder of the Company with respect to any Shares purchasable or issuable upon the exercise of the Option, in whole or in part, prior to the date of exercise of the Option.

9. Acceleration of Exercisability of Option.

(a) This Option shall become immediately fully exercisable upon the termination of the Optionee’s Continuous Service other than a termination by the Company or a Related Entity for Cause.

3

(b) This Option shall become immediately fully exercisable in the event that, prior to the termination of the Option pursuant to Section 6 hereof, and during the Optionee’s Continuous Service, there is a “Change in Control,” as defined in Section 9(b) of the Plan.

(c) This Option shall become immediately fully exercisable in the event that, prior to the termination of the Option pursuant to Section 6 hereof, (i) the Option is terminated pursuant to Section 6(b)(i) hereof, or (ii) the Company exercises its discretion to provide a cancellation notice with respect to the Option pursuant to Section 6(b)(ii) hereof.

10. No Right to Continued Employment or Service with the Company. Neither the Option nor this Agreement shall confer upon the Optionee any right to continued employment or service with the Company.

11. Law Governing. This Agreement shall be governed in accordance with and governed by the internal laws of the State of Delaware.

12. Interpretation / Provisions of Plan Control. This Agreement is subject to all the terms, conditions and provisions of the Plan, including, without limitation, the amendment provisions thereof, and to such rules, regulations and interpretations relating to the Plan adopted by the Committee as may be in effect from time to time. If and to the extent that this Agreement conflicts or is inconsistent with the terms, conditions and provisions of the Plan, the Plan shall control, and this Agreement shall be deemed to be modified accordingly. The Optionee accepts the Option subject to all of the terms and provisions of the Plan and this Agreement. The undersigned Optionee hereby accepts as binding, conclusive and final all decisions or interpretations of the Committee upon any questions arising under the Plan and this Agreement, unless shown to have been made in an arbitrary and capricious manner.

13. Notices. Any notice under this Agreement shall be in writing and shall be deemed to have been duly given when delivered personally or when deposited in the United States mail, registered, postage prepaid, and addressed, in the case of the Company, to the Company’s Secretary at 555 Turnpike Street, Canton, MA 02021, or if the Company should move its principal office, to such principal office, and, in the case of the Optionee, to the Optionee’s last permanent address as shown on the Company’s records, subject to the right of either party to designate some other address at any time hereafter in a notice satisfying the requirements of this Section.

4

IN WITNESS WHEREOF, the undersigned have executed this Agreement as of the      day of                     , 200  .

The Optionee acknowledges receipt of a copy of the Plan and represents that he or she has reviewed the provisions of the Plan and this Option Agreement in their entirety, is familiar with and understands their terms and provisions, and hereby accepts this Option subject to all of the terms and provisions of the Plan and the Option Agreement. The Optionee further represents that he or she has had an opportunity to obtain the advice of counsel prior to executing this Option Agreement.

5