CASUAL MALE RETAIL GROUP, INC.

Exhibit 10.5

THIS NOTE HAS NOT BEEN REGISTERED UNDER THE U.S. SECURITIES ACT OF 1933, AS AMENDED (THE “SECURITIES ACT”), AND, ACCORDINGLY, MAY NOT BE OFFERED, SOLD, PLEDGED OR OTHERWISE TRANSFERRED, EXCEPT AS SET FORTH IN THE NEXT SENTENCE. BY ITS ACQUISITION HEREOF OR OF A BENEFICIAL INTEREST HEREIN, THE HOLDER: (1) REPRESENTS THAT IT IS AN “ACCREDITED INVESTOR” (AS DEFINED IN RULE 501(a)(1), (2), (3) OR (7) OF REGULATION D UNDER THE SECURITIES ACT (AN “AI”), (2) AGREES THAT IT WILL NOT RESELL OR OTHERWISE TRANSFER THIS NOTE EXCEPT (A) TO THE COMPANY, (B) IN A TRANSACTION MEETING THE REQUIREMENTS OF RULE 144 UNDER THE SECURITIES ACT, (C) TO AN AI THAT, PRIOR TO SUCH TRANSFER, FURNISHES THE COMPANY A SIGNED LETTER CONTAINING CERTAIN CUSTOMARY REPRESENTATIONS AND AGREEMENTS RELATING TO THE TRANSFER OF THIS NOTE (THE FORM OF WHICH CAN BE OBTAINED FROM THE COMPANY) AND AN OPINION OF COUNSEL ACCEPTABLE TO THE COMPANY THAT SUCH TRANSFER IS IN COMPLIANCE WITH THE SECURITIES ACT, OR (D) IN ACCORDANCE WITH ANOTHER EXEMPTION FROM THE REGISTRATION REQUIREMENTS OF THE SECURITIES ACT (AND BASED UPON AN OPINION OF COUNSEL ACCEPTABLE TO THE COMPANY) AND, IN EACH CASE, IN ACCORDANCE WITH THE APPLICABLE SECURITIES LAWS OF ANY STATE OF THE UNITED STATES OR ANY OTHER APPLICABLE JURISDICTION, AND (3) AGREES THAT IT WILL DELIVER TO EACH PERSON TO WHICH THIS NOTE OR AN INTEREST HEREIN IS TRANSFERRED A NOTICE SUBSTANTIALLY TO THE EFFECT OF THIS LEGEND, AND (4) FURTHER AGREES THAT IT MAY NOT RESELL OR OTHERWISE TRANSFER THIS NOTE EXCEPT (X) IN DENOMINATIONS OF NOT LESS THAN $250,000 AND (Y) WITH THE PRIOR WRITTEN CONSENT OF THE COMPANY, WHICH CONSENT SHALL NOT UNREASONABLY BE WITHHELD.

THIS NOTE IS SUBJECT AND SUBORDINATE TO THE LIABILITIES OF CASUAL MALE RETAIL GROUP, INC. DUE OR TO BECOME DUE TO FLEET RETAIL FINANCE INC., AS AGENT, PURSUANT TO A SUBORDINATION AGREEMENT DATED                              , 2003.

CASUAL MALE RETAIL GROUP, INC.

12% Senior Subordinated Note

due 2010

CASUAL MALE RETAIL GROUP, INC., a Delaware corporation (the “Company”, which term includes any successor corporation), for value received, promises to pay to                             , or registered assigns, the principal sum of                              ($                    ) on July 2, 2010.

Interest Payment Dates: July 31 and January 31.

Reference is made to the further provisions of this Security contained herein, which will for all purposes have the same effect as if set forth at this place.

IN WITNESS WHEREOF, the Company has caused this Security to be signed manually or by facsimile by its duly authorized officers.

CASUAL MALE RETAIL GROUP, INC.

12% Senior Subordinated Note

due 2010

CASUAL MALE RETAIL GROUP, INC., a Delaware corporation (the “Company”), promises to pay interest on the principal amount of this Security at the rate per annum shown above. The Company will pay interest semi-annually on July 31 and January 31 of each year (each an “Interest Payment Date”), commencing July 31, 2003. Interest on the Securities will accrue from the most recent date on which interest has been paid in full or, if interest has not been paid in full, from                     , 2003. Interest will be computed on the basis of a 360-day year of twelve 30-day months or in the case of a partial month, the actual number of days elapsed.

The Company shall pay interest at the rate of interest then borne by the Securities on overdue installments of principal and on overdue installments of interest to the extent lawful as provided in the Note Agreement. The interest rate in respect of any overdue installment of interest on the Securities which is not paid when due by virtue of Article 8 of the Note Agreement (as defined below) shall be increased by 500 basis points, to a rate of 17% per annum.

The Company shall pay interest on the Securities (except defaulted interest) to the persons who are the registered Holders at the close of business on the Interest Payment Date. Holders must surrender Securities to the Company at its principal place of business to collect principal payments. The Company shall pay principal and interest in money of the United States that at the time of payment is legal tender for payment of public and private debts (“U.S. Legal Tender”). However, the Company may pay principal and interest by wire transfer of Federal funds, or interest by check payable in such U.S. Legal Tender. The Company shall deliver any such interest payment to a Holder at the Holder’s registered address.

The Company issued the Securities under a Note Agreement, dated as of July 2, 2003 (the “Note Agreement”), by and among the Company, the Guarantors named therein and the Initial Purchasers. Capitalized terms herein are used as defined in the Note Agreement unless otherwise defined herein. The terms of the Securities include those stated in the Note Agreement and as it may be amended from time to time.

The Securities will be redeemable, at the Company’s option, in whole at any time or in part from time to time after July 2, 2004, upon not less than 15 nor more than 30 days’ notice, at the redemption prices set forth below (expressed as percentages of the principal

amount of Securities so redeemed), plus, in each case, accrued and unpaid interest to the date of redemption.

Notice of redemption will be sent, by first class mail, postage prepaid, at least 15 days but not more than 30 days before the Redemption Date to each Holder of Securities to be redeemed at such Holder’s registered address.

Except as set forth in the Note Agreement, unless the Company defaults in the payment of such Redemption Price plus accrued and unpaid interest, if any, the Securities called for redemption will cease to bear interest from and after such Redemption Date and the only right of the Holders of such Securities will be to receive payment of the Redemption Price plus accrued and unpaid interest, if any to the Redemption Date.

Section 4.13 of the Note Agreement provides that, upon the occurrence of a Change of Control and subject to further limitations contained therein, the Company will make an offer to purchase the Securities in accordance with the procedures set forth in the Note Agreement.

The Securities are in registered form, without coupons. A Holder shall register the transfer of or exchange of Securities in accordance with the Note Agreement. The Company may require a Holder, among other things, to furnish appropriate endorsements and transfer documents and to pay certain transfer taxes or similar governmental charges payable in connection therewith as permitted by the Note Agreement. The Company need not register the transfer of or exchange of any Securities or portions thereof (i) during a period beginning at the opening of business 15 days before the mailing of a notice of redemption of Securities and ending at the close of business on the day of such mailing and (ii) selected for redemption, except the unredeemed portion of any Security being redeemed in part.

The registered Holder of a Security shall be treated as the owner of it for all purposes.

Subject to certain exceptions, the Note Agreement and the Securities may be amended or supplemented with the written consent of the Holders of at least 75% in aggregate principal amount of the Securities then outstanding, and any existing Default or Event of Default or compliance with any provision may be waived with the consent of the Holders of a majority in aggregate principal amount of the Securities then outstanding. Without notice to or consent of any Holder, the Company may amend or supplement the Note Agreement and the Securities to, among other things, cure any ambiguity, defect or inconsistency or make any other change that does not adversely affect the rights of any Holder of a Security in any material respect.

The Note Agreement contains certain covenants that, among other things, limit the ability of the Company and the Subsidiaries to incur additional Indebtedness, create certain Liens, pay dividends or make certain other Restricted Payments, consummate certain Asset Sales, enter into certain transactions with Affiliates and merge or consolidate with any other Person or sell, assign, transfer, lease, convey or otherwise dispose of all or substantially all of the assets of the Company. The limitations are subject to a number of important qualifications and exceptions.

The Indebtedness evidenced by the Securities is, to the extent and in the manner provided in the Note Agreement, subordinated in right of payment to the prior payment in full in cash of all Senior Debt, and this Security is issued subject to such provisions. Each Holder of this Security, by accepting the same, agrees to and shall be bound by such provisions.

If an Event of Default occurs and is continuing, the Holders of at least 25% in aggregate principal amount of Securities then outstanding may declare the principal of and accrued interest on all the Securities to be due and payable immediately in the manner and with the effect provided in the Note Agreement. Holders of Securities may not enforce the Note Agreement or the Securities except as provided in the Note Agreement. The Note Agreement permits, subject to certain limitations therein provided, Holders of a majority in aggregate principal amount of the Securities then outstanding to exercise any power.

No stockholder, director, officer, employee or incorporator, as such, of the Company shall have any liability for any obligation of the Company under the Securities or the Note Agreement or for any claim based on, in respect of or by reason of, such Obligations or their creation. Each Holder of a Security by accepting a Security waives and releases all such liability. The waiver and release are part of the consideration for the issuance of the Securities.

This Security will be entitled to the benefits of certain Guarantees made for the benefit of the Holders. Reference is hereby made to the Note Agreement for a statement of the

respective rights, limitations of rights, duties and Obligations thereunder of the Guarantors and the Holders.

Customary abbreviations may be used in the name of a Holder of a Security or an assignee, such as: TEN COM (= tenants in common), TEN ENT (= tenants by the entireties), JT TEN (= joint tenants with right of survivorship and not as tenants in common), CUST (= Custodian), and U/G/M/A (= Uniform Gifts to Minors Act).

The Company will furnish to any Holder of a Security upon written request and without charge a copy of the Note Agreement. Requests may be made to: Casual Male Retail Group, Inc., 555 Turnpike Street, Canton, Massachusetts 02021, Attn: Chief Financial Officer.

This Security and the Note Agreement shall be governed by and construed in accordance with the laws of the State of New York, as applied to contracts made and performed within the State of New York, without regard to principles of conflict of laws. Each of the parties hereto agrees to submit to the jurisdiction of the courts of the State of New York in any action or proceeding arising out of or relating to this Security.

ASSIGNMENT FORM

I or we assign and transfer this Security to:

Participant in a recognized Signature Guarantee Medallion Program (or other signature guarantor program reasonably acceptable to the Company)

OPTION OF HOLDER TO ELECT PURCHASE

If you want to elect to have this Security purchased by the Company pursuant to Section      of the Note Agreement, check the box:

If you want to elect to have only part of this Security purchased by the Company pursuant to      of the Note Agreement, state the amount: $                    

Dated:                                                 

Signed:                                                 

(Sign exactly as name appears on the other side of this Security)

Signature Guarantee:                                                             

Participant in a recognized Signature Guarantee Medallion Program (or other signature guarantor program reasonably acceptable to the Company)