THIRD AMENDMENT TO FOURTH AMENDED AND RESTATED LOAN AND SECURITY AGREEMENT

THIRD AMENDMENT TO

FOURTH AMENDED AND RESTATED LOAN AND SECURITY AGREEMENT

This Third Amendment to Fourth Amended and Restated Loan and Security Agreement (the “Third Amendment”) is made as of this 31st day of January, 2006 by and among

CASUAL MALE RETAIL GROUP, INC., and DESIGNS APPAREL, INC. (referred to individually as a “Borrower” and collectively as the “Borrowers”); and

CASUAL MALE RETAIL GROUP, INC., as Borrowers’ Representative for the Borrowers; and

FLEET RETAIL GROUP, LLC, (formerly known as FLEET RETAIL GROUP, INC.), HELLER FINANCIAL, INC., NATIONAL CITY BUSINESS CREDIT, INC., WELLS FARGO FOOTHILL, INC., WELLS FARGO BUSINESS CREDIT, INC., LASALLE RETAIL FINANCE, A DIVISION OF LASALLE BUSINESS CREDIT, INC., AGENT FOR STANDARD FEDERAL BANK NATIONAL ASSOCIATION, and WEBSTER BUSINESS CREDIT CORPORATION (together with each of their successors and assigns, referred to individually as a “Revolving Credit Lender” and collectively as the “Revolving Credit Lenders”); and

FLEET RETAIL GROUP, LLC, (formerly known as FLEET RETAIL GROUP, INC.), as SwingLine Lender; and

FLEET RETAIL GROUP, LLC, (formerly known as FLEET RETAIL GROUP, INC.), as Tranche B Lender (together with the Revolving Credit Lenders and the SwingLine Lender, the “Lenders”); and

FLEET RETAIL GROUP, LLC, (formerly known as FLEET RETAIL GROUP, INC.), as Administrative Agent and Collateral Agent for the Lenders; and

WELLS FARGO FOOTHILL, INC., as Syndication Agent; and

NATIONAL CITY BUSINESS CREDIT, INC. and HELLER FINANCIAL, INC., as Co-Documentation Agents (together with the Administrative Agent, Collateral Agent and Syndication Agent, the “Agents”).

in consideration of the mutual covenants herein contained and benefits to be derived herefrom.

W I T N E S S E T H

A. Reference is made to the Fourth Amended and Restated Loan and Security Agreement dated as of October 29, 2004 by and among the Borrowers, the Borrowers’ Representative, the Lenders and the Agents, as amended by that certain First Amendment to Fourth Amended and Restated Loan and Security Agreement dated March 16, 2005, as amended by that certain Second Amendment to Fourth Amended and Restated Loan and Security Agreement dated December 15, 2005 (as amended and in effect the “Credit Agreement”).

B. The Borrowers have requested that the Agents and the Lenders agree to amend the Credit Agreement to allow the Loan Parties to (i) enter into a certain sale leaseback transaction; (ii) prepay the Convertible Notes; and (iii) prepay the Tranche B Note.

C. The Agents and the Lenders, subject to the terms and conditions of this Third Amendment have agreed to modify the Credit Agreement.

Accordingly, the Agents, the Lenders, the Loan Parties , and the Borrowers’ Representative agree as follows:

1. Definitions. Capitalized terms used herein and not otherwise defined herein shall have the meanings assigned to such terms in the Credit Agreement.

2. Amendment to Credit Agreement.

“Permitted Sale Leaseback Transaction”: A sale leaseback transaction entered in connection with the Canton Property, provided that (i) (A) such sale is made for fair market value, as determined by the Administrative Agent, in its reasonable discretion, (B) the lease is on market terms, as determined by the Administrative Agent, in its reasonable discretion, (C) sale is made to a Non-Affiliate, (D) the sale is for cash consideration, and (E) a portion of the proceeds are used to retire all the outstanding Obligations pursuant to the Tranche B Note, subject to the term of Section 3.3(b) ; and (ii) in connection with the lease, lessor enters into a landlord waiver and access agreement reasonably satisfactory to the Administrative Agent.

2

Except as specifically set forth in Section 3.3(a) with respect to the Tranche B Amortization Payment, the Borrowers may not prepay all or any portion of the principal balance of the Tranche B Loan prior to the Maturity Date or Acceleration unless each of the Tranche B Loan Prepayment Conditions is satisfied in connection with such prepayment; provided, however, that notwithstanding the foregoing, so long as no Default shall have occurred and be continuing on the date on which such prepayment is made and immediately after giving effect to such prepayment, the Tranche B Loan may be prepaid without satisfaction of the Tranche B Loan Prepayment Conditions if (A) the sole source of funds for such prepayment shall be proceeds from either (i) the sale of equity in Casual Male on terms and conditions and subject to execution of documentation satisfactory to the Administrative Agent; or (ii) the Permitted Sale Leaseback Transaction, and (B) the sales proceeds from such Permitted Sale Leaseback Transaction are equal to or greater than $50,000,000. Subject to satisfaction of the Tranche B Loan Prepayment Conditions or the exception described in the preceding sentence, any prepayment of the Tranche B Loan shall be without penalty or premium.

“(iii) with respect to the Convertible Notes (a) regularly scheduled payments of interest so long as of the date of such payment, and after giving effect thereto, there exists no Default; (b) payment, prepayment and/or retirement of Indebtedness thereunder, so long as (x) no more than fifteen days prior thereto, the Borrowers’ Representative has provided the Administrative Agent with a forecast for the next succeeding six (6) month period which reflects that Excess Availability will never be less than $30,000,000; and (y) on the date of any such payment, and immediately after giving effect thereto,(1) there exist no Default; and (2) Excess Availability is greater than $30,000,000.00.”

3

3. Ratification of Loan Documents. Except as otherwise provided for herein, the terms and conditions of the Credit Agreement and of the other Loan Documents remain in full force and effect, and each Loan Party hereby ratifies, confirms and reaffirms, all and singular, the terms and conditions of, and the warranties and representations set forth, therein.

4. Conditions Precedent to Effectiveness. This Third Amendment shall not be effective until each of the following conditions precedent have been fulfilled to the satisfaction of the Administrative Agent:

a. This Third Amendment shall have been duly executed and delivered by the respective parties hereto, and, shall be in full force and effect.

b. All action on the part of the Loan Parties necessary for the valid execution, delivery and performance by the Loan Parties of this Third Amendment shall have been duly and effectively taken and evidence thereof satisfactory to the Administrative Agent shall have been provided to the Administrative Agent.

c. The Loan Parties shall have provided such additional instruments and documents to the Administrative Agent as the Administrative Agent and the Administrative Agent’s counsel may have reasonably requested.

d. The Loan Parties shall have paid to the Administrative Agent, for the benefit of the Lenders, an amendment fee in the amount of $45,000.00.

5. Miscellaneous.

a. This Third Amendment may be executed in several counterparts and by each party on a separate counterpart, each of which when so executed and delivered shall be an original, and all of which together shall constitute one instrument.

b. This Third Amendment expresses the entire understanding of the parties with respect to the transactions contemplated hereby. No prior negotiations or discussions shall limit, modify, or otherwise affect the provisions hereof.

c. Any determination that any provision of this Third Amendment or any application hereof is invalid, illegal or unenforceable in any respect and in any instance shall not effect the validity, legality or enforceability of such provision in any other instance, or the validity, legality or enforceability of any other provisions of this Third Amendment.

4

d. The Loan Parties shall pay on demand all reasonable costs and expenses of the Administrative Agent, including, without limitation, reasonable attorneys’ fees in connection with the preparation, negotiation, execution and delivery of this Third Amendment.

e. Each Loan Party warrants and represents that the Loan Party has consulted with independent legal counsel of each Loan Party’s selection in connection with this Third Amendment and is not relying on any representations or warranties of the Administrative Agent or its counsel in entering into this Third Amendment.

[The remainder of this page is intentionally left blank]

5

IN WITNESS WHEREOF, the parties have duly executed this Third Amendment as of the day and year first above written.

6

7

8

9

10