SECOND AMENDMENT TO THIRD AMENDED AND RESTATED LOAN AND SECURITY AGREEMENT

Exhibit 10.2

SECOND AMENDMENT TO

THIRD AMENDED AND RESTATED LOAN AND SECURITY AGREEMENT

This Second Amendment to Third Amended and Restated Loan and Security Agreement (the “Second Amendment”) is made as of this 3rd day of November, 2003 by and among

CASUAL MALE RETAIL GROUP, INC. (formerly known as Designs, Inc.), a Delaware corporation and DESIGNS APPAREL, INC. (referred to individually as a “Borrower” and collectively as the “Borrowers”); and

CASUAL MALE RETAIL GROUP, INC., a Delaware corporation, as Borrowers’ Representative for the Borrowers; and

FLEET RETAIL FINANCE INC., HELLER FINANCIAL, INC., NATIONAL CITY COMMERCIAL FINANCE, INC., WELLS FARGO RETAIL FINANCE LLC, WELLS FARGO BUSINESS CREDIT, INC., LASALLE RETAIL FINANCE, A DIVISION OF LASALLE BUSINESS CREDIT, INC., AGENT FOR STANDARD FEDERAL BANK NATIONAL ASSOCIATION, THE PROVIDENT BANK, and WEBSTER WHITEHALL BUSINESS CREDIT CORPORATION (together with each of their successors and assigns, referred to individually as a “Revolving Credit Lender” and collectively as the “Revolving Credit Lenders”); and

FLEET RETAIL FINANCE INC., as SwingLine Lender; and

BACK BAY CAPITAL FUNDING LLC, as Tranche B Lender (together with the Revolving Credit Lenders and the SwingLine Lender, the “Lenders”); and

FLEET RETAIL FINANCE INC., as Administrative Agent and Collateral Agent for the Lenders; and

WELLS FARGO RETAIL FINANCE, LLC, as Syndication Agent; and

NATIONAL CITY COMMERCIAL FINANCE, INC. and HELLER FINANCIAL, INC., as Co-Documentation Agents (together with the Administrative Agent, Collateral Agent and Syndication Agent, the “Agents”).

in consideration of the mutual covenants herein contained and benefits to be derived herefrom.

W I T N E S S E T H

A. Reference is made to the Third Amended and Restated Loan and Security Agreement dated as of May 14, 2002 by and among the Borrowers, the Borrowers’ Representative, the Lenders and the Agents, as amended by a certain First Amendment to Third Amended and Restated Loan and Security Agreement dated as of October 16, 2002 (the “Credit Agreement”).

B. The Agents, the Lenders, the Borrowers and the Borrowers’ Representative desire to modify the Credit Agreement as set forth herein.

Accordingly, the Agents, the Lenders, the Borrowers, and the Borrowers’ Representative agree as follows:

1. Definitions. Capitalized terms used herein and not otherwise defined herein shall have the meanings assigned to such terms in the Credit Agreement.

2. Amendments to Credit Agreement:

“Applicable Inventory Advance Rate”: The following rates for the following periods:

“Applicable Margin” The following percentages for Base Margin Loans and Libor Loans based upon the following criteria:

The Applicable Margin shall initially be established at Level III. Thereafter, the Applicable Margin shall be adjusted quarterly on the first day of each calendar quarter, commencing with the Fiscal quarter June 2003, based upon the average Excess Availability during the prior quarter,

provided that in no event shall the Applicable Margin be established at Level I or Level II during the first twelve (12) months subsequent to the Closing Date. Upon the occurrence of an Event of Default, the Applicable Margin may, at the option of the Administrative Agent, be immediately increased to the percentages set forth in Level IV (even if the Excess Availability requirements for another Level have been met) and interest shall be determined in the manner set forth in Section 2.12(f).

and inserting in lieu thereof the following:

“Designs Store”: any store operated by any one of the Designs, Inc. Companies.

“Maturity Date”: May 14, 2006.

“Permitted Asset Disposition”: The following:

(a) A sale or other disposition of the assets of any Loan Party (other than as specified in clauses (b), (c) and (d) of this definition), not in the ordinary course, so long as the following conditions are satisfied:

(i) The sale, liquidation or other disposition of Inventory at any locations from which a Loan Party determines to cease the conduct of its business, (x) shall be on terms satisfactory to the Administrative Agent and (y) notwithstanding the Administrative Agent’s furnishing of any such consent, the Administrative Agent may, in the exercise of its reasonable discretion, impose Inventory Reserves as a result of the occurrence of any such sale, liquidation, or disposition;

(ii) The aggregate of all such sales or other dispositions of assets during the term of this Agreement shall not exceed five percent (5%) of the value of all assets of Casual Male Retail Group, Inc. as of the Closing Date;

(iii) Each such sale or other disposition shall be for fair consideration in an arm’s length transaction; and

(iv) On the date on which any sale or other disposition of assets is consummated, no Default shall have occurred and be continuing or will occur as a result of such consummation.

(b) The rejection of up to fifteen (15) leases as contemplated by section 6.16 of the Casual Male Acquisition Agreement.

(c) The sale, spin-off or other disposition of Designs LPI Corp., and its ownership interests in Securex, on terms reasonably satisfactory to the Administrative Agent. The Administrative Agent shall execute and deliver such releases as shall be reasonably requested in order for such disposition to be consummated.

(d) The sale of any Collateral located in any Designs Store, provided that the conditions set forth in (a)(i), (iii) and (iv), above, are satisfied.

“Revolving Credit Casual Male Companies Inventory Advance Rate”: The following rates for the following periods:

“Revolving Credit Designs, Inc. Companies Inventory Advance Rate”: The following rates for the following periods:

“Revolving Credit Ceiling”: $90,000,000.00

(a) In the event that the Termination Date occurs, for any reason (whether by virtue of Acceleration or otherwise), prior to the Maturity Date, then except as provided in Section 2.16(b), the Borrowers shall pay the Administrative Agent, for the Pro-Rata account of the Revolving Credit Lenders, the “Revolving Credit Early Termination Fee” (so referred to herein) consisting of (i) one and one-half percent (1 ¹/2%) of the Revolving Credit Ceiling in effect as of the date of this Agreement if the Termination Date shall occur at any time prior to the first anniversary of the Closing Date and (i) one percent (1%) of the Revolving Credit Ceiling in effect as of the date of this Agreement if the Termination Date shall occur at any time after the first anniversary of the Closing Date and more than 90 days prior to the Maturity Date.

and inserting in lieu thereof the following:

(a) In the event that the Termination Date occurs, for any reason (whether by virtue of Acceleration or otherwise), prior to May 14, 2005, then except as provided in Section 2.16(b), the Borrowers shall pay the Administrative Agent, for the Pro-Rata account of the Revolving Credit Lenders, the “Revolving Credit Early Termination Fee” (so referred to herein) consisting of (i) one and one-half percent (1 ¹/2%) of the Revolving Credit Ceiling if the Termination Date shall occur at any time

prior to the first anniversary of the Closing Date and (ii) one percent (1%) of the Revolving Credit Ceiling if the Termination Date shall occur after the first anniversary of the Closing Date and on or before May 14, 2005.

(iii) investments in the ECKO Joint Venture not to exceed $5,000,000 in the aggregate during the term of this agreement.

and inserting in lieu thereof the following:

(iii) investments in the ECKO Joint Venture not to exceed $10,000,000 in the aggregate during the term of this agreement.

6.11 Financial Performance Covenants.

(a) The Loan Parties shall maintain at all times Excess Availability of not less than the lesser of: (i) $6,000,000, and (ii) 8.5% of the Borrowing Base.

(b) The Loan Parties shall not permit or suffer Capital Expenditures for the

(i) Fiscal year ending January 25, 2003, to be greater than $12,000,000;

(ii) Fiscal year ending January 30, 2004, to be greater than $15,000,000;

(iii) Fiscal year ending January 29, 2005, to be greater than $20,000,000;

(iv) Fiscal year ending January 28, 2006, to be greater than $20,000,000; and

(v) period from January 29, 2006, through the Maturity Date, to be greater than $10,000,000.

Notwithstanding the foregoing, any amount set forth in clauses (i) through (iv) which is not committed or spent in such Fiscal year may be carried over for Capital Expenditures during the next Fiscal year.

(c) In the event that the Loan Parties’ Excess Availability is less than the aggregate of (A) $10,000,000, and (B) the Excess Availability requirement pursuant to Section 6.11(a) above, the Loan Parties shall maintain EBITDA of not less than:

(i) $21,900,000 through November 30, 2003;

(ii) $22,800,000 as of December 31, 2003;

(iii) $23,000,000 as of January 31, 2004;

(iv) $23,000,000 as of February 29, 2004;

(v) $24,100,000 as of March 31, 2004;

(vi) $23,500,000 as of April 30, 2004;

(vii) $24,400,000 as of May 31, 2004;

(viii) $25,500,000 as of June 30, 2004;

(ix) $25,000,000 as of July 31, 2004;

(x) $25,000,000 as of August 31, 2004;

(xi) $25,000,000 as of September 30, 2004;

(xii) $25,500,000 as of October 31, 2004;

(xiii) $25,500,000 as of November 30, 2004;

(xiv) $26,500,000 as of December 31, 2004;

(xv) $26,500,000 as of January 31, 2005;

(xvi) $27,000,000 as of February 28, 2005;

(xvii) $27,500,000 as of March 31, 2005;

(xviii) $28,500,000 as of April 30, 2005;

(xix) $29,000,000 as of May 31, 2005;

(xx) $30,000,000 as of June 30, 2005;

(xxi) $31,000,000 as of July 31, 2005;

(xxii) $31,000,000 as of August 31, 2005;

(xxiii) $32,000,000 as of September 30, 2005;

(xxiv) $33,000,000 as of October 31, 2005;

(xxv) $34,500,000 as of November 30, 2005;

(xxvi) $36,000,000 as of December31, 2005;

(xxvii) $36,500,000 as of January 31, 2006;

(xxviii) $37,101,000 as of February 28, 2006;

(xxix) $37,490,000 as of March 31, 2006;

(xxx) $38,006,000 as of April 30, 2006; and

(xxxi) $38,434,000 as of May 31, 2006.

EBITDA will be tested on a rolling twelve (12) month basis as of last day of the month prior to any date on which the Loan Parties’ Excess Availability is less than the aggregate of (A) $10,000,000, and (B) the Excess Availability requirement pursuant to Section 6.11(a).

The Administrative Agent may determine the Loan Parties’ compliance with such covenants based upon financial reports and statements provided by the Borrowers’ Representative to the Administrative Agent (whether or not such financial reports and statements are required to be furnished pursuant to this Agreement) as well as by reference to interim financial information provided to, or developed by, the Administrative Agent.

3. Amendment Fee. Borrowers shall pay to Administrative Agent an amendment fee equal to One Hundred Fifty Thousand Dollars ($150,000.00), which fee shall be due on the date hereof and shall be deemed fully earned as of the date hereof.

4. Ratification of Loan Documents. Except as otherwise provided for herein, the terms and conditions of the Credit Agreement and of the other Loan Documents remain in full force and effect, and each Borrower hereby ratifies, confirms and reaffirms, all and singular, the terms and conditions of, and the warranties and representations set forth, therein.

5. Conditions Precedent to Effectiveness. This Second Amendment shall not be effective until each of the following conditions precedent have been fulfilled to the satisfaction of the Administrative Agent:

6. Miscellaneous.

[The remainder of this page is intentionally left blank]

IN WITNESS WHEREOF, the parties have duly executed this Second Amendment as of the day and year first above written.