Letter Agreement Regarding Restrictions on Excluded Collateral – Aegis Communications Group, Inc. and The CIT Group/Business Credit, Inc.

Exhibit 10.23

May 11, 2006

The CIT Group/Business Credit, Inc., as agent

Two Lincoln Centre

5420 LBJ Freeway, Suite 200

Dallas, Texas 75240

Attention: Regional Credit Manager

Ladies and Gentlemen:

AEGIS COMMUNICATIONS GROUP, INC.; ADVANCED TELEMARKETING CORPORATION, IQI, INC., LEXI INTERNATIONAL, INC. and INTERSERV SERVICES CORPORATION (individually and collectively, the “Company”), THE CIT GROUP/BUSINESS CREDIT, INC., as agent (in such capacity, “Agent”) for itself and certain other lenders (collectively, “Lenders”) and Lenders are parties to a certain Financing Agreement dated as of May 11, 2006 (as the same has been amended and may be amended, modified or supplemented from time to time, the “Loan Agreement”). Capitalized terms not defined shall have the meaning assigned to such terms in the Loan Agreement.

In consideration of the agreement of Agent and Lenders to enter into the Loan Agreement and to enter into the financial accommodations evidenced thereby, Company hereby covenants and agrees with the Agent, that until the indefesible payment in full of the Obligations (as defined in the Loan Agreement) and until all Obligations pursuant to the Loan Agreement have terminated whether direct or indirect, primary or secondary, joint or several, fixed or contingent, Company will not create, assume or permit to exist any lien, mortgage, pledge, charge, security interest or other encumbrance of any kind (each of which is herein called a “Lien”) on any Equipment or Real Estate now or hereinafter owned by Company, (the “Excluded Collateral”) except Permitted Encumbrances.

Company covenants and agrees that except to the extent expressly permitted by the Loan Agreement: (a) without the prior written consent of Agent, Company will not exchange, assign, sell, transfer, hypothecate, pledge or otherwise dispose of the Excluded Collateral, any of its interests in the Excluded Collateral or take any other action similar to or having a result similar to any of the foregoing (each a “Transfer Event”). Any purported Transfer Event which occurs, absent compliance with the terms of this letter agreement, shall be deemed void ab initio.

Company hereby authorizes Agent to file, in all uniform commercial code filing offices deemed necessary or desirable by Agent, uniform commercial code financing statements for notice purposes, substantially in the form of Exhibit B hereto.

This letter agreement represents the entire agreement between the parties with respect to the subject matter contained herein and supersedes any prior or contemporaneous agreement, whether written or oral on such subject matter. No provision of this letter agreement may be waived or modified except in writing signed by Agent and the other parties hereto. This letter agreement shall be binding upon and inure to the benefit of the successors and assigns of the parties hereto, provided, that, the Company may assign this letter agreement or delegate any of their obligations hereunder without Agent’s prior written consent. This letter agreement will be construed in accordance with the laws of the State of New York

without regard to conflict of law principles. This letter agreement may be executed in any number of counterparts, which, taken together, shall constitute one original document. Signature by facsimile will bind the parties hereto.

Any communications or notices to be given or sent to any party hereto pursuant to the terms hereof shall be in writing and shall be sent to the appropriate party at the address of such party and in the manner set forth in the Loan Agreement.

Please evidence your agreement to the foregoing terms and conditions by signing and returning a copy of this letter agreement to the undersigned.

[Remainder of page intentionally blank; signatures follow]

[Signatures continue on next page.]

ACKNOWLEDGED AND AGREED TO

as of the date first written above:

AEGIS COMMUNICATIONS GROUP, INC.

ADVANCED TELEMARKETING CORPORATION

IQI, INC.

LEXI INTERNATIONAL, INC.

INTERSERV SERVICES CORPORATION