AMENDMENT TO RIGHTS AGREEMENT

Exhibit 4.1

AMENDMENT TO RIGHTS AGREEMENT

Amendment, dated as of October 10, 2008 (the “Amendment”), between Imperial Sugar Company, a Texas corporation (the “Company”), and The Bank of New York, a New York trust company (the “Rights Agent”).

W I T N E S S E T H:

– – – – – – – – – –

WHEREAS, the Company and the Rights Agent are parties to a Rights Agreement dated as of December 31, 2002 (the “Agreement”);

WHEREAS, pursuant to Section 27 of the Agreement, the Company has directed the Rights Agent to enter into this Amendment;

WHEREAS, as reported in a Form 3 filed with the Securities and Exchange Commission on October 2, 2008, Barclays PLC beneficially owns an aggregate of 3,338,051 shares of Common Stock of the Company;

WHEREAS, such shares were previously owned by Lehman Brothers Inc. and were acquired by a subsidiary of Barclays PLC on September 22, 2008 pursuant to a bankruptcy court supervised sale under the Securities Investor Protection Act of 1970; and

WHEREAS, capitalized terms used but not defined herein shall have the respective meanings ascribed to them in the Agreement.

NOW, THEREFORE, in consideration of the premises and the mutual agreements herein set forth, the parties hereby agree as follows:

Section 1. Amendment to Definition of Acquiring Person. Clause (A) of the second paragraph of the definition of “Acquiring Person” in Section 1 of the Agreement is amended to read in its entirety as follows:

(A) so long as Barclays PLC, together with all Affiliates and Associates thereof, remains the beneficial owner of the greater of (i) the Barclays Permitted Ownership Amount and (ii) 15% of the outstanding shares of Common Stock, Barclays PLC and any Affiliate or Associate thereof shall not become an Acquiring Person unless and until, after the date of this Amendment, Barclays PLC, together with all Affiliates and Associates thereof, becomes the Beneficial Owner of shares of Common Stock in excess of the sum of the then-current Barclays Permitted Ownership Amount and shares of Common Stock constituting 1% or more of the then-outstanding shares of Common Stock. Notwithstanding the foregoing, or any provision in this Agreement to the contrary, shares of Common Stock held by an Affiliate or Associate of Barclays PLC, in an aggregate amount from time to time not to exceed 7.5% of the outstanding shares of Common Stock, (i) that are held for the sole purpose of maintaining an index or model-driven fund, (ii) that may be “disaggregated” from securities holdings of Barclays Capital Inc. for purposes of reporting under Sections 13 and 16 of the Exchange Act pursuant to interpretive positions of the Securities and Exchange Commission thereunder, and (iii) as to which Barclays PLC, Barclays Bank PLC and Barclays Capital Inc. exercise no voting or investment authority or power, shall not be included for purposes of the foregoing calculation.

Section 2. Amendment to Definition of Affiliate. The definition of “Affiliate” in Section 1 of the Agreement is amended to read in its entirety as follows:

“Affiliate” shall have the meaning ascribed to such term in Rule 12b-2 of the General Rules and Regulations under the Exchange Act, as in effect on the date of this Agreement; provided, however, that no Person shall be deemed an Affiliate of Barclays PLC solely by virtue of being an officer or director of Barclays PLC or of any Associate or Affiliate thereof unless and until such officer or director, as the case may be, and Barclays PLC (or an Affiliate or Associate of Barclays PLC) (i) have any agreement, arrangement or understanding (whether or not in writing) for the purpose of acquiring, holding, voting (except pursuant to a revocable proxy or consent as described in the proviso to subparagraph (i) of the definition of “Beneficial Owner”) or disposing of any voting securities of the Company or (ii) are members of any group (as that term is used in

2

Rule 13d-5(b) of the General Rules and Regulations under the Exchange Act, as in effect on the date of this Agreement) with respect to the Company or securities of the Company.

Section 3. Definition of Barclays Permitted Ownership Amount.

Section 1 of the Agreement is amended by adding the following definition:

“Barclays Permitted Ownership Amount” shall mean 3,338,051 shares of Common Stock, representing all shares of Common Stock Beneficially Owned by Barclays PLC and any Affiliate and Associate thereof as of the date of this Amendment (computed in accordance with the provisions of clause (A) of the definition of Acquiring Person) or such lesser amount Beneficially Owned by them as a result of sales, dispositions or transfers occurring after the date of this Amendment.

Section 4. Amendment to Duties of Rights Agent. Section 20(c) of the Agreement is amended to read in its entirety as follows:

(c) The Rights Agent shall be liable hereunder only for its own gross negligence or willful misconduct (which gross negligence or willful misconduct must be determined by a final, non-appealable order, judgment, decree or ruling of a court of competent jurisdiction). In no event shall the Rights Agent be liable for special, indirect, consequential, punitive or incidental loss or damage of any kind whatsoever (including but not limited to lost profits), even if the Rights Agent has been advised of the likelihood of such loss or damage and regardless of the form of action. Any liability of the Rights Agent under this Agreement will be limited to the amount of annual fees paid by the Company to the Rights Agent.

Section 5. Certification. The officer of the Company executing this Amendment on behalf of the Company hereby certifies on behalf of the Company that this Amendment complies with the terms of Section 27 of the Agreement.

Section 6. Severability. If any term, provision, covenant or restriction of this Amendment is held by a court of competent jurisdiction or other authority to be invalid, void or unenforceable, the remainder of the terms, provisions, covenants and restrictions of this Amendment shall remain in full force and effect and shall in no way be affected, impaired or invalidated.

3

Section 7. Governing Law. This Amendment shall be deemed to be a contract made under the laws of the State of Texas and for all purposes shall be governed by and construed in accordance with the laws of such State applicable to contracts made and to be performed entirely within such State, except that the rights, duties and obligations of the Rights Agent shall be governed by the laws of the State of New York applicable to contracts made and to be performed entirely within New York.

Section 8. Counterparts. This Amendment may be executed in any number of counterparts and each of such counterparts shall for all purposes be deemed to be an original, and all such counterparts shall together constitute but one and the same instrument.

Section 9. Descriptive Headings. Descriptive headings of the several Sections of this Amendment are inserted for convenience only and shall not control or affect the meaning or construction of any of the provisions hereof.

Section 10. Confirmation of Agreement. Except to the extent specifically amended hereby, the provisions of the Agreement shall remain unmodified, and the Agreement as amended hereby is confirmed as being in full force and effect.

4

IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be duly executed all as of the day and year first above written.

5