VOTING AGREEMENT

Exhibit 10.4

VOTING AGREEMENT

This Voting Agreement (this “Agreement”), dated as of March 26, 2012 between the undersigned stockholder (“Stockholder”) of Carrols Restaurant Group, Inc., a Delaware corporation (the “Company”), and Burger King Corporation, a Florida corporation (“BK”).

WHEREAS, the Company and BK have entered into that certain Asset Purchase Agreement, dated as of the date hereof (as the same may be amended from time to time, the “Purchase Agreement”), providing for, among other things, the acquisition by the Company of certain assets of BK in consideration for the issuance by the Company to BK of shares of newly-designated Series A Convertible Preferred Stock, in each case pursuant to the terms and conditions of the Purchase Agreement;

WHEREAS, the Company has agreed in the Purchase Agreement to provide each stockholder of the Company entitled to vote at the annual meeting of the stockholders of the Company to be held in 2012, or at any special meeting of the stockholders of the Company held prior to such date, a proxy statement soliciting each such stockholder’s affirmative vote at the Stockholder Meeting for the stockholder approval contemplated by Section 6(b) of the Certificate of Designation (the “Issuance Resolutions”);

WHEREAS, in order to induce BK to enter into the Purchase Agreement, Stockholder agreed to execute and deliver this Agreement and to make certain representations, warranties, covenants and agreements with respect to the shares of common stock, par value $.01 per share, of the Company (“Company Common Stock”) beneficially owned by Stockholder and set forth below Stockholder’s signature on the signature page hereto (the “Original Shares”) and any additional “Shares” (as defined in Section 1 below).

NOW, THEREFORE, in consideration of the premises and for other good and valuable consideration, the receipt, sufficiency and adequacy of which are hereby acknowledged, the parties hereto agree as follows:

For purposes of this Agreement, capitalized terms used and not defined herein shall have the respective meanings ascribed to them in the Purchase Agreement.

The term “Amendment” shall have the meaning set forth in Section 11(h).

The term “Existing Voting Agreement” means that certain Voting Agreement dated as of July 27, 2011 by and among Carrols Restaurant Group, Inc., Jefferies Capital Partners IV L.P., Jefferies Employee Partners IV LLC, and JCP Partners IV LLC.

The term “Options” shall have the meaning set forth in Section 2(b).

The term “Shares” shall mean the Original Shares, as they may be adjusted due to the acquisition of additional shares of Company Common Stock or the transfer of Original Shares pursuant to Section 5 hereof, provided that neither the “Original Shares” nor the “Shares” shall include shares of Company Common Stock held or acquired by Jefferies & Company, Inc. (other than such shares acquired from Stockholder otherwise than in the ordinary course of business), it being understood that (i) Jefferies & Company, Inc. may be deemed to be under common control with Stockholder, (ii) such company may hold or acquire shares of Company Common Stock in the ordinary course of its business, (iii) Stockholder disclaims any beneficial ownership of such shares, and (iv) none of such shares shall be subject to this Agreement except to the extent they are acquired from Stockholder otherwise than in the ordinary course of business.

The term “Transfer” shall have the meaning set forth in Section 5.

Stockholder represents and warrants to BK that:

2

3

Stockholder agrees that Stockholder will not, and will not permit any entity under Stockholder’s control to, deposit any of the Shares in a voting trust, grant any proxies with respect to the Shares or subject any of the Shares to any arrangement with respect to the voting of the Shares other than agreements entered into with BK and the Existing Voting Agreement.

Stockholder agrees that all shares of Company Common Stock that Stockholder purchases, acquires the right to vote or otherwise acquires beneficial ownership (as defined in Rule 13d-3 under the Exchange Act) of after the execution of this Agreement shall be subject to the terms of this Agreement and shall constitute Shares for all purposes of this Agreement.

The parties agree that nothing in this Agreement shall be deemed to limit or restrict the Stockholder’s ability to directly or indirectly, transfer, sell, offer, exchange, assign, pledge or otherwise dispose of or encumber (“Transfer”) any of the Shares or enter into any contract, option or other agreement with respect to, or consent to, a Transfer of, any of the Shares or Stockholder’s voting or economic interest therein; provided that such Transfer is made either (i) in an open market transaction or (ii) the transferee agrees in a writing, reasonably satisfactory in form and substance to BK, to be bound by all of the terms of this Agreement.

This Agreement shall terminate upon the earliest to occur of (i) the date on which the Issuance Resolutions are approved by the stockholders of the Company, (ii) the date on which the Purchase Agreement is terminated in accordance with its terms, (iii) the date of any amendment to the Purchase Agreement or any change to or modification of the certificate of designation of the Series A Convertible Preferred Stock attached as Exhibit B to the Purchase Agreement, in each case which change is materially adverse to the Company or Stockholder, and (iv) December 31, 2013.

Stockholder makes no agreement or understanding in this Agreement in Stockholder’s capacity as a director or officer of the Company or any of its subsidiaries (if Stockholder holds such office), and nothing in this Agreement: (a) will limit or affect any actions or omissions taken by Stockholder in stockholder’s capacity as such a director or officer, including in exercising rights under the Purchase Agreement, and no such actions or omissions shall be deemed a breach of this Agreement or (b) will be construed to prohibit, limit or restrict Stockholder from exercising Stockholder’s fiduciary duties as an officer or director of the Company or to its stockholders.

4

Each party hereto acknowledges that it will be impossible to measure in money the damage to the other party if a party hereto fails to comply with any of the obligations imposed by this Agreement, that every such obligation is material and that, in the event of any such failure, the other party will not have an adequate remedy at law or damages. Accordingly, each party hereto agrees that injunctive relief or other equitable remedy, in addition to remedies at law or damages, is the appropriate remedy for any such failure and will not oppose the seeking of such relief on the basis that the other party has an adequate remedy at law. Each party hereto agrees that it will not seek, and agrees to waive any requirement for, the securing or posting of a bond in connection with the other party’s seeking or obtaining such equitable relief.

This Agreement supersedes all prior agreements, written or oral, between the parties hereto with respect to the subject matter hereof and contains the entire agreement between the parties with respect to the subject matter hereof. This Agreement may not be amended or supplemented, and no provisions hereof may be modified or waived, except by an instrument in writing signed by both of the parties hereto. No waiver of any provisions hereof by either party shall be deemed a waiver of any other provisions hereof by such party, nor shall any such waiver be deemed a continuing waiver of any provision hereof by such party.

All notices, requests, claims, demands, and other communications hereunder shall be in writing and shall be deemed to have been given (a) when delivered by hand (with written confirmation of receipt), (b) when received by the addressee if sent by a nationally recognized overnight courier (receipt requested), (c) on the date sent by facsimile or e-mail of a PDF document (with confirmation of transmission) if sent during normal business hours of the recipient, and on the next Business Day if sent after normal business hours of the recipient, or (d) on the third day after the date mailed, by certified or registered mail, return receipt requested, postage prepaid. Such communications must be sent to the respective parties at the following addresses (or at such other address for a party as shall be specified in a notice given in accordance with this Section 10):

If to BK, to the address or facsimile number set forth for BK in Section 12.2 of the Purchase Agreement.

If to Stockholder, to the address or facsimile number set forth for Stockholder on the signature page hereof.

5

6

[SIGNATURE PAGE FOLLOWS]

7

IN WITNESS WHEREOF, the parties hereto have executed and delivered this Agreement as of the date first written above.

8