FIRST AMENDMENT TO VOTING AGREEMENT

Exhibit 10.5

Execution Version

FIRST AMENDMENT

TO

VOTING AGREEMENT

This FIRST AMENDMENT TO VOTING AGREEMENT (the “Amendment”), is dated effective as of May 30, 2012, and entered into by and among Carrols Restaurant Group, Inc., a Delaware corporation (the “Company”), Jefferies Capital Partners IV L.P., a Delaware limited partnership (“Jefferies Capital Partners IV”), Jefferies Employee Partners IV LLC, a Delaware limited liability company (“Jefferies Employee Partners”), and JCP Partners IV LLC, a Delaware limited liability company (together with Jefferies Capital Partners IV and Jefferies Employee Partners, “Jefferies Capital Partners”).

RECITALS:

WHEREAS, the Company and Jefferies Capital Partners entered into that certain Voting Agreement, dated as of July 27, 2011 (the “Original Agreement”);

WHEREAS, the Spin-Off has been consummated;

WHEREAS, the Company, Carrols LLC, a Delaware limited liability company and wholly-owned subsidiary of the Company (“Purchaser”) and Burger King Corporation, a Florida corporation (“Seller”) have entered into that certain Asset Purchase Agreement, dated as of March 26, 2012 (the “APA”), pursuant to which, among other things, the Purchaser shall purchase certain assets from and assume certain liabilities of Seller and, as partial consideration therefore, the Company shall issue to Seller 100 shares (the “Seller Shares”) of Series A Convertible Preferred Stock of the Company, par value $0.01 per share (“Preferred Stock”) (the transactions contemplated by the APA, the “Transaction”);

WHEREAS, upon conversion, the Seller Shares shall be convertible into a number of shares of common stock of the Company (the “Conversion Shares”) equal to 28.9% of the issued and outstanding common stock of the Company, par value par value $0.01 per share (“Common Stock”), on a fully diluted basis as of the closing of the Transaction (after giving effect to the issuance of the Seller Shares);

WHEREAS, pursuant to the Certificate of Designation of Series A Convertible Preferred Stock filed by the Company with the Secretary of State of the State of Delaware on May 29, 2012 (the “Certificate of Designation”), the Company may not issue any Conversion Shares to the extent that such shares, after giving effect to such issuance after conversion and when added to the number of shares of Common Stock previously issued upon the conversion of any shares of Preferred Stock, would exceed the Maximum Number (as such term is defined in the Certificate of Designation), unless and until the Company obtains stockholder approval removing such restrictions and permitting such issuances in accordance with applicable NASDAQ Stock Market Rules (“Stockholder Approval”);

WHEREAS, entry into this Amendment is a condition to closing of the Transaction and a material inducement for Seller entering into the Purchase Agreement; and

WHEREAS, the parties hereto desire to amend the Original Agreement as hereinafter set forth.

NOW, THEREFORE, in consideration of the mutual covenants and agreements herein contained and other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto, intending to be legally bound, hereby agree as follows:

1. Capitalized Terms. Capitalized terms used and defined in this Amendment shall have the meanings assigned to them in this Amendment (including those in the recital paragraphs), and capitalized terms used in this Amendment and not defined herein shall have the meanings assigned to them in the Original Agreement, in each case, unless the context clearly requires otherwise.

2. Amendments. The Original Agreement is hereby amended as follows:

(a) Section 2.1 of the Original Agreement shall be deleted in its entirety and replaced with the following:

2.1 VOTING ON PARTICULAR MATTERS. Jefferies Capital Partners hereby agrees at all times on and after the Spin-Off Effective Date to vote, or cause to be voted, all of the Subject Shares, or give consent in lieu thereof, as follows:

(a) if the matter concerned is a proposed merger, consolidation, reorganization, dissolution or similar transaction of the Company, or a proposed sale or disposition of all or substantially all of the assets or business of the Company (in each case requiring the vote of the shareholders of the Company pursuant to the General Corporation Law of the State of Delaware or otherwise), in the sole and absolute discretion of Jefferies Capital Partners;

(b) if the matter concerned is the Stockholder Approval, in the sole and absolute discretion of Jefferies Capital Partners; and

(c) for all other matters, in the same proportions as the shares of Common Stock voted on such matters, or as to which consent shall have been given in lieu of such vote, by the holders of Common Stock other than Jefferies Capital Partners, taken as a whole.

(b) All references to “Section 2.1(b)” in the Original Agreement are hereby amended to read “Section 2.1(c).”

3. Ratification and Confirmation. Except to the extent specifically amended by this Amendment, the parties hereby ratify and confirm the terms and provisions of the Original

Agreement and the Original Agreement remains in full force and effect. All references to “this Agreement” in the Original Agreement shall be deemed as references to this Agreement as amended by this Amendment.

4. Governing Law. This Amendment shall be governed and construed in accordance with the laws of the State of Delaware, without reference to conflicts of laws principles.

5. Counterparts. This Amendment may be executed in any number of counterparts, each of which shall be deemed an original and all of which taken together shall constitute but one and the same instrument which may be sufficiently evidenced by one counterpart, and any of the parties hereto may execute this Amendment by signing any such counterpart.

[Signatures Begin on Next Page]

IN WITNESS WHEREOF, the parties have executed this Agreement effective as of the date first above written.

[Countersignature Page Follows]

Signature Page to First Amendment to Voting Agreement

Countersignature Page to First Amendment to Voting Agreement