CHANGE IN CONTROL AGREEMENT

EXHIBIT 10.20

CHANGE IN CONTROL AGREEMENT

THIS AGREEMENT (the “Agreement”) is made as of January 3, 2006 (the “Effective Date”) by and between Darrell R. Rains (the “Employee”) and Georgia Bank & Trust Company, a Georgia corporation (the “Bank”).

WHEREAS, the Employee is currently employed by the Bank, a wholly-owned subsidiary of Southeastern Bank Financial Corporation (the “Company”); and

WHEREAS, the Bank desires to induce the Employee to continue in the employ of the Bank by offering this agreement providing severance benefits to the Employee upon a Change in Control (as defined below) of the Company or the Bank.

NOW, THEREFORE, in consideration of the mutual covenants herein contained, the parties agree as follows:

1. Definitions. For purposes of this Agreement, the following terms and conditions shall have the meanings set forth in this Section 1:

(a) “Area” means the geographic area within the boundaries of Richmond and Columbia Counties, Georgia. It is the express intent of the parties that the Area as defined herein is the area where the Employee performs services on behalf of the Bank as of the Effective Date.

(b) “Board of Directors” means the Board of Directors of the Company or the Bank, as the context implies.

(c) “Business of the Company” means the business conducted by the Company and the Bank which is the business of banking, including the solicitation of time and demand deposits and the making of residential, consumer, commercial and corporate loans.

(d) “Cause” means the occurrence of any of the following events: (i) conduct by the Employee that is demonstrably likely to lead to material financial harm to the Company or the Bank; (ii) conduct by the Employee of a fraudulent nature against the Company or the Bank that resulted or was intended to result in direct or indirect gain to or personal enrichment of the Employee; (iii) conduct resulting in the conviction of the Employee of a felony; and (iv) conduct by the Employee that results in the permanent removal of the Employee from his position as an officer or an employee of the Company or the Bank pursuant to written order by any regulatory agency with authority and jurisdiction over the Company or the Bank, as the case may be.

(e) “Change in Control” means the occurrence of any of the following events on or after the Effective Date:

154

(i) within any twelve-month period (beginning on or after the Effective Date) the persons who constitute the Board of Directors of the Company or the Bank immediately before such twelve-month period (the “Incumbent Directors”) cease for any reason to constitute at least a majority of such Board of Directors; provided, however, that any person becoming a director subsequent to the Effective Date shall be deemed to be an Incumbent Director if that director was elected to such Board of Directors by, or on the recommendation of or with the approval of, at least two-thirds (²/₃) of the directors who then qualified as Incumbent Directors; and provided further that no director whose initial assumption of office is in connection with an actual or threatened election contest relating to the election of directors shall be deemed an Incumbent Director;

(ii) the consummation of a reorganization, merger or consolidation of the Company or the Bank, with respect to which persons who were the stockholders of the Company or the Bank, as the case may be, immediately prior to such reorganization, merger or consolidation do not, immediately thereafter, own more than fifty percent (50%) of the combined voting power entitled to vote in the election of directors of the reorganized, merged or consolidated entity’s then outstanding voting securities; or

(iii) the sale, transfer or assignment of all or substantially all of the assets of the Company and the Bank to any third party.

(f) “Confidential Information” means data and information relating to the Business of the Company (which does not rise to the status of a Trade Secret) which is or has been disclosed to the Employee or of which the Employee became aware as a consequence of or through its relationship to the Company and the Bank and which has value to the Company and the Bank and is not generally known to its competitors. Without limiting the foregoing, Confidential Information shall include the following:

(i) all items of information that could be classified as a trade secret pursuant to Georgia law;

(ii) the names, addresses and banking requirements of the customers of the Company and the Bank and the nature and amount of business done with such customers;

(iii) the names and addresses of employees and other business contacts of the Company and the Bank;

(iv) the particular names, methods and procedures utilized by the Company and the Bank in the conduct and advertising of their business;

(v) the applications, operating system, communication and other computer software and derivatives thereof, including, without limitation, sources and object codes, flow charts, coding sheets, routines, subrouting and related documentation and manuals of the Company and the Bank; and

155

(vi) marketing techniques, purchasing information, pricing policies, loan policies, quoting procedures, financial information, customer data and other materials or information relating to the Company’s and the Bank’s manner of doing business.

Confidential Information shall not include any data or information that has been voluntarily disclosed to the public by the Company or the Bank (except where such public disclosure has been made by the Employee without authorization) or that has been independently developed and disclosed by others, or that otherwise enters the public domain through lawful means.

(g) “Good Reason” means, with respect to a voluntary resignation by the Employee, any one of the following events, but only if such event first arose within forty-five (45) days of such resignation, the Employee provided the Employer with written notice of the event within fifteen (15) days after the event occurred and an opportunity to cure for at least ten (10) business days from its receipt of the notice and the circumstances continued, uncured, through the effective date of the Employee’s resignation:

(i) a material diminution in the Employee’s position, authority or duties effected by the Employer;

(ii) a material reduction in the Employee’s base salary rate or annual bonus opportunity effected by the Employer; or

(iii) a requirement by the Employer that the Employee’s services be rendered primarily at a location more than fifty (50) miles from August, Georgia.

(h) “Trade Secrets” means information, without regard to form, including, but not limited to, technical or nontechnical data, formulas, patterns, compilations, programs, devices, methods, techniques, drawings, processes, financial data, financial plans, product plans or lists of actual or potential customers or suppliers which (i) derives economic value, actual or potential, from not being generally known to, and not being readily ascertainable by proper means by, other persons who can obtain economic value from its disclosure or use, and (ii) is the subject of efforts that are reasonable under the circumstances to maintain its secrecy.

2. Term. The term of this Agreement (the “Term”) shall become effective as of the Effective Date and shall remain in effect until the earliest of (a) the Employee’s termination of employment with the Bank prior to a Change in Control; (b) the twenty-fourth-month anniversary of the effective date of a Change in Control if there has been no Qualifying Termination of Employment (as defined in Section 3); (c) a termination of this Agreement pursuant to Section 18; or (d) until all amounts that may be payable to the Employee pursuant to Section 3 below have been paid.

156

3. Severance Benefits Upon Termination of Employment. If, while this Agreement is in effect, the Employee (a) is involuntarily terminated by the Bank without Cause within twenty (24) months following a Change in Control; or (b) resigns his employment with the Bank for Good Reason within twenty-four (24) months following such Change in Control (each, a “Qualifying Termination of Employment”), the Bank shall pay to the Employee in a lump sum an amount equal to two (2) times the sum of (i) the Employee’s annual base salary in effect at the time of termination of employment and (ii) the largest annual bonus amount paid to the Employee within the three calendar years preceding the calendar year in which occurs the Change in Control (the “Lump Sum Payment. The Lump Sum Payment is sometimes referred to in this Agreement as the “Severance Benefits.” In no event shall the payment described in this Section 3 exceed the amount permitted by Section 280G of the Internal Revenue Code, as amended (the “Code”). Therefore, if the aggregate present value (determined as of the date of the Change in Control in accordance with the provisions of Section 280G of the Code) of both the Severance Benefits and all other payments to the Employee in the nature of compensation which are contingent on a change in ownership or effective control of the Company or the Bank or in the ownership of a substantial portion of the assets of the Company or the Bank (the “Aggregate Severance”) would result in a “parachute payment,” as defined under Section 280G of the Code, then the Aggregate Severance shall not be greater than an amount equal to 2.99 multiplied by Employee’s “base amount” for the “base period,” as those terms are defined under Section 280G. In the event the Aggregate Severance is required to be reduced pursuant to this Section 3, the Employee shall be entitled to determine which portions of the Aggregate Severance are to be reduced so that the Aggregate Severance satisfies the limit set forth in the preceding sentence.

The Lump Sum Payment shall be paid to the Employee as soon as practicable following the effective date of the Qualifying Termination of Employment. The Bank shall be entitled to withhold appropriate employment and income taxes, if required by applicable law, from any Severance Benefits that become payable.

4. Confidentiality.

(a) All Confidential Information and Trade Secrets and all physical embodiments thereof received or developed by the Employee while employed by the Bank are confidential to and are and will remain the sole and exclusive property of the Company and the Bank. Except to the extent necessary to perform the duties assigned to him by the Bank, the Employee will hold such Confidential Information and Trade Secrets in trust and strictest confidence, and will not use, reproduce, distribute, disclose or otherwise disseminate the Confidential Information and Trade Secrets or any physical embodiments thereof and may in no event take any action causing or fail to take the action necessary to prevent, any Confidential Information and Trade Secrets disclosed to or developed by the Employee to lose its character or cease to qualify as Confidential Information or Trade Secrets.

157

(b) The covenants of confidentiality set forth herein will apply during the term of the Employee’s employment to any Confidential Information and Trade Secrets disclosed by the Bank or developed by the Employee prior to or after the date hereof. The covenants restricting the use of Confidential Information will continue and be maintained by the Employee for a period of twelve (12) months following termination of this Agreement. The covenants restricting the use of Trade Secrets will continue and be maintained by the Employee following termination of this Agreement for so long as permitted by the then-current Georgia Trade Secrets Act of 1990, O.C.G.A. § 10-1-760, et. seq.

5. Noncompetition. The Employee agrees that, for twelve (12) months following the Employee’s termination of employment, regardless of the reason, the Employee will not (except on behalf of or with the prior written consent of the Bank), within the Area, either directly or indirectly, on his own behalf or in the service or on behalf of others, as an employee or in any other capacity which involves duties and responsibilities similar to those undertaken for the Bank, engage in any business which is the same as or essentially the same as the Business of the Company.

6. Nonsolicitation. The Employee agrees that, for twelve (12) months following the Employee’s termination of employment, regardless of the reason:

(a) the Employee will not (except on behalf of or with the prior written consent of the Bank), on the Employee’s own behalf or in the service or on behalf of others, solicit, divert or appropriate or attempt to solicit, divert or appropriate, directly or by assisting others, any business from any of the customers of the Bank, including actively sought prospective customers, with whom the Employee has or had material contact during the last two (2) years of the Employee’s employment, for purposes of providing products or services that are competitive with those provided by the Company and the Bank; and

(b) the Employee will not on the Employee’s own behalf or in the service or on behalf of others, solicit, recruit or hire away or attempt to solicit, recruit or hire away, directly or by assisting others, any employee of the Bank with whom the Employee has or had material contact during the last two (2) years of the Employee’s employment, whether or not such employee is a full-time employee or a temporary employee of the Bank and whether or not such employment is pursuant to a written agreement and whether or not such employment is for a determined period or is at will.

7. Remedies. The Employee agrees that, in addition to all remedies provided by law or in equity, the Bank shall be entitled to specific performance of this Agreement and to both temporary and permanent injunctions to prevent a breach or contemplated breach by the Employee of the covenants in Sections 4, 5 and 6 hereof. If the Employee breaches his obligations pursuant to Sections 4, 5 or 6 hereof, the Employee will forfeit any amounts owed to the Employee under Section 3 hereof which have not been paid to the Employee.

8. No Mitigation. No amounts or benefits payable to the Employee hereunder shall be subject to mitigation or reduction by income or benefits the Employee receives from other sources.

158

9. Continued Employment. Nothing herein shall entitle Employee to continued employment with the Bank or to continued tenure in any specific office or position. The Employee’s employment with the Bank shall be terminable at the will of the Bank, with or without Cause, subject to the terms of any other written agreement as may be in effect between the parties.

10. Assignment. The rights and obligations of the Bank under this Agreement shall inure to the benefit of the Bank’s successors and assigns. This Agreement may be assigned by the Bank to any legal successor of the Bank or to an entity which purchases all or substantially all of the assets of the Bank. In the event the Bank assigns this Agreement as permitted by this Agreement and the Employee remains employed by the assignee, the “Bank” as defined herein will refer to the assignee and the Employee will not be deemed to have terminated employment hereunder until the Employee terminates employment from the assignee.

11. Arbitration. Any controversy or claim arising out of or relating to this Agreement, or the breach thereof, shall be adjudicated through binding arbitration before a single arbitrator in accordance with the Commercial Arbitration Rules of the American Arbitration Association. The Bank and the Employee agree that they will seek to enforce any arbitration award in the Superior Court of Richmond County, Georgia. The decision of the arbitration panel shall be final and binding upon the parties and judgment upon the award rendered by the arbitration panel may be entered by any court having jurisdiction. The Bank and the Employee agree to share equally the fees and expenses associated with the arbitration proceedings, except as otherwise provided by Section 12 below. [Employee must initial here: DRR]

12. Attorneys’ Fees. With respect to arbitration of disputes and if litigation ensues between the parties concerning the enforcement of an arbitration award and the Employee prevails in the dispute, the Bank will pay and be financially responsible for all costs, expenses, reasonable attorneys’ fees and reasonable expenses incurred by the Employee (or the Employee’s estate in the event of his death) in connection with the dispute.

13. Notice. All notices, consents, waivers and other communications required or permitted by this Agreement shall be in writing and shall be deemed given to a party when (a) delivered to the appropriate address by hand or by nationally recognized overnight courier service (costs prepaid); (b) sent by facsimile with confirmation of transmission by the transmitting equipment; or (c) received or rejected by the addressee, if sent by certified mail, return receipt requested, in each case to the following addresses or facsimile numbers and marked to the attention of the person (by name or title) designated below (or to such other address, facsimile number or person as a party may designate by written notice to the other parties):

159

14. Headings. Sections or other headings contained herein are for reference purposes only and shall not affect in any way the meaning or interpretation of this Agreement.

15. Entire Agreement. This Agreement contains the entire understanding of the parties with respect to the subject matter hereof.

16. Severability. In the event that one or more of the provisions of this Agreement shall be or become invalid, illegal or unenforceable in any respect, the validity, legality and enforceability of the remaining provisions contained herein shall not be affected thereby.

17. Applicable Law and Choice of Forum. This Agreement shall be construed and enforced under and in accordance with the laws of the State of Georgia. The parties agree that any appropriate state or federal court located in Richmond County, Georgia shall have exclusive jurisdiction of any case or controversy arising under or in connection with Sections 4 through 7 of this Agreement shall be a proper forum in which to adjudicate such case or controversy. The parties consent and waive any objection to the jurisdiction or venue of such courts.

18. Amendment. This Agreement may not be modified, amended, supplemented or terminated except by a written agreement between the Bank and the Employee.

19. Survival. The provisions of Sections 4, 5, 6, 7, 11, 12, 17 and 20 above shall survive, as necessary, the expiration of the Term or any other termination of this Agreement.

20. Confidentiality and Professionalism. Employee represents and agrees that Employee will keep the terms, amount, value, and nature of consideration paid to Employee, and the fact of this Agreement completely confidential, and that Employee will not hereafter disclose any information concerning this Agreement to anyone other than Employee’s immediate family and professional representatives who will be informed of and bound by this confidentiality clause.

[Remainder of page intentionally left blank]

160

IN WITNESS WHEREOF, each of the parties has executed this Agreement as of the date and year first above written.

Georgia Bank & Trust Company

161