MAKE-WHOLE AGREEMENT

Exhibit 10.4

MAKE-WHOLE AGREEMENT

This Make-Whole Agreement (this “Agreement”) is made as of June 7, 2004 by and between Flag Financial Corporation, a Georgia corporation (“Flag”), and Joseph W. Evans, a Georgia resident (“Evans”).

Preliminary Statements

(1) Evans is the chairman, president and chief executive officer of Flag. Prior to his election as chairman of Flag, Evans was president and chief executive officer of Century South Banks, Inc. (“CSBI”). On June 7, 2001, CSBI was acquired by BB&T Corporation. As a condition of the Acquisition, Evans entered into an Employment Agreement with Branch Banking and Trust Company (“BB&T”), a wholly owned subsidiary of BB&T Corporation, whereby Evans received $27,538.62 per month for his services through May 31, 2006. The Employment Agreement also contained restrictions on Evans’ future conduct;

(2) Flag wishes to enter into an agreement with Riverside Bank to acquire its mortgage subsidiary, Riverside Residential Lending, Inc. (“Riverside”);

(3) The restriction on future conduct contained in the Employment Agreement may restrict Flag’s ability to acquire Riverside and conduct its business;

(4) To remove any uncertainty regarding Flag’s ability to own and operate Riverside, Flag requested that Evans terminate the Employment Agreement; and

(5) In order to induce BB&T to terminate the Employment Agreement, Evans will forfeit $660,926.93 in future payments from BB&T.

For good and valuable consideration, the receipt and adequacy of which are hereby acknowledged, and as a material inducement to Evans to terminate the Employment Agreement, Flag hereby promises Evans the prompt and full payment and performance of the obligations described below in this Agreement, upon the following terms and conditions:

1. Make-Whole Payments. Flag promises to pay to Evans $11,015 per month on the tenth day of each month beginning June 10, 2004 and ending May 10, 2009 (the amounts described to be paid under this Section 1 herein collectively called the “Make-Whole Amounts”). Flag shall be liable for the payment of all of the Make-Whole Amounts whether or not the Riverside acquisition closes. It is the intent of Flag for these payments to become part of the goodwill for the Riverside transaction. If for any reason it is later determined that these payments are not considered goodwill, Flag may prepay to Evans the amounts then due hereunder discounted to the present value of such payments using a discount rate of 6%. In the event there occurs a change in control of Flag, Evans may cause Flag to prepay to Evans the amounts then due hereunder discounted to the present value of such payments using a discount rate of six percent. For purposes of this Agreement, “Change of Control” means any transaction, whether by merger, consolidation, asset sale, tender offer, reverse stock split, or otherwise, which results in the acquisition or beneficial ownership (as such term is defined under the rules and regulations promulgated under the Securities Exchange Act of 1934, as amended) by any person or entity or any

group of persons or entities acting in concert, of 50% or more of the outstanding shares of common stock of Flag, or the sale of substantially all of the assets of Flag.

2. Liability of Flag. Flag shall be unconditionally and irrevocably obligated to make such payments on any such date, time being of the essence. In the event Flag fails to pay in full any portion of the Make-Whole Amounts as and when due and payable under this Agreement, Evans shall have all rights and remedies available at law or in equity including, without limitation, the right to pursue a suit for specific performance against Flag.

3. Representations and Warranties of Flag. Flag hereby represents, warrants, and covenants that (a) Flag has derived substantial benefit, directly or indirectly, from the making of this Agreement by Flag and has received good, valuable and sufficient consideration for Flag’s execution and delivery of this Agreement; (b) this Agreement is duly authorized and valid, and is binding upon and enforceable against Flag, (c) Flag is duly organized, validly existing, and in good standing under the laws of the State of Georgia, and has full power to enter into and perform this Agreement; and (d) Flag’s execution, delivery or performance of this Agreement does not (i) violate or conflict with any provision of law, governmental rule or regulation, order, writ, judgment, injunction, decree, determination or award of any court, arbitrator, governmental department, commission, board, bureau, or agency by which Flag or any of Flag’s assets are bound, or (ii) conflict with, result in breach of or constitute a default under any indenture, agreement or other instrument to which Flag is a party or by which any of its assets are bound.

4. Binding Effect. This Agreement is binding on Flag, its successors and assigns.

5. Governing Law. This Agreement, and its validity, enforcement, and interpretation, shall for all purposes be governed by and construed in accordance with the laws of the State of Georgia and applicable United States federal law, and is intended to be performed in accordance with, and only to the extent permitted by, such laws. All obligations of Flag hereunder are payable and performable at the place or places where the Make-Whole Obligations are payable and performable.

6. Invalidity of Certain Provisions. If any provision of this Agreement or the application thereof to any party or circumstance shall, for any reason and to any extent, be declared to be invalid or unenforceable, neither the remaining provisions of this Agreement nor the application of such provision to any other party or circumstance shall be affected thereby, and the remaining provisions of this Agreement, or the applicability of such provision to other parties or circumstances, as applicable, shall remain in effect and be enforceable to the maximum extent permitted by applicable law.

7. Attorneys’ Fees and Costs of Collection. Flag shall pay on demand all reasonable attorneys’ fees and all other costs and expenses incurred by Evans in the enforcement of or preservation of Evans’s rights under this Agreement.

8. Payments. All sums payable under this Agreement shall be paid in lawful money of the United States of America that at the time of payment is legal tender for the payment of public and private debts. If any sums payable under this Agreement shall not

2

be paid within five days of when due, then Flag hereby agrees that the amounts due shall bear interest at its prime interest rate, plus 2% per annum, from the date such payment is due until such payment is paid in full.

9. Notices. Any notice, consent, approval, or other communication which is provided for or required by this Agreement must be in writing and may be delivered in person to any party, or may be sent by Federal Express, UPS, or other reputable courier service regularly providing evidence of delivery. Any such notice or other written communications shall be deemed received by the party to whom it is sent (i) in the case of personal delivery, on the date of delivery to the party to whom such notice is addressed as evidenced by a written receipt signed on behalf of such party, and (ii) in the case of courier delivery, the date receipt is acknowledged by the party to whom such notice is addressed as evidenced by a written receipt signed on behalf of such party. For purposes of notices, the addresses of the parties hereto shall be as follows, which addresses may be changed at any time by written notice given in accordance with this provision:

Failure of, or delay in delivery of any copy of a notice or other written communication shall not impair the effectiveness of such notice or written communication given to any party to this Agreement as specified herein.

10. Cumulative Rights. The exercise by Evans of any right or remedy hereunder or at law or in equity shall not preclude the concurrent or subsequent exercise of any other right or remedy. Evans shall have all rights, remedies and recourses afforded to him by reason of this Agreement or by law or equity. The exercise of, discontinuance of the exercise of, or failure to exercise any of such rights, remedies, or recourses shall in no event be construed as a waiver or release thereof or of any other right, remedy, or recourse. No waiver of any default on the part of Flag or of any breach of any of the provisions of this Agreement or of any other document shall be considered a waiver of any other or subsequent default or breach, and no delay or omission in exercising or enforcing the rights and powers granted herein or in any other document shall be construed as a waiver of such rights and powers, and no exercise or enforcement of any rights or powers hereunder or under any other document shall be held to exhaust such rights and powers, and every such right and power may be exercised from time to time.

3

11. Term. This Agreement shall continue in effect until the Make-Whole Amounts are fully and finally paid, performed, and discharged.

12. Interpretation. Whenever the context of any provisions hereof shall require it, words in the singular shall include the plural, words in the plural shall include the singular, and pronouns of any gender shall include the other genders. Captions and headings are for convenience only and shall not affect the construction of this Agreement. The terms “herein”, “hereof”, “hereto”, “hereunder” and similar terms refer to this Agreement and not to any particular Section or subsection of this Agreement. The terms “include” and “including” shall be interpreted as if followed by the words “without limitation”. All references in this Agreement to sums denominated in dollars or with the symbol “$” refer to the lawful currency of the United States of America, unless such reference specifically identifies another currency.

13. Execution. This Agreement may be executed in multiple counterparts, each of which, for all purposes, shall be deemed an original, and all of which together shall constitute one and the same agreement.

IN WITNESS WHEREOF, Flag and Evans have duly executed this Agreement as of the date first written above.

4