EMPLOYMENT AGREEMENT

Exhibit 10.4

[Execution Copy]

EMPLOYMENT AGREEMENT

     This Employment Agreement (this “Agreement”) is made and entered into effective as of July 10, 2006 (the “Effective Date”), by and among The Peachtree Bank, a Georgia banking corporation (“Employer”); and Jim Olson, a resident of the State of Georgia (“Executive”).

Recitals

     WHEREAS, pursuant to that certain Agreement and Plan of Merger (the “Merger Agreement”) dated as of May 24, 2006, between Alabama National BanCorporation, a Delaware corporation (“ANB”) and Employer’s parent holding company, Employer will become a wholly-owned subsidiary of ANB; and

     WHEREAS, Executive is a valuable member of Employer’s management team, and, as a condition to the consummation of the transactions provided for in the Merger Agreement, Executive and Employer have agreed to enter into this Agreement.

Agreement

     NOW THEREFORE, in consideration of the mutual recitals and covenants contained herein, and for other good and valuable consideration, the sufficiency of which is hereby acknowledged, the parties hereby agree as follows:

     1.     Employment. Employer agrees to employ Executive and Executive agrees to be employed by Employer, subject to the terms and provisions of this Agreement. The period of Executive’s employment by Employer under the terms of this Agreement will be coterminous with the Agreement Term (as defined below), unless such employment terminates earlier in accordance with the provisions of Section 9 hereof.

     2.     Agreement Term. The term of this Agreement (the “Agreement Term”) shall be for a period of five (5) years, commencing on the Effective Date. Any termination of Executive’s employment prior to the expiration of the Agreement Term shall be independent from, and shall not serve to terminate or shorten, the Agreement Term.

     3.     Duties; Extent of Services. Executive shall perform for Employer all duties incident to the position of Senior Credit Officer, under the direction of the Board of Directors, the Chief Executive Officer, and/or the President of Employer or their designee(s). In addition, Executive shall engage in such other services for Employer or its affiliated companies as Employer from time to time shall direct. The duties, services and reporting relationship of Executive and the title of Executive’s position may be extended, reduced, re-assigned, curtailed or modified unilaterally by Employer from time to time without breaching or affecting the enforceability of the terms of this Agreement so long as Executive continues to occupy an officer position with Employer (or its successor or assignee). Executive shall use Executive’s best efforts in, and devote Executive’s entire time, attention and energy, during regular business hours, to Employer’s business, and Executive shall not conduct any other activities that are or may be detrimental to Employer’s business.

     4.     Compensation. The compensation due to Executive shall be as follows and, except as otherwise provided herein, shall be due from the Effective Date until the termination of Executive’s employment:

     (a)     Base Salary. Executive’s total annual base salary shall be not less than $160,000, payable with the same frequency as the salaries of other employees of Employer.

     (b)     Annual Bonus Opportunity. Executive will be eligible to receive an annual bonus, the amount of which, if any, would be determined by Employer’s Board of Directors or its designee after an annual review of the performance of Executive and Employer for the prior calendar year.

     (c)     Benefits. Executive is entitled to vacation days, paid holidays and sick days, and to participate in Employer’s health and retirement plans, as provided in Employer’s Personnel Policy, as such may be amended from time to time. Executive is entitled to a $750 per month car allowance and a $400 per month allowance for club dues.

     5.     Compliance With Rules and Policies. Executive shall comply in all material respects with all of the rules, regulations, and respective policies of Employer and ANB now or hereinafter in effect. Executive shall promptly and faithfully do and perform any and all other duties and responsibilities which Executive may, from time to time, be directed to do by the Board of Directors of Employer, the Chief Executive Officer of Employer, or the President of Employer, or their respective designee(s).

     6.     Representation of Executive. Executive represents to Employer that Executive is not subject to any rule, regulation or agreement, including without limitation, any non-compete agreement, that purports to, or which reasonably could be expected to, limit, restrict or interfere with Executive’s ability to engage in the activities provided for in this Agreement. Executive further represents that Executive has carefully read and fully understands all the provisions of this Agreement and has had an opportunity to discuss this Agreement with Executive’s private attorney.

     7.     Disclosure of Information. Executive acknowledges that all data, information and documentation relating to the business of Employer (whether constituting a trade secret or not) which is or has been disclosed to Executive or of which Executive became aware as a consequence of or through Executive’s relationship to Employer and which has value to Employer and is not generally known to its competitors, including without limitation information about customers and potential customers and the business methods, sales, services, techniques, financial and business conditions, goals and operations of Employer and ANB (collectively, “Confidential Information”), are valuable, special and unique assets of Employer’s business. Executive will not, during Executive’s employment and for a period of three (3) years after such employment with Employer ends, (a) disclose any Confidential Information to any person, firm, corporation, association, or other entity not employed by or affiliated with Employer for any reason or purpose whatsoever, or (b) use any Confidential Information for any reason other than to further the business of Employer. Executive agrees to return any written Confidential Information (including without limitation all Confidential Information stored in electronic format), and all copies thereof, immediately upon the termination of Executive’s employment for any reason (whether hereunder or otherwise). In the event of a breach or threatened breach by Executive of the provisions of this Section 7, in addition to all other remedies available to Employer, Employer shall be entitled to an injunction restraining Executive from disclosing any Confidential Information or from rendering any services to any person, firm, corporation, association or other entity to whom any Confidential Information has been disclosed or is threatened to be disclosed.

     8.     Competition.

     (a)     During the period of Executive’s employment and, depending on the circumstances of employment termination, for the period following the termination of Executive’s employment provided for in Section 9 below, Executive shall not, directly or indirectly, other than on behalf of Employer: (i) perform the

same or substantially the same job duties and responsibilities as Executive performed for Employer (including as an organizer, director or proposed executive officer of a new financial institution) on behalf of a business or enterprise that provides products or services that compete with Employer’s products or services; provided, however, that this restriction is limited to the following Georgia counties: Fulton, DeKalb, and Gwinnett (the “Territory”); (ii) solicit any customer of Employer for the purpose of offering services or products that compete with the services or products offered by Employer; provided, however, that following the termination of Executive’s employment with Employer, this restriction is limited to customers with whom Executive had business contact during the last 24 months of Executive’s employment with Employer; (iii) solicit any director, officer or employee of Employer to leave his or her position or employment with Employer; provided, however, that following the termination of Executive’s employment with Employer, this restriction is limited to directors, officers and employees with whom Executive had business contact during the last 24 months of Executive’s employment.

     (b)      Executive represents that Executive’s experience and capabilities, together with the compensation paid by Employer and payable under this Agreement, are such that the provisions of this Section 8 will not impose an undue hardship on Executive or prevent Executive from earning a livelihood.

     (c)      In the event the enforceability of any of the covenants in Sections 7 or 8 is challenged in court and Executive is not enjoined from breaching any of the covenants, then if a court of competent jurisdiction thereafter finds that the challenged protective covenant is enforceable, the applicable time period as to such covenant shall be deemed tolled upon the filing of the lawsuit addressing the enforceability of Sections 7 and/or 8 until the dispute is finally resolved and all periods of appeal have expired.

     (d)      In the event of any conduct or threatened conduct by Executive violating any provision of this Section 8, Employer shall be entitled, in addition to other available remedies, to injunctive relief and/or specific performance of such provision.

     (e)      Executive acknowledges that (i) Executive has occupied, and will continue to occupy, a position of trust and confidence with Employer prior to the date hereof and has and will become familiar with Confidential Information, including without limitation trade secrets, as that term is defined in Georgia statutes; (ii) Executive has entered into this Agreement ancillary the sale of a business; therefore, the parties agree and expect that the “sale of the business” standard for non-compete agreements that has been established under Georgia law will apply to the interpretation and enforcement of Sections 7 and 8; (iii) ANB has required that Executive make the covenants set forth in Sections 7 and 8 of this Agreement as a material condition to Employer’s employment of Executive and ANB’s acquisition of the capital stock of Employer’s parent holding company, including any capital stock owned by Executive; (iv) the provisions of Sections 7 and 8 of this Agreement are reasonable in geographic scope, duration, and scope and are necessary to protect and preserve Employer’s legitimate business interests, including, without limitation, its trade secrets, valuable confidential business information, relationships with specific prospective and existing customers, customer goodwill, and specialized training provided to Executive; and (v) Employer would be irreparably damaged if Executive were to breach the covenants set forth in Sections 7 or 8 of this Agreement.

     9.     Termination of Employment.

     (a)      Employer may terminate Executive’s employment prior to the expiration of the Agreement Term “For Cause.” In such event, all rights and obligations specified in Section 8(a) shall survive any such termination until the expiration of the Agreement Term, and Executive shall not be entitled to any further compensation or benefits from Employer under this Agreement or otherwise, except as provided pursuant to the express terms of any employee benefit plan in which Executive is a participant as of the effective date of such

termination of employment; provided, however, that the rights and obligations in Section 8(a) shall not extend past the 2^nd anniversary of the termination date of Executive’s employment unless Employer makes payments to Executive based on the base salary amount set forth in Section 4(a) above (to be paid with the same frequency as Executive’s salary was paid prior to termination) during the period beginning on the 2^nd anniversary of the termination date of Executive’s employment and ending upon the expiration of the rights and obligations in Section 8(a) as provided for in this Section 9(a) above. Subject to Section 8(c), the rights and obligations specified in Section 8(a) shall not extend past the expiration of the Agreement Term. “For Cause” means (i) abuse of or addiction to intoxicating drugs (including alcohol); (ii) any act or omission on the part of Executive which constitutes fraud, misrepresentation, embezzlement, misappropriation of corporate assets, breach of a duty owed to Employer, or conduct grossly inappropriate to Executive’s office; (iii) indictment or conviction of a felony or a crime of moral turpitude; (iv) the suspension or removal of Executive by federal or state banking regulatory authorities; or (v) a breach by Executive of any of the material terms of this Agreement. In addition, the services of Executive and the obligations of Employer under this Agreement may be terminated For Cause by Employer due to the death or total disability of Executive. For purposes of this Section 9, the term “total disability” means Executive’s inability, as a result of illness or injury, to perform the essential functions of Executive’s job, with or without a reasonable accommodation, for a reasonable period of time, which the parties generally anticipate would be one hundred eighty (180) consecutive days.

     (b)      Employer may terminate Executive’s employment at any time prior to the expiration of the Agreement Term for any reason other than “For Cause”; provided, however, if Employer terminates Executive other than For Cause, or if Executive terminates employment for “Good Reason” (as defined below): (i) Executive shall continue to receive the amount of Executive’s then current base salary until the expiration of the Agreement Term (to be paid with the same frequency as Executive’s salary was paid prior to termination) as if Executive’s employment had continued during such period; and (ii) all rights and obligations specified in Section 8(a) shall survive such termination until the expiration of the Agreement Term. Subject to Section 8(c), the rights and obligations specified in Section 8(a) shall not extend past the expiration of the Agreement Term. Other than the payments and benefits provided for in this Section 9(b), and except as provided pursuant to the express terms of any employee benefit plan in which Executive is a participant as of the effective date of such termination of employment, Executive acknowledges that Executive shall not be entitled to any other payments, benefits or damages from Employer under this Agreement or otherwise in connection with a termination of Executive by Employer other than For Cause or a termination by Executive for Good Reason, and Executive hereby waives all rights and claims with respect thereto. “Good Reason” means a material breach of this Agreement by Employer, after Executive has provided written notice of such breach to Employer within 30 days, and Employer has been afforded at least 30 days to remedy such breach from the date such notice is received by Employer.

     (c)      If Executive resigns or terminates Executive’s employment hereunder for any reason (other than Good Reason) prior to the expiration of the Agreement Term, (i) Executive must provide at least 30 days prior written notice of such resignation or termination, (ii) all rights and obligations specified in Section 8(a) shall survive any such termination until the expiration of the Agreement Term; provided, however, that the rights and obligations in Section 8(a) shall not extend past the 2^nd anniversary of the termination date of Executive’s employment unless Employer, at its sole option and election, makes payments to Executive based on the base salary amount set forth in Section 4(a) above (to be paid with the same frequency as Executive’s salary was paid prior to termination) during the period beginning on the 2^nd anniversary of the termination date of Executive’s employment and ending upon the expiration of the rights and obligations in Section 8(a) as provided for in this Section 9(c) above; provided, further, that subject to Section 8(c), the rights and obligations specified in Section 8(a) shall not extend past the expiration of the Agreement Term, (iii) Executive shall not be entitled to any further compensation or benefits from Employer under this Agreement or otherwise (except for any period beyond the 2^nd anniversary of the termination date as described in subsection (ii) hereof, and

except as provided pursuant to the express terms of any employee benefit plan in which Executive is a participant as of the effective date of such termination of employment), and (iv) Employer shall be entitled to all remedies available under this Agreement and applicable law. Upon receipt of any such notice of termination, Employer may elect, at its sole option, to have Executive’s resignation or termination effective immediately. Notwithstanding the foregoing, if after the occurrence of a Change in Control (as defined below), (A) Executive is assigned duties materially inconsistent with Executive’s authorities, duties and responsibilities as in effect immediately prior to such Change in Control, and (B) Executive thereafter voluntarily terminates employment hereunder, the rights and obligations specified in Section 8(a) shall terminate and shall not survive any such termination of employment. “Change in Control” means (1) any transaction, whether by merger, consolidation, tender offer or otherwise, which results in the acquisition of beneficial ownership (as such term is defined under rules and regulations promulgated under the Securities Exchange Act of 1934, as amended) by any person or entity or any group of persons or entities acting in concert, of 50% or more of the outstanding shares of common stock of ANB; or (2) the sale of all or substantially all of the assets of ANB.

     (d)      If Executive’s employment does not terminate prior to the expiration of the Agreement Term and is not renewed, extended or continued on an “at will” basis but, rather, terminates upon the natural expiration of the Agreement Term, then the restrictive covenants in Section 8(a) shall not apply following the expiration of the Agreement Term; provided, however, at Employer’s sole option and election, Employer may continue to make payments of Executive’s then current base salary and, in such event, the restrictive covenants in Section 8(a) shall last for so long as such payments are being made, but not past the first (1st) anniversary of the expiration of the Agreement Term.

     10.      Notice. For the purposes of this Agreement, notices and demands shall be deemed given when mailed by United States mail, addressed in the case of Employer to The Peachtree Bank, 9570 Medlock Bridge Road, Duluth, Georgia 30097, Attention: Chief Executive Officer, with a copy to ANB at Alabama National BanCorporation, 1927 First Avenue North, Birmingham, Alabama 35203, Attention: Chief Executive Officer; or in the case of Executive, to Executive’s last known address of record contained in Employer’s personnel files.

     11.      Miscellaneous. No provision of this Agreement may be modified, waived or discharged unless such modification, waiver or discharge is agreed to in writing. The validity, interpretation, construction and performance of this Agreement shall be governed by Title 9 of the U.S. Code and the laws of the State of Georgia without regard to principles of conflicts of laws.

     12.      Validity. Should any court of competent jurisdiction, arbitrator or other judicial body decide, hold, adjudge or decree that any provision, clause or term of this Agreement is invalid, void or unenforceable, such determination shall not affect any other provision of this Agreement, and all other provisions of this Agreement shall remain in full force and effect as if such void or unenforceable provision, clause or term had not been included herein. Such determination shall not be deemed to affect the validity or enforceability of this entire Agreement in any other situation or circumstance, and the parties agree that the scope of this Agreement is intended to extend to Employer the maximum protection permitted by law. The parties expressly deem the scope, length of time and the size of the Territory provided for in Sections 7 and 8 of this Agreement to be reasonable. If, however, any judicial body or arbitrator decides, holds, adjudges or decrees that the scope, length of time and/or the size of the Territory provided for in Section 7 and/or 8 of this Agreement is/are unreasonable, then it is the express intent of the parties that such court determine the scope, length of time and/or size of the territory that is/are reasonable and that such court enforce the terms of this Agreement in accordance with such determination.

     13.      Arbitration.

     (a)      Except as otherwise provided herein, any dispute or controversy arising under or in connection with this Agreement or Executive’s employment, between Executive and Employer, ANB or any of Employer’s or ANB’s officers, directors, agents or employees, shall be settled exclusively by arbitration in Atlanta, Georgia, in accordance with the rules of the American Arbitration Association then in effect. The agreement set forth herein to arbitrate shall be specifically enforceable under the prevailing arbitration law. There are two exceptions to this mandatory arbitration provision. First, Employer shall have the right to seek enforcement by temporary restraining order and preliminary/interlocutory injunction, specific performance or other equitable relief of the provisions of Section 7 and/or 8 hereof. Second, Executive shall have the right to enforce any right to compensation due to him under this Agreement in any state or federal court of competent jurisdiction without regard to whether any such claim has been or can be referred to arbitration.

     (b)      The parties hereto (i) acknowledge that they have read and understood the provisions of this Section regarding arbitration and (ii) that performance of this Agreement will be incident to interstate commerce as that term is used in the Federal Arbitration Act (“FAA”), and the parties contemplate substantial interstate activity in the performance of this Agreement including, but not limited to, interstate travel, the use of interstate phone lines, the use of the U.S. mail services, the use of the internet, the use of email, and other interstate courier services. Consequently, the FAA shall interpret, govern and control any arbitration hereunder.

     (c)      Notice of the demand for arbitration shall be filed in writing with the other party to this Agreement and with the American Arbitration Association. The demand for arbitration shall be made within a reasonable time after the claim, dispute or other matter in question has arisen, and in no event shall it be made after the date when institution of legal or equitable proceedings based on such claim, dispute or other matter in question would be barred by the applicable statute of limitations. The award rendered by the arbitrator shall be final and judgment may be entered upon it in accordance with applicable law in any court having jurisdiction thereof.

     14.      Parties. This Agreement shall be binding upon and shall inure to the benefit of any successors or assigns of Employer and/or ANB. Employer may assign this Agreement without the consent of Executive, and Employer’s successors and assigns may enforce any and all terms and conditions of this Agreement, including but not limited to the confidentiality, non-competition and non-solicitation provisions contained in this Agreement. This Agreement shall be binding upon Executive’s heirs, estate and personal representative, and Executive may not assign any of Executive’s rights or delegate any of Executive’s duties or obligations under this Agreement or any portion hereof.

     15.      Waiver of Claims. In consideration of the obligations of Employer hereunder, Executive acknowledges that, at the request and election of Employer, as a condition precedent to receiving any severance payments or benefits from Employer otherwise payable hereunder, Executive will be required to execute and deliver a release whereby Executive unconditionally releases Employer, its directors, officers, employees, agents and shareholders, from any and all claims, liabilities and obligations of any nature pertaining to the termination of Executive’s employment by Employer for any reason, including but not limited to (a) any claims under federal, state or local laws prohibiting discrimination, including without limitation the Age Discrimination in Employment Act of 1967, as amended, or (b) any claims growing out of any alleged legal restrictions on Employer’s right to terminate Executive’s employment, such as any alleged implied contract of employment or termination contrary to public policy.

     16.      Taxes; Withholding.

     (a)      Executive shall be solely responsible for the payment of all income and other taxes in connection with any and all payments received from Employer or made by Employer on Executive’s behalf. All amounts payable pursuant to this Agreement, including without limitation severance compensation, shall be subject to reduction by all applicable withholding, social security and other federal, state and local taxes and deductions.

     (b)      If, in the reasonable estimation of Employer and/or its tax advisors, any payment or benefit received or to be received by Executive, whether pursuant to the terms of this Agreement or any other plan, arrangement or agreement with Employer or its affiliates (collectively the “Total Payments”) would not be deductible (in whole or in part) as a result of Section 280G of the Internal Revenue Code of 1986, as amended (the “Code”), by Employer or its affiliates, the parties hereby agree, to the extent possible, to take all action and execute all documents necessary to insure that none of the payments made to Executive shall be treated as “parachute payments” for the purposes of disallowance of deductions under Section 280G; provided, however, that to the extent the foregoing is not possible, payments or benefits shall be so reduced until no portion of the Total Payments is not deductible by Employer (or its affiliate, as the case may be). Executive shall be entitled to elect which payments or benefits shall be so reduced.

     17.      Survival. The rights, obligations, agreements and provisions contained in Sections 7 through 20 of this Agreement shall survive the termination of Executive’s employment for any reason and/or the expiration of the Agreement Term.

     18.      American Jobs Creation Act of 2004. To the extent the American Jobs Creation Act of 2004, as amended, and the regulations thereunder (collectively, the “Act”) apply to any payment to be made to Executive hereunder, the parties’ intent is that such payment, unless expressly provided otherwise (such as in the case of severance payments) or unless deferred pursuant to the terms of a written deferred compensation plan maintained by Employer or one of its affiliates, will be paid no later than (a) 2¹/2 months after the end of Executive’s first taxable year in which the amount is no longer subject to a “substantial risk of forfeiture” or (b) 2¹/2 months after the end of Executive’s first taxable year in which the amount is no longer subject to a “substantial risk of forfeiture.” The purpose of this provision is to reflect the parties’ desire and intent to comply with the Act, to the extent applicable.

     19.      Entire Agreement; Reinstatement. This Agreement supersedes and cancels any prior employment agreement, change in control agreement or understanding entered into between Executive and Employer and/or The PB Financial Services Corporation (“PBF”). Executive acknowledges and agrees that, except as provided for in Section 19 above, Executive is not eligible to receive any change in control or similar payment from either Employer, PBF or ANB, and Executive waives all rights thereto. Notwithstanding the foregoing, the parties acknowledge that this Agreement has been entered into as a condition to, but prior the consummation of, the transactions provided for in the Merger Agreement. If the Merger Agreement is terminated pursuant to the terms thereof, then this Agreement shall simultaneously and automatically terminate as well, without any further action on the part of the parties hereto, both parties shall be released of their respective obligations hereunder (including without limitation Employer’s obligation to pay the Stay Bonus), and Executive’s employment by Employer shall revert to the status in effect immediately prior to the execution of this Agreement.

     IN WITNESS WHEREOF, this Agreement has been duly executed and delivered by Executive and by a duly authorized officer of Employer as of the date first above written.

Witness		Jim Olson
Witness