RETENTION, CONFIDENTIALITY, NONSOLICITATION AND NONACCEPTANCE AGREEMENT

Exhibit 10.2

RETENTION, CONFIDENTIALITY,

NONSOLICITATION AND NONACCEPTANCE AGREEMENT

This AGREEMENT (the “Agreement”) is made as of March 2, 2007 by and between Boston Private Financial Holdings, Inc., a Massachusetts corporation with its headquarters located in Boston, Massachusetts (the “Employer”), Charter Bank, a Washington chartered bank (the “Bank”) and Terry A. Peterson (the “Executive”), which shall be effective at the Effective Time of the Merger (as defined in the Agreement and Plan of Merger by and between the Employer and Charter Financial Corporation dated as of March 3, 2007 (the “Merger Agreement”)). In the event that the Effective Time shall not occur, this Agreement shall be void ab initio and of no further force and effect.

WHEREAS, at the Effective Time, the Employer will acquire Charter Financial Corporation and its wholly-owned subsidiary, the Bank;

WHEREAS, following the Effective Time, the Employer will continue to operate the Bank under the name Charter Bank;

WHEREAS, following the Effective Time, the Bank desires to continue to employ the Executive as President and Chief Operating Officer of the Bank, and the Executive desires to continue to be employed by the Bank;

WHEREAS, the Executive acknowledges that, in his position he will be given access to and will help develop trade secrets, valuable confidential business or professional information, substantial customer relationships and customer good will on behalf of the Bank and the Employer;

NOW THEREFORE, in consideration of the mutual promises contained in this Agreement, the Employer, the Bank and the Executive agree as follows:

1. Restricted Stock Grants.

(a) Conditions of Grants. On the Effective Date and the first anniversary thereof, provided that the Executive is employed by the Employer and/or the Bank on such date, the Executive will receive a grant of a number of restricted shares of the Employer (the “Restricted Shares”) pursuant to the terms of the Employer’s 2004 Stock Option and Incentive Plan or any successor plan (the “Plan), to the extent not inconsistent with the terms herewith, as follows:

(i) Initial Grant of Restricted Shares. The number of Restricted Shares issued on the Effective Date shall be equal to the quotient obtained by dividing (x) $98,750 by (y) the closing price of the Employer’s common stock on the Effective Date,

(ii) Second Grant of Restricted Shares. The number of Restricted Shares issued to the Executive on the first anniversary date of the Effective Date shall be equal to the quotient obtained by dividing (A) $98,750 by (B) the closing price of the Employer’s common stock on the date of grant.

(b) Vesting of Restricted Stock Grant. Subject to Section 1(c) below of this Agreement, each grant of Restricted Shares will vest over a five-year period, with vesting at 33 ¹/3 percent on each of the third, fourth and fifth anniversary of the date of grant, subject to the Executive’s continuous employment with the Employer and/or the Bank on each applicable vesting date.

(c) Potential for Accelerated Vesting of Restricted Shares. Notwithstanding the foregoing and any provision in the Plan (including without limitation Section 7(c) thereof) to the contrary, the Restricted Shares shall immediately vest upon a Change of Control (as defined in the Plan) or upon the Executive’s death or disability.

2. Confidential Information, Nonsolicitation and Nonacceptance.

(a) Confidential Information. As used in this Agreement, “Confidential Information” means information belonging to the Employer and the Bank which is of value to the Employer and the Bank in the course of conducting their business and the disclosure of which could result in a competitive or other disadvantage to the Employer or the Bank. Confidential Information includes, without limitation, financial information, reports, and forecasts; inventions, improvements and other intellectual property; trade secrets; know-how; designs, processes or formulae; software; market or sales information or plans; customer lists; and business plans, prospects and opportunities (such as possible acquisitions or dispositions of businesses or facilities) which is of value to the Employer and the Bank in the course of conducting its business and the disclosure of which could result in a competitive or other disadvantage to the Employer and the Bank. Confidential Information includes information developed by the Executive in the course of the Executive’s employment by the Bank, as well as other information to which the Executive may have access in connection with the Executive’s employment. Confidential Information also includes the confidential information of others with which the Employer has a business relationship. Notwithstanding the foregoing, Confidential Information does not include information in the public domain, unless due to breach of the Executive’s duties under Section 2(b).

(b) Confidentiality. The Executive understands and agrees that the Executive’s employment creates a relationship of confidence and trust between the Executive and the Employer and the Bank with respect to all Confidential Information. At all times, both during the Executive’s employment with the Employer and the Bank and after its termination, the Executive will keep in confidence and trust all such Confidential Information, and will not use or disclose any such Confidential Information without the written consent of the Employer or the Bank, except as may be necessary in the ordinary course of performing the Executive’s duties to the Employer and the Bank or as may be required by applicable law or pursuant to legal process.

(c) Documents, Records, etc. All documents, records, data, apparatus, equipment and other physical property, whether or not pertaining to Confidential Information, which are furnished to the Executive by the Employer and or Bank or are produced by the Executive in connection with the Executive’s employment will be and remain the sole property of the Employer and the Bank. The Executive will return to the Employer and the Bank all such materials and property as and when requested by the Employer and/or the Bank. In any event,

2

the Executive will return all such materials and property immediately upon termination of the Executive’s employment for any reason. The Executive will not retain with the Executive any such material or property or any copies thereof after such termination.

(d) Nonsolicitation and Nonacceptance Agreement. During the Executive’s employment with the Bank and for two (2) years thereafter, the Executive (i) will refrain from directly or indirectly employing, attempting to employ, recruiting or otherwise soliciting, inducing or influencing any person to leave employment with the Employer or the Bank (other than terminations of employment of subordinate employees undertaken in the course of the Executive’s employment with the Employer and the Bank); (ii) will refrain from directly or indirectly soliciting or encouraging any client or supplier of the Employer or the Bank to terminate or otherwise modify adversely its business relationship with the Employer or the Bank; and (iii) will refrain from directly or indirectly accepting banking business from a client doing business with the Bank at the time of the Executive’s termination of employment in King County, Washington and in any county in which the Executive last worked. The provisions of this Agreement are intended to prohibit actions taken directly by the Executive or indirectly by the Executive through another entity with which the Executive is associated, either as an owner, partner, shareholder, consultant, agent, employee, co-venturer or otherwise. The Executive understands that the restrictions set forth in this Section 2(d) are intended to protect the Employer’s and the Bank’s interest in its Confidential Information and established employee, client and supplier relationships and goodwill, and agrees that such restrictions are reasonable and appropriate for this purpose. In the event the Employer or the Bank terminates the Executive without Cause, or the Executive terminates for Good Reason, the restrictions set forth in this Section 2(d) shall cease to apply.

Only the following shall constitute “Cause” for such termination:

(i) use of alcohol or a controlled substance which is materially and demonstrably injurious to the Employer or the Bank;

(ii) the conviction of the Executive for, or a plea of guilty or nolo contendere by the Executive to, a felony (other than traffic offenses);

(iii) intentional refusal by the Executive to perform his stated and lawful duties to the Employer or the Bank in any material respect, after written notice to the Executive and an opportunity to cure;

(iv) willful breach of fiduciary duty owed by the Executive to the Employer or the Bank, which is materially and demonstrably injurious to the Employer or the Bank;

(v) the Executive’s gross negligence or willful misconduct in the performance of his duties hereunder or the material breach of any of the terms of this Agreement by the Executive, after written notice to the Executive and an opportunity to cure; or

3

(vi) willful failure to cooperate with a bona fide internal investigation or an investigation by regulatory or law enforcement authorities, after being instructed by the Employer and the Bank to cooperate, or the willful destruction or failure to preserve documents or other materials known to be relevant to such investigation or the willful inducement of others to fail to cooperate or to produce documents or other materials.

Only the following shall constitute “Good Reason” for termination by the Executive:

(A) a substantial adverse change, without the prior written consent of the Executive, in the nature or scope of the Executive’s authority, duties or responsibilities from the authority, duties or responsibilities exercised by the Executive prior to the Effective Time;

(B) any decrease in the base salary of the Executive except for across-the-board decreases similarly affecting all or substantially all management employees of the Bank;

(C) the failure to include the Executive in any bonus or material benefit plan of the Bank; or

(D) any requirement by the Employer or the Bank that the Executive’s services be rendered primarily at a location or locations more than 35 miles away from the Executive’s office location immediately prior to the Effective Time.

The Executive shall provide the Employer and the Bank with reasonable notice and an opportunity to cure any of the events listed in subparts (A) through (D) above and the Executive shall not be entitled to terminate for Good Reason unless the Employer and the Bank, as applicable, fails to cure within a reasonable period, which shall in no event exceed five (5) business days.

(e) Injunction. The Executive agrees that it would be difficult to measure any damages caused to the Employer which might result from any breach by the Executive of the promises set forth in this Section 2, and that in any event money damages would be an inadequate remedy for any such breach. Accordingly, subject to Section 4 of this Agreement, the Executive agrees that if the Executive breaches, or proposes to breach, any portion of this Agreement, the Employer and the Bank shall be entitled, in addition to all other remedies that they may have, to an injunction or other appropriate equitable relief to restrain any such breach without showing or proving any actual damage to the Employer or the Bank.

3. Arbitration of Disputes. Any controversy or claim arising out of or relating to this Agreement or the breach thereof or otherwise arising out of the Executive’s employment or the termination of that employment (including, without limitation, any claims of unlawful employment discrimination whether based on age or otherwise) shall, to the fullest extent permitted by law, be settled by arbitration in any forum and form agreed upon by the parties or, in the absence of such an agreement, under the auspices of the American Arbitration Association (“AAA”) in Seattle, Washington in accordance with the Employment Dispute Resolution Rules of the AAA, including, but not limited to, the rules and procedures applicable to the selection of arbitrators. In the event that any person or entity other than the Executive, the Employer or the

4

Bank may be a party with regard to any such controversy or claim, such controversy or claim shall be submitted to arbitration subject to such other person or entity’s agreement. Judgment upon the award rendered by the arbitrator may be entered in any court having jurisdiction thereof. This Section 3 shall be specifically enforceable. Notwithstanding the foregoing, this Section 3 shall not preclude any party from pursuing a court action for the sole purpose of obtaining a temporary restraining order or a preliminary injunction in circumstances in which such relief is appropriate; provided that any other relief shall be pursued through an arbitration proceeding pursuant to this Section 3.

4. Consent to Jurisdiction. To the extent that any court action is permitted consistent with or to enforce Section 2 of this Agreement, the parties hereby consent to the jurisdiction of the Superior Court of the State of Washington and the United States District Court for the District of Washington. Accordingly, with respect to any such court action, the Executive (a) submits to the personal jurisdiction of such courts; (b) consents to service of process; and (c) waives any other requirement (whether imposed by statute, rule of court, or otherwise) with respect to personal jurisdiction or service of process.

5. Enforceability. If any portion or provision of this Agreement (including, without limitation, any portion or provision of any section of this Agreement) shall to any extent be declared illegal or unenforceable by a court of competent jurisdiction, then the remainder of this Agreement, or the application of such portion or provision in circumstances other than those as to which it is so declared illegal or unenforceable, shall not be affected thereby, and each portion and provision of this Agreement shall be valid and enforceable to the fullest extent permitted by law.

6. Waiver. No waiver of any provision hereof shall be effective unless made in writing and signed by the waiving party. The failure of any party to require the performance of any term or obligation of this Agreement, or the waiver by any party of any breach of this Agreement, shall not prevent any subsequent enforcement of such term or obligation or be deemed a waiver of any subsequent breach.

7. Notices. Any notices, requests, demands and other communications provided for by this Agreement shall be sufficient if in writing and delivered in person or sent by a nationally recognized overnight courier service or by registered or certified mail, postage prepaid, return receipt requested, to the Executive at the last address the Executive has filed in writing with the Bank or, in the case of the Employer, at its main offices, attention of the Employer’s Chief Executive Officer, and shall be effective on the date of delivery in person or by courier or three (3) days after the date mailed.

8. Amendment. This Agreement may be amended or modified only by a written instrument signed by the Executive and by a duly authorized representative of the Employer.

9. Governing Law. This is a Washington contract and shall be construed under and be governed in all respects by the laws of the State of Washington, without giving effect to the conflict of laws principles of such State. With respect to any disputes concerning federal law, such disputes shall be determined in accordance with the law as it would be interpreted and applied by the United States Court of Appeals for the Ninth Circuit.

5

This Agreement has been executed as a sealed instrument by the Employer, by its duly authorized officer, and by the Executive, as of the Effective Date.

6