EXHIBIT 10.5.1 MIDWESTONE FINANCIAL GROUP, INC.

EXHIBIT 10.5.1

MIDWESTONE FINANCIAL GROUP, INC.

FOURTH AMENDMENT TO SECOND AMENDED

AND RESTATED CREDIT AGREEMENT

This Fourth Amendment to Second Amended and Restated Credit Agreement (herein, the “Amendment”) is entered into as of April 28, 2006, between MidWestOne Financial Group, Inc., an Iowa corporation (the “Borrower”), and Harris N.A. (the “Bank”).

PRELIMINARY STATEMENTS

A. The Borrower and the Bank are parties to that certain Second Amended and Restated Credit Agreement, dated as of November 30, 2003, as amended (the “Credit Agreement”). All capitalized terms used herein without definition shall have the same meanings herein as such terms have in the Credit Agreement.

B. The Borrower has requested that the Bank extend the Revolving Credit Termination Date and amend Sections 7.12(a) (Non-Performing Assets; Adjusted Non-Performing Assets) and 7.14 (Return of Assets) of the Credit Agreement, and the Bank is willing to do so under the terms and conditions set forth in this Amendment.

NOW, THEREFORE, for good and valuable consideration, the receipt and sufficiency of which is hereby acknowledged, the parties hereto agree as follows:

SECTION 1. AMENDMENTS.

Subject to the satisfaction of the conditions precedent set forth in Section 2 below, the Credit Agreement shall be and hereby is amended as follows:

1.1. The definition of “Revolving Credit Termination Date” set forth in Section 4 of the Credit Agreement (Definitions) shall be and hereby is amended and restated in its entirety to read as follows:

“Revolving Credit Termination Date” means April 30, 2007, or such earlier date on which the Revolving Credit Commitment is terminated in whole pursuant to Section 2.4, 8.2 or 8.3 hereof.

1.2. Section 7.14 of the Credit Agreement (Return on Assets) shall be and hereby is amended and restated in its entirety to read as follows:

Section 7.14. Return on Assets. As of the last day of each June and December in each year, the Borrower shall maintain a

ratio of Consolidated Net Income for the 12-month period then ended to Consolidated Total Assets as at the last day of such 12-month period then ended of not less than .0085 to 1.0.

1.3. On or about January 1, 2006, each of the following Subsidiaries of the Borrower, Pella State Bank, Central Valley Bank and MidWestOne Bank, merged with and into MidWestOne Bank & Trust, also a Subsidiary of the Borrower (the “Merger”). Subsequent to the Merger, MidWestOne Bank & Trust changed its name to MidWestOne Bank. As a result of the Merger, Schedule 5.2 of the Credit Agreement shall be amended and restated in its entirety to read as set forth on Schedule 5.2 attached hereto and made a part hereof.

SECTION 2. CONDITIONS PRECEDENT.

The effectiveness of this Amendment is subject to the satisfaction of all of the following conditions precedent:

2.1. The Borrower and the Bank shall have executed and delivered this Amendment.

2.2. The Borrower shall deliver to the Bank (a) original stock certificate number 1 representing 50,400 shares of MidWestOne Bank (f/k/a MidWestOne Bank & Trust), (b) original stock certificate number 1 representing 5,000 shares of MidWest One Investment Services, Inc. and (c) original stock certificate number 6 representing 250 shares of Cook & Son Agency, Inc. (and undated stock powers related thereto executed in blank).

2.3. Legal matters incident to the execution and delivery of this Amendment shall be satisfactory to the Bank and its counsel.

SECTION 3. REPRESENTATIONS.

In order to induce the Bank to execute and deliver this Amendment, the Borrower hereby represents to the Bank that as of the date hereof, after giving effect to amendments set forth above, the representations and warranties set forth in Section 5 of the Credit Agreement are and shall be and remain true and correct (except that the representations contained in Section 5.5 shall be deemed to refer to the most recent financial statements of the Borrower delivered to the Bank) and the Borrower is in compliance with the terms and conditions of the Credit Agreement and no Default or Event of Default exists under the Credit Agreement or shall result after giving effect to this Amendment.

SECTION 4. MISCELLANEOUS.

4.1. The Borrower heretofore executed and delivered to the Bank various Collateral Documents. The Borrower hereby acknowledges and agrees that the Liens created and provided

-2-

for by the Collateral Documents continue to secure, among other things, the Obligations arising under the Credit Agreement as amended hereby; and the Collateral Documents and the rights and remedies of the Bank thereunder, the obligations of the Borrower thereunder, and the Liens created and provided for thereunder remain in full force and effect and shall not be affected, impaired or discharged hereby. Nothing herein contained shall in any manner affect or impair the priority of the liens and security interests created and provided for by the Collateral Documents as to the indebtedness which would be secured thereby prior to giving effect to this Amendment.

4.2. Except as specifically amended herein, the Credit Agreement shall continue in full force and effect in accordance with its original terms. Reference to this specific Amendment need not be made in the Credit Agreement, the Notes, or any other instrument or document executed in connection therewith, or in any certificate, letter or communication issued or made pursuant to or with respect to the Credit Agreement, any reference in any of such items to the Credit Agreement being sufficient to refer to the Credit Agreement as amended hereby.

4.3. The Borrower agrees to pay on demand all costs and expenses of or incurred by the Bank in connection with the negotiation, preparation, execution and delivery of this Amendment, including the fees and expenses of counsel for the Bank.

4.4. This Amendment may be executed in any number of counterparts, and by the different parties on different counterpart signature pages, all of which taken together shall constitute one and the same agreement. Any of the parties hereto may execute this Amendment by signing any such counterpart and each of such counterparts shall for all purposes be deemed to be an original. Delivery of executed counterparts of this Amendment by telecopy shall be effective as an original. This Amendment shall be governed by the internal laws of the State of Illinois.

[SIGNATURE PAGE TO FOLLOW]

-3-

This Fourth Amendment to Second Amended and Restated Credit Agreement is entered into as of the date and year first above written.

Accepted and agreed to.

-4-

SCHEDULE 5.2

SUBSIDIARIES AS OF APRIL 28, 2006