VOTING AGREEMENT

Exhibit 10.4

VOTING AGREEMENT

This Voting Agreement (this “Agreement”) is dated as of January 20, 2014, by and between Center Bancorp, Inc., a New Jersey corporation and registered bank holding company (“Parent”), and the shareholder of ConnectOne Bancorp, Inc., a New Jersey corporation and registered bank holding company (the “Company”), executing this Agreement on the signature page hereto (the “Shareholder”).

RECITALS

A. Concurrently with the execution of this Agreement, Parent and the Company have entered into an Agreement and Plan of Merger (the “Merger Agreement”) that provides, among other things, for the merger (the “Merger”) of the Company with and into Parent upon the terms and subject to the conditions set forth therein.

B. As of the date hereof, the Shareholder is the record and Beneficial Owner (as defined below) of that number of shares of Company Common Stock (including, for purposes of this Agreement, all shares or other voting securities into which any shares of Company Common Stock may be reclassified, sub-divided, consolidated or converted and any rights and benefits arising therefrom (including any dividends or distributions of securities that may be declared in respect of such shares of Company Common Stock), the “Company Common Shares”) set forth below the Shareholder’s name on the signature page hereto.

C. As a condition to Parent’s willingness to enter into and perform its obligations under the Merger Agreement, the Shareholder has agreed to enter into this Agreement.

NOW THEREFORE, the parties hereto agree as follows:

I. CERTAIN DEFINITIONS

1.1. Capitalized Terms. Capitalized terms used in this Agreement and not defined herein shall have the meanings ascribed to such terms in the Merger Agreement.

1.2. Other Definitions. For the purposes of this Agreement:

“Beneficial Owner” or “Beneficial Ownership” with respect to any securities means having “beneficial ownership” of such securities (as determined pursuant to Rule 13d-3 under the Securities Exchange Act of 1934, as amended).

“Jointly Owned Shares” means the Company Common Shares Beneficially Owned by the Shareholder as of the applicable record date (including any Company Common Shares that the Shareholder may acquire after the date hereof) for which the Shareholder has joint or shared voting power with such Shareholder’s spouse.

“Owned Shares” means the Company Common Shares Beneficially Owned by the Shareholder as of the applicable record date (including any Company Common Shares that the Shareholder may acquire after the date hereof) for which the Shareholder has sole voting power.

“Restricted Transfer Termination Date” means the soonest of (i) the date that is six (6) months after the Merger Agreement is terminated in accordance with its terms, (ii) the Effective Time, (iii) the date, if any, on which the Parent releases the Shareholder from the Shareholder’s

obligations hereunder and (iv) the date immediately following the date, if any, on which Company’s shareholders approve all of the Company Shareholder Matters.

“Transfer” means, with respect to a security, the sale, grant, assignment, transfer, pledge, hypothecation, encumbrance, constructive sale, or other disposition of such security or the Beneficial Ownership thereof (including by operation of law), or the entry into of any contract, agreement or other obligation to effect any of the foregoing, including, for purposes of this Agreement, the transfer or sharing of any voting, investment or dispositive power of such security.

II. SUPPORT OBLIGATIONS OF THE SHAREHOLDER

2.1. Agreement to Vote. The Shareholder irrevocably and unconditionally agrees that from and after the date hereof, at any meeting (whether annual or special, and at each adjourned or postponed meeting) of shareholders of the Company called to vote for approval of the Merger, however called, or in connection with any written consent of the Company’s shareholders relating to the Merger, the Shareholder will (x) appear at each such meeting, cause all of the Shareholder’s Owned Shares, and use the Shareholder’s reasonable best efforts to cause all of the Shareholder’s Jointly Owned Shares, to be counted as present thereat for purposes of calculating a quorum, and respond to each request by the Company for written consent, if any, (y) vote (or consent) or cause to be voted (or validly execute and return and cause a consent to be granted with respect to) all of the Owned Shares and use the Shareholder’s reasonable best efforts to cause to be voted (or validly execute and return and use the Shareholder’s reasonable best efforts to cause a consent to be granted with respect to) all of the Jointly Owned Shares, in each case, in favor of all Company Shareholder Matters, including the adoption of the Merger Agreement and the Merger and, if it shall be necessary for any such meeting to be adjourned or postponed due to a lack of a quorum, in favor of such adjournment or postponement and (z) vote (or consent) or cause to be voted (or validly execute and return and cause a consent to be granted with respect to) all of the Owned Shares and use the Shareholder’s reasonable best efforts to cause to be voted (or validly execute and return and use the Shareholder’s reasonable best efforts to cause a consent to be granted with respect to) all of the Jointly Owned Shares, in each case, against any Company Acquisition Proposal.

2.2. Restrictions on Transfer. Except as otherwise consented to in writing by Parent, the Shareholder agrees from and after the date hereof, not to tender, or cause to be tendered, into any tender or exchange offer or otherwise directly or indirectly Transfer, or cause to be Transferred, any Owned Shares or Jointly Owned Shares (or any rights, options or warrants to acquire any Company Common Shares), except for transfers to charities, charitable trusts, or other charitable organizations under Section 501(c)(3) of the Internal Revenue Code of 1986, as amended, lineal descents or the spouse of the Shareholder, or to a trust or other entity for the benefit of one or more of the foregoing persons, or by means of an in-kind distribution of all or part of the Shareholder’s Company Common Shares to the Shareholder’s direct or indirect equityholders; provided that the transferee of any transfer described in this Section 2.2 agrees in writing to be bound by the terms of this Agreement. If so requested by Parent, the Shareholder agrees that the certificates representing Owned Shares and Jointly Owned Shares shall bear a legend stating that they are subject to this Agreement.

2.3 Company Acquisition Proposal. The Shareholder agrees that from and after the date hereof, the Shareholder will not, and will use the Shareholder’s reasonable best efforts to not permit any of the Shareholder’s affiliates to, directly or indirectly, solicit, initiate, encourage or facilitate, or furnish or disclose non-public information in furtherance of, or comment publicly in favor of, any inquiries or the making of any proposal with respect to any Company Acquisition Proposal, or negotiate, explore or otherwise engage in discussions with any person (other than Parent or its directors, officers, employees, agents and representatives) with respect to any Company Acquisition Proposal or enter into any agreement, arrangement or understanding with respect to any Company Acquisition Proposal or agree to or otherwise assist in the effectuation of any Company Acquisition Proposal or comment publicly in favor of any Company Acquisition Proposal; provided, however, that nothing herein shall prevent the Shareholder from taking any action, or omitting to take any action, (i) if applicable, as a member of the Board of Directors of the Company required so as not to act inconsistently with the Shareholder’s fiduciary obligations as a Director of the Company after consultation with outside counsel or (ii) if applicable, as an officer of the Company required so as not to act inconsistently with the Shareholder’s fiduciary obligations, if any, as an officer of the Company after consultation with outside counsel, in each case to the extent, and only to the extent, permitted by Section 5.3 of the Merger Agreement.

III. GENERAL

3.1. Governing Law; Jurisdiction. This Agreement and any controversies arising with respect hereto shall be construed in accordance with and governed by the laws of the State of New Jersey (without regard to principles of conflict of laws that would apply the law of another jurisdiction). Each of the parties hereto hereby irrevocably and unconditionally submits, for itself and its property, to the jurisdiction of any New Jersey State court or federal court of the United States of America sitting in New Jersey, and any appellate court thereof, in any action or proceeding arising out of or relating to this Agreement, or for recognition or enforcement of any judgment, and each of the parties hereto hereby irrevocably and unconditionally agrees that all claims in respect of any such action or proceeding may be heard and determined in any such New Jersey State court or, to the extent permitted by law, in such federal court.

3.2. Amendments. This Agreement may not be amended except by written agreement signed by Parent and by the Shareholder.

3.3. Entire Agreement. This Agreement constitutes the entire agreement and supersedes all other prior agreements, understandings, representations and warranties, both written and oral, among the parties to this Agreement with respect to the subject matter of this Agreement.

3.4. Counterparts; Execution. This Agreement may be executed in any number of counterparts, all of which are one and the same agreement. This Agreement may be executed by facsimile or pdf signature by any party and such signature is deemed binding for all purposes hereof, without delivery of an original signature being thereafter required.

3.5. Effectiveness and Termination. This Agreement will become effective when Parent has received the counterparts signed by the Shareholder and itself and shall terminate on the Restricted Transfer Termination Date Upon such termination, except for any rights any party

may have in respect of any breach by any other party of its obligations hereunder, neither party hereto shall have any further obligation or liability hereunder.

3.6 Proxy. The Shareholder hereby constitutes and appoints the President of Parent, with full power of substitution, as the Shareholder’s proxy with respect to the matters set forth herein, including without limitation, each of the matters described in Sections 2.1 and 2.3 of this Agreement, and hereby authorizes such proxy to represent and to vote, if and only if the Shareholder (i) fails to vote or (ii) attempts to vote (whether by proxy, in person or by written consent) in a manner that is inconsistent with the terms of this Agreement, all of such Shareholder’s Owned Shares in the manner contemplated by Sections 2.1 and 2.3 of this Agreement. The proxy granted pursuant to the immediately preceding sentence is given to induce Parent to execute the Merger Agreement and, as such, is coupled with an interest and shall be irrevocable unless and until this Agreement or any such rights granted hereunder terminate or expire pursuant to the terms hereof. The Shareholder hereby revokes any and all previous proxies with respect to the Shareholder’s Owned Shares and shall not hereafter, unless and until this Agreement or any rights granted hereunder terminate or expire pursuant to the terms hereof, purport to grant any other proxy or power of attorney with respect to any of the Shareholder’s Owned Shares, deposit any of the Shareholder’s Owned Shares into a voting trust or enter into any agreement (other than this Agreement), arrangement or understanding with any person or entity, directly or indirectly, to vote, grant any proxy or give instructions with respect to the voting of any of any of the Shareholder’s Owned Shares, in each case, with respect to any of the matters set forth herein.

3.7 Equitable Remedies. The parties hereto agree that irreparable harm would occur in the event that any of the agreements and provisions of this Agreement were not performed fully by the parties hereto in accordance with their specific terms or conditions or were otherwise breached, and that money damages are an inadequate remedy for breach of this Agreement because of the difficulty of ascertaining and quantifying the amount of damage that will be suffered by the parties hereto in the event that this Agreement is not performed in accordance with its terms or conditions or is otherwise breached. It is accordingly hereby agreed that the parties hereto shall be entitled to an injunction or injunctions to restrain, enjoin and prevent breaches of this Agreement by the other parties and to enforce specifically the terms and provisions hereof in any court referred to in Section 3.1 hereof, such remedy being in addition to, and not in lieu of, any other rights and remedies to which the other parties are entitled to at law or in equity.

3.8 Waiver of Jury Trial. Each party hereto hereby waives to the fullest extent permitted by applicable law, any right it may have to a trial by jury in respect to any suit, action or other proceeding directly or indirectly arising out of, under or in connection with this Agreement or any transaction contemplated hereby. Each party hereto (a) certifies that no representative of any other party hereto has represented, expressly or otherwise, that such other party would not, in the event of any suit, action or other proceeding, seek to enforce that foregoing waiver and (b) acknowledges that it and the other parties hereto have been induced to enter into this Agreement, by, among other things, the mutual waivers and certifications in this Section 3.8.

3.9 Construction. This Agreement shall be deemed to have been drafted by each of the parties hereto and, consequently, when construing its terms, none of the parties will be deemed to have been the draftsperson.

[signature pages follow]

IN WITNESS WHEREOF, each party hereto has caused this Agreement to be signed as of the date first above written.

(Shareholder signature page follows)

[Parent Signature Page to Voting Agreement]

[Shareholder Signature Page to Voting Agreement]