AMENDMENT TO EMPLOYMENT AGREEMENT

Exhibit 10.4

AMENDMENT TO

EMPLOYMENT AGREEMENT

THIS AMENDMENT TO EMPLOYMENT AGREEMENT (“Amendment”) is made and entered into as of the 26th day of November 2012, by and among Provident New York Bancorp, a Delaware corporation (the “Company), Provident Bank, a savings bank organized and existing under the laws of the United States of America (the “Bank”; and together with the Company, “Provident”), and James R. Peoples (“Executive”).

WHEREAS, Executive, the Company and the Bank are parties to an Employment Agreement, dated as of November 30, 2011 (the “Employment Agreement”); and

WHEREAS, Provident and Executive desire to amend the Employment Agreement to extend the term of the Employment Agreement until November 30, 2015 and make certain changes relating to the payment of severance after a Change in Control;;

NOW, THEREFORE, in consideration of Executive’s continued employment by Provident and other good and valuable consideration the receipt of which Executive acknowledges, the Company, the Bank and Executive agree that the Employment Agreement is amended as follows:

1. The text of Section 2(a) of the Employment Agreement is hereby amended to read as follows:

Executive’s period of employment with Provident shall begin on the Effective Date and shall continue until November 30, 2015 (or if a Change in Control occurs prior to November 30, 2015 not earlier than the one year anniversary of the date of the Change in Control), unless terminated prior thereto by either Provident or Executive in accordance with Section 6 hereof (such period of employment being the “Employment Period”).

2. The text of clause (B) of Section 6(a)(ii) of the Employment Agreement is hereby amended to read as follows:

If such termination of employment occurs within one year after a Change in Control, subject to Section 6(g), Provident shall (I) pay to Executive within sixty (60) days following the date of termination a lump sum cash payment (the “CIC Severance Payment”) equal to the sum of (x) Executive’s Base Salary immediately prior to termination of employment and (y) Executive’s target bonus for the fiscal year that includes the date of termination, and (II) pay to Executive on a monthly basis commencing with the first month following Executive’s termination of employment and continuing until the eighteenth month following Executive’s termination of employment a cash payment (subject to reduction for applicable withholding taxes) equal to the monthly COBRA premium in effect as of the date of Executive’s termination of employment for the level of coverage in effect for Executive under Provident’s group health plan (the “CIC COBRA Payments”);

IN WITNESS WHEREOF, the Company and the Bank have caused this Amendment to be executed and Executive has hereunto set his hand, all as of the date first written above.