PROMISSORY NOTE AMENDMENT

Exhibit 10.1

PROMISSORY NOTE AMENDMENT

This Promissory Note Amendment by and between Impac Funding Corporation (“Maker”) and Impac Mortgage Holdings, Inc. (“Holder”) is made and executed effective as of June 30, 2004 (the “Amendment”).

RECITALS:

WHEREAS, Maker executed and issued that certain Note dated June 30, 1999 in the original principal amount of $14,500,000 in favor of Holder, a copy of which is attached hereto as Exhibit A (the “Note”);

WHEREAS, the parties acknowledge and agree that Maker has timely made all minimum required payments under, and currently is not in default of any of the terms or conditions of, the Note; and

WHEREAS, notwithstanding its rights under the Note, Holder has agreed to extend the term of the Note until June 30, 2005 and to accept monthly interest only payments in accordance with the terms of this Amendment for an additional 12-month period ending June 30, 2005 (the “Extended Interest Only Period”).

NOW THEREFORE, the parties intending to be legally bound, hereby agree that the foregoing recitals are incorporated herein by reference and as follows:

1. Extension of Term. The parties hereby agree to extend the term of the Note until June 30, 2005 (the “Maturity Date”), at which time any and all remaining principal and unpaid interest shall be due and payable.

2. Interest Rate. For each calendar month during the Extended Interest Only Period, the Note shall bear interest (computed on the basis of a 360-day year, for the actual number of days occurring in the period for which such interest is payable) at a rate per annum equal to the Prime Rate (as defined below) on the principal amount from time to time remaining unpaid. For purposes of this Amendment, the “Prime Rate” in effect for any particular calendar month during the Extended Interest Only Period shall be the “prime rate” established in the Wall St. Journal on the first business day of such calendar month.

3. Minimum Interest Only Payments. Commencing July 31, 2004 and continuing thereafter on the last day of each calendar month until the Maturity Date, Maker shall pay in arrears, and Holder agrees to accept, monthly interest only payments calculated in accordance with Section 2 above.

4. Conflict of Terms. All other terms and conditions of the Note shall remain unchanged and in full force and effect. In the event of a conflict between the terms of the Note and this Amendment, this Amendment shall control. Nothing contained in this Amendment shall effect [sic] the right of Maker to prepay in part or in full without penalty or premium all amounts owed under the Note.

5. Complete Agreement. This Amendment, along with the Note, constitutes the entire understanding of the parties and supersede [sic] in their entirety all prior oral or written understandings related thereto.

6. Consideration. The parties acknowledge and agree the mutual covenants contained herein shall be sufficient consideration for all purposes.

7. Notice. Any notice required or permitted to be sent pursuant to this Amendment shall be sent by United States certified mail, return receipt requested, to the last known address of such party.

[SIGNATURES ON FOLLOWING PAGE]

2

IN WITNESS WHEREOF, the parties have executed this Amendment effective as of June 30, 2004.

3