SECOND AMENDMENT TO CREDIT AGREEMENT

EXHIBIT 4.1

SECOND AMENDMENT TO CREDIT AGREEMENT

SECOND AMENDMENT TO CREDIT AGREEMENT, dated as of February 13, 2004 (this “Agreement”), among Huttig Building Products, Inc., a Delaware corporation (“Huttig”), the domestic subsidiaries of Huttig party hereto, as joint and several borrowers, the Lenders party hereto, and JPMorgan Chase Bank, as administrative and collateral agent for the Lenders (in such capacities, together with its successors in such capacities, the “Agent”).

W I T N E S S E T H:

WHEREAS, Huttig, certain domestic subsidiaries of Huttig, the Lenders named therein, the Agent, and certain other parties thereto have entered into that certain Credit Agreement dated as of August 12, 2002, as amended by the First Amendment thereto, dated as of May 30, 2003 (as amended, supplemented, or otherwise modified from time to time, the “Credit Agreement”; capitalized terms used herein but not otherwise defined herein shall have the meanings given such terms in the Credit Agreement); and

WHEREAS, the Borrowers have requested that the Lenders and the Agent amend certain provisions of the Credit Agreement, and the Lenders and the Agent are willing so to amend the Credit Agreement on the terms and subject to the conditions set forth herein.

NOW, THEREFORE, in consideration of the premises and the agreements herein contained, the Borrowers, the Lenders, and the Agent hereby agree as follows:

Section 1. Amendments to the Credit Agreement. Upon satisfaction of the conditions set forth in Section 3 hereof, the Credit Agreement is hereby amended, effective as of the date hereof (the “Second Amendment Effective Date”), as follows:

(a) The following definitions are hereby added to Section 1.01 of the Credit Agreement in appropriate alphabetical order:

“Availability” means, as of any date of determination, the amount by which (a) the lesser of (i) the Borrowing Base as of such date and (ii) the aggregate Revolving Credit Commitments exceeds (b) the Total Exposure as of such date.

“Dominion Event” means the existence of any of the following situations: (i) Monthly Average Availability for any Fiscal Month is less than $20,000,000 or (ii) a Default or Event of Default has occurred and is continuing.

“Monthly Average Availability” means, for any Fiscal Month, (a) the sum of the Availability at the end of each day of such fiscal month divided by (b) the number of days in such fiscal month.

“Second Amendment” means that certain Second Amendment to Credit Agreement, dated as of February [—], 2004, among Huttig, the domestic subsidiaries of Huttig party thereto, the Lenders party thereto, and the Agent.

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

“Second Amendment Effective Date” has the meaning set forth in Section 1 of the Second Amendment.

(b) The definition of “Collateral Account” set forth in Section 1.01 of the Credit Agreement is hereby amended and restated in its entirety as follows:

“Collateral Account” means any account established and maintained by the Borrowers or any one of them pursuant to Section 2.03(b) at the Agent or at another financial institution reasonably acceptable to the Agent (each, a “Bank”) as an account into which all proceeds of Collateral shall be deposited pursuant to any of the Security Document, and pursuant to any Lock Box Agreement or Controlled Account Agreement which the Borrowers or any of their Subsidiaries may enter into with the Agent or with another financial institution reasonably acceptable to the Agent.

(c) The definition of “Monthly Average Collateral Availability” set forth in Section 1.01 of the Credit Agreement is hereby deleted in its entirety.

(d) Section 2.03 of the Credit Agreement is hereby amended by replacing the phrase “Section 2.03(b)” contained in subsection (b) thereof with the phrase “Section 2.03(c)”, re-labeling subsection (b) thereof as subsection (c), re-labeling subsection (c) thereof as subsection (d), and adding the following new subsection as subsection (b):

(b) The Borrowers shall maintain one or more Collateral Accounts providing for control and, in the event a Dominion Event occurs and is continuing, springing dominion over all assets deposited therein. Upon the occurrence and throughout the continuation of a Dominion Event, Agent shall be entitled to deliver notice to each Bank that such Bank shall comply with the instructions of the Agent in respect of such Collateral Account (a “Dominion Notice”). The Agent shall deliver a copy of any such Dominion Notice to the Borrowers at or about the same time as the Agent delivers such Dominion Notice to the respective Bank. On or before the Second Amendment Effective Date, the Borrowers shall issue to each Bank an irrevocable letter of instruction (which the Borrowers will cause such Bank to agree to) directing such Bank, upon receipt of written instructions from Agent, to comply with the instructions of Agent in respect of the Collateral Accounts maintained by such Bank, including, without limitation, any instruction to transfer all payments or other remittances received in each such Collateral Account to Agent’s account for application on account of the Obligations as provided in Section 2.03(c). After the occurrence and during the continuance of a Dominion Event, all funds deposited in any Collateral Account shall immediately become the property of Agent, for the ratable benefit of the Lenders, regardless of whether an instruction notice has been delivered by the Agent pursuant to this Section, and the Borrowers shall obtain the agreement by each Bank in favor of the Agent to waive any recoupment, set-off rights, and any security interest in, or against, the funds so deposited (other than for customary items, such as returned items and reasonable bank charges). The Agent assumes

2

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

no responsibility for any Collateral Account deposit arrangements, including, without limitation, any claim of accord and satisfaction or release with respect to deposits accepted by any Bank thereunder. If the Agent has delivered a Dominion Notice to any Bank and, at any time after the effective date of such Dominion Notice, Monthly Average Availability equals or exceeds $30,000,000 for any three consecutive Fiscal Months and no Default or Event of Default has occurred during such three consecutive Fiscal Month period, then the Agent agrees to rescind such Dominion Notice. Upon a subsequent occurrence of a Dominion Event, the Agent’s rights under this Section 2.03, including, without limitation, its right to deliver Dominion Notices pursuant to this Section, shall be reinstated.

(e) Section 2.03 of the Credit Agreement is hereby further amended by amending and restating the subsection thereof identified as subsection (c) after giving effect to this Agreement in its entirety to read as follows:

(c) Except to the extent otherwise expressly provided in any Security Document, the Agent shall, not later than 2:00 p.m. on each Banking Day when a Dominion Event has occurred and is continuing, transfer out of the Collateral Accounts all moneys remitted thereto by Account Debtors of the Borrowers or by financial institutions at which Lock Boxes or Controlled Accounts are maintained. Any such transfers by the Agent shall be applied, first, to the payment of accrued interest in respect of outstanding Base Rate Loans to the extent then due and payable in accordance with the terms hereof, second, to the principal amount of outstanding Base Rate Loans, third, to accrued interest in respect of outstanding Eurodollar Loans to the extent then due and payable in accordance with the terms hereof, fourth, to the principal amount of outstanding Eurodollar Loans (provided, that except upon the occurrence and during the continuance of an Event of Default, no payment of a Eurodollar Loan shall be made under this Section on a date other than the last day of a Eurodollar Interest Period or the Revolving Credit Termination Date), fifth, to cash collateralize outstanding Letters of Credit in an amount not to exceed one hundred ten percent (110%) of the aggregate amount available to be drawn under all outstanding Letters of Credit, sixth, to all other outstanding Obligations, and seventh, any amounts remaining after application pursuant to the foregoing provisions shall be paid to or at the direction of the Borrowers.

(f) Section 2.04 of the Credit Agreement is hereby amended by replacing the phrase “Section 2.03(c)” contained in the introductory sentence thereof with the phrase “Section 2.03(d)”.

(g) Each of Sections 2.08 and 2.11 of the Credit Agreement is hereby amended by replacing the phrase “Section 2.03(b)” contained therein with the phrase “Section 2.03(c)”.

3

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

(h) Section 8.01 of the Credit Agreement is hereby amended by deleting the word “Collateral” each time it appears therein.

(i) Section 9.01(l) of the Credit Agreement is hereby amended by inserting the following text immediately before the period contained in such section:

; provided, however, that it shall not constitute an Event of Default under this Section 9.01(l) for the Borrowers to withdraw or attempt to withdraw funds from a Collateral Account that at the time of such withdrawal or attempted withdrawal is not subject to the dominion of the Agent pursuant to Section 2.03(b)

Section 2. Conditions Precedent. The effectiveness of this Agreement is subject to the satisfaction of the following conditions:

(a) Second Amendment. Each of the Borrowers and the Required Lenders shall have delivered a duly executed counterpart of this Agreement to the Agent.

(b) Officer’s Certificate. The Borrowers shall have delivered to the Agent a certificate of a duly authorized officer of each of the Borrowers dated the date of this Agreement, in form and substance satisfactory to the Agent, to the effect that, after giving effect to this Agreement, (a) all of the representations and warranties set forth in the Facility Documents are true and correct in all material respects on and as of the date hereof with the same effect as though made on the date hereof, except to the extent that such representations and warranties expressly relate to an earlier date, (b) none of the Borrowers is in violation of any of the covenants contained in this Agreement and the other Facility Documents, and (c) no Default or Event of Default has occurred and is continuing.

(c) Delivery of Instruction to Banks. The Borrowers shall have (i) issued to each Bank (as defined in Section 1(b) of this Agreement) an irrevocable letter of instruction directing such Bank, upon receipt of written instructions from Agent, to comply with the instructions of Agent in respect of the Collateral Accounts maintained by such Bank, including, without limitation, any instruction to transfer all payments or other remittances received in each such Collateral Account to Agent’s account for application on account of the Obligations as provided in Section 2.03(c), and (ii) caused each such Bank to agree to such letter of instruction in writing.

(d) Amendments to Controlled Account Agreements. The Borrowers shall cause each of the Controlled Account Agreements governing any Collateral Accounts to be amended to provide for springing dominion by the Agent over all assets deposited therein in accordance with the provisions of the Credit Agreement as amended hereby, each such amendment to be in form and substance reasonably acceptable to the Agent.

(e) Amendment Fee. The Borrowers shall have paid to the Agent for the pro rata account of the Lenders party hereto a nonrefundable amendment fee of $50,000.

(f) Agent Fees and Expenses. The Borrowers shall have paid all fees and expenses (including, without limitation, legal fees and expenses) incurred by the Agent in connection with this Agreement.

4

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

Section 3. Miscellaneous.

(a) No Waiver. Except as expressly set forth herein, this Agreement shall not by implication or otherwise limit, impair, constitute a waiver of, or otherwise affect the rights and remedies of the Agent or any Lender under the Facility Documents, and shall not alter, modify, amend or in any way affect any of the terms, conditions, obligations, covenants or agreements contained in the Facility Documents, all of which are ratified and affirmed in all respects and shall continue in full force and effect. Nothing herein shall be deemed to entitle the Borrowers to a consent to, or a waiver, amendment, modification or other change of, any of the terms, conditions, obligations, covenants or agreements contained in the Facility Documents in similar or different circumstances.

(b) No Representations by Lenders or Agent. Each Borrower hereby acknowledges that it has not relied on any representation, written or oral, express or implied, by any Lender or the Agent, other than those expressly contained herein, in entering into this Agreement.

(c) Representations of the Borrowers. Each Borrower represents and warrants to the Agent and the Lenders that, after giving effect to this Agreement, (a) the representations and warranties set forth in the Facility Documents are true and correct in all material respects on and as of the date hereof with the same effect as though made on the date hereof, except to the extent that such representations and warranties expressly relate to an earlier date, (b) none of the Borrowers is in violation of any of the covenants contained in this Agreement and the other Facility Documents, and (c) no Default or Event of Default has occurred and is continuing.

(d) General Release. In consideration of, among other things, the agreements provided for herein, each of the Borrowers, on behalf of itself and its Subsidiaries and its and their successors and assigns (collectively, the “Releasing Persons”), hereby forever waives, releases and discharges to the fullest extent permitted by law any and all claims (including, without limitation, cross-claims, counterclaims, defenses, rights of set-off and recoupment), causes of action, demands, suits, costs, expenses and damages (collectively, the “Claims”), that any Releasing Person now has, of whatever nature and kind, whether known or unknown, whether arising at law or in equity, against any or all of (i) the Agent, the Syndication Agent, the Managing Agent, the Co-Agent, the Documentation Agent, and each Lender, (ii) each of the Affiliates of the Persons identified in the foregoing clause (i), (iii) the respective successors and assigns of each of the Persons identified in the foregoing clauses (i) and (ii), and (iv) each and all of the officers, directors, employees, agents, attorneys and other representatives of each of the Persons identified in the foregoing clauses (i), (ii), and (iii) (collectively, the “Released Persons”), based in whole or in part on facts, whether or not now known, existing on or before the execution of this Agreement relating to the Credit Agreement or any of the other Facility Documents or any act or omission of any Released Person in connection with the credit extended thereunder; provided that the releases contained herein shall not affect, excuse or absolve the Agent, the Syndication Agent, the Managing Agent, the Co-Agent, the Documentation Agent, or any Lender from performing its obligations under the Credit Agreement or any of the other Facility Documents. The execution and delivery by the Borrowers of this Agreement, and the acceptance by them of any Loans or other financial accommodations made by the Agent or any Lender after the date hereof shall constitute a ratification, adoption and confirmation by the Releasing Persons of the foregoing general release of all Claims against any Released Person that are based in whole or in part on facts, whether or not now known or unknown, existing on or prior to the date of this Agreement. In entering into this Agreement, each of the Borrowers has consulted with and been represented by counsel and expressly disclaims any reliance on any representations, acts or omissions by any of the Released Persons and hereby agrees and

5

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

acknowledges that the validity and effectiveness of the release set forth above do not depend in any way on any such representations, acts and/or omissions or the accuracy, completeness or validity thereof. The provisions of this Section shall be operative as of the Second Amendment Effective Date and shall survive the termination of this Agreement, the Credit Agreement and the other Facility Documents and payment in full of all amounts owing thereunder.

(e) Successors and Assigns. This Agreement shall be binding upon the parties hereto and their respective successors and assigns and shall inure to the benefit of the parties hereto and the successors and assigns of the Lenders and the Agent.

(f) Headings. The headings and captions hereunder are for convenience only and shall not affect the interpretation or construction of this Agreement.

(g) Severability. The provisions of this Agreement are intended to be severable. If for any reason any provision of this Agreement shall be held invalid or unenforceable in whole or in part in any jurisdiction, such provision shall, as to such jurisdiction, be ineffective to the extent of such invalidity or unenforceability without in any manner affecting the validity or enforceability thereof in any other jurisdiction or the remaining provisions hereof in any jurisdiction.

(h) Counterparts. This Agreement may be executed in any number of counterparts, all of which taken together shall constitute one and the same instrument, and any party hereto may execute this Agreement by signing any such counterpart. Delivery of an executed counterpart of a signature page to this Agreement by telecopier shall be effective as delivery of a manually executed counterpart of this Agreement.

(i) Governing Law. The whole of this Agreement and the rights and obligations of the parties hereto shall be governed, construed and interpreted in accordance with, the laws of the State of New York without regard to any conflicts-of-laws rules which would require the application of the laws of any other jurisdiction.

(j) Jurisdiction, Venue and Service. Each of the Borrowers hereby irrevocably consents and submits to the nonexclusive jurisdiction and venue of all federal and state courts located in the County of New York, State of New York and consents that any order, process, notice of motion or other application to or by any of said courts or a judge thereof may be served within or without such court’s jurisdiction by registered mail or by personal service, provided a reasonable time for appearance is allowed, in connection with any action or proceeding arising out of, under or relating to this Agreement. At the option of the Agent, upon the instructions of the Required Lenders, the Borrowers may be joined in any action or proceeding commenced by the Agent or the Lenders against any Borrower or Guarantor in connection with or based on this Agreement, and recovery may be had against the Borrowers in such action or proceeding or in any independent action or proceeding against the Borrowers, without any requirement that the Agent or the Lenders first assert, prosecute or exhaust any remedy or claim against any Borrower or Guarantor. Each of the Borrowers hereby irrevocably waives (to the fullest extent permitted by applicable law) any objection that it may now or hereafter have to the laying of venue of any suit, action or proceeding arising out of, under or relating to this Agreement brought in any federal or state court located in the County of New York, State of New York, and hereby further irrevocably waives (to the fullest extent permitted by applicable law) any claim that any such suit, action or proceeding brought in any such court has been brought in an inconvenient forum.

6

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

(k) Waiver of Jury Trial. EACH OF THE AGENT, THE LENDERS, AND THE BORROWERS HEREBY KNOWINGLY, VOLUNTARILY AND INTENTIONALLY WAIVES (TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW) ANY RIGHT IT MAY HAVE TO A TRIAL BY JURY OF ANY DISPUTE ARISING UNDER OR RELATING TO THIS AGREEMENT, AND AGREES THAT ANY SUCH DISPUTE SHALL BE TRIED BEFORE A JUDGE SITTING WITHOUT A JURY. IN ADDITION, EACH OF THE AGENT, THE LENDERS, AND THE BORROWERS WAIVES THE RIGHT TO INTERPOSE ANY DEFENSE BASED UPON ANY STATUTE OF LIMITATIONS OR ANY CLAIM OF LACHES AND ANY SET-OFF OR COUNTER CLAIM OF ANY NATURE OR DESCRIPTION. EACH OF THE AGENT, THE LENDERS, AND THE BORROWERS ACKNOWLEDGES THAT THE FOREGOING WAIVERS ARE FREELY MADE.

(Signature Pages Follow)

7

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

IN WITNESS WHEREOF, the undersigned have caused this Agreement to be duly executed and delivered as of the date first above written.

8

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

9

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

10

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

11

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

12

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

13

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

14

SECOND AMENDMENT

TO HUTTIG CREDIT AGREEMENT

15