SECOND AMENDMENT

EXHIBIT 10.78

SECOND AMENDMENT

TO

LOAN AND SUBORDINATED DEBENTURE PURCHASE AGREEMENT

BETWEEN

LASALLE BANK NATIONAL ASSOCIATION

AND

TAYLOR CAPITAL GROUP, INC.

Second Amendment dated as of June 8, 2004

First Amendment dated as of November 27, 2003

Original Loan and Subordinated Debenture Purchase Agreement dated as of November 27, 2002

SECOND AMENDMENT TO

LOAN AND SUBORDINATED DEBENTURE PURCHASE AGREEMENT

This SECOND AMENDMENT TO LOAN AND SUBORDINATED DEBENTURE PURCHASE AGREEMENT (“Second Amendment”), dated as of June 8, 2004, is entered into by and between TAYLOR CAPITAL GROUP, INC., a Delaware corporation (“Borrower”), and LASALLE BANK NATIONAL ASSOCIATON, a national banking association (“Lender”).

R E C I T A L S :

A. The parties hereto have entered into that certain Loan and Subordinated Debenture Purchase Agreement, dated as of November 27, 2002, as previously amended, restated, supplemented or modified from time to time, including by that certain First Amendment to Loan and Subordinated Debenture Purchase Agreement, dated as of November 27, 2003 (the “2002 Loan Agreement”).

B. The parties hereto desire to amend and modify the 2002 Loan Agreement in accordance with the terms and subject to the conditions set forth in this Second Amendment. As amended and modified by this Second Amendment, the 2002 Loan Agreement may be referred to as the “Agreement.”

C. The parties desire to amend the terms of the 2002 Loan Agreement to: (i) modify the debt-to-equity covenant contained in subsection 7.6 of the 2002 Loan Agreement; (ii) permit and reflect the possible issuance by Borrower of junior subordinated debentures in connection with the issuance by a special purpose subsidiary of Borrower of trust preferred securities; and (iii) otherwise modify the 2002 Loan Agreement as provided in this Second Amendment. The parties agree to undertake such modifications in accordance with the terms, subject to the conditions, and in reliance upon the recitals, representations, warranties, and covenants set forth herein, in the Agreement, and in the other Loan Documents, irrespective of whether entered into or delivered on or after November 27, 2002.

D. Capitalized terms used but not otherwise defined in this Second Amendment shall have the meanings respectively ascribed to them in the 2002 Loan Agreement.

NOW, THEREFORE, in consideration of the mutual representations, warranties, covenants, and agreements hereinafter set forth, and for other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the parties hereto hereby agree as follows:

A G R E E M E N T :

A. Amendments to Subsection 1.1 of the 2002 Loan Agreement.

(i) The following definition is hereby added to subsection 1.1 of the 2002 Loan Agreement in the appropriate alphabetical order:

““2004 Trust Preferred Transaction” means the issuance by Borrower, in a transaction to be consummated prior to July 1, 2004, of $41,238,000 of junior subordinated debentures in connection with the issuance by TAYC Capital Trust II, a Subsidiary of Borrower, of $40,000,000 of undivided preferred interests in such trust and $1,238,000 of undivided common interests in such trust, which junior subordinated debentures are subordinate and junior in all respects to all of Borrower’s obligations to Lender as provided by the express terms of the indenture related to such junior subordinated debentures.”

1

(ii) The term “Indenture” is hereby deleted from subsection 1.1 of the 2002 Loan Agreement and replaced in its entirety with the following:

““Indenture” means (a) that certain indenture dated as of October 21, 2002, between Borrower and LaSalle Bank National Association, as trustee, and (b) after the consummation of the 2004 Trust Preferred Transaction, the indenture executed and delivered in connection with the 2004 Trust Preferred Transaction.”

(iii) The term “Junior Subordinated Debentures” is hereby deleted from subsection 1.1 of the 2002 Loan Agreement and replaced in its entirety with the following:

““Junior Subordinated Debentures” means (a) the 9¾% junior subordinated debentures issued by Borrower pursuant to the Indenture referenced in clause (a) of the definition of the term “Indenture”, and (b) after the consummation of the 2004 Trust Preferred Transaction, the junior subordinated debentures issued by Borrower in connection with the 2004 Trust Preferred Transaction.”

(iv) The term “Permitted Distributions” is hereby deleted from subsection 1.1 of the 2002 Loan Agreement and replaced in its entirety with the following:

““Permitted Distributions” means either or both of (a) during any calendar year, either or both of (i) cash dividends of Borrower (x) with respect to its preferred stock designated as 9% Non-cumulative Perpetual Preferred Stock, Series A in an amount not to exceed $3,400,000 plus (y) additional cash dividends that do not exceed, in the aggregate, 25% of the net after tax income (determined in accordance with GAAP) of Borrower, if any, for such calendar year, and (ii) cash dividends by a Subsidiary (other than a Trust) that do not exceed, in the aggregate, 60% of the net after tax income (determined in accordance with GAAP) of such Subsidiary, if any, for such calendar year, and (b) cash distributions by either Trust (with respect to its outstanding securities (i) as of the date of this Agreement in the case of the Trust referenced in clause (a) of the definition of the term “Trust” and (ii) as of the date of consummation of the 2004 Trust Preferred Transaction in the case of the Trust referenced in clause (b) of the definition of the term “Trust”).”

(v) The term “Trust” is hereby deleted from subsection 1.1 of the 2002 Loan Agreement and replaced in its entirety with the following:

““Trust” means (a) that certain Delaware statutory trust known as “TAYC Capital Trust I,” which is maintained by Borrower in accordance with that certain Amended and Restated Trust Agreement dated as of October 21, 2002, and (b) after the consummation of the 2004 Trust Preferred Transaction, the Delaware statutory trust to be known as “TAYC Capital Trust II”, maintained by Borrower in connection with the 2004 Trust Preferred Transaction.”

B. Amendment to Subsection 4.4.5 of the 2002 Loan Agreement. Subsection 4.4.5 of the 2002 Loan Agreement is hereby deleted and replaced in its entirety with the following:

“4.4.5 Subordination. All Junior Subordinated Debentures are expressly subordinate and junior in all respects (including, without limitation, with respect to the right of payment) to the Loans to the extent provided in each Indenture. The Loans constitute “Senior Indebtedness” as defined in each Indenture.”

C. Amendment to Subsection 5.2.5 of the 2002 Loan Agreement. Subsection 5.2.5 of the 2002 Loan Agreement is hereby deleted and replaced in its entirety with the following:

“5.2.5 Making Loans. Borrower shall not itself, nor shall it cause, permit or allow any Subsidiary to make any loans or advances, whether secured or unsecured, to any Person, other than (a) loans or advances to employees (made in accordance with applicable laws, at arms-length and pursuant to terms that are no more favorable than may be obtained from unrelated

2

third-party institutional lenders) up to the aggregate sum of $500,000 at any time outstanding, (b) loans or advances made by Subsidiary Bank in the ordinary course of business and in accordance with applicable laws and regulations and safe and sound banking practices, and (c) loans made by either Trust to Borrower that are evidenced by Junior Subordinated Debentures.”

D. Amendment to Subsection 7.6 of the 2002 Loan Agreement. Subsection 7.6 of the 2002 Loan Agreement is hereby deleted and replaced in its entirety with the following:

“7.6 Debt to Equity. Borrower (on a non-consolidated, parent-company-only basis) shall maintain its ratio of (a) Indebtedness (on a non-consolidated, parent-company-only basis) minus the amount of any Junior Subordinated Debentures to (b) stockholder’s equity (on a non-consolidated, parent-company-only basis, as determined in accordance with GAAP) at no more than 35% at all times, measured quarterly and derived from the financial statements filed by Borrower with its primary federal regulator, which are consistent with the financial information and reports contemplated in Section 6 hereof.”

E. Amendment to Subsection 8.1.1 of the 2002 Loan Agreement. Subsection 8.1.1.22 of the 2002 Loan Agreement is hereby deleted and replaced in its entirety with the following:

“8.1.1.22. Any of the Junior Subordinated Debentures are no longer junior and subordinate in all respects to the Loans; or

8.1.1.23. The failure of Borrower to deliver to Lender within 45 days of the consummation of the 2004 Trust Preferred Transaction the written approval or non-objection of the FRB relating to the 2004 Trust Preferred Transaction.”

F. Consent and Waiver with Respect to Certain Matters. In addition to Borrower’s desire to consummate the 2004 Trust Preferred Transaction, Borrower has notified Lender of Borrower’s desire to redeem 1,530,000 shares of Series A 9% noncumulative perpetual preferred stock, $25.00 stated value per share, being all of the outstanding shares of such series of preferred stock, following the consummation of the 2004 Trust Preferred Transaction (the “Redemption”). Effective immediately upon receipt by Lender of the documents referenced in clauses (i) through (iii) below, Lender consents to the 2004 Trust Preferred Transaction and the Redemption and waives the Potential Events of Default under subsections 5.2.3 and 5.3.3 of the 2002 Loan Agreement that would otherwise result from the consummation of the 2004 Trust Preferred Transaction and the Redemption, respectively. In accordance with subsection 9.5 of the 2002 Loan Agreement, this wavier shall be effective only with respect to the matters described in this Section (F), and Lender reserves all rights not expressly waived herein. Borrower shall deliver the following documents to Lender in connection with the consummation of the 2004 Trust Preferred Transaction: (i) a written opinion of counsel for Borrower, substantially in the form of Exhibit A attached hereto; (ii) true, accurate, and complete copies of the final trust agreement, guaranty, indenture and other documentation evidencing or relating to the 2004 Trust Preferred Transaction; and (iii) a true, accurate, and complete copy of the notice to the FRB relating to the 2004 Trust Preferred Transaction.

G. Representations and Warranties. Borrower hereby represents and warrants to the Lender as follows:

(i) No Event of Default or Potential Event of Default has occurred and is continuing (or would result from the amendments contemplated hereby).

(ii) The execution, delivery and performance by the Borrower of this Second Amendment have been duly authorized by all necessary corporate and other action and do not and will not require any registration with, consent or approval of, or notice to or action by any Person (including any Governmental Agency) in order to be effective and enforceable.

3

(iii) This Second Amendment, and the other Loan Documents (as amended by this Second Amendment) constitute the legal, valid and binding obligations of the Borrower, enforceable against the Borrower in accordance with their respective terms.

(iv) All representations and warranties of the Borrower in the 2002 Loan Agreement are true and correct, except, for the purposes of this Second Amendment only, all references in Section 4 of the 2002 Loan Agreement to (x) the term “Borrower 2001 Audited Financial Statements Date” shall be deemed to refer to “December 31, 2003”; (y) the term “Borrower 2001 Audited Financial Statements” shall be deemed to refer to “the consolidated audited financial statements of the Borrower as of the year ending December 31, 2003”; and (z) the term “Interim Financial Statements Date” shall be deemed to refer to call reports and regulatory filings (including Form FRY-9C filings) by the Subsidiary Bank for the period ending “March 30, 2004.”

(v) The Borrower’s obligations under the Agreement and under the other Loan Documents are not subject to any defense, counterclaim, set-off, right to recoupment, abatement or other claim.

H. Conditions. Notwithstanding anything to the contrary contained elsewhere in the Agreement, the obligation of Lender to amend the debt-to-equity covenant and otherwise modify the 2002 Loan Agreement as contemplated by this Second Amendment shall be subject to the performance by the Borrower prior to the date on which this Second Amendment is executed (the “Amendment Closing Date”) of all of its agreements theretofore to be performed under the Agreement and to the satisfaction of the following conditions precedent. The obligations to continue to make disbursements of proceeds under the Loans are, and shall remain, subject to the conditions precedent in the 2002 Loan Agreement and to the receipt by the Lender of all the following in form and substance satisfactory to the Lender and its counsel, and, where appropriate, duly executed and dated the Amendment Closing Date:

(i) a certificate of good standing of the Borrower, certified by the appropriate governmental official in its jurisdiction of incorporation and dated within the five business days preceding the date hereof;

(ii) copies, certified by the Secretary or Assistant Secretary of the Borrower, of the (a) resolutions duly adopted by the board of directors of the Borrower authorizing the execution, delivery and performance of this Second Amendment and the other documents to be delivered by the Borrower pursuant to this Second Amendment (the “Amendment-Related Documents”), and (b) the Bylaws of the Borrower as currently in effect; and

(iii) a written opinion of counsel for the Borrower, addressed to the Lender, substantially in the form of Exhibit B attached hereto.

I. Additional Terms.

(i) Acknowledgment of Indebtedness under Agreement. The Borrower acknowledges and confirms that, as of the date hereof, (a) the Borrower is indebted to the Lender, without defense, setoff, or counterclaim, in the aggregate principal amount of (1) Five Hundred Thousand and No/100 Dollars ($500,000.00) under the Term Loan and (2) Ten Million and No/100 Dollars ($10,000,000.00) under the Subordinated Debenture; and (b) there is no amount outstanding under the Revolving Loan.

(ii) Effectiveness. The provisions of this Second Amendment will become effective upon its execution by the parties hereto.

(iii) The Agreement. All references in the 2002 Loan Agreement to the term “Agreement” shall be deemed to refer to the Agreement referenced in this Second Amendment.

4

(iv) Second Amendment and 2002 Loan Agreement to be Read Together. This Second Amendment supplements and is hereby made a part of the 2002 Loan Agreement, and the 2002 Loan Agreement and this Second Amendment shall from and after the date hereof be read together and shall constitute the Agreement. Except as otherwise set forth herein, the 2002 Loan Agreement shall remain in full force and effect.

(v) Loan Documents. The term “Loan Documents,” as used in the Agreement, shall from and after the date hereof include the Amendment-Related Documents.

(vi) Counterparts. This Second Amendment may be executed by facsimile in one or more counterparts, each of which shall be deemed an original and all of which taken together shall constitute one and the same document.

[Remainder of Page Intentionally Left Blank]

5

IN WITNESS WHEREOF, the Borrower and the Lender have executed this Second Amendment as of the date first written above.

Certain schedules, exhibits and similar attachments to this Agreement have not been filed with this exhibit. The Registrant agrees to furnish supplementally any omitted schedule, exhibit or similar attachment to the Securities and Exchange Commission upon request.