AMENDED AND RESTATED SECURITY AGREEMENT

Exhibit 10.24(b)

AMENDED AND RESTATED SECURITY AGREEMENT

THIS AMENDED AND RESTATED SECURITY AGREEMENT (this “Agreement”), dated as of June 13, 2011 is from Duluth Holdings Inc. and Duluth Trading Company, LLC, a Wisconsin limited liability company (each a “Company” and jointly, the “Companies”), in favor of Harris N.A. (“Secured Party”).

1. Definitions. All terms defined in Articles 1 through 9 of the applicable Uniform Commercial Code, as it may be amended from time, shall have the meanings specified therein unless otherwise defined herein or unless the context otherwise requires. As used herein, the following terms have the following meanings:

“Accounts” has the meaning provided in the applicable Uniform Commercial Code.

“Collateral” means all of any Company’s right, title and interest in and to the following, whether now owned and existing or hereafter created or acquired, wherever located, together with all additions and accessions and all proceeds and products thereof: Accounts; Instruments; supporting obligations, including all guaranties and letter of credit rights; all of the any Company’s life insurance policies and their cash surrender values; Investment Property; deposit accounts; chattel paper; General Intangibles; computer and other data processing hardware, software programs, whether owned, licensed or leased, and, all documentation for such hardware and software; leases, rents, issues and profits; Equipment; Inventory; any insurance coverage relating to the foregoing, including casualty insurance coverage and credit insurance coverage, and all books and records of the Companies pertaining to any of the foregoing, including books and records stored or maintained on any type of computer and/or data processing system or equipment (including but not limited to all related discs, tapes, printouts and media). Collateral shall include all licenses and permits of the Companies.

“Environmental Laws” shall mean all federal, state and local laws including statutes, regulations, ordinances, codes, rules and other governmental restrictions and requirements relating to the discharge of air pollutants, water pollutants or process waste water or otherwise relating to the environment or hazardous substances including, but not limited to, the Federal Solid Waste Disposal. Act, the Federal Clean Air Act, the Federal Clean Water Act, the Federal Resource Conservation and Recovery Act of 1976, the Federal Comprehensive Environmental Response, Compensation, and Liability Act of 1980, regulations of the Environmental Protection Agency, regulations of the Nuclear Regulatory Agency, and regulations of any state department of natural resources or state environmental protection agency now or at any time hereafter in effect.

“Equipment” means all machinery, equipment, motor vehicles, furniture and fixtures owned by any Company and, to the extent legally assignable, all leases and agreements for use of machinery, equipment and fixtures leased by any Company, and all modifications, alterations, repairs, substitutions and replacements thereof or thereto.

1

“Event of Default” means the occurrence of any of the following: (a) an Event of Default under the Loan Agreement or any other agreement between any Company and Secured Party, (b) any representation made by any Company in this Agreement is false in any material respect on the date as of which made or as of which the same is to be effective or (c) any Company fails to timely comply with any of its obligations under this Agreement.

“General Intangibles” means any personal property owned by any Company (other than Accounts, Instruments, chattel paper, Equipment or Inventory) including, but not limited to, general intangibles, causes of action, contract rights, rights to insurance claims and proceeds, tax refunds, claims for tax refunds, rights of indemnification, contribution and subrogation, payment intangibles, goodwill, patents, know-how, trademarks, copyrights, trade names, patent, trademark, trade name and copyright registrations and applications, trade secrets, customer lists, licenses and franchises, and license agreements related to any of the foregoing (and income derived therefrom).

“Instrument” means a negotiable instrument owned by any Company, a certificated security owned by any Company or any other writing owned by any Company which evidences a right to the payment of money, other than chattel paper.

“Inventory” means all of any Company’s inventory, including all goods held for sale, lease or demonstration or to be furnished under contracts of service, all goods leased to others, trade-ins and repossessions, raw materials, work in process and materials or supplies used or consumed in any Company’s business.

“Investment Property” means all of any Company’s investment property, including but not limited to all stock and other interest of any Company in its subsidiaries, if any.

“Lien” means any mortgage, pledge, security interest, encumbrance, lien or charge of any kind.

“Loan Agreement” means the Amended and Restated Loan Agreement among the Companies and Secured Party dated the date hereof, as it may be amended, restated or otherwise modified from time to time.

“Notes” means the Revolving Note and the Term Note.

“Revolving Note” means the Amended and Restated Revolving Note of even date herewith executed by the Companies in favor of Secured Party in the principal amount of $13,000,000.00, as it may be amended, restated or replaced from time to time.

“Term Note” means the Amended and Restated Term Note of even date herewith executed by the Companies in favor of Secured Party in the principal amount of $684,960.05, as it may be amended, restated or replaced from time to time.

“Obligations” means all of any Company’s debts, obligations and liabilities to Secured Party or any affiliate of Secured Party, including but not limited to all debts, obligations and liabilities arising pursuant to the Notes and the Loan Agreement, of whatever nature (contingent or otherwise), and all amendments and extensions or renewals thereof, whether

2

arising out of existing, contemporaneous or future credit granted by Secured Party to any Company, to any Company and another, or to another guaranteed by any Company. “Obligations” includes all Letter of Credit Liabilities (as defined in the Loan Agreement), Hedging Agreement Liabilities (as defined in the Loan Agreement), and all other Obligations (as defined in the Loan Agreement).

2. Grant of Security Interest.

2.1 Security Interest. Each Company grants Secured Party a security interest in the Collateral to secure the payment of the Obligations.

2.2 Right of Set Off. Each Company also grants Secured Party a security interest and lien in any credit balance or other money now or hereafter owed such Company by Secured Party, and, in addition, agrees that Secured Party may at any time after an occurrence of an Event of Default, without notice or demand, set off against such credit balance or other money any amount unpaid under the Obligations.

3. Representations and Warranties of the Companies. Each Company, jointly and severally, represents and warrants to Secured Party that:

3.1 Ownership. Except as permitted under the Loan Agreement, a Company owns the Collateral free of all Liens and no financing statement (other than those in favor of Secured Party) is on file covering any of the Collateral.

3.2 Sale of Goods or Services Rendered. Each Account and chattel paper constituting Collateral as of this date arose from the performance of services by a Company or from a bonafide sale or lease of goods which have been delivered or shipped to the account debtor and for which a Company has genuine invoices, shipping documents or receipts.

3.3 Location of Collateral. Duluth Holdings Inc.’s (“Duluth Holdings”) place of business or, if more than one, its chief executive office, and the place where Duluth Holdings keeps its records concerning Accounts, is 170 Countryside Drive, PO Box 409, Belleville, Wisconsin 53508. Duluth Trading Company, LLC’s (“Duluth Trading”) place of business or, if more than one, its chief executive office, and the place where Duluth Trading keeps its records concerning Accounts, is 170 Countryside Drive, PO Box 409, Belleville, Wisconsin 53508. No Company will not change, the location of its chief executive office, the place where it keeps records concerning Accounts or the place where Equipment or Inventory is kept unless such change is permitted by the Loan Agreement and 30 days’ advance written notice of such change, describing the new location, has been given to Secured Party.

3.4 Fixtures. Exhibit A contains the description of all real estate to which any Collateral is affixed.

3.5 Intellectual Property. Exhibit B contains a correct and complete list and description of all federally registered patents, trademarks and copyrights owned by any Company.

3

3.6 Environmental Compliance. To the knowledge of the Companies, there are no conditions existing currently or likely to exist during the term of the Loan Agreement which would subject any Company to damages, penalties, injunctive relief or cleanup costs under any Environmental Laws or which require or are likely to require cleanup, removal, remedial action or other response pursuant to Environmental Laws by any Company.

3.7 Effectiveness of Representations and Warranties. The representations and warranties contained in this Section 3 shall be true and correct on and as of the date hereof and until the Obligations have been paid in full, with such changes as are approved by Secured Party or are permitted by the Loan Documents (as defined in the Loan Agreement).

4. Covenants of the Company. Each Company agrees that while any credit is available to any Company under the Notes and while any of the Obligations remain unpaid:

4.1 Maintenance of Collateral. Each Company shall: (a) maintain the Collateral in good condition and repair and not permit its value to be impaired (ordinary wear and tear excepted); (b) keep the Collateral free from all Liens except Liens in favor of Secured Party and Liens permitted by the Loan Agreement; (c) defend the Collateral against all claims and legal proceedings by persons other than Secured Party; (d) pay and discharge when due all taxes, levies and other charges or fees upon the Collateral; (e) not sell, lease or otherwise dispose of the Collateral or permit the Collateral to become a fixture or an accession to other goods, except as permitted by this Agreement; (f) not permit the Collateral to be used in violation of any applicable law or regulation or policy of insurance; (g) operate its business in compliance with all applicable provisions of the federal Fair Labor Standards Act; and, (h) as to Collateral consisting of Instruments and chattel paper, preserve the rights of any Company therein against prior parties.

4.2 Insurance. Each Company will keep all the Collateral insured against loss by fire, extended coverage perils and such other hazards as Secured Party requires in amounts not less than the replacement cost thereof. All insurance policies shall be issued by an insurance company or companies acceptable to Secured Party.

Each Company shall cause the issuer of each insurance policy to issue a certificate of insurance naming Secured Party as an additional insured, lender’s loss payee and mortgagee and containing an agreement by the insurer that the policy shall not be terminated or modified without at least 30 days’ prior written notice to Secured Party, and each Company shall deliver each such certificate to Secured Party. In the event of any loss or casualty exceeding $250,000 for any loss or $1,000,000 in the aggregate for all losses in any fiscal year of any Company (the “floor amount”) which is covered by insurance, the Company experiencing such loss shall give immediate notice thereof to Secured Party and each Company grants to Secured Party the right to make proof of such loss or damage. Secured Party is hereby authorized and empowered by and on behalf of each Company to settle, adjust or compromise any claims for loss, damage or destruction under any such insurance policy.

The proceeds of any loss in excess of the floor amount shall be paid to Secured Party and, if initially received by any Company, shall be immediately turned over to Secured Party. The proceeds of any loss which is less than the floor amount shall be paid to the

4

Company entitled to such proceeds; provided, however, that if an Event of Default has occurred and is continuing, all proceeds of each such insurance policy shall be deposited with Secured Party. Each Company authorizes Secured Party to endorse in the name of such Company on any instrument evidencing such proceeds.

All proceeds of any such insurance received by Secured Party shall be held by Secured Party and shall be applied by it either to the Obligations or to the repair or replacement of the lost, stolen, damaged or destroyed property with respect to which such proceeds were received.

4.3 Maintenance of Security Interest. Each Company authorizes Secured Party to file financing statements describing the Collateral. Each Company shall, at such Company’s expense, cooperate in Secured Party’s efforts to comply with or address any amendments to Article 9 of the Uniform Commercial Code that may be in effect from time to time. Each Company shall, at such Company’s expense, take any action requested by Secured Party to preserve and protect the rights of Secured Party in the Collateral or to establish, determine priority of, perfect, continue perfected, terminate and/or enforce Secured Party’s interest in the Collateral. Each Company shall execute and deliver to Secured Party any and all documents which Secured Party reasonably requests to protect its security interest in the Collateral.

4.4 Books and Records; Inspection.

(a) Each Company will keep proper books of record and account in which full, true and proper entries will be made with respect to the Collateral. Without limiting the generality of the foregoing, each Company agrees that it will at all times keep accurate and complete records with respect to the Accounts and the Inventory including, but not limited to, a record of all payments received on account thereof and of all credits granted.

(b) Each Company agrees that Secured Party and its representatives shall have the right during normal business hours from time to time to call at any Company’s place of business where its records concerning the Accounts are kept and any other place where any of the Collateral is located, examine the Collateral and all records concerning the Collateral and make extracts therefrom or copies thereof.

4.5 Chattel Paper; Instruments. Chattel paper, Instruments and other documents which constitute Collateral shall be on forms satisfactory to Secured Party. Each Company shall promptly mark all such forms of Collateral to indicate conspicuously Secured Party’s interest therein and, upon request, deliver them to Secured Party.

4.6 Compliance with Environmental Laws. Each Company shall timely comply with all applicable Environmental Laws.

5. Possession and Use of Collateral. Until notice is given by Secured Party to the Companies after the occurrence and during the continuance of an Event of Default, the Companies (a) shall have the right to remain in possession and to use and to retain exclusive control of the Collateral with power to manage, operate, develop, use and enjoy the Collateral; (b) may sell, lease, use or dispose of Inventory in the ordinary course of business; (c) shall

5

diligently collect (at its own expense) the Accounts in accordance with sound business practices; (d) may sell or otherwise dispose of, free from the Lien of this Security Agreement, any Equipment which may have become obsolete, inadequate or worn-out if such Equipment is no longer necessary in the conduct of the business of the Companies (provided, however, that no Company may dispose of Equipment having an aggregate net book value greater than $250,000 during any 12-month period unless the prior written consent of Secured Party is obtained; (e) as to Equipment which is necessary in the conduct of any Company’s business, may replace any item of such Equipment which has become obsolete,. inadequate or worn-out with another item of Equipment which has a value and useful life at least equal to that of such replaced Equipment immediately prior to the time such Equipment became obsolete, inadequate or worn-out, and is suitable for a use which is the same or similar to such item of replaced Equipment; (f) may alter, add to, repair or replace any and all Equipment; and (g) may dispose of any Collateral as otherwise expressly permitted under the Loan Documents.

6. Remedies of Secured Party.

6.1 Remedies Upon Default. Upon the occurrence and during the continuance of an Event of Default, Secured Party may exercise any or all of the following rights and remedies:

(a) Collection of Accounts. Secured Party may at any time notify the account debtor under any Account of Secured Party’s security interest therein and direct such account debtor to make payments directly to Secured Party. Secured Party may enforce collection of, settle, compromise or renew any such Account. Any proceeds of Accounts received or collected by any Company shall not be commingled with any other funds or property of any Company and shall be turned over to Secured Party in precisely the form received (but endorsed by such Company for collection, if necessary) not later than the Business Day following the day of receipt. Each Company hereby irrevocably appoints Secured Party as such Company’s attorney with power to ask for, demand, sue for, collect, receive and receipt for any monies due or to become due under any Account and to endorse checks, drafts, orders and other instruments for the payment of money to such Company with respect to an Account; provided that Secured Party shall not be obligated to make any demand for payment; to inquire concerning the nature or sufficiency of any payment received by Secured Party or to take any other action regarding any Account and no action taken or not taken by Secured Party with respect to an Account shall give rise to any defense, counterclaim or offset in favor of such Company or to any claim or proceeding against Secured Party.

(b) Wisconsin Uniform Commercial Code. Secured Party shall have all of the rights and remedies for default provided by the Wisconsin Uniform Commercial Code as well as any other applicable law. With respect to such rights and remedies:

(i) Secured Party may take possession of the Collateral without a hearing, the right to or necessity of which each Company waives;

(ii) Secured Party may require any Company to assemble the Collateral and make it available to Secured Party at any convenient place designated by Secured Party and Secured Party shall have the right to take immediate possession of the Collateral, and may enter any of the premises of any Company or wherever the Collateral shall be located and keep the Collateral at such locations without charge until sold; and

(iii) Written notice, when required by law, sent to any Company at least ten calendar days (counting the day of sending) before a proposed disposition of the Collateral is reasonable notice.

(c) Protective Advances. Secured Party is authorized, at its option, in any Company’s name or otherwise, to take such action as may be necessary or desirable to remedy any failure by such Company to comply with its obligations hereunder including, without limitation, signing such Company’s name or paying any amount so required, and any amount so paid shall be payable by such Company to Secured Party upon demand with interest from the date of payment by Secured Party at the applicable default interest rate stated in the Loan Agreement.

6

6.2 No Marshaling. The Collateral may be sold in such parcels and in such order as Secured Party shall determine. Each Company, for itself and all other persons claiming by, through or under it, hereby waives and releases, to the extent permitted by applicable law, any right to have the Collateral or any part thereof, marshaled upon any sale, foreclosure or other disposition thereof.

6.3 Instruments of Sale. Secured Party may execute and deliver to each purchaser of Collateral bills of sale, deeds or other instruments conveying or transferring the property sold. Each Company irrevocably appoints Secured Party as its attorney to execute and deliver all such instruments and ratifies and confirms all actions taken by Secured Party pursuant to such appointment. If so requested by Secured Party, each Company shall execute and deliver to Secured Party or to any such purchaser of Collateral such instruments of conveyance as deemed necessary or convenient by Secured Party.

6.4 Application of Proceeds. All amounts received by Secured Party in exercising its rights hereunder shall, unless otherwise required by law, be applied by Secured Party to expenses incurred by Secured Party in protecting or enforcing its rights under this Agreement (including without limitation reasonable attorneys’ fees and all expenses of taking possession, storing, holding, repairing, restoring, preparing for disposition and disposing of the Collateral) and to the Obligations as set forth in Section 4.1 of the Loan Agreement.

6.5 Remedies Cumulative. No remedy granted herein to Secured Party is exclusive of any other remedy granted hereunder or by applicable law.

6.6 Waiver. Secured Party may permit a Company to cure any default hereunder without waiving the default so cured and Secured Party may waive any default without waiving any subsequent or prior default by such Company.

6.7 Protection or Preservation of Collateral. Secured Party has no duty to protect, insure, collect or realize upon the Collateral or preserve rights in it against prior parties. Secured Party shall not be responsible or liable for any shortage, discrepancy, damage, loss or destruction of any part of the Collateral regardless of the cause thereof. Secured Party has no obligation to clean the Collateral or otherwise prepare the Collateral for sale.

7

6.8 Compliance with Other Laws. Secured Party may comply with any applicable state or federal law requirements in connection with a disposition of the Collateral and compliance will not be considered adversely to affect the commercial reasonableness of any sale of the Collateral.

6.9 Warranties. Secured Party may sell the Collateral without giving any warranties as to the Collateral. Secured Party may specifically disclaim any warranties of title or the like. This procedure will not be considered to adversely affect the commercial reasonableness of any sale of the Collateral.

7. Miscellaneous.

7.1 Notices. Any notices, communications and waivers under this Agreement shall be in writing and shall be (i) delivered in person, (ii) mailed, postage prepaid, either by registered or certified mail, return receipt requested, or (iii) by overnight express carrier, addressed in each case as follows:

or to any other address as to any of the parties hereto; as such party shall designate in a written notice to the other party hereto. All notices sent pursuant to the terms of this Section 7.1 shall be deemed received (i) if personally delivered, then on the date of delivery, (ii) if sent by overnight, express carrier, then on the next Business Day (as defined in the Loan Agreement) immediately following the day sent, or (iii) if sent by registered or certified mail, then on the earlier of the third Business Day following the day sent or when actually received.

7.2 Governing Law. This Agreement is being delivered in and shall be deemed to be a contract governed by the laws of the State of Wisconsin and shall be interpreted and enforced in accordance with the laws of that state without regard to the principles of conflicts of laws.

7.3 Submission to Jurisdiction. As a material inducement to Secured Party making the loans evidenced by the Notes:

(a) EACH COMPANY AGREES THAT ALL ACTIONS OR PROCEEDINGS IN ANY MANNER RELATING TO OR ARISING OUT OF THIS AGREEMENT OR THE NOTE MAY BE BROUGHT ONLY IN COURTS OF THE STATE

8

OF WISCONSIN LOCATED IN MILWAUKEE COUNTY OR THE FEDERAL COURT FOR THE EASTERN DISTRICT OF WISCONSIN AND EACH COMPANY CONSENTS TO THE JURISDICTION OF SUCH COURTS.

(b) EACH COMPANY WAIVES ANY OBJECTION IT MAY NOW OR HEREAFTER HAVE TO THE VENUE OF ANY SUCH COURT AND ANY RIGHT IT MAY HAVE NOW OR HEREAFTER HAVE TO CLAIM THAT ANY SUCH ACTION OR PROCEEDING IS IN AN INCONVENIENT COURT.

7.4 Waiver of Jury Trial. EACH COMPANY HEREBY KNOWINGLY AND VOLUNTARILY WAIVES THE RIGHT IT MAY HAVE TO A JURY TRIAL WITH RESPECT TO ANY ACTION OR CLAIM BASED ON OR ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE NOTE, ANY COURSE OF CONDUCT, COURSE OF DEALING, STATEMENTS (WHETHER VERBAL OR WRITTEN) OR ANY OTHER ACTION OF EITHER PARTY. THIS PROVISION IS A MATERIAL INDUCEMENT TO SECURED PARTY TO MAKE THE LOANS EVIDENCED BY THE NOTES.

7.5 Limitation of Liability. EACH COMPANY HEREBY WAIVES ANY RIGHT IT MAY HAVE TO CLAIM OR RECOVER FROM SECURED PARTY ANY SPECIAL, EXEMPLARY, PUNITIVE, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES, OF WHATEVER NATURE, OTHER THAN ACTUAL, COMPENSATORY DAMAGES.

7.6 Severability. The invalidity of any provision of this Agreement shall not affect the validity of any other provision.

7.7 Persons Bound. This Agreement is for the benefit of Secured Party and its successors and assigns and binds each Company and such Company’s successors and assigns.

7.8 Entire Agreement Amendment and Restatement. This Agreement, the Loan Agreement, the Notes and the other Loan Documents (as defined in the Loan Agreement) embody the entire agreement and understanding between the Companies and the Secured Party with respect to the subject matter hereof and thereof. This Agreement is an amendment and restatement of the following documents and is not a novation: (i) the Commercial Security Agreement executed by Duluth Holdings Inc. in favor of Secured Party dated May 4, 2010, and (ii) the Commercial Security Agreement executed by Duluth Trading Company, LLC in favor of Secured Party dated May 4, 2010.

9

IN WITNESS WHEREOF, the parties hereto have executed this Agreement on the day and year first above written.

10

EXHIBIT A

LEGAL DESCRIPTIONS

Property Address: 170 Countryside Drive, Belleville, Wisconsin 53508

Tax Key Number: 0508 ###-###-####-4

Legal Description: Lot 5, Plat of Belleville Industrial Park Number 2, Village of Belleville, Dane County, Wisconsin.

Property Address: 100 West Main Street, Mount Horeb, Wisconsin 53572

Tax Key Number: 157/0606-123-2025-5

Legal Description: Lots Five (5), Six (6), Seven (7) and Eight (8), Block One (1), Carl Boeck’s Survey and Plat of Mount Horeb Station, in the Village of Mount Horeb, Dane County, Wisconsin.

11

EXHIBIT B

INTELLECTUAL PROPERTY

[See Attached]

12

Page 1 of 2

Page 2 of 2