PATENT AND TRADEMARK SECURITY AGREEMENT

Exhibit 10.4

PATENT AND TRADEMARK SECURITY AGREEMENT

This Patent and Trademark Security Agreement (the “Agreement”), dated as of June 10, 2008, is made by and between Phoenix Footwear Group, Inc., a Delaware corporation having a business location at the address set forth below next to its signature (the “Debtor”), and Wells Fargo Bank, National Association (“Wells Fargo”), and having a business location at the address set forth below next to its signature.

Recitals

A. Company, certain affiliates of Company, and Wells Fargo are parties to a Credit and Security Agreement (as amended, supplemented or restated from time to time, the “Credit Agreement”) dated the same date as this Agreement, setting forth the terms on which Wells Fargo may now or hereafter extend credit to or for the account of Company.

B. As a condition to extending credit to or for the account of Company, Wells Fargo has required the execution and delivery of this Agreement by Company.

ACCORDINGLY, in consideration of the mutual covenants contained in the Loan Documents and herein, the parties hereby agree as follows:

“Patents” means all of Company’s right, title and interest in and to patents or applications for patents, fees or royalties with respect to each, and including without limitation the right to sue for past infringement and damages therefor, and licenses thereunder, all as presently existing or hereafter arising or acquired, including without limitation the patents listed on Exhibit A.

“Security Interest” has the meaning given in Section 2.

“Trademarks” means all of Company’s right, title and interest in and to: (i) trademarks, service marks, collective membership marks, registrations and applications for registration for each, and the respective goodwill associated with each, (ii) licenses, fees or royalties with respect to each, (iii) the right to sue for past, present and future infringement, dilution and damages therefor, and (iv) licenses thereunder, all as presently existing or hereafter arising or acquired, including, without limitation, the marks listed on Exhibit B.

(a) Existence; Authority. Company is a corporation duly organized, validly existing and in good standing under the laws of its state of incorporation, and this Agreement has been duly and validly authorized by all necessary corporate action on the part of Company.

(b) Patents. Exhibit A accurately lists all Patents owned or controlled by Company as of the date hereof, or to which Company has a right as of the date hereof to have assigned to it, and accurately reflects the existence and status of applications and letters patent pertaining to the Patents as of the date hereof. If after the date hereof, Company owns, controls or has a right to have assigned to it any Patents not listed on Exhibit A, or if Exhibit A ceases to accurately reflect the existence and status of applications and letters patent pertaining to the Patents, then Company shall within 60 days provide written notice to Wells Fargo with a replacement Exhibit A, which upon acceptance by Wells Fargo shall become part of this Agreement.

(c) Trademarks. Exhibit B accurately lists all Trademarks owned or controlled by Company as of the date hereof and accurately reflects the existence and status of Trademarks and all applications and registrations pertaining thereto as of the date hereof; provided, however, that Exhibit B need not list common law marks (i.e., Trademarks for which there are no applications or registrations) which are not material to Company’s or any Affiliate’s business(es). If after the date hereof, Company owns or controls any Trademarks not listed on Exhibit B (other than common law marks which are not material to Company’s or any Affiliate’s business(es)), or if Exhibit B ceases to accurately reflect the existence and status of applications and registrations pertaining to the Trademarks, then Company shall promptly provide written notice to Wells Fargo with a replacement Exhibit B, which upon acceptance by Wells Fargo shall become part of this Agreement.

(d) Affiliates. As of the date hereof, no Affiliate owns, controls, or has a right to have assigned to it any items that would, if such item were owned by Company, constitute Patents or Trademarks. If after the date hereof any Affiliate owns, controls, or has a right to have assigned to it any such items, then Company shall promptly either: (i) cause such Affiliate to assign all of its rights in such item(s) to Company; or (ii) notify Wells Fargo of such item(s) and cause such Affiliate to execute and deliver to Wells Fargo a patent and trademark security agreement substantially in the form of this Agreement.

-2-

Patent and Trademark Security Agreement

WFBC/Phoenix Footwear Group

(e) Title. Company has absolute title to each Patent and each Trademark listed on Exhibits A and B, free and clear of all Liens except Permitted Liens. Company (i) will have, at the time Company acquires any rights in Patents or Trademarks hereafter arising, absolute title to each such Patent or Trademark free and clear of all Liens except Permitted Liens, and (ii) will keep all Patents and Trademarks free and clear of all Liens except Permitted Liens.

(f) No Sale. Except as permitted in the Credit Agreement, Company will not assign, transfer, encumber or otherwise dispose of the Patents or Trademarks, or any interest therein, without Wells Fargo’s prior written consent.

(g) Defense. Company will at its own expense and using commercially reasonable efforts, protect and defend the Patents and Trademarks against all claims or demands of all Persons other than those holding Permitted Liens.

(h) Maintenance. Company will at its own expense maintain the Patents and the Trademarks to the extent reasonably advisable in its business including, but not limited to, filing all applications to obtain letters patent or trademark registrations and all affidavits, maintenance fees, annuities, and renewals possible with respect to letters patent, trademark registrations and applications therefor. Company covenants that it will not abandon nor fail to pay any maintenance fee or annuity due and payable on any Patent or Trademark, nor fail to file any required affidavit or renewal in support thereof, without first providing Wells Fargo: (i) sufficient written notice, of at least 30 days, to allow Wells Fargo to timely pay any such maintenance fees or annuities which may become due on any Patents or Trademarks, or to file any affidavit or renewal with respect thereto, and (ii) a separate written power of attorney or other authorization to pay such maintenance fees or annuities, or to file such affidavit or renewal, should such be necessary or desirable.

(i) Wells Fargo’s Right to Take Action. If Company fails to perform or observe any of its covenants or agreements set forth in this Section 3, and if such failure continues for a period of ten (10) calendar days after Wells Fargo gives Company written notice thereof (or, in the case of the agreements contained in subsection (h), immediately upon the occurrence of such failure, without notice or lapse of time), or if Company notifies Wells Fargo that it intends to abandon a Patent or Trademark, Wells Fargo may (but need not) perform or observe such covenant or agreement or take steps to prevent such intended abandonment on behalf and in the name, place and stead of Company (or, at Wells Fargo’s option, in Wells Fargo’s own name) and may (but need not) take any and all other actions which Wells Fargo may reasonably deem necessary to cure or correct such failure or prevent such intended abandonment.

-3-

Patent and Trademark Security Agreement

WFBC/Phoenix Footwear Group

(j) Costs and Expenses. Except to the extent that the effect of such payment would be to render any loan or forbearance of money usurious or otherwise illegal under any applicable law, Company shall pay Wells Fargo on demand the amount of all moneys expended and all costs and expenses (including reasonable attorneys’ fees and disbursements) incurred by Wells Fargo in connection with or as a result of Wells Fargo’s taking action under subsection (i) or exercising its rights under Section 6, together with interest thereon from the date expended or incurred by Wells Fargo at the Default Rate.

(k) Power of Attorney. To facilitate Wells Fargo’s taking action under subsection (i) and exercising its rights under Section 6, Company hereby irrevocably appoints (which appointment is coupled with an interest) Wells Fargo, or its delegate, as the attorney-in-fact of Company with the right (but not the duty) from time to time to create, prepare, complete, execute, deliver, endorse or file, in the name and on behalf of Company, any and all instruments, documents, applications, financing statements, and other agreements and writings required to be obtained, executed, delivered or endorsed by Company under this Section 3, or, necessary for Wells Fargo, after an Event of Default and at any time thereafter during the Default Period commenced thereby, to enforce or use the Patents or Trademarks or to grant or issue any exclusive or non-exclusive license under the Patents or Trademarks to any third party, or to sell, assign, transfer, pledge, encumber or otherwise transfer title in or dispose of the Patents or Trademarks to any third party. Company hereby ratifies all that such attorney shall lawfully do or cause to be done by virtue hereof. The power of attorney granted herein shall terminate upon the termination of the Credit Agreement as provided therein and the payment and performance of all Indebtedness.

-4-

Patent and Trademark Security Agreement

WFBC/Phoenix Footwear Group

(a) Wells Fargo may exercise any or all remedies available under the Credit Agreement.

(b) Wells Fargo may sell, assign, transfer, pledge, encumber or otherwise dispose of the Patents and Trademarks.

(c) Wells Fargo may enforce the Patents and Trademarks and any licenses thereunder, and if Wells Fargo shall commence any suit for such enforcement, Company shall, at the request of Wells Fargo, do any and all lawful acts and execute any and all proper documents required by Wells Fargo in aid of such enforcement.

-5-

Patent and Trademark Security Agreement

WFBC/Phoenix Footwear Group

(a) Arbitration. The parties hereto agree, upon demand by any party, to submit to binding arbitration all claims, disputes and controversies between or among them (and their respective employees, officers, directors, attorneys, and other agents), whether in tort, contract or otherwise arising out of or relating to in any way this Agreement and its negotiation, execution, collateralization, administration, repayment, modification, extension, substitution, formation, inducement, enforcement, default or termination.

(b) Governing Rules. Any arbitration proceeding will (i) proceed in a location in California selected by the American Arbitration Association (“AAA”); (ii) be governed by the Federal Arbitration Act (Title 9 of the United States Code), notwithstanding any conflicting choice of law provision in any of the documents between the parties; and (iii) be conducted by the AAA, or such other administrator as the parties shall mutually agree upon, in accordance with the AAA’s commercial dispute resolution procedures, unless the claim or counterclaim is at least $1,000,000.00 exclusive of claimed interest, arbitration fees and costs in which case the arbitration shall be conducted in accordance with the AAA’s optional procedures for large, complex commercial disputes (the commercial dispute resolution procedures or the optional procedures for large, complex commercial disputes to be referred to, as applicable, as the “Rules”). If there is any inconsistency between the terms hereof and the Rules, the terms and procedures set forth herein shall control. Any party who fails or refuses to submit to arbitration following a demand by any other party shall bear all costs and expenses incurred by such other party in compelling arbitration of any dispute. Nothing contained herein shall be deemed to be a waiver by any party that is a bank of the protections afforded to it under 12 U.S.C. §91 or any similar applicable state law.

(c) No Waiver of Provisional Remedies, Self-Help and Foreclosure. The arbitration requirement does not limit the right of any party (if not otherwise restricted by the terms and conditions of this Agreement) to (i) foreclose against real or personal property collateral; (ii) exercise self-help remedies relating to collateral or proceeds of collateral such as setoff or repossession; or (iii) obtain provisional or ancillary remedies such as replevin, injunctive relief, attachment or the appointment of a receiver, before during or after the pendency of any arbitration proceeding. This exclusion does not constitute a waiver of the right or obligation of any party to submit any dispute to arbitration or reference hereunder, including those arising from the exercise of the actions detailed in sections (i), (ii) and (iii) of this paragraph.

(d) Arbitrator Qualifications and Powers. Any arbitration proceeding in which the amount in controversy is $5,000,000.00 or less will be decided by a single arbitrator selected according to the Rules, and who shall not render an award of greater than $5,000,000.00. Any dispute in which the amount in controversy exceeds $5,000,000.00 shall be decided by majority vote of a panel of three arbitrators; provided however, that all three arbitrators must actively participate in all hearings and deliberations. The arbitrator will be a neutral attorney licensed in the State of California or a neutral retired judge of the state or federal judiciary of California, in either case with a minimum of ten years experience in the substantive law applicable to the subject matter of the dispute to be arbitrated. The arbitrator will determine whether or not an issue is arbitratable and will give effect to the statutes of limitation in determining any claim. In any arbitration proceeding the arbitrator will decide (by documents

-6-

Patent and Trademark Security Agreement

WFBC/Phoenix Footwear Group

only or with a hearing at the arbitrator’s discretion) any pre-hearing motions which are similar to motions to dismiss for failure to state a claim or motions for summary adjudication. The arbitrator shall resolve all disputes in accordance with the substantive law of California and may grant any remedy or relief that a court of such state could order or grant within the scope hereof and such ancillary relief as is necessary to make effective any award. The arbitrator shall also have the power to award recovery of all costs and fees, to impose sanctions and to take such other action as the arbitrator deems necessary to the same extent a judge could pursuant to the Federal Rules of Civil Procedure, the California Rules of Civil Procedure or other applicable law. Judgment upon the award rendered by the arbitrator may be entered in any court having jurisdiction. The institution and maintenance of an action for judicial relief or pursuit of a provisional or ancillary remedy shall not constitute a waiver of the right of any party, including the plaintiff, to submit the controversy or claim to arbitration if any other party contests such action for judicial relief.

(e) Discovery. In any arbitration proceeding discovery will be permitted in accordance with the Rules. All discovery shall be expressly limited to matters directly relevant to the dispute being arbitrated and must be completed no later than 20 days before the hearing date and within 180 days of the filing of the dispute with the AAA. Any requests for an extension of the discovery periods, or any discovery disputes, will be subject to final determination by the arbitrator upon a showing that the request for discovery is essential for the party’s presentation and that no alternative means for obtaining information is available.

(f) Class Proceedings and Consolidations. The resolution of any dispute arising pursuant to the terms of this Agreement shall be determined by a separate arbitration proceeding and such dispute shall not be consolidated with other disputes or included in any class proceeding.

(g) Payment Of Arbitration Costs And Fees. The arbitrator shall award all costs and expenses of the arbitration proceeding.

(h) Real Property Collateral; Judicial Reference. Notwithstanding anything herein to the contrary, no dispute shall be submitted to arbitration if the dispute concerns indebtedness secured directly or indirectly, in whole or in part, by any real property unless (i) the holder of the mortgage, lien or security interest specifically elects in writing to proceed with the arbitration, or (ii) all parties to the arbitration waive any rights or benefits that might accrue to them by virtue of the single action rule statute of California, thereby agreeing that all indebtedness and obligations of the parties, and all mortgages, liens and security interests securing such indebtedness and obligations, shall remain fully valid and enforceable. If any such dispute is not submitted to arbitration, the dispute shall be referred to a referee in accordance with California Code of Civil Procedure Section 638 et seq., and this general reference agreement is intended to be specifically enforceable in accordance with said Section 638. A referee with the qualifications required herein for arbitrators shall be selected pursuant to the AAA’s selection procedures. Judgment upon the decision rendered by a referee shall be entered in the court in which such proceeding was commenced in accordance with California Code of Civil Procedure Sections 644 and 645.

-7-

Patent and Trademark Security Agreement

WFBC/Phoenix Footwear Group

(i) Miscellaneous. To the maximum extent practicable, the AAA, the arbitrators and the parties shall take all action required to conclude any arbitration proceeding within 180 days of the filing of the dispute with the AAA. No arbitrator or other party to an arbitration proceeding may disclose the existence, content or results thereof, except for disclosures of information by a party required in the ordinary course of its business or by applicable law or regulation. If more than one agreement for arbitration by or between the parties potentially applies to a dispute, the arbitration provision most directly related to the Loan Documents or the subject matter of the dispute shall control. This arbitration provision shall survive termination, amendment or expiration of any of the Loan Documents or any relationship between the parties.

[signatures continued on next page]

-8-

Patent and Trademark Security Agreement

WFBC/Phoenix Footwear Group

IN WITNESS WHEREOF, the parties have executed this Patent and Trademark Security Agreement as of the date written above.

The foregoing instrument was acknowledged before me this          day of             , 200__, by                                     , the President of                                              , a                      corporation, on behalf of the corporation.

The foregoing instrument was acknowledged before me this          day of             , 200    , by                                              , a Vice President of Wells Fargo Bank, National Association, on behalf of the national association.

EXHIBIT A

PATENTS AND PATENT APPLICATIONS

A-1

EXHIBIT B

TRADEMARKS AND TRADEMARK APPLICATIONS

B-1

B-2