SWINGLINE NOTE

Exhibit 10.4

SWINGLINE NOTE

FOR VALUE RECEIVED, OMEGA PROTEIN CORPORATION, a Nevada corporation, and OMEGA PROTEIN, INC., a Virginia corporation, each having its principal place of business at 2105 Citywest Blvd., Suite 500, Houston, Harris County, Texas 77042 (collectively, the “Borrowers”), jointly and severally promise to pay to the order of WELLS FARGO BANK, NATIONAL ASSOCIATION, a national banking association (the “Lender”), the principal sum of Five Million and No/100 Dollars ($5,000,000.00) or, if less, the unpaid principal amount of all Swingline Loans made by the Lender from time to time pursuant to that Amended and Restated Loan Agreement, dated as of March 21, 2012 (as further amended, restated, supplemented or otherwise modified from time to time, the “Loan Agreement”) by and among the Borrowers, the Lenders who are or may become a party thereto, as Lenders, and Wells Fargo Bank, National Association, as Administrative Agent, Issuing Lender and Swingline Lender, together with interest on the unpaid principal balance as set forth below. All sums hereunder are payable to the Administrative Agent on behalf of the Lender at the Administrative Agent’s principal office in Houston, Harris County, Texas.

1. Definitions. Unless the context hereof otherwise requires or provides, the terms used herein have the same meanings as defined in the Loan Agreement.

2. Interest Rate. The unpaid principal balance from time to time outstanding from the date hereof until maturity (whether by acceleration or otherwise) shall bear interest as provided in the Loan Agreement.

3. Payment of Interest and Principal. The principal of and interest on this Swingline Note shall be due and payable as provided in the Loan Agreement. Subject to the terms of the Loan Agreement, the principal and interest due hereunder shall be evidenced by the Administrative Agent’s and the Lender’s records which, absent manifest error, shall be conclusive evidence of the computation of principal and interest balances owed by the Borrowers to the Lender. Swingline Loans refunded as Revolving Credit Loans in accordance with Section 2.02(b) of the Loan Agreement shall be payable by the Borrowers as Revolving Credit Loans pursuant to the Revolving Credit Notes, and shall not be payable under this Swingline Note as Swingline Loans.

4. Default. Upon the occurrence of an Event of Default set forth in the Loan Agreement, the Administrative Agent and the Lenders shall have the rights and remedies provided in the Loan Agreement.

5. Waiver. Each surety, endorser, guarantor and any other party now or hereafter liable for the payment of this Swingline Note in whole or in part (“Surety”) and the Borrowers hereby severally (a) waive grace, demand, presentment for payment, notice of nonpayment, protest, notice of protest, non-payment or dishonor, notice of intent to accelerate, notice of acceleration and all other notices, filing of suit and diligence in collecting this Swingline Note or enforcing any other security with respect to same, (b) agree to any substitution, surrender,

subordination, waiver, modification, change, exchange or release of any security or the release of the liability of any parties primarily or secondarily liable hereon, (c) agree that the Administrative Agent or any Lender is not required first to institute suit or exhaust their remedies hereon against any Borrower, any Surety or others liable or to become liable hereon or to enforce their rights against them or any security with respect to same or to join any of them in any suit against any others of them, and (d) consent to any extension or postponement of time of payment of this Swingline Note and to any other indulgence with respect hereto without notice thereof to any of them. No failure or delay on the part of the Administrative Agent or any Lender in exercising any right, power or privilege hereunder shall operate as a waiver thereof.

6. Attorneys’ Fees. If this Swingline Note is not paid at maturity, regardless of how such maturity may be brought about, or is collected or attempted to be collected through the initiation or prosecution of any suit or through any probate, bankruptcy or any other judicial proceedings, or through any arbitration proceeding, or is placed in the hands of an attorney for collection, the Borrowers shall pay, in addition to all other amounts owing hereunder, all actual expenses of collection, all court costs and reasonable attorney’s fees incurred by the holder hereof.

7. Limitation on Agreements. All agreements among the Borrowers and the Lender, whether now existing or hereafter arising, are hereby limited so that in no event shall the amount paid, or agreed to be paid to or charged or demanded by the Lender for the use, forbearance, or detention of money or for the payment or performance of any covenant or obligation contained herein or in any other document evidencing, securing or pertaining to this Swingline Note, exceed the Maximum Rate. If any circumstance otherwise would cause the amount paid, charged or demanded to exceed the Maximum Rate, the amount paid or agreed to be paid to or charged or demanded by the Lender shall be reduced to the Maximum Rate, and if the Lender ever receives interest which otherwise would exceed the Maximum Rate, such amount which would be excessive interest shall be applied to the reduction of the principal of this Swingline Note and not to the payment of interest, or if such excessive interest otherwise would exceed the unpaid balance of principal of this Swingline Note, such excess shall be applied first to other indebtedness of the Borrowers to the Lender, and the balance, if any, shall be refunded to the Borrowers. In determining whether the interest paid, agreed to be paid, charged or demanded hereunder exceeds the highest amount permitted by Applicable Law, all sums paid or agreed to be paid to or charged or demanded by the Lender for the use, forbearance or detention of the indebtedness of the Borrowers to the Lender shall, to the extent permitted by Applicable Law, (i) be amortized, prorated, allocated and spread throughout the full term of such indebtedness until payment in full so that the actual rate of interest on account of such indebtedness is uniform throughout such term, (ii) be characterized as a fee, expense or other charge other than interest, and (iii) exclude any voluntary prepayments and the effects thereof. This Section shall be subject to the terms and provisions of Section 11.04 of the Loan Agreement.

8. Governing Law and Venue.

(a) THIS SWINGLINE NOTE SHALL BE CONSTRUED IN ACCORDANCE WITH AND GOVERNED BY THE LAWS (BUT NOT THE RULES GOVERNING CONFLICTS OF LAWS) OF THE STATE OF TEXAS. Chapter 346 of the Texas Finance Code does not apply to this Swingline Note. The Borrowers irrevocably submit

2

to the jurisdiction of any Texas state court or any United States court located in the State of Texas (or any court having jurisdiction over appeals from any such court) in any proceeding between or among them arising out of or in any way relating to this Swingline Note or the Loan Documents whether arising in contract, tort or otherwise. Any suit, action or proceeding may be brought in the courts of the State of Texas, County of Harris, or in the United States District Court for the Southern District of Texas, Houston Division. The Borrowers irrevocably consent to the service of process in any suit, action or proceeding in said court by the mailing thereof, by registered or certified mail, postage prepaid, to its address for notices set forth in the Loan Agreement. Service shall be deemed effective five (5) days after such mailing. If requested to do so by any party, the Borrowers agree to waive service of process and to execute any and all documents necessary to implement such waiver in accordance with the Texas Rules of Civil Procedure. The Borrowers irrevocably waive any objections which they may now or hereafter have (including any based on the grounds of forum non conveniens) to the laying of venue of any suit, action or proceeding arising out of or relating to this Swingline Note or the Loan Documents brought in the courts located in Harris County, Texas. Nothing herein impairs the right to bring proceedings in the courts of any other jurisdiction or to effect service of process in any other manner permitted.

(b) The Borrowers recognize that courts outside of Harris County, Texas, may also have jurisdiction over suits, actions or proceedings arising out of this Swingline Note and the Loan Documents. In the event any party shall institute a proceeding involving this Swingline Note or the Loan Documents in a jurisdiction outside Harris County, Texas, the party instituting such litigation shall indemnify the other parties for any losses and expenses that may result from the breach of the foregoing covenant to institute such proceeding only in a state or federal court in Harris County, Texas, including without limitation any additional expenses incurred as the result of litigating in another jurisdiction; such as the expenses and reasonable fees of local counsel and travel and lodging expenses of the indemnified parties, its witnesses, experts and support personnel.

9. Business Day. Subject to the Loan Agreement, if any action is required or permitted to be taken hereunder on a day which is not a Business Day, such action shall be taken on the next succeeding day which is a Business Day, and, to the extent applicable, interest on the unpaid principal balance shall continue to accrue at the applicable rate.

10. Agreement. This Swingline Note is a Note referred to in the Loan Agreement, and is entitled to the benefits thereof and the security as provided for therein. Reference is made to the Loan Agreement and the Loan Documents for a statement of the rights and obligations of the Borrowers, a description of the nature and extent of the security and the rights of the parties with respect to such security, and a statement of the terms and conditions under which the due date of this Swingline Note may be accelerated.

11. Relationship of Parties. The Borrowers and the Lender agree that the relationship among them shall be solely that of debtor and creditor. Nothing contained in this Swingline Note or in any other Loan Document shall be deemed to create a partnership, tenancy-in-common, joint tenancy, joint venture or co-ownership by or among the Borrowers and the Lender. The Lender shall not be in any way responsible or liable for debts, losses, obligations or duties of the Borrowers with respect to the collateral described in the Loan Documents or otherwise. The Borrowers, at all times consistent with the terms and provisions of this Swingline Note and the Loan Documents, shall be free to determine and follow its own policies and practices in the conduct of their business.

3

12. Arbitration.

(a) Arbitration. The parties hereto agree, upon demand by any party, to submit to binding arbitration all claims, disputes and controversies between or among them (and their respective employees, officers, directors, attorneys, and other agents), whether in tort, contract or otherwise in any way arising out of or relating to (i) any credit subject hereto, or any of the Loan Documents, and their negotiation, execution, collateralization, administration, repayment, modification, extension, substitution, formation, inducement, enforcement, default or termination; or (ii) requests for additional credit.

(b) Governing Rules. Any arbitration proceeding will (i) proceed in a location in Texas selected by the American Arbitration Association (“AAA”); (ii) be governed by the Federal Arbitration Act (Title 9 of the United States Code), notwithstanding any conflicting choice of law provision in any of the documents between the parties; and (iii) be conducted by the AAA, or such other administrator as the parties shall mutually agree upon, in accordance with the AAA’s commercial dispute resolution procedures, unless the claim or counterclaim is at least $1,000,000.00 exclusive of claimed interest, arbitration fees and costs in which case the arbitration shall be conducted in accordance with the AAA’s optional procedures for large, complex commercial disputes (the commercial dispute resolution procedures or the optional procedures for large, complex commercial disputes to be referred to herein, as applicable, as the “Rules”). If there is any inconsistency between the terms hereof and the Rules, the terms and procedures set forth herein shall control. Any party who fails or refuses to submit to arbitration following a demand by any other party shall bear all costs and expenses incurred by such other party in compelling arbitration of any dispute. Nothing contained herein shall be deemed to be a waiver by any party that is a bank of the protections afforded to it under 12 U.S.C. §91 or any similar applicable state law.

(c) No Waiver of Provisional Remedies, Self-Help and Foreclosure. The arbitration requirement does not limit the right of any party to (i) foreclose against real or personal property collateral; (ii) exercise self-help remedies relating to collateral or proceeds of collateral such as setoff or repossession; or (iii) obtain provisional or ancillary remedies such as replevin, injunctive relief, attachment or the appointment of a receiver, before during or after the pendency of any arbitration proceeding. This exclusion does not constitute a waiver of the right or obligation of any party to submit any dispute to arbitration or reference hereunder, including those arising from the exercise of the actions detailed in sections (i), (ii) and (iii) of this paragraph.

(d) Arbitrator Qualifications and Powers. Any arbitration proceeding in which the amount in controversy is $5,000,000.00 or less will be decided by a single arbitrator selected according to the Rules, and who shall not render an award of greater than $5,000,000.00. Any dispute in which the amount in controversy exceeds $5,000,000.00 shall be decided by majority vote of a panel of three (3) arbitrators; provided however, that all three (3) arbitrators must actively participate in all hearings and deliberations. The arbitrator will be a neutral attorney

4

licensed in the State of Texas with a minimum of ten (10) years experience in the substantive law applicable to the subject matter of the dispute to be arbitrated. The arbitrator will determine whether or not an issue is arbitratable and will give effect to the statutes of limitation in determining any claim. In any arbitration proceeding the arbitrator will decide (by documents only or with a hearing at the arbitrator’s discretion) any pre-hearing motions which are similar to motions to dismiss for failure to state a claim or motions for summary adjudication. The arbitrator shall resolve all disputes in accordance with the substantive law of Texas and may grant any remedy or relief that a court of such state could order or grant within the scope hereof and such ancillary relief as is necessary to make effective any award. The arbitrator shall also have the power to award recovery of all costs and fees, to impose sanctions and to take such other action as the arbitrator deems necessary to the same extent a judge could pursuant to the Federal Rules of Civil Procedure, the Texas Rules of Civil Procedure or other Applicable Law. Judgment upon the award rendered by the arbitrator may be entered in any court having jurisdiction. The institution and maintenance of an action for judicial relief or pursuit of a provisional or ancillary remedy shall not constitute a waiver of the right of any party, including the plaintiff, to submit the controversy or claim to arbitration if any other party contests such action for judicial relief.

(e) Discovery. In any arbitration proceeding, discovery will be permitted in accordance with the Rules. All discovery shall be expressly limited to matters directly relevant to the dispute being arbitrated and must be completed no later than twenty (20) days before the hearing date. Any requests for an extension of the discovery periods, or any discovery disputes, will be subject to final determination by the arbitrator upon a showing that the request for discovery is essential for the party’s presentation and that no alternative means for obtaining information is available.

(f) Class Proceeding and Consolidations. No party hereto shall be entitled to join or consolidate disputes by or against others in any arbitration, except parties who have executed any Loan Document, or to include in any arbitration any dispute as a representative or member of a class, or to act in any arbitration in the interest of the general public or in a private attorney general capacity.

(g) Payment of Arbitration Costs and Fees. The arbitrator shall award all costs and expenses of the arbitration proceeding.

(h) Miscellaneous. To the maximum extent practicable, the AAA, the arbitrators and the parties shall take all action required to conclude any arbitration proceeding within one hundred eighty (180) days of the filing of the dispute with the AAA. No arbitrator or other party to an arbitration proceeding may disclose the existence, content or results thereof, except for disclosures of information by a party required in the ordinary course of its business or by Applicable Law or regulation. If more than one agreement for arbitration by or between the parties potentially applies to a dispute, the arbitration provision most directly related to the Loan Documents or the subject matter of the dispute shall control. This arbitration provision shall survive termination, amendment or expiration of any of the Loan Documents or any relationship between the parties.

5

13. WAIVER OF JURY TRIAL. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, THE BORROWERS HEREBY IRREVOCABLY AND EXPRESSLY WAIVE ALL RIGHT TO A TRIAL BY JURY IN ANY ACTION, PROCEEDING, OR COUNTERCLAIM (WHETHER BASED UPON CONTRACT, TORT, OR OTHERWISE) ARISING OUT OF OR RELATING TO THIS SWINGLINE NOTE OR ANY OF THE OTHER LOAN DOCUMENTS OR THE TRANSACTIONS CONTEMPLATED HEREBY OR THEREBY OR THE ACTIONS OF THE ADMINISTRATIVE AGENT OR THE LENDER IN THE NEGOTIATION, ADMINISTRATION, OR ENFORCEMENT HEREOF OR THEREOF.

[Signature page follows.]

6

IN WITNESS WHEREOF, the undersigned has executed this Swingline Note under seal as of the day and year first above written.

Signature Page to Swingline Note