SUBORDINATION AGREEMENT

Exhibit 10.20.04

SUBORDINATION AGREEMENT

THIS AGREEMENT is entered into by and among SALEM COMMUNICATIONS CORPORATION (“Borrower”), EDWARD G. ATSINGER III (“Creditor”), and WELLS FARGO BANK, NATIONAL ASSOCIATION (“Agent”), in its capacity as administrative agent for itself, the swing line lender, L/C issuer and each lender from time to time (collectively, the “Senior Parties”) under the Credit Agreement dated as of December 1, 2009, as amended from time to time (the “Senior Credit Agreement,” together with all of the Loan Documents as defined therein, the “Senior Debt Documents”), by and among the Borrower, the Agent and the other Senior Parties thereto.

RECITALS

A. Borrower is indebted to Creditor, and Borrower proposes to obtain credit or has obtained credit from Senior Parties; and

B. Senior Parties have indicated that they will extend or continue credit to Borrower if certain conditions are met, including without limitation, the requirement that Creditor execute this Agreement.

NOW, THEREFORE, as an inducement to Senior Parties to extend or continue credit and for other valuable consideration, the parties hereto agree as follows:

1. INDEBTEDNESS SUBORDINATED. Creditor subordinates all Indebtedness now or at any time hereafter owing from Borrower to Creditor (“Junior Debt”) to all Indebtedness now or at any time hereafter owing from Borrower to any of the Senior Parties (“Senior Debt”). Creditor irrevocably consents and directs that all Senior Debt shall be paid in full in cash or other immediately available funds prior to Borrower making any payment on any Junior Debt, except such payments as are expressly permitted by this Agreement. Creditor will, and Agent is authorized in the name of Creditor from time to time to, execute and file such financing statements and other documents as Agent may require in order to give notice to other persons and entities of the terms and provisions of this Agreement. As long as this Agreement is in effect, Creditor will not take any action or initiate any proceedings, judicial or otherwise, to enforce Creditor’s rights or remedies with respect to any Junior Debt, including without limitation, any action to enforce remedies with respect to any collateral securing any Junior Debt or to obtain any judgment or prejudgment remedy against Borrower or any such collateral.

2. INDEBTEDNESS DEFINED. The word “Indebtedness” is used herein in its most comprehensive sense and includes any and all advances, debts, obligations and liabilities of Borrower heretofore, now or hereafter made, incurred or created, whether voluntary or involuntary and however arising, whether due or not due, absolute or contingent, liquidated or unliquidated, determined or undetermined, including under any swap, derivative, foreign exchange, hedge, deposit, treasury management or other similar transaction or arrangement, and whether Borrower may be liable individually or jointly with others, including without limitation, obligations and liabilities arising from notes, repurchase agreements and trust receipts, and any and all interest and other amounts thereon which may accrue subsequent to Borrower becoming subject to any state or federal debtor-relief statute. Notwithstanding the foregoing or anything in this Subordination Agreement to the contrary, the definition of Junior Debt shall not include and this Subordination Agreement shall not apply to any salary, wages, bonuses, stock options, employee benefits, employment contract obligations, split dollar life insurance programs, board fees and other similar compensation paid or owing by Borrower to

-1-

Creditor in his capacity as an employee or director of Borrower and any reimbursements in the ordinary course related thereto.

3. RESTRICTION OF PAYMENT OF JUNIOR DEBT; DISPOSITION OF PAYMENTS RECEIVED BY CREDITOR. Borrower will not make, and Creditor will not accept or receive, any payment or benefit in cash, by setoff or otherwise, directly or indirectly, on account of principal, interest or any other amounts owing on any Junior Debt, except such payments as are expressly permitted herein. Borrower is permitted to make and Creditor to receive any payments of principal and interest on that certain Letter of Intent to Lend dated as of November 17, 2011 between Borrower and Creditor, in the principal amount of $6,000,000 (as existing on the date hereof, the “Note”); provided however, that no payment of principal or interest on the Note shall be made by Borrower, or received by Creditor, after notice from Agent to Creditor that a Default (as such term is defined in the Senior Debt Documents), or any condition, event or act which with the giving of notice or the passage of time or both would constitute a Default, has occurred under the terms of any Senior Debt. If any payment is made in violation of this Agreement, Creditor shall promptly deliver the same to Agent in the form received, with any endorsement or assignment necessary for the transfer of such payment or amounts setoff from Creditor to Agent, to be either (in Agent’s sole discretion) held as cash collateral securing the Senior Debt or applied in reduction of the Senior Debt in such order as Agent shall determine, and until so delivered, Creditor shall hold such payment in trust for and on behalf of, and as the property of, Agent.

4. DISPOSITION OF EVIDENCE OF INDEBTEDNESS. If there is any existing promissory note or other evidence of any of the Junior Debt, including the Note, or if any promissory note or other evidence of Indebtedness is executed at any time hereafter with respect thereto, then Borrower and Creditor will mark the same with a legend stating that it is subject to this Agreement, and if asked to do so, will deliver the same to Agent. Creditor shall not, without Agent’s prior written consent, assign, transfer, hypothecate or otherwise dispose of any claim it now has or may at any time hereafter have against Borrower at any time that any Senior Debt remains outstanding and/or any Senior Party remains committed to extend any credit to Borrower.

5. AGREEMENT TO BE CONTINUING; APPLIES TO BORROWER’S EXISTING INDEBTEDNESS AND ANY INDEBTEDNESS HEREAFTER ARISING. This Agreement shall be a continuing agreement and shall apply to any and all Indebtedness of Borrower to any Senior Party or Creditor now existing or hereafter arising, including any Indebtedness arising under successive transactions, related or unrelated, and notwithstanding that from time to time all Indebtedness theretofore existing may have been paid in full.

6. TERMINATION BY CREDITOR. Creditor may, to the extent provided herein, terminate this Agreement by delivering written notice to Agent. Any such notice must be sent to Agent by registered U.S. mail, postage prepaid, addressed to its Ventura/Santa Barbara Commercial Banking Office at 601 E. Daily Drive, Suite 110, Camarillo, California 93010, or at such other address as Agent shall from time to time designate. If such notice is received by Agent, this Agreement shall terminate as of the date of receipt, except that the obligations of Creditor and the rights of Senior Parties hereunder shall continue with respect to all Senior Debt which existed at the time of Agent’s receipt of such notice, or thereafter arose pursuant to any agreement to extend credit by which any Senior Party is bound at the time of its receipt of such notice, and any extensions, renewals or modifications of any such then existing or committed Senior Debt, including without limitation, modifications to the amount of principal or interest payable on any Senior Debt and the release of any security for or any guarantors of all or any portion of any Senior Debt.

-2-

7. REPRESENTATIONS AND WARRANTIES; INFORMATION. Borrower and Creditor represent and warrant to Senior Parties that: (a) no interest in the Junior Debt has been assigned or otherwise transferred to any person or entity; (b) payment of the Junior Debt has not been heretofore subordinated to any other creditor of Borrower; and (c) Creditor has the requisite power and authority to enter into and perform its obligations under this Agreement. Creditor further represents and warrants to Senior Parties that Creditor has established adequate, independent means of obtaining from Borrower on a continuing basis financial and other information pertaining to Borrower’s financial condition. Creditor agrees to keep adequately informed from such means of any facts, events or circumstances which might in any way affect Creditor’s risks hereunder, and Creditor agrees that no Senior Party shall have any obligation to disclose to Creditor information or material about Borrower which is acquired by any Senior Party in any manner. Any Senior Party may, at such Senior Party’s sole option and without obligation to do so, disclose to Creditor any information or material relating to Borrower which is acquired by such Senior Party by any means, and Borrower hereby agrees to and authorizes any such disclosure by such Senior Party.

8. TRANSFER OF ASSETS OR REORGANIZATION OF BORROWER. If any petition is filed or any proceeding is instituted by or against Borrower under any provisions of the Bankruptcy Reform Act, Title 11 of the United States Code, or any other or similar law relating to bankruptcy, insolvency, reorganization or other relief for debtors, or generally affecting creditors’ rights, or seeking the appointment of a receiver, trustee, custodian or liquidator of or for Borrower or any of its assets, any payment or distribution of any of Borrower’s assets, whether in cash, securities or any other property, which would be payable or deliverable with respect to any Junior Debt, shall be paid or delivered to Agent until all Senior Debt is paid in full in cash or other immediately available funds. Creditor grants to Agent the right to enforce, collect and receive any such payment or distribution and to give releases or acquittances therefor, and Creditor authorizes Agent as its attorney-in-fact to vote and prove the Junior Debt in any of the above-described proceedings or in any meeting of creditors of Borrower relating thereto.

9. OTHER AGREEMENTS; NO THIRD PARTY BENEFICIARIES. Agent shall have no direct or indirect obligations to Creditor of any kind with respect to the manner or time in which Agent exercises (or refrains from exercising) any of its rights or remedies with respect to the Senior Debt, Borrower or any of Borrower’s assets. Creditor understands that there may be various agreements among Senior Parties and Borrower evidencing and governing the Senior Debt, and Creditor acknowledges and agrees that such agreements are not intended to confer any benefits on Creditor. Creditor further acknowledges that Agent may administer the Senior Debt and any of Senior Parties’ agreements with Borrower in any way Agent deems appropriate, without regard to Creditor or the Junior Debt. Creditor waives any right Creditor might otherwise have to require a marshalling of any security held by Agent for all or any part of the Senior Debt or to direct or affect the manner or timing with which Agent enforces any of its security. Nothing in this Agreement shall impair or adversely affect any right, privilege, power or remedy of Agent with respect to the Senior Debt, Borrower or any assets of Borrower, including without limitation, Agent’s right to: (a) waive, release or subordinate any of Agent’s security or rights; (b) waive or ignore any defaults by Borrower; and/or (c) restructure, renew, modify or supplement the Senior Debt, or any portion thereof, or any agreement with Borrower relating to any Senior Debt. All rights, privileges, powers and remedies of Agent may be exercised from time to time by Agent without notice to or consent of Creditor.

10. BREACH OF AGREEMENT BY BORROWER OR CREDITOR. In the event of any breach of this Agreement by Borrower or Creditor, then and at any time thereafter Agent shall have the right to declare immediately due and payable all or any portion of the Senior Debt

-3-

without presentment, demand, notice of nonperformance, protest, notice of protest or notice of dishonor, all of which are hereby expressly waived by Borrower and Creditor. No delay, failure or discontinuance of Agent in exercising any right, privilege, power or remedy hereunder shall be deemed a waiver of such right, privilege, power or remedy; nor shall any single or partial exercise of any such right, privilege, power or remedy preclude, waive or otherwise affect the further exercise thereof or the exercise of any other right, privilege, power or remedy. Any waiver, permit, consent or approval of any kind by Agent with respect to this Agreement must be in writing and shall be effective only to the extent set forth in such writing.

11. LIQUIDATED DAMAGES. Inasmuch as the actual damages which could result from a breach by Creditor of its duties under this Agreement are uncertain and would be impractical or extremely difficult to fix, Creditor shall pay to Agent, in the event of any such breach by Creditor, as liquidated and agreed damages, and not as a penalty, all sums received by Creditor in violation of this Agreement on account of the Junior Debt, which sums represent a reasonable endeavor to estimate a fair compensation for the foreseeable losses that might result from such a breach.

12. COSTS, EXPENSES AND ATTORNEYS’ FEES. If any party hereto institutes any arbitration or judicial or administrative action or proceeding to enforce any provisions of this Agreement, or alleging any breach of any provision hereof or seeking damages or any remedy, the losing party or parties shall pay to the prevailing party or parties all costs and expenses, including reasonable attorneys’ fees (to include outside counsel fees and all allocated costs of such prevailing party’s in-house counsel), expended or incurred by the prevailing party or parties in connection therewith, whether incurred at the trial or appellate level, in an arbitration proceeding or otherwise, and including any of the foregoing incurred in connection with any bankruptcy proceeding (including without limitation, any adversary proceeding, contested matter or motion brought by Agent or any other person) relating to Borrower, Creditor or any other person or entity.

13. SUCCESSORS; ASSIGNS; AMENDMENT. This Agreement shall be binding upon and inure to the benefit of the heirs, executors, administrators, legal representatives, successors and assigns of the parties. This Agreement may be amended or modified only in writing signed by all parties hereto.

14. OBLIGATIONS JOINT AND SEVERAL; CONSTRUCTION. If this Agreement is executed by more than one Creditor, it shall bind them jointly and severally. All words used herein in the singular shall be deemed to have been used in the plural where the context so requires.

15. SEVERABILITY OF PROVISIONS. If any provision of this Agreement shall be held to be prohibited by or invalid under applicable law, such provision shall be ineffective only to the extent of such prohibition or invalidity, without invalidating the remainder of such waiver or other provision or any remaining provisions of this Agreement.

16. GOVERNING LAW. This Agreement shall be governed by and construed in accordance with the laws of the State of California.

17. ARBITRATION.

(a) Arbitration. The parties hereto agree, upon demand by any party, to submit to binding arbitration all claims, disputes and controversies between or among them (and their respective employees, officers, directors, attorneys, and other agents), whether in tort, contract

-4-

or otherwise, in any way arising out of or relating to this Agreement and its negotiation, execution, collateralization, administration, repayment, modification, extension, substitution, formation, inducement, enforcement, default or termination.

(b) Governing Rules. Any arbitration proceeding will (i) proceed in a location in California selected by the American Arbitration Association (“AAA”); (ii) be governed by the Federal Arbitration Act (Title 9 of the United States Code), notwithstanding any conflicting choice of law provision in any of the documents between the parties; and (iii) be conducted by the AAA, or such other administrator as the parties shall mutually agree upon, in accordance with the AAA’s commercial dispute resolution procedures, unless the claim or counterclaim is at least $1,000,000.00 exclusive of claimed interest, arbitration fees and costs in which case the arbitration shall be conducted in accordance with the AAA’s optional procedures for large, complex commercial disputes (the commercial dispute resolution procedures or the optional procedures for large, complex commercial disputes to be referred to herein, as applicable, as the “Rules”). If there is any inconsistency between the terms hereof and the Rules, the terms and procedures set forth herein shall control. Any party who fails or refuses to submit to arbitration following a demand by any other party shall bear all costs and expenses incurred by such other party in compelling arbitration of any dispute. Nothing contained herein shall be deemed to be a waiver by any party that is a bank of the protections afforded to it under 12 U.S.C. §91 or any similar applicable state law.

(c) No Waiver of Provisional Remedies, Self-Help and Foreclosure. The arbitration requirement does not limit the right of any party to (i) foreclose against real or personal property collateral; (ii) exercise self-help remedies relating to collateral or proceeds of collateral such as setoff or repossession; or (iii) obtain provisional or ancillary remedies such as replevin, injunctive relief, attachment or the appointment of a receiver, before during or after the pendency of any arbitration proceeding. This exclusion does not constitute a waiver of the right or obligation of any party to submit any dispute to arbitration or reference hereunder, including those arising from the exercise of the actions detailed in sections (i), (ii) and (iii) of this paragraph.

(d) Arbitrator Qualifications and Powers. Any arbitration proceeding in which the amount in controversy is $5,000,000.00 or less will be decided by a single arbitrator selected according to the Rules, and who shall not render an award of greater than $5,000,000.00. Any dispute in which the amount in controversy exceeds $5,000,000.00 shall be decided by majority vote of a panel of three arbitrators; provided however, that all three arbitrators must actively participate in all hearings and deliberations. The arbitrator will be a neutral attorney licensed in the State of California or a neutral retired judge of the state or federal judiciary of California, in either case with a minimum of ten years experience in the substantive law applicable to the subject matter of the dispute to be arbitrated. The arbitrator will determine whether or not an issue is arbitratable and will give effect to the statutes of limitation in determining any claim. In any arbitration proceeding the arbitrator will decide (by documents only or with a hearing at the arbitrator’s discretion) any pre-hearing motions which are similar to motions to dismiss for failure to state a claim or motions for summary adjudication. The arbitrator shall resolve all disputes in accordance with the substantive law of California and may grant any remedy or relief that a court of such state could order or grant within the scope hereof and such ancillary relief as is necessary to make effective any award. The arbitrator shall also have the power to award recovery of all costs and fees, to impose sanctions and to take such other action as the arbitrator deems necessary to the same extent a judge could pursuant to the Federal Rules of Civil Procedure, the California Rules of Civil Procedure or other applicable law. Judgment upon the award rendered by the arbitrator may be entered in any court having jurisdiction. The

-5-

institution and maintenance of an action for judicial relief or pursuit of a provisional or ancillary remedy shall not constitute a waiver of the right of any party, including the plaintiff, to submit the controversy or claim to arbitration if any other party contests such action for judicial relief.

(e) Discovery. In any arbitration proceeding, discovery will be permitted in accordance with the Rules. All discovery shall be expressly limited to matters directly relevant to the dispute being arbitrated and must be completed no later than 20 days before the hearing date. Any requests for an extension of the discovery periods, or any discovery disputes, will be subject to final determination by the arbitrator upon a showing that the request for discovery is essential for the party’s presentation and that no alternative means for obtaining information is available.

(f) Class Proceedings and Consolidations. No party hereto shall be entitled to join or consolidate disputes by or against others in any arbitration, except parties who have executed this Agreement or any other contract, instrument or document relating to any Indebtedness, or to include in any arbitration any dispute as a representative or member of a class, or to act in any arbitration in the interest of the general public or in a private attorney general capacity.

(g) Payment Of Arbitration Costs And Fees. The arbitrator shall award all costs and expenses of the arbitration proceeding.

(h) Real Property Collateral; Judicial Reference. Notwithstanding anything herein to the contrary, no dispute shall be submitted to arbitration if the dispute concerns indebtedness secured directly or indirectly, in whole or in part, by any real property unless (i) the holder of the mortgage, lien or security interest specifically elects in writing to proceed with the arbitration, or (ii) all parties to the arbitration waive any rights or benefits that might accrue to them by virtue of the single action rule statute of California, thereby agreeing that all indebtedness and obligations of the parties, and all mortgages, liens and security interests securing such indebtedness and obligations, shall remain fully valid and enforceable. If any such dispute is not submitted to arbitration, the dispute shall be referred to a referee in accordance with California Code of Civil Procedure Section 638 et seq., and this general reference agreement is intended to be specifically enforceable in accordance with said Section 638. A referee with the qualifications required herein for arbitrators shall be selected pursuant to the AAA’s selection procedures. Judgment upon the decision rendered by a referee shall be entered in the court in which such proceeding was commenced in accordance with California Code of Civil Procedure Sections 644 and 645.

(i) Miscellaneous. To the maximum extent practicable, the AAA, the arbitrators and the parties shall take all action required to conclude any arbitration proceeding within 180 days of the filing of the dispute with the AAA. No arbitrator or other party to an arbitration proceeding may disclose the existence, content or results thereof, except for disclosures of information by a party required in the ordinary course of its business or by applicable law or regulation. If more than one agreement for arbitration by or between the parties potentially applies to a dispute, the arbitration provision most directly related to the documents between the parties or the subject matter of the dispute shall control. This arbitration provision shall survive termination, amendment or expiration of any of the documents or any relationship between the parties.

(j) Small Claims Court. Notwithstanding anything herein to the contrary, each party retains the right to pursue in Small Claims Court any dispute within that court’s jurisdiction. Further, this arbitration provision shall apply only to disputes in which either party seeks to recover an amount of money (excluding attorneys’ fees and costs) that exceeds the jurisdictional limit of the Small Claims Court.

-6-

IN WITNESS WHEREOF, the parties hereto have executed this Agreement as of May 21, 2012.

-7-