PREFERRED STOCK EXCHANGE AGREEMENT

Exhibit 10.4

PREFERRED STOCK EXCHANGE AGREEMENT

This Preferred Stock Exchange Agreement (this “Agreement”) is made as of May 9, 2013, by and among Institutional Financial Markets, Inc., a corporation organized under the laws of the State of Maryland (the “Company”), Cohen Bros. Financial, LLC (“Cohen”), and Daniel G. Cohen.

RECITALS:

WHEREAS, Cohen is the owner of an aggregate of Four Million Nine Hundred Eighty-Three Thousand Five Hundred Fifty-Seven (4,983,557) shares of Series D Voting Non-Convertible Preferred Stock of the Company (collectively, the “Series D Shares”);

WHEREAS, Cohen desires to exchange (the “Exchange”) the Series D Shares for an aggregate of Four Million Nine Hundred Eighty-Three Thousand Five Hundred Fifty-Seven (4,983,557) newly issued shares of Series E Voting Non-Convertible Preferred Stock of the Company (collectively, the “Series E Shares” and, together with the Series D Shares, the “Shares”);

WHEREAS, the Shares have substantially identical rights, preferences, privileges and restrictions other than with respect to the Company’s obligation to redeem the Shares, and accordingly, the terms of the Series E Shares effectively serve as an amendment of the terms of the Series D Shares solely with respect to when the Company has an obligation to redeem the Series D Shares; and

WHEREAS, pursuant to this Agreement, Cohen and the Company desire to, among other things, set forth the terms and conditions of the Exchange.

NOW, THEREFORE, in consideration of the premises and the mutual covenants contained herein, and for other good and valuable consideration, the receipt and sufficiency of which is hereby acknowledged, the parties to this Agreement, intending to be legally bound, hereby agree as follows:

1. The Exchange. Cohen hereby assigns and conveys to the Company the Series D Shares, free and clear of all liens, claims and encumbrances. On the date of this Agreement, Cohen shall deliver to the Company the stock certificate representing the Series D Shares. In exchange for the Series D Shares, the Company shall issue on the date of this Agreement the Series E Shares to Cohen and shall deliver to Cohen the stock certificate representing the Series E Shares. Cohen and the Company hereby acknowledge and agree that the Series E Shares shall have all of the rights, preferences, privileges and restrictions described in the Institutional Financial Markets, Inc. Articles Supplementary, Series E Voting Non-Convertible Preferred Stock, and as described in the Articles of Incorporation of the Company, as amended from time to time. Immediately upon execution and delivery of this Agreement, each of the Series D Shares shall be deemed cancelled in exchange for the Series E Shares.

2. Representations and Warranties.

2.1 Representations and Warranties of Cohen. To induce the Company to enter into the Exchange, Cohen hereby represents and warrants to the Company as follows:

(a) Cohen (i) is a limited liability company duly organized and validly existing and in good standing under the laws of the State of Delaware, and (ii) has the limited liability company power and authority to own its property and assets and to transact the business in which it is engaged.

(b) Cohen has the limited liability company power to execute, deliver and carry out the terms and provisions of this Agreement.

(c) Cohen is the sole owner of the Series D Shares and has good and marketable title thereto, free and clear of all liens, claims and encumbrances whatsoever.

(d) This Agreement constitutes a legal, valid and binding obligation of Cohen, enforceable against Cohen in accordance with its terms.

2.2 Representations and Warranties of the Company. To induce Cohen to enter into the Exchange, the Company represents and warrants to Cohen as follows:

(a) The Company (i) is a corporation duly organized and validly existing and in good standing under the laws of the State of Maryland, and (ii) has the corporate power and authority to own its property and assets and to transact the business in which it is engaged.

(b) The Series E Shares that are being issued by the Company to Cohen hereunder have been duly authorized, and, upon the issuance of the Series E Shares to Cohen in accordance with the terms and provisions of this Agreement, the Series E Shares will be validly issued, fully paid and nonassessable.

(c) The Company has the corporate power to execute, deliver and carry out the terms and provisions of this Agreement and to issue the Series E Shares to Cohen hereunder, and the Company has taken all necessary corporate action to authorize the Company’s execution, delivery and performance of this Agreement.

(d) This Agreement constitutes a legal, valid and binding obligation of the Company, enforceable against the Company in accordance with its terms.

3. Miscellaneous.

3.1 Termination of December 28, 2012 Exchange Agreement. The parties hereto agree that the Preferred Stock Exchange Agreement, dated December 28, 2012, by and among the Company, Cohen and Daniel G. Cohen, is hereby terminated in its entirety.

3.2 Descriptive Headings. The descriptive headings herein are inserted for convenience only and are not intended to be part of or to affect the meaning or interpretation of this Agreement.

2

3.3 Counterparts. This Agreement may be executed in one or more counterparts, each of which shall be deemed an original, but all of which, together, shall constitute one and the same agreement. Facsimile or electronically transmitted signature pages shall be deemed an original for purposes of this Agreement.

3.4 Entire Agreement. This Agreement constitutes the entire agreement and supersede all prior agreements and understandings, both written and oral, between the parties hereto with respect to the subject matter hereof.

3.5 Interpretation. In the event of an ambiguity or question of intent or interpretation arises, this Agreement shall be construed as if drafted jointly by the parties hereto and no presumption or burden of proof shall arise favoring or disfavoring any party hereto by virtue of the authorship of any provisions of this Agreement.

3.6 Successors and Assigns. This Agreement shall be binding upon and inure to the benefit of the successors and assigns of the parties hereto.

3.7 Severability. If any provisions of this Agreement shall be held invalid or unenforceable, such invalidity or unenforceability shall not affect any other provision hereof.

3.8 Governing Law. The validity, interpretation and enforcement of this Agreement and any dispute arising out of the relationship between the parties hereto, whether in contract, tort, equity or otherwise, shall be governed by the laws of the State of New York, but excluding any principles of conflicts of law or other rule of law that would cause the application of the law of any jurisdiction other than the laws of the State of New York.

[REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]

3

IN WITNESS WHEREOF, the undersigned have executed this Preferred Stock Exchange Agreement on the day and year first above-written.

[Signature page to Preferred Stock Exchange Agreement]