FIM Holdings LLC c/o Cerberus Capital Management, L.P. 299 Park Avenue New York, NY 10171

Exhibit 10.3

FIM Holdings LLC

c/o Cerberus Capital Management, L.P.

299 Park Avenue

New York, NY 10171

August 19, 2013

Ally Financial Inc.

1177 Avenue of the Americas, Floor 16

New York, New York 10036

Ladies and Gentlemen:

The purpose of this letter agreement is to set forth certain understandings and agreements between Ally Financial Inc. (“Ally”) and FIM Holdings LLC (“FIM”), relating to the Amended and Restated Governance Agreement, dated May 21, 2009 (the “Governance Agreement”) made by and among Ally, FIM, the United States Department of the Treasury (“Treasury”) and the other parties thereto.

In connection with the consummation of the private offering (the “Offering”) of not less than $1.0 billion of Ally’s common stock, par value $0.01 per share (“Common Stock”), the repurchase of Ally’s Fixed Rate Cumulative Mandatorily Convertible Preferred Stock, Series F-2 from Treasury and the termination of Treasury’s right to receive additional shares of Common Stock pursuant to Section 6(a)(i)(B) of Schedule A of the certificate of designations set forth as Exhibit H to Ally’s Amended and Restated Certificate of Incorporation, as amended, in each case as contemplated by the Agreement in Respect of Securities Repurchase and Share Adjustment Provision, dated as of the date hereof, by and between Treasury and Ally (collectively, the “Contemplated Transactions”), each of Ally and FIM hereby irrevocably relinquishes and surrenders all rights, privileges and powers afforded to it and waives and releases all obligations and duties owed or required to be performed under the Governance Agreement, including, without limitation, Sections 3(a), 3(b), 4(a), 4(b), 4(c), 4(d), 4(j) and 4(l) of the Governance Agreement.

This letter agreement shall be effective and binding only upon the satisfaction of the following conditions:

This letter agreement will be governed by and construed in accordance with the State of Delaware applicable to contracts made and to be performed entirely within such State. Each of the parties hereto agrees to submit to the exclusive jurisdictions and venue of the Court of Chancery of the State of Delaware (or, if the Court of Chancery of the State of Delaware lacks jurisdiction, then in the applicable Delaware state court), or if under applicable law exclusive jurisdiction of such suit, action or proceeding is vested in the federal courts, then the United States District Court for the District of Delaware, for any and all civil actions, suits or proceedings arising out of or relating to this letter agreement or the transactions contemplated hereby. This letter agreement may be executed by each of the parties hereto on separate counterparts.

[Signature Page Follows]

2

[Signature Page to Relinquishment Agreement]

UNITED STATES DEPARTMENT OF THE TREASURY

1500 PENNSYLVANIA AVENUE, N.W.

WASHINGTON, D.C. 20220

August 19, 2013

Ally Financial Inc.

1177 Avenue of the Americas, Floor 16

New York, New York 10036

Re: Governance Agreement

Ladies and Gentlemen:

The purpose of this letter agreement is to set forth certain understandings and agreements between Ally Financial Inc. (“Ally”) and the United States Department of the Treasury (“Treasury”), relating to the Amended and Restated Governance Agreement, dated May 21, 2009 (the “Governance Agreement”) made by and among Ally, Treasury, FIM Holdings LLC (“FIM”) and the other parties thereto.

In connection with the consummation of the private offering of not less than $1.0 billion of Ally’s common stock, par value $0.01 per share (“Common Stock”), the repurchase of Ally’s Fixed Rate Cumulative Mandatorily Convertible Preferred Stock, Series F-2 from Treasury and the termination of Treasury’s right to receive additional shares of Common Stock pursuant to Section 6(a)(i)(B) of Schedule A of the certificate of designations set forth as Exhibit H to Ally’s Amended and Restated Certificate of Incorporation, as amended, in each case as contemplated by the Agreement in Respect of Securities Repurchase and Share Adjustment Provision, dated as of the date hereof, by and between Treasury and Ally (collectively, the “Contemplated Transactions”), each of Ally and Treasury hereby irrevocably relinquishes and surrenders all rights, privileges and powers afforded to it and waives and releases all obligations and duties owed or required to be performed under the Governance Agreement, including, without limitation, Sections 3(a), 3(b), 4(a), 4(b), 4(c), 4(j) and 4(l) of the Governance Agreement.

This letter agreement shall be effective and binding only upon the satisfaction of the following conditions:

This letter agreement will be governed by and construed in accordance with the State of Delaware applicable to contracts made and to be performed entirely within such State. Each of the parties hereto agrees to submit to the exclusive jurisdictions and venue of the Court of Chancery of the State of Delaware (or, if the Court of Chancery of the State of Delaware lacks jurisdiction, then in the applicable Delaware state court), or if under applicable law exclusive jurisdiction of such suit, action or proceeding is vested in the federal courts, then the United States District Court for the District of Delaware, for any and all civil actions, suits or proceedings arising out of or relating to this letter agreement or the transactions contemplated hereby. This letter agreement may be executed by each of the parties hereto on separate counterparts.

[Signature Page Follows]

2

[Signature Page to Relinquishment Agreement]