RECITALS:

Execution Copy

Exhibit 10.20

AMENDED AND RESTATED UNIT PLEDGE AGREEMENT

THIS AMENDED AND RESTATED UNIT PLEDGE AGREEMENT (the “Agreement”) is made and entered into this 11th day of June 2004, by and between LANGUAGE LINE, INC., a Delaware corporation (“Secured Party”), and MATTHEW T. GIBBS, II and KATHY GIBBS (collectively, “Borrower”) and amends and restates the Stock Pledge Agreement dated 17 June 2002, (the “Prior Pledge”) by and between Language Line, LLC, a predecessor of the Secured Party (“Language Line”) and the Borrower, and the Prior Pledge is hereby amended and replaced in its entirety with this Agreement.

RECITALS:

A. On April 14, 2004, Language Line Holdings, Inc., a Delaware corporation (“Holdings Inc.”), entered into an Agreement and Plan of Merger (“Agreement and Plan of Merger”) with Language Line Acquisition, Inc., a Delaware corporation, pursuant to which Holdings Inc. merged with and into the Secured Party in accordance with the Agreement and Plan of Merger (the “Merger”).

B. In connection with the Merger, the Borrower has entered into an Amended and Restated Promissory Note secured by Unit Pledge Agreement dated as of the date hereof (as it may be amended or otherwise modified from time to time, the “New Note”) to evidence Borrower’s obligation to repay the sum of One Hundred Thousand Dollars $100,000 (the “Loan”). The New Note amends and restates the Promissory Note secured by Stock Pledge Agreement between the Borrower and Language Line dated June 17, 2002 (the “Prior Note”), and the Prior Note has been amended and replaced in its entirety by the New Note.

C. The Secured Party agreed to accept the New Note on the condition that the Borrower secure the New Note by pledging with Secured Party all of Borrower’s interest in 1,500,000 Class A Common Units and 11,927,273 Class C Common Units in Language Line Holdings, LLC (“Holdings LLC”), a Delaware limited liability company and the parent of the Secured Party (the “Units”).

NOW, THEREFORE, for good and valuable consideration the receipt and adequacy of which are hereby acknowledged, the parties hereto agree as follows.

1. Pledge.

(a) Borrower hereby assigns, transfers and pledges the Units to the Secured Party as security for payment of the New Note.

(b) On execution of this Agreement, the shares pledged under the Prior Pledge (the “Shares”) are hereby released and all Share certificates, assignments or other documents of transfer relating to the Shares and provided under the Prior Pledge will be returned to the Borrower by the Secured Party.

(c) Borrower agrees that he will deposit with the Secured Party pursuant to the provisions of Section 8313(a) of the Commercial Code of the State of California, the certificates representing the Units with two executed Unit assignments (with date blank) for each

certificate, accompanied by such documents of transfer as may be necessary to authorize the Secured Party or its transfer agent to transfer the Units to the Secured Party if required to do so by the provisions of this Agreement, it being understood that the Units are, as of the date hereof, uncertified. Such documents are to be held by the Secured Party for the benefit of both Secured Party and Debtor during the period of this Agreement.

(d) Borrower shall have the right to execute all unit rights and rights to subscribe, and to receive all liquidating dividends, cash dividends, units, new securities or other rights or property which the Borrower is or may hereafter become entitled to receive on account of the Units pledged hereunder; provided, however, that in the event the Borrower receives any such property, other than cash dividends, he will immediately deliver such property to the Secured Party to be held as collateral in the same manner as the Units originally pledged hereunder. As used in this Agreement, the term “Units” refers to all the Units assigned, transferred, and pledged hereunder, and all other property received in respect thereof, other than cash dividends.

(e) Borrower, at his option, may transfer to the Secured Party upon execution of this Agreement (or as soon thereafter as practicable), collateral other than the Units (“Substitute Collateral”), which shall be acceptable in form to the Secured Party and adequate to secure part or all of Borrower’s obligations under the New Note, in lieu of part or all of the Units, and shall thereupon be entitled to retain, free from the pledge hereunder, an amount of Units having a fair market value equivalent, in the judgment of the Secured Party’s Board of Directors, to the value of the Substitute Collateral, taking into account fluctuations in the value of the Substitute Collateral over the term of the New Note and the Secured Party’s need to have the New Note fully secured. Borrower must maintain the Substitute Collateral at a value equal to the aggregate purchase price of the Units for which it serves as substitute Collateral. The Secured Party shall have sole discretion to determine the value of Substitute Collateral at all times. Borrower shall pledge such additional Substitute Collateral as the Secured Party deems necessary to adequately secure the New Note promptly upon receipt of a written demand to do so by the Secured Party. All Substitute Collateral and additions thereto shall be deemed transferred to the Secured Party at the time the original collateral (for which it serves as substitute) was transferred to the Secured Party. Borrower agrees to take all actions, execute all instruments, agreements and notices and do all other things necessary for the Secured Party to perfect it Security interest in the Substitute Collateral and all additions thereto whenever requested by the Secured Party.

(f) In the event that Holdings LLC is involved in a merger reorganization, exchange reorganization, sale-of-assets reorganization or other event requiring the transfer of a part or all of the Units, Borrower shall, within ten days after demand by the Secured Party, execute any documents necessary to insure the continued secured status of the New Note by the Units, any securities or property issued in respect thereto and the Substitute Collateral.

(g) As used in this Agreement, the term “Collateral” refers to the Units and/or the Substitute Collateral.

2. Rights in the Collateral. Unless and until the ownership of the Collateral is transferred to the Secured Party pursuant to the provisions hereof, the Secured Party shall collect

2

and receive all property, other than cash dividends distributed in respect of the Units and other than rents or interest payable with respect to the Substitute Collateral. The Secured Party shall hold the same as collateral under this Agreement. Borrower shall retain all incidents of ownership in the Collateral not specifically limited herein and not in derogation of the Secured Party’s security interest in the Collateral, including the right to vote the Units or other Unit held as Collateral, the right to lease any real property used as Substitute Collateral, subject to the terms of this Agreement, the right to receive all notices sent with respect to the Collateral, and the right to grant subordinate secured interests in the Collateral with the Secured Party’s prior written consent, which may be withheld for any reason.

3. Taxes, Charges and Expenses.

(a) Borrower agrees to pay, prior to delinquency, all taxes, charges, liens and assessments against the Collateral. In the event Borrower fails to make any such payment, the Secured party may at its option pay any such charges and shall be the sole judge of the legality or validity thereof and the amount necessary to discharge the same.

(b) Borrower will defend the Collateral against any and all claims and demands of all persons at any time claiming an interest therein.

(c) All advances, charges, taxes, assessments, costs and expenses, including reasonable attorneys’ fees, incurred or paid by the Secured Party in exercising any right, power or remedy conferred by this Agreement, or any enforcement thereof, or to preserve the value of the Collateral, shall become a part of the indebtedness secured hereunder and shall be paid to the Secured Party by Borrower immediately upon demand.

4. Default. The occurrence of any of the following shall be a default under this Agreement:

(a) Borrower fails to make payment when due of any part or installment of principal or interest, and such default is not cured within ten (10) days of the Secured Party’s giving notice of such default to Borrower;

(b) Borrower becomes insolvent in that either a petition is filed by or against Borrower under any bankruptcy law, or he is unable to pay his debts as they fall due, or he makes a general assignment for the benefit of his creditors or takes any other action to take advantage of any insolvency laws;

(c) Borrower fails to perform any of his obligations or to comply with any of the terms under the New Note; or

(d) Borrower is in default under or fails to comply with the provisions of any agreement, instrument, decree, judgment, order, obligation, covenant, bond, lien, encumbrance, security interest, article of incorporation or bylaw pertaining to the Collateral or affecting Borrower’s or the Secured Party’s rights in the Collateral.

3

5. Remedies of Secured Party.

(a) Should any default, as provided in paragraph 4 above, continue for a period of five (5) days or more following written notice from Secured Party of the default, the New Note shall become immediately due and payable at the option of the Secured Party, without notice or demand, the Secured Party shall have the right to take possession and proceed against the Collateral in accordance with this Agreement or the New Note, and the Secured Party shall have all the rights and remedies provided by law, particularly the provisions of the Commercial Code of the State of California, Investment Securities and Secured Transactions.

(b) Borrower waives the benefit of any statute of limitations affecting his liability under this Agreement or the New Note, or the enforcement thereof, and agrees that any payment of any indebtedness or other act which shall toll any statute of limitations applicable thereto shall similarly operate to toll such statute of limitations applicable to this Agreement or the New Note. Borrower waives all presentments, demands for performance, notices of non-performance, protests, notices of protest, notices of dishonor and notices of acceptance of this Agreement or the New Note, with respect to any default and liability under this Agreement and the New Note.

(c) Should the Secured Party proceed against all or any part of the Collateral, it may proceed to do so by sale, public or private, and in the market or in private or negotiated sale or sales, and subject to such terms and conditions, all as the Secured Party in its sole discretion deems proper; provided, however, that should the Secured Party purchase all or part of the Collateral at a private sale, it is expressly agreed by Borrower that fair market value of the Collateral may be established by the Secured Party using the most recent sales price for Units of the same Class. It is agreed and understood that sale of the Units under investment letter is a commercially reasonable disposition. The aggregate proceeds of such sale or sales shall be applied by the Secured Party as follows:

(i) The Secured Party shall first pay itself all reasonable costs and expenses of preparing for and conducting such sale or sales, including without limitation its legal expenses and fees incurred;

(ii) The unpaid balance of the New Note plus seven and one-half percent (7.5%) per annum simple interest on such balance for the period between default on the New Note and the date the Secured Party consummates the sale, shall be paid to the Secured Party;

(iii) Any further balance shall be applied to other indebtedness, if any, then owing from Borrower to the Secured Party; and

(iv) The remaining balance, if any, after application of items (i), (ii) and (iii) above shall be paid and set over to Borrower.

6. Release of Collateral. The Secured Party shall release the Collateral from this pledge upon the payment by the Borrower to Secured Party of the full amount owing under the New Note as therein provided.

7. Non-Waiver. The rights, powers and remedies given to the Secured Party by this Agreement will be in addition to all rights, powers and remedies given the Secured Party by

4

virtue of any statute or rule or law. The Secured Party shall have the right to enforce one or more of such remedies, successively or concurrently, and any action to enforce the same shall not bar the Secured Party from pursuing any further remedy which it may have hereunder, under the New Note, or otherwise as provided by law, including, without limitation, the absolute right on the part of the Secured Party to commence an action against Borrower or his spouse for a judgment in the amount of all sums due and collectible under this Agreement and the New Note without first foreclosing on the security interest granted hereunder and selling the Collateral. Any forbearance, failure or delay by the Secured Party in the exercise of any right, power or remedy hereunder, or under the New Note, shall not be deemed to be a waiver of such right, power or remedy and any single or partial exercise of any right, power or remedy shall not preclude the further exercise thereof. Every right, power and remedy of the Secured Party shall continue in full force and effect until the same is specifically waived by an instrument in writing executed by the Secured Party.

8. Binding Effect. The rights and remedies of this Agreement shall inure to the benefit of, and be binding upon, the heirs, successors and assigns of the parties. Borrower agrees that the Secured Party can assign its security interest hereunder and all its rights, including its rights to receive payment, under the New Note to any natural person or entity. In the event of such assignment, Borrower agrees that he will not assert against the assignee any claim or defense which he may have against the Secured Party if the assignee takes such assignment for value, in good faith and without notice of such claim or defense.

9. Notices. Any notice, approval, disapproval, Consent, waiver or other communication (“Notice”) required or permitted to be given under this Agreement shall be in writing and shall be delivered (a) personally, or (b) mailed, certified or registered United States mail, postage prepaid, return receipt requested, or (c) sent by overnight courier, or (d) sent by telecopier or facsimile. All such Notices shall be deemed given (i) if personally served or sent by overnight courier, when delivered to the person to whom such Notice is addressed, or (ii) if by mail, three (3) days following deposit in the United States mail, or (iii) if by telecopier or facsimile, when the sender’s telecopy machine has confirmed that it was received. All Notices shall be addressed to the party to whom such Notice is to be given at the party’s address (or telecopier number, as the case may be) set forth below or as such party shall otherwise direct by Notice sent pursuant to this section:

10. Attorneys’ Fees. In the event of any action at law or in equity between Borrower and Secured Party arising out of Borrower’s responsibilities under this Agreement, the

5

unsuccessful party to such action shall pay to the prevailing party all costs and reasonable attorneys’ fees incurred by the prevailing party.

11. Governing Law; Terms. This Security Agreement shall be governed by and construed in accordance with the laws of the State of California.

6

IN WITNESS WHEREOF, this AMENDED AND RESTATED UNIT PLEDGE AGREEMENT has been executed at Monterey, California, on the date first above written.

7