PROMISSORY NOTE

Exhibit 10.4

PROMISSORY NOTE

FOR VALUE RECEIVED, American Apparel (USA), LLC, a California limited liability company (the “Maker”), hereby promises to pay to the order of Dov Charney (the “Payee”), his successors and assigns, on or before the Maturity Date (as hereinafter defined), the principal sum of TWO MILLION FIVE HUNDRED THOUSAND DOLLARS ($2,500,000), in lawful money of the United States of America, together with accrued and unpaid interest hereon from the date hereof plus all other unpaid amounts to be paid or reimbursed by the Maker pursuant to this Note, in accordance with the terms hereinafter set forth.

1. Interest Rate. The outstanding principal amount of this Note, together with all accrued and unpaid interest hereon, shall bear interest at 6.00% per annum (the “Base Rate”). Interest shall be payable quarterly in arrears on March 31, June 30, September 30 and December 31 of each year, commencing on March 31, 2009. Following the occurrence and during the continuance of an Event of Default, Maker further promises to pay interest on the outstanding balance under this Note at a rate of interest (the “Default Rate”) equal to the Base Rate plus two percent (2.0%) per annum, which amount shall accrue from the date of such Event of Default until the earlier of the date on which all amounts due under this Note shall have been paid in full or the date on which such Event of Default is cured or waived, and shall be paid in kind in accordance with the terms hereof (“Default Interest”). Except as otherwise provided herein, interest shall be calculated on the basis of a 360-day year of twelve 30-day months.

All interest that accrues on this Note shall be paid in kind and added to the outstanding principal amount of this Note. Interest that is paid in kind and added to the outstanding principal amount of this Note, together with any Default Interest accruing hereunder from time to time, is collectively referred to herein as “Additional PIK Principal”. Additional PIK Principal shall be added to the principal amount of this Note, and shall be considered principal for all purposes under this Note, and without limiting the foregoing, shall bear interest at the Base Rate as provided herein, beginning on the date such interest is paid in kind and the Additional PIK Principal it reflects is added to the principal amount hereof.

2. Maturity Date. The principal amount of this Note (or, if less, the unpaid principal balance of such amount) together with all accrued and unpaid interest hereon plus all other unpaid amounts to be paid or reimbursed by the Maker hereunder shall be due and payable by the Maker in full on January 2, 2013 (the “Maturity Date”).

3. Application of Payments; Overdue Payments. Each payment on this Note shall be credited first to the payment of all fees, costs and expenses for which the Payee is entitled to reimbursement pursuant to Section 10 hereof, then to accrued but unpaid interest, and then to the principal. All payments of the unpaid principal balance and interest will be made without withholding or deduction for or on account of any present or future taxes, duties, assessments or governmental charges of whatever nature, unless the withholding of such taxes or duties is required by law.

4. Manner of Payment. All payments hereunder shall be made in immediately available funds to the Payee or to such other payee or address as the Payee may designate from time to time. If any payment of principal or interest on this Note is due on a day which is not a Business Day, as

1

defined below, such payment shall be due on the next succeeding Business Day, and such extension of time shall not be taken into account in calculating the amount of interest payable under this Note. “Business Day” means any day, other than a Saturday or Sunday, that is neither a legal holiday nor a day on which banking institutions in New York are authorized or required by law, regulation or executive order to close.

5. Prepayment. Unless such prepayment is then prohibited to be made pursuant to the terms of the Amended and Restated Subordination Agreement, dated as of December 19, 2008 (as amended, restated, amended and restated, supplemented, modified or replaced from time to time, the “BOA Subordination Agreement”), among the Holder, Maker and Bank of America, N.A. (successor by merger to LaSalle Business Credit, LLC, as Agent for LaSalle Bank Midwest National Association, acting through its division, LaSalle Retail Finance), as Administrative Agent or the Amended and Restated Subordinated Agreement, dated as of December 19, 2008 (as amended, restated, amended and restated, supplemented, modified or replaced from time to time, the “SOF Subordination Agreement” and, together with the BOA Subordination Agreement, the “Subordination Agreements”), among the Holder, Maker and SOF Investments, L.P. – Private IV, the Maker shall have the right to prepay the principal amount hereof in full or in part, together with all accrued interest on the amount prepaid to the date of such prepayment, at any time and from time to time (the “Prepayment Right”). Any prepayment amount received by the Payee in connection with the Prepayment Right shall be applied first to the payment of all fees, costs and expenses for which the Payee is entitled to reimbursement pursuant to Section 10 hereof, then to accrued but unpaid interest on this Note through the date of such prepayment, then to principal. Any such prepayment shall be payable without any premium or penalty of any kind.

6. Covenant. The Maker agrees that, until this Note shall have been repaid in full, the Maker shall pay the principal amount of and interest on this Note on the dates and in the manner provided herein.

7. Further Assurances. At the Maker’s reasonable expense, the Maker shall do all acts, furnish to the Payee all agreements, consents, instruments or other documents, and do or cause to be done all such other things as the Payee may reasonably request from time to time in order to give full effect to the purpose and provisions of this Note. If the Maker fails to perform any act required by this Note, the Payee may perform, or cause performance of, such act, and the expenses of the Payee therewith shall be reimbursed upon demand by the Maker in accordance with Section 10 hereof.

8. Events of Default. Each of the following events shall constitute an “Event of Default” hereunder (whether it shall be voluntary or involuntary or occur or be effected by operation of law or otherwise):

(a) a failure on the part of the Maker to make any payment of principal when due under this Note (except to the extent such payment is then prohibited to be made pursuant to the terms of either Subordination Agreement);

(b) a failure on the part of the Maker to make any payment of interest when due under this Note and such failure shall continue for 30 days (except to the extent such payment is then prohibited to be made pursuant to the terms of either Subordination Agreement);

(c) the Maker shall commence (or take any action for the purpose of commencing) any proceeding under any bankruptcy, insolvency, readjustment of debt, moratorium or similar law or statute or make (or take any action for the purpose of making) a general assignment for the benefit of its creditors or shall admit in writing its inability to pay its debts generally as they become due;

2

(d) a proceeding shall be commenced against the Maker under any bankruptcy, insolvency, readjustment of debt, moratorium or similar law or statute and relief is ordered against it, or the proceeding is controverted but is not dismissed within thirty (30) days after the commencement thereof;

(e) the Maker consents to or suffers the appointment of a receiver, trustee or custodian to any substantial part of its assets that is not vacated within thirty (30) days after such appointment; or

(f) any event occurs in relation to the Maker which under the law of any relevant jurisdiction has an analogous or equivalent effect to any of the events mentioned in clause (c), (d) or (e) of this Section 8.

The Maker shall notify the Payee promptly (but in no event later than five (5) Business Days following the occurrence of any Event of Default) in writing of the occurrence of such Event of Default.

9. Remedies.

(a) If an Event of Default shall occur and be continuing, then the Payee may, by notice in writing to the Maker, (i) declare the then correct principal of and accrued interest on this Note to be, and the same shall upon such notice forthwith become, immediately due and payable and (ii) in addition to all rights and remedies available to it at law, exercise any or all of its rights and remedies under this Note; provided, however, that when any Event of Default described in clause (c), (d), (e) or (f) of Section 8 has occurred and is continuing, then all amounts in respect of principal and interest under this Note shall immediately become due and payable, without presentment, demand of payment, protest, notice of intent to accelerate, notice of acceleration or notice of any kind, all of which are hereby expressly waived.

(b) No failure or delay by the Payee in exercising any remedy, right, power or privilege under this Note shall operate as a waiver of such remedy, right, power or privilege, nor shall any single or partial exercise of such remedy, right, power or privilege preclude any other or further exercise of such remedy, right, power or privilege. No remedy, right, power or privilege conferred upon or reserved to the Payee by this Note is intended to be exclusive of any other remedy, right, power or privilege provided or permitted by law, but each shall be cumulative and in addition to every other remedy, right, power or privilege so provided or permitted and each may be exercised concurrently or independently from time to time and as often as may be deemed expedient by the Payee.

10. Expenses. From time to time, the Maker shall reimburse the Payee upon demand for all reasonable fees, costs and expenses (including the fees, costs and expenses of counsel and court costs) incurred after the date of this Note in connection with (a) the collection of any outstanding principal and interest accrued hereunder (whether or not suit is filed to enforce the terms hereof) and (b) the enforcement of any rights or remedies provided for pursuant to this Note. The provisions of this Section 10 shall survive the termination of this Note.

11. Waivers. The Maker hereby waives diligence, presentment, demand of payment, protest, notice of intent to accelerate, notice of acceleration and all other notices and demands whatsoever with respect to such liabilities and obligations or to any action under this Note, except as specifically provided for in this Note. The Maker further agrees that it shall remain liable for all amounts due hereunder notwithstanding any extension of time or any change in the terms of payment of this Note granted by the Payee or any delay or failure by the Payee to exercise any rights hereunder.

3

12. Governing Law; Severability. This Note shall be governed by and construed in accordance with the internal laws of the State of New York, including, without limitation, Section 5-1401 of the New York General Obligations Law. All claims and proceedings arising from or related to this Note, shall be adjudicated solely by the federal or state courts of New York, and the parties hereby attorn to the jurisdiction thereof. Whenever possible, each provision of this Note shall be interpreted in such manner as to be effective and valid under applicable law, but if any provision hereof is in conflict with any applicable law or is otherwise unenforceable for any reason whatsoever, then such invalidity, illegality or unenforceability shall not affect any other provision hereof, and this Note shall be construed as if such invalid, illegal or unenforceable provision had never been contained herein.

13. Amendment; Waiver. No amendment to this Note shall be effective unless in writing signed by the Maker and the Payee. No waiver of any provision of this Note and no consent to any departure by the Maker hereunder shall be effective unless in writing signed by the Payee.

14. Registration. The Maker will maintain, at its principle place of business, a register of the Payee(s) of this Note (the “Register”), and will update the Register to reflect any permitted assignments or transfers subsequent to the date hereof. The Maker will make payments of principal and interest as specified hereunder to the Payee(s) named as such in the Register. The Payee shall notify the Maker in writing prior to any assignment, transfer or other disposition of this Note (or any portion hereof) or such Payee’s rights or interests hereunder, with such written notice to be delivered to the Maker not later than one (1) Business Day prior to any such assignment, transfer or disposition and which notice shall specify the principal amount hereunder that is the subject of such assignment, transfer or disposition. Notwithstanding anything to the contrary herein, (i) the registered owner of this Note (or any portion hereof) as indicated on the Register shall be the party with the exclusive right to receive payment of any principal amount and accrued and unpaid interest on this Note and (ii) payments required to be made hereunder shall not be subject to counter-claim or offset of any nature whatsoever.

15. Loss, Mutilation, Etc. Upon notice from the Payee to the Maker of the loss, theft, destruction or mutilation of this Note, and upon receipt of an indemnity reasonably satisfactory to the Maker from the Payee or, in the case of mutilation hereof, upon surrender of the mutilated Note, the Maker will make and deliver a new note of like tenor in lieu of this Note.

16. Notices. All notices, consents, waivers or other communications required or permitted hereunder to be delivered to the Maker shall be in writing and shall be mailed by registered or certified mail, return receipt requested, postage prepaid or otherwise delivered by hand, by messenger or an internationally recognized courier (charges prepaid) or facsimile transmission, addressed to the Maker at 747 Warehouse St., Los Angeles, CA 90021. Each such notice, consent, waiver or other communication shall for all purposes hereunder be treated as effective or as having been given when delivered, if delivered by hand or by messenger (or internationally recognized courier), 24 hours after confirmed receipt if sent by facsimile transmission or at the earlier of its receipt or on the second (2^nd) Business Day after mailing, if mailed, as aforesaid.

17. Subordination Agreements. Each of the Maker and the Holder (by the Holder’s acceptance hereof) expressly acknowledges and agrees that the rights and obligations of the Maker and the Holder hereunder are subject to the terms and conditions of the Subordination Agreements. Notwithstanding the foregoing, in the event of any conflict between the terms of this Note and any Subordination Agreement, the terms of such Subordination Agreement shall govern, and in the event of any conflict between the terms of the BOA Subordination Agreement and the SOF Subordination Agreement, the terms of the BOA Subordination Agreement shall govern.

4

IN WITNESS WHEREOF, this Note has been executed and delivered by the undersigned on the 19th day of December, 2008.

5