Actually, Dolly Madison almost starved to death, after James died. Paul Jennings a former slave who had been freed by the Madison's, gave Dolly food and money so she would not starve to death. This is even though Dolly had once owned Paul Jennings.
As I said before, James Madison almost bankrupted himself, trying to support 100 slaves, 2/3 of whom were too old, too young or too infirm to work.
Madison, unlike Jefferson, was not an extravangant spender, so you cannot blame Madison for what happened.
Will of President Madison
I, James Madison of Orange County do make this my last will and testament, hereby revoking all wills by me heretofore made.
http://www.jamesmadisonmus.org/textpages/will.htm