As he increasingly pushes his Justice Department to protect voting rights and end unfair prison sentences and police brutality, Holder has drawn on personal history to make the case that the nation has much work to do to achieve justice for all. It's a legacy he'll likely draw on when he travels Wednesday to Ferguson, Missouri, to supervise the federal investigation of the fatal shooting of a black 18-year-old by a white police officer.
Holder tells how his father, an immigrant from Barbados proudly wearing his World War II uniform, was ejected from a whites-only train car. How his future sister-in-law, escorted by U.S. marshals, integrated the University of Alabama in spite of a governor who stood in the schoolhouse door to block her. How as a college student, he was twice pulled over, his car searched, even though he wasn't speeding.
And Holder recalls that the slaying of black teen Trayvon Martin in 2012 prompted him to sit down with his own 15-year-old son for a talk about the way a young black male must act and speak if confronted by police — the same talk his father had given him decades earlier.
"I had to do this to protect my boy," the nation's first black attorney general said at an NAACP convention last year.
President Barack Obama is sending Holder to Ferguson to bring the full weight of the federal government into the investigation of the death of another young black man, Michael Brown, who was unarmed when a white police officer shot him multiple times Aug. 9. Daily and nightly protests, sometimes marred by rioting and looting and met with tear gas and rubber-coated bullets from police, have rocked the suburban St. Louis community since.
Holder has led an unusually fast and aggressive Justice Department response to the local case, sending teams of prosecutors and dozens of FBI agents to investigate and arranging a federal autopsy on top of one by local authorities.
Still, protesters in the streets say they aren't convinced justice will be done. Holder's record on civil rights and personal commitment may help reassure the community when he visits.
"It's a powerful message," said William Yeomans, a law school fellow at American University who worked in the Justice Department's Civil Rights Division for more than two decades. "He's the embodiment of law enforcement, and the positive contribution he can make here is to assure the community that the federal government is taking very seriously the quest for justice in this incident."
Holder reinvigorated a civil rights force at Justice, Yeomans said, that had been scaled back and demoralized during President George W. Bush's administration.
Holder's department has been especially strong in going after police misconduct, both through criminal civil rights cases and lawsuits against police departments, Yeomans said.
His civil rights push got off to a difficult start, however.
Shortly after taking office in February 2009, Holder called the United States "a nation of cowards" when it comes to talking about race in a Black History Month speech. Conservative backlash was swift. Holder quickly toned down his rhetoric while quietly rebuilding the division.
For much of Holder's early tenure, his public profile was shaped by battles over how to prosecute terror cases, the use of armed drones to kill terror suspects overseas and his handling of various Obama administration controversies. A 2012 vote in the Republican-controlled House made Holder the first sitting Cabinet member ever held in contempt of Congress over his refusal to hand over without preconditions documents involving the Fast and Furious gun investigation.
More than a dozen Republican lawmakers have called for his impeachment for not prosecuting anyone in the Internal Revenue Service for targeting conservative groups and for his department's probes of journalists linked to news leaks.
But over the last three years, civil rights has moved to the forefront, starting with Holder's opposition to state voter ID laws that make it harder for the poor to cast ballots. He compared Texas' voter ID law to a poll tax, the now-illegal fees imposed across the South for decades to block African-Americans from voting. The Justice Department is now suing Texas and North Carolina over their voting restrictions.
"Over the past three years, the department's Civil Rights Division has filed more criminal civil rights cases than during any other period in its history - including record numbers of hate-crimes cases," Holder noted in April.
Holder has indicated he's unlikely to stay on as attorney general through the end of Obama's second term, but says he has more to accomplish before departing. That may partly explain his accelerated push for equal treatment under the law.
He has worked on easing mandatory sentences, especially for nonviolent drug offenses that have a disproportionate impact on black men.
Holder ordered his prosecutors to stop charging many drug defendants with offenses that carry mandatory minimum sentences, pushed a U.S. Sentencing Commission proposal to lower guideline penalties and backed legislation to give judges more discretion in sentencing.
"This focused reliance on incarceration is not just financially unsustainable," he said in March, "it comes with human and moral costs that are impossible to calculate."
He used the Martin killing two years ago to criticize "stand your ground" gun laws in states like Florida. The Justice Department is investigating the 17-year-old's death but has yet to say if it will file federal civil rights charges against George Zimmerman, the neighborhood watch volunteer who said he killed Martin in self-defense and was acquitted in a state trial.
Holder even partnered with the Education Department to try to change the so-called "school-to-prison pipeline," where minority children — especially black students — are suspended and expelled at a rate that's three times higher than that of white children.