Johnson, acquired in a trade with the Baltimore Orioles, was 4-2 with a 6.92 ERA in 38 games with the A's.
Brought in to replace Grant Balfour as the team's closer, Johnson faltered early, blowing his first two chances before losing his job.
He allowed four runs on three hits without retiring a batter in Wednesday night's 9-7 victory over the Astros.
Johnson led the AL in saves in each of the past two seasons, with 50 last year and 51 in 2012.
Evan Scribner was recalled to replace Johnson on the 25-man roster.