In a written response included in the report, the IRS commissioner of the Tax Exempt and Government Entities Division said there was no criminal behavior behind the actions of the agents, but rather inefficient management.
"We believe the front-line career employees that made the decisions acted out of a desire for efficiency and not out of any political and partisan viewpoint," the commissioner wrote.
Among the recommendations made by the Treasury inspector general: The IRS must better document reasons why applications are chosen for review, develop a process to track requests for assistance, develop and provide training to employees before each election cycle and immediately resolve outstanding cases.
The report also called on Treasury to develop guidelines to explain social welfare activity -- the primary factor in obtaining tax-exempt status.
In the case of subpoenaed phone records, Holder said Wednesday that his decision to remove himself from the Justice Department investigation into a leak that led it to the surreptitious move left him unable to respond to questions about it.
"I don't know what happened there with the intersection between the AP and the Justice Department," Holder told the House Judiciary Committee. "I was recused from the case."
Asked about the AP case Thursday, Obama said he was unable to discuss specifics because of an ongoing investigation.
He added that he doesn't apologize for cracking down on leaks of classified information that can jeopardize U.S. troops and intelligence officers, but emphasized support for consideration of a new "shield law" for the media. The president also expressed full confidence in Holder.
Democratic Sen. Charles Schumer of New York previously introduced the proposed shield law that passed the Senate Judiciary Committee in 2009 but never advanced.
Federal shield legislation would protect journalists from revealing their sources and beef up protections for reporters and their sources caught up in such probes.
According to AP, the investigation into its records appears related to a story revealing that the CIA had thwarted an al Qaeda plot to blow up a U.S.-bound jetliner in May 2012, around the anniversary of the killing of Osama bin Laden.
The seized phone records covered a two-month period beginning in May 2012 and included more than 20 AP lines, including personal phones and AP phone numbers in New York; Hartford, Connecticut; and Washington.
"A shield law would keep lazy prosecutors from going after reporters' notes and phone records and compel them to actually conduct investigations that do not step all over the First Amendment," Teri Hayt, the First Amendment chairwoman of the Associated Press Media Editors, said in a statement issued before the White House announcement.
In the past decade, Congress has come close to passing a federal shield law. But support for the measure shrank during the WikiLeaks scandal in which thousands of classified U.S. diplomatic cables were released.
Obama has pledged transparency and openness since taking office, but legal analysts contend his administration's actions suggest otherwise.
For example, the Obama administration has used the Espionage Act, which was passed in 1917, to target suspected leakers in six cases, twice the number undertaken by all previous administrations combined.
In an example of Washington politics, many Republicans now criticizing the crackdown on AP have been the loudest advocates for stronger federal action against classified leaks.
The biggest pushback by Obama and the White House has been against a relentless Republican effort to vilify the administration's response to the Benghazi attack last September 11, particularly talking points used by U.N. Ambassador Susan Rice in the days that followed that erroneously contended the assault resulted from a demonstration over an anti-Islam film.
Republicans, who accuse the administration of failing bolster security prior to the attack and botching the response to it, contend the talking points amounted to misleading the public for political gain less than two months before the November election by removing references to a planned terrorist assault.
On Wednesday, the White House released more than 100 pages of e-mails in a bid to quell critics who say Obama and his aides played politics with national security.
The e-mails detail the complex back and forth between the CIA, State Department, and the White House in developing the unclassified talking points used by Rice.
Obama has called Republican concentration on the talking points a political "side show," and senior administration officials contend the e-mails demonstrate the process of developing the talking points was not focused on politics but rather on events.
The White House and its allies in Congress have made the case that any confusion and conflicting information in the early hours and days after the attacks stemmed from the "fog of war" -- not any deliberate effort to mislead the American people about the source of the attacks.
Analysis: CIA role in Benghazi underreported For instance, some of the e-mails expressed caution about what should be said publicly during an FBI investigation, while others focused on the strength of intelligence at the time.
The e-mails indicate the CIA was likely the lead organization in developing the talking points with the State Department recommending significant changes.
Rep. Darrell Issa, the chairman of the House Oversight Committee which is investigating the matter, told CNN's "Situtation Room" that his staff wants to digest the e-mails. He stressed that they were a selected set of documents as released and the committee is still seeking a range of other information.
However, a letter to Issa by the co-chairmen of an independent review of the Benghazi attack expressed irritation over his portrayal of the panel's work and requested a public hearing at which they can testify.
"The public deserves to hear your questions and our answers," wrote former U.S. ambassador, Thomas Pickering and retired chairman of the Joint Chiefs of Staff, Adm. Mike Mullen, who led the Accountability Review Board convened by Clinton to investigate the attack.
Eight months after their report cited "systemic failures and leadership and management deficiencies" at the State Department," Issa continues to be a leading critic of the accountability board, calling its review "a failure" and asking for further investigations into the Obama administration's response during the attack and its aftermath.