OTTAWA -- Prime Minister Stephen Harper is steadfastly refusing to acknowledge any question about his leadership during the months last winter when an expense scandal involving three of his Conservative Senate appointees was in the public eye.

Harper has spent the past three days insisting that he first learned that his chief of staff had helped pay off Sen. Mike Duffy's $90,000 in improper expense claims only after the public was told of the matter by CTV in mid-May.

But he has failed to address repeated questions about how he instructed his staff to handle the politically damaging expense revelations after they first aired late last fall.

Neither Harper nor his NDP and Liberal counterparts were in question period Thursday, so it fell to Heritage Minister James Moore to bear the brunt of the opposition onslaught.

"Duffy keeps the $90,000 while the Receiver General gets $90,000 from an illicit deal that was so wrong it cost Mr. Wright his job," said Liberal MP Ralph Goodale.

"Will the government repudiate the dirty money and instead garnishee Duffy's wages and seize his assets so he pays for his wrongdoing, not some deal-maker in the PMO?"

Moore pointed out the matter is in the hands of the RCMP, then chastised the Liberals for failing to address reports that Liberal Sen. Pana Merchant is the beneficiary of her husband's secret offshore bank account worth $1.7 million.

During an appearance earlier Thursday with Chilean President Sebastian Pinera, Harper was again asked what direction he gave his office to manage the political fallout when it first became public last year.

Harper said it is up to the Senate to investigate the matter, and that he learned only on May 15 that Duffy had not repaid his expenses from his own pocket.

Harper said the facts in the expense scandal are absolutely clear -- adding that the facts "are not good, but they are clear and they are simple."

The prime minister said Nigel Wright, his now-departed chief of staff, "decided to use his own personal money to assist Mr. Duffy to reimburse to reimburse the taxpayers of Canada."

"That's what he decided to do and he decided not to tell me until the 15th of May, after speculation about the source of funds appeared in the media."

There was also evidence Thursday of the bruising impact the ongoing scandal is having on the Conservative caucus.

"Power has slowly been transferred from (the House of Commons) first to the cabinet and then from the cabinet ultimately to the PMO. And I think it's a problem," said Brent Rathgeber, a Conservative backbencher with a reputation for being blunt and straightforward.

The Conservatives were elected in part on a promise to change that, Rathgeber acknowledged.

"That's why there's a lot of disappointment amongst our base and why we're all eagerly awaiting answers and a resolution to ensure that these types of incidents don't happen again."

Harper faced more than 40 direct questions on the affair in the House of Commons earlier this week but has not provided any information on who in his office was asked to handle the ongoing expense scandal, what instructions they were given, or why his chief of staff would take it upon himself to give $90,000 to a senator accused of improper housing expense claims.

Duffy stopped co-operating with an independent audit of his expenses once the $90,000 was paid back. Conservatives in the Senate cited that repayment in declaring the matter closed shortly after receiving the audit.

Duffy and Sen. Pamela Wallin are no longer part of the Conservative caucus, along with Sen. Patrick Brazeau, another Harper appointee accused of improper expense claims.

Sen. Mac Harb has left the Liberal caucus while fighting what he says are unfair accusations that he too abused his Senate housing allowance.