The 68-year-old premier, under pressure from the opposition to call snap elections, has come under fire for frequent evening visits to hotel bars at a time when fears of recession are mounting.
The TBS poll showed 61 per cent of respondents want him to cut back on such excursions, and 43 per cent do not expect much from his policies.
Aso took office with a mission to lead the ruling Liberal Democratic Party (LDP) to victory in elections which must take place by September 2009.
He has put off high-risk general elections for now because of the global financial crisis, saying a bigger priority is to tackle an economic slump by pushing through a $US300 billion ($438.5 billion) fiscal stimulus package.
The flamboyant premier has tried to boost his popularity by demonstrating his down-to-earth side to voters.
Last night, he turned up in a Tokyo pub where he chatted to university students over a beer - before heading to a hotel bar.
Since taking office, Mr Aso has regularly gone out for dinner and drinks at restaurants and bars of luxury hotels in downtown Tokyo with his close friends and aides.
"Fortunately, I have money. I pay by myself,'' Mr Aso said.
Japan has had four prime ministers in the past two years amid a series of scandals and legislative deadlocks.