The pullback from Bajaur, an area of Pakistan's tribal region where the Taliban and al-Qaida have forged close ties, came after the military began an offensive there late last week.
The military said six soldiers were killed; the Pakistani Taliban put the number at 22. It was unclear how many civilians died.
The clash was the second in two weeks between government forces and the Taliban. The army has been trying to push the Taliban out of Swat, an area east of the tribal region, where a 2-month-old peace deal between the government of the North-West Frontier province and the Taliban is in shreds.
There was speculation among Pakistanis that the sudden offensive in Bajaur was aimed at satisfying the Bush administration, which has criticized Pakistan for doing too little to stop Taliban fighters from crossing into Afghanistan to attack American soldiers.
Never miss a local story.
The Frontier Corps, a paramilitary force commanded by the Pakistani army, tried to take back a strategic military post in Bajaur that the Taliban had captured last winter.
The post, Loe Sam, is 10 miles from Damadola, a Pakistani town on the border, which the U.S. bombed in January 2006 in the belief that it would hit Ayman al-Zawahri, the al-Qaida deputy. The strike set off protests across Pakistan. Loe Sam has strategic significance because it provides access to a pass that leads to Kunar province in Afghanistan.
The area is used as an operating base by Faqir Muhammad, a senior member of the Pakistani Taliban umbrella group, Tehrik-i-Taliban. Muhammad is second in command to group leader Baitullah Mehsud.
The military used airstrikes to protect soldiers as they retreated to Khar, the capital of Bajaur, said Mahmood Shah, a retired brigadier of the Pakistani army who until 2006 was in charge of security in the tribal areas.