IS said it was part of its recently declared campaign to "avenge" the death of its leader Abu Bakr al-Baghdadi during a US raid in Syria in October.
No details were given about the victims, who were all male, but IS says they were "captured in the past weeks" in Nigeria's north-eastern Borno State.
The 56-second video was produced by the IS "news agency" Amaq.
It was released on 26 December and analysts say it was clearly timed to coincide with Christmas celebrations.
President Muhammadu Buhari has condemned the apparent killings and urged Nigerians not to let themselves be divided along religious lines.
"We should, under no circumstance, let the terrorists divide us by turning Christians against Muslims because these barbaric killers don't represent Islam and millions of other law-abiding Muslims around the world," he said in a statement.
He did not mention the identity of the victims.
The aid agencies whose staff are being held by the "Islamic State West Africa Province" (Iswap) have not yet commented.
What do we see in the video?
The footage was filmed in an unidentified outdoor area.
One captive in the middle is shot dead while the other 10 are pushed to the ground and beheaded.
"We killed them as revenge for the killing of our leaders, including Abu Bakr al-Baghdadi and [IS spokesman] Abul-Hasan al-Muhajir," said a member of the group's media unit, according to Ahmad Salkida, a journalist who was first sent the video.
How active is Islamic State in Nigeria?
One faction of Nigerian Islamist group Boko Haram broke away in 2016 and now fights under the banner of Iswap.
Earlier this month, the group said it had killed four aid workers it had abducted in north-eastern Nigeria.
Last year, Iswap killed two midwives it had previously taken hostage.
It is also active in neighbouring countries, including Burkina Faso, Cameroon, Chad, Niger and Mali.
The video once again highlights the brutal tactics of Iswap, while the timing of the release - over the Christmas period - is also designed to get maximum attention.
This video may be intended to increase tension between Christians and Muslims in Nigeria, as well as put more pressure on the government to respond to their demands, says security analyst Kabiru Adamu.
Earlier this month, the group released a video of the captives, appealing to the Nigerian authorities and the Christian Association of Nigeria (CAN) to intervene.
Iswap has used hostage-taking as a bargaining tool - either for ransom or in exchange for their arrested members, although the authorities have never confirmed carrying out a prisoner swap.
The militant group has previously killed a number of hostages, including members of the security forces and aid workers but this is the largest group to be killed at one time.
It is not clear how many captives Iswap is currently holding in Nigeria, but there are believed to be dozens - mainly security forces, aid workers and those perceived to be associated with government institutions.