India's Home Minister Amit Shah said he was "anguished" at the news of civilians being killed in the incident late on Saturday night.
Nagaland's chief minister Neiphiu Rio told Reuters a probe will be conducted and the guilty punished in the incident, which he ascribed to intelligence failure.
At least a dozen civilians and some members of the security forces were injured in the attack, said a federal defence ministry official based in New Delhi.
Locals in Nagaland have frequently accused forces of wrongly targeting innocent locals in their counterinsurgency operations against rebel groups.
The incident took place in and around Oting village in Mon district, bordering Myanmar, during a counterinsurgency operation conducted by members of the Assam Rifles, a part of Indian security forces deployment in the state, said a senior police official based in Nagaland.
Firing began when a truck carrying 30 or more coal-mine labourers were passing the Assam Rifles camp area.
"The troopers had intelligence inputs about some militant movement in the area and on seeing the truck they mistook the miners to be rebels and opened fire killing six labourers," the senior police official told Reuters, requesting anonymity as he is not authorised to speak with journalists.
"After the news of firing spread in the village, hundreds of tribal people surrounded the camp. They burnt Assam Rifles vehicles and clashed with the troopers using crude weapons," he said.
Members of the Assam Rifles retaliated, and in the second attack eight more civilians and a security member were among those killed, the official said.
In recent years India has tried to persuade Myanmar to evict rebels from bases in the thick jungles of the region, which borders Nagaland, Manipur and Arunachal Pradesh