Abu Dujana, of the Lashkar-e-Taiba militant group, was gunned down with an accomplice in Pulwama district to the south of the region's summer capital Srinagar, police spokesman Manoj Pandita said.
Acting on a tipoff, police launched a search in the village of Hadripora in the early hours and killed the two in a fierce encounter, Pandita said.
Dujana was implicated in several militant attacks in South Kashmir and had a cash reward of 1.5 million rupees ($23,400) on his head.
Unrest has simmered in Kashmir since a popular militant leader was killed just over a year ago, sparking clashes that have left more than 90 civilians dead.
India has been struggling to restore normality in Muslim-majority Kashmir, which both India and Pakistan claim in full but rule in part.
India accuses Pakistan of infiltrating militants across a de facto border to carry out attacks, a charge denied by Islamabad. ($1 = 64.0825 Indian rupees)