Rizwan and Irfan both suffered hamstring injuries in the third match in Rawalpindi, where a second-string New Zealand side won by seven wickets to level the five-match series 1-1
Boult (4-51) dismissed Khawaja (88), Mitchell Starc and Jason Behrendorff in successive deliveries to claim the tournament's second hat-trick after India's Mohammed Shami achieved the feat against Afghanistan.
Electing to bat, Australia had slumped to 92-5 in the 22nd over when Khawaja, who was dropped in the slip before he could open his account, and Carey (71) combined to arrest the slide.
Lockie Ferguson and fellow quick Jimmy Neesham claimed two wickets apiece after Boult had dismissed Aaron Finch to draw first blood.
Table-toppers Australia have already secured a place in the semi-finals with 12 points, one ahead of New Zealand, from seven games.