A 9.8 AFR is way fat and will kill the rings. I beleave what ever afr makes max power on the motor is what you want to shot for on nitrous. It is best to look at the plugs.

If the afr is decent and it is still breaking up it could be bad wires or a weak coil.

 

Something weird on my rx7 that has never happened on any of my nitrous setups before is the car will break up if the motor is cold(below 170 degrees), but above that it runs fine. Weirdest thing I've seen, but I just get the motor up to around 185-190 degrees now and no problems.

 

I believe that in the instuctions that came with my kit it says to make sure the motor is up to temp before spraying. I don't think that's the OPs problem, just thought I'd throw that info out there.