Page 1 of 1

Dithering or Sigma Clipping not working?

Posted: Wed Mar 13, 2019 7:07 pm
by Astrogate
Hi Robin

Last night I was out doing some live stacking and noticed issues with either dithering, sigma clipping or both? I enclosed two images of M 64 with the white point lowered a lot in order to show the background flaws. One image is 26 x 30 secs and the other is 40 x 30 secs both at gain 350 with a matching master dark and flat under sever light pollution and brightness level at 10. I was using a C9.25 sct at f 7.2 and guiding. As you can clearly see in the images the hot pixels are not being removed and grouping in the bottom right corners. Also with the longer stack of 40 sub-images a satellite streak has developed on right side and at about 7 o'clock below M 64 a line of warm pixels has developed. I have dithering set to dither at 12 pixels for my imaging camera according to the image camera scale and guide camera scale. Should either using sigma clipping with ample sub-images, dithering or both combine remove at least these hot/warm pixels, cosmic rays and possibly the satellite streak when pixel scale is set to dither high enough?

Thank you!

Chris A

Re: Dithering or Sigma Clipping not working?

Posted: Wed Mar 13, 2019 8:55 pm
by admin
Hi Chris,

I'm not aware of any bugs in this area, but that's not to say that there couldn't be new bags that I haven't found yet.

You should be able to watch the progress of dithering if you keep an eye on the PHD2 screen while SharpCap is working – that will let you confirm whether the dithering commands are getting through to the guiding application or not.

The Sigma clipping algorithm is not active for about the first 10 frames that are captured in a stack as SharpCap has to have stacked a good number of frames in order to work out the statistics of variation at each pixel to be able to apply the Sigma clipping algorithm. So it is possible that the satellite streak came through early in the stack before the Sigma clipping algorithm could become active. If you're sure that's not the case please let me know. If you happen to have saved the raw frames during that stack then it might be interesting if you shared with me the one with the satellite streak in and about 15 to 20 other frames and I can see if I can reproduce the issue and work out where the bug is.

Cheers, Robin

Re: Dithering or Sigma Clipping not working?

Posted: Thu Mar 14, 2019 1:05 am
by Astrogate
Hi Robin

Thank you for the fast reply! I was watching PHD2 on and off and did notice the dithering commands coming from SharpCap to PHD2. Very good point about the satellite streak, but unfortunately I did not capture any raw's last night. Perhaps it was added to the 40 stacked towards the end of the live stacking. I am surprised though that in both images (26 and 40 stacked sub-images at 30 secs each sub) there is a blue and green group of hot pixels forming bottom right corner which I thought the dithering would have eliminated. Thank you for your help!

Chris A

Re: Dithering or Sigma Clipping not working?

Posted: Thu Mar 14, 2019 9:15 pm
by admin
Hi Chris,

Dithering will help spread around the effect of a hot pixel, but if you only dither a small number of times during the stack then you will still see the effects of hot pixels although they will not be as bright. I would usually imagine using dark subtraction to remove the worst effects of hot pixels and then dithering would help remove any pattern or raining nois by the dark subtraction.

Cheers, Robin

Re: Dithering or Sigma Clipping not working?

Posted: Fri Mar 15, 2019 12:59 am
by Astrogate
Thank you Robin for the tips and I will try it out. I was using 30 second subs and dithering set to every 90 secs. I was using a matching dark master but it is a month old, and perhaps I need to take a new set.

Cheers,

Chris

Re: Dithering or Sigma Clipping not working?

Posted: Tue Apr 21, 2020 6:39 am
by rhombus
I am having similar issues with Sharpcap Pro. I have dithering enabled (and it is working) every 60sec. I have hot pixels that do not ever disappear even after an hour of stack time and satellite streaks that may or may not. I have all raw files saved including individual frames that I can share.

Re: Dithering or Sigma Clipping not working?

Posted: Tue Apr 21, 2020 6:08 pm
by admin
Hi,

The sigma clipping in SharpCap as to work in a slightly different way to normal sigma clipping. In a normal processing program, all the frames are available to be processed once and the application can work through them twice – once to calculate the mean and standard deviation of each pixel and then a second time to stack the images while ignoring the pixel values that fall outside the appropriate sigma range. SharpCap can't do this as it has to process the data on the fly as each new frame is added. To get around this, SharpCap doesn't apply any sigma clipping to the first 10 frames – they are stacked without any limits on pixel values. Once 10 frames have been stacked, SharpCap starts to get an idea of the mean and sigma values for each pixel and can start applying the sigma clipping algorithm. However this means that if you have artefacts that get into the image in those first 10 frames they may not be removable later. In fact if they are far enough away from the correct value for a particular pixel, the sigma clipping algorithm may actually reject the new (better) values for those pixels in later frames, leading to the effect you are seeing where certain artefacts just don't go away.

So far I haven't managed to think of a way around this problem – at least not a workable one anyway!

Some practical advice would be to ensure that dithering frequency ( at least initially) is high enough to ensure that dithering occurs once or twice during those first 10 frames to help spread hot pixel is around. Additionally, using a dark frame should help – not only does dark subtraction remove the majority of hot pixels anyway, but SharpCap also statistically spots the pixels in the dark frame that are 'hot' and ensures that data from those pixels is substituted with data from a nearby non-hot pixel.

You can also adjust if you want the number of frames that are used for the initialisation phase of sigma clipping. Setting this value to low may lead to interesting results – the process relies on having reasonable statistics of the value at each pixel and you can't get reasonably good statistics from only five or six samples!

Hope this helps, Robin

Re: Dithering or Sigma Clipping not working?

Posted: Mon Jun 01, 2020 3:24 pm
by rhombus
Thank you Robin! After reading your response I realized that my initial sigma settings were set to 5 frames not 10. I ran a test after making this change and it appears to have resolved the issue.

Re: Dithering or Sigma Clipping not working?

Posted: Mon Jun 01, 2020 5:49 pm
by admin
Hi,

Glad to hear that that sorted the problem for you :-)

Cheers, Robin