Dithering or Sigma Clipping not working?

A place to report problems and bugs in SharpCap
Forum rules
Please read about Troubleshooting USB Issues before posting.

*** Please do not post license keys - please report any problems with licensing to 'admin' by PM ***

Please include the following details in any bug report:

* Version of SharpCap
* Camera and other hardware being user
* Operating system version
* Contents of the SharpCap log after the problem has occurred.
[If SharpCap crashes, please send the bug report when prompted instead of including the log]
Post Reply
Astrogate
Posts: 71
Joined: Wed May 24, 2017 6:52 pm

Dithering or Sigma Clipping not working?

Post by Astrogate » Wed Mar 13, 2019 7:07 pm

Hi Robin

Last night I was out doing some live stacking and noticed issues with either dithering, sigma clipping or both? I enclosed two images of M 64 with the white point lowered a lot in order to show the background flaws. One image is 26 x 30 secs and the other is 40 x 30 secs both at gain 350 with a matching master dark and flat under sever light pollution and brightness level at 10. I was using a C9.25 sct at f 7.2 and guiding. As you can clearly see in the images the hot pixels are not being removed and grouping in the bottom right corners. Also with the longer stack of 40 sub-images a satellite streak has developed on right side and at about 7 o'clock below M 64 a line of warm pixels has developed. I have dithering set to dither at 12 pixels for my imaging camera according to the image camera scale and guide camera scale. Should either using sigma clipping with ample sub-images, dithering or both combine remove at least these hot/warm pixels, cosmic rays and possibly the satellite streak when pixel scale is set to dither high enough?

Thank you!

Chris A
Attachments
M64 40 X 30 secs
M64 40 X 30 secs
M64 40 X 30 secs (1036 x 705).jpg (262.86 KiB) Viewed 934 times
M64 26 X 30 secs
M64 26 X 30 secs
M64 26 X 30 secs (1036 x 705).jpg (239.54 KiB) Viewed 934 times

User avatar
admin
Site Admin
Posts: 4081
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Dithering or Sigma Clipping not working?

Post by admin » Wed Mar 13, 2019 8:55 pm

Hi Chris,

I'm not aware of any bugs in this area, but that's not to say that there couldn't be new bags that I haven't found yet.

You should be able to watch the progress of dithering if you keep an eye on the PHD2 screen while SharpCap is working – that will let you confirm whether the dithering commands are getting through to the guiding application or not.

The Sigma clipping algorithm is not active for about the first 10 frames that are captured in a stack as SharpCap has to have stacked a good number of frames in order to work out the statistics of variation at each pixel to be able to apply the Sigma clipping algorithm. So it is possible that the satellite streak came through early in the stack before the Sigma clipping algorithm could become active. If you're sure that's not the case please let me know. If you happen to have saved the raw frames during that stack then it might be interesting if you shared with me the one with the satellite streak in and about 15 to 20 other frames and I can see if I can reproduce the issue and work out where the bug is.

Cheers, Robin

Astrogate
Posts: 71
Joined: Wed May 24, 2017 6:52 pm

Re: Dithering or Sigma Clipping not working?

Post by Astrogate » Thu Mar 14, 2019 1:05 am

Hi Robin

Thank you for the fast reply! I was watching PHD2 on and off and did notice the dithering commands coming from SharpCap to PHD2. Very good point about the satellite streak, but unfortunately I did not capture any raw's last night. Perhaps it was added to the 40 stacked towards the end of the live stacking. I am surprised though that in both images (26 and 40 stacked sub-images at 30 secs each sub) there is a blue and green group of hot pixels forming bottom right corner which I thought the dithering would have eliminated. Thank you for your help!

Chris A

User avatar
admin
Site Admin
Posts: 4081
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Dithering or Sigma Clipping not working?

Post by admin » Thu Mar 14, 2019 9:15 pm

Hi Chris,

Dithering will help spread around the effect of a hot pixel, but if you only dither a small number of times during the stack then you will still see the effects of hot pixels although they will not be as bright. I would usually imagine using dark subtraction to remove the worst effects of hot pixels and then dithering would help remove any pattern or raining nois by the dark subtraction.

Cheers, Robin

Astrogate
Posts: 71
Joined: Wed May 24, 2017 6:52 pm

Re: Dithering or Sigma Clipping not working?

Post by Astrogate » Fri Mar 15, 2019 12:59 am

Thank you Robin for the tips and I will try it out. I was using 30 second subs and dithering set to every 90 secs. I was using a matching dark master but it is a month old, and perhaps I need to take a new set.

Cheers,

Chris

rhombus
Posts: 5
Joined: Tue Apr 21, 2020 6:37 am

Re: Dithering or Sigma Clipping not working?

Post by rhombus » Tue Apr 21, 2020 6:39 am

I am having similar issues with Sharpcap Pro. I have dithering enabled (and it is working) every 60sec. I have hot pixels that do not ever disappear even after an hour of stack time and satellite streaks that may or may not. I have all raw files saved including individual frames that I can share.

User avatar
admin
Site Admin
Posts: 4081
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Dithering or Sigma Clipping not working?

Post by admin » Tue Apr 21, 2020 6:08 pm

Hi,

The sigma clipping in SharpCap as to work in a slightly different way to normal sigma clipping. In a normal processing program, all the frames are available to be processed once and the application can work through them twice – once to calculate the mean and standard deviation of each pixel and then a second time to stack the images while ignoring the pixel values that fall outside the appropriate sigma range. SharpCap can't do this as it has to process the data on the fly as each new frame is added. To get around this, SharpCap doesn't apply any sigma clipping to the first 10 frames – they are stacked without any limits on pixel values. Once 10 frames have been stacked, SharpCap starts to get an idea of the mean and sigma values for each pixel and can start applying the sigma clipping algorithm. However this means that if you have artefacts that get into the image in those first 10 frames they may not be removable later. In fact if they are far enough away from the correct value for a particular pixel, the sigma clipping algorithm may actually reject the new (better) values for those pixels in later frames, leading to the effect you are seeing where certain artefacts just don't go away.

So far I haven't managed to think of a way around this problem – at least not a workable one anyway!

Some practical advice would be to ensure that dithering frequency ( at least initially) is high enough to ensure that dithering occurs once or twice during those first 10 frames to help spread hot pixel is around. Additionally, using a dark frame should help – not only does dark subtraction remove the majority of hot pixels anyway, but SharpCap also statistically spots the pixels in the dark frame that are 'hot' and ensures that data from those pixels is substituted with data from a nearby non-hot pixel.

You can also adjust if you want the number of frames that are used for the initialisation phase of sigma clipping. Setting this value to low may lead to interesting results – the process relies on having reasonable statistics of the value at each pixel and you can't get reasonably good statistics from only five or six samples!

Hope this helps, Robin

rhombus
Posts: 5
Joined: Tue Apr 21, 2020 6:37 am

Re: Dithering or Sigma Clipping not working?

Post by rhombus » Mon Jun 01, 2020 3:24 pm

Thank you Robin! After reading your response I realized that my initial sigma settings were set to 5 frames not 10. I ran a test after making this change and it appears to have resolved the issue.

User avatar
admin
Site Admin
Posts: 4081
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Dithering or Sigma Clipping not working?

Post by admin » Mon Jun 01, 2020 5:49 pm

Hi,

Glad to hear that that sorted the problem for you :-)

Cheers, Robin

Post Reply