Hi Robin,
I have a question about live stacking and sigma rejection. I am currently using a sigma of 2 with very low threshold. My frame count is set to 10 and I dither 10 pixels with my asi071 mc pro. PhD2 is set to dither in spiral. What I am seeing is my dark doesn’t remove two strong hot pixels. I end up with 9 hot pixels in a grid during the initial 10 frames. This leaves me with a grid of 9 hot pixels and some random cold pixels. The sigma settings discard about 0.02% of pixels unless a satellite goes through a frame. The settings remove all future hot pixels and satellite streaks except for the initial 9 in the spiral. The intensity of these pixels do not lessen after all future dithers.
I dither every frame so sometimes 50 or so 2 minute frames. If I reduce the initial frame count to 5 I get a grid of 4 hot pixels. Will these always be there because of how sigma rejection is calculated? Is it possible to adjust the algorithm to reject these hot pixels if doing it offline using previously acquired frames like dss does? By this I mean looking at all frames initially and then stacking with sigma clipping? Thanks!
Steve
Stacking and hot pixels
-
admin
- Site Admin
- Posts: 13271
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Stacking and hot pixels
Hi Steve,
Because SharpCap is trying to work on a live stream of incoming images it can't process them in quite the same way that other stacking programs do – programs like DSS can look at all the pixel values for a particular pixel to work out the statistics before deciding which ones to remove. If SharpCap wanted to work that way it would have to save the data for every single frame and re-stack all of the frames every time a new one arrives – the process would quickly slow down to the point of being unusable after a few frames.
SharpCap's approach is to treat the first 10 frames as the source of its initial statistics for pixel values – that means if there is a hot pixel or pixels in the first 10 frames and they are consistent then the values get baked into the statistics and future data for those pixels gets thrown away because it's inconsistent with the statistics that exist (beyond the sigma threshold).
Two things that you could do that might help hard to increase the number of initial frames that are used to set up the statistics and to increase the sigma threshold. Using a threshold of two is a bit low – in randomly distributed data it would reject about 5% of values.
Cheers, Robin
Because SharpCap is trying to work on a live stream of incoming images it can't process them in quite the same way that other stacking programs do – programs like DSS can look at all the pixel values for a particular pixel to work out the statistics before deciding which ones to remove. If SharpCap wanted to work that way it would have to save the data for every single frame and re-stack all of the frames every time a new one arrives – the process would quickly slow down to the point of being unusable after a few frames.
SharpCap's approach is to treat the first 10 frames as the source of its initial statistics for pixel values – that means if there is a hot pixel or pixels in the first 10 frames and they are consistent then the values get baked into the statistics and future data for those pixels gets thrown away because it's inconsistent with the statistics that exist (beyond the sigma threshold).
Two things that you could do that might help hard to increase the number of initial frames that are used to set up the statistics and to increase the sigma threshold. Using a threshold of two is a bit low – in randomly distributed data it would reject about 5% of values.
Cheers, Robin
Re: Stacking and hot pixels
Hi Robin,
I recently started using SharpCap Live Stacking combined with PHD2 dithering and faced similar issues with hot/cold pixels using Sigma Clipped Stacking.
I also read a further post on that topic in this forum from April this year (viewtopic.php?t=1496) in which you in addition explained the following regarding the stacked frames during the initial frame count:
„ … However this means that if you have artefacts that get into the image in those first 10 frames they may not be removable later. In fact if they are far enough away from the correct value for a particular pixel, the sigma clipping algorithm may actually reject the new (better) values for those pixels in later frames, leading to the effect you are seeing where certain artefacts just don't go away.“
Probably I am overlooking something essential, but couldn't one solution be to introduce the option „Skip stack of Initial Frame Cound phase“ during which Sigma Clipping is not applied? So if the value is set to 10, then with the 11th frame start a new stack using the statistics gained with the previous initial frame stack? That way all issues that could be introduced in the stack during the first initial phase as you described, would not go into the „real“ stack. With the usual short exposure times for live stacking in the range of seconds to max a few minutes it should not be a problem to „loose“ the first 5 or 10 frames to get the necessary statistic data for the Sigma Clipping algorithm. Anyway skip of initial frame stack could be an option and not the default.
I have not tried, but thought that this could be possible already somehow by using the Advanced Live Stack option „Save and Reset every xx min of total exposure“. But according to the manual it will make a complete reset each time, which would mean that also the statistic data get lost and the initial frame analysis is repeated every xx min.
Regardless from above question, I wonder if one should avoid starting with dithering already during this Initial Frame Count? Would it be better to start dithering after this initial phase (Pause – start dithering - Resume)?
Cheers,
Paul
I recently started using SharpCap Live Stacking combined with PHD2 dithering and faced similar issues with hot/cold pixels using Sigma Clipped Stacking.
I also read a further post on that topic in this forum from April this year (viewtopic.php?t=1496) in which you in addition explained the following regarding the stacked frames during the initial frame count:
„ … However this means that if you have artefacts that get into the image in those first 10 frames they may not be removable later. In fact if they are far enough away from the correct value for a particular pixel, the sigma clipping algorithm may actually reject the new (better) values for those pixels in later frames, leading to the effect you are seeing where certain artefacts just don't go away.“
Probably I am overlooking something essential, but couldn't one solution be to introduce the option „Skip stack of Initial Frame Cound phase“ during which Sigma Clipping is not applied? So if the value is set to 10, then with the 11th frame start a new stack using the statistics gained with the previous initial frame stack? That way all issues that could be introduced in the stack during the first initial phase as you described, would not go into the „real“ stack. With the usual short exposure times for live stacking in the range of seconds to max a few minutes it should not be a problem to „loose“ the first 5 or 10 frames to get the necessary statistic data for the Sigma Clipping algorithm. Anyway skip of initial frame stack could be an option and not the default.
I have not tried, but thought that this could be possible already somehow by using the Advanced Live Stack option „Save and Reset every xx min of total exposure“. But according to the manual it will make a complete reset each time, which would mean that also the statistic data get lost and the initial frame analysis is repeated every xx min.
Regardless from above question, I wonder if one should avoid starting with dithering already during this Initial Frame Count? Would it be better to start dithering after this initial phase (Pause – start dithering - Resume)?
Cheers,
Paul
-
admin
- Site Admin
- Posts: 13271
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Stacking and hot pixels
Hi Paul,
the purpose of the initial phase is to establish the (supposedly) correct statistics for each pixel in the image. That is, the mean and standard deviation of the values seen for each pixel so far. If new incoming data is too far from the mean established during the initial phase then it will be discarded. Because of this, restarting after the first phase would get you anywhere – the problem is embedded in the statistical data which would need to be carried forward over the restart otherwise you would be starting from scratch again and have to perform another initialisation phase.
I think the approach that I might be tempted to take is to judiciously increase the 'Sigma low limit' slider. This will force SharpCap to accept a wider range of values for all pixels in the image. Because the things that you are trying to exclude like satellite trails are usually very different from the normal image levels it should be possible to find a value for this that will help reduce the problems while still excluding satellite/aeroplane trails.
Cheers, Robin
the purpose of the initial phase is to establish the (supposedly) correct statistics for each pixel in the image. That is, the mean and standard deviation of the values seen for each pixel so far. If new incoming data is too far from the mean established during the initial phase then it will be discarded. Because of this, restarting after the first phase would get you anywhere – the problem is embedded in the statistical data which would need to be carried forward over the restart otherwise you would be starting from scratch again and have to perform another initialisation phase.
I think the approach that I might be tempted to take is to judiciously increase the 'Sigma low limit' slider. This will force SharpCap to accept a wider range of values for all pixels in the image. Because the things that you are trying to exclude like satellite trails are usually very different from the normal image levels it should be possible to find a value for this that will help reduce the problems while still excluding satellite/aeroplane trails.
Cheers, Robin
Re: Stacking and hot pixels
Hi Robin,
thank you for your fast reply. It was to be assumed, that it would not be that easy.
Regarding my second question (more related to hot pixels), already dither or not during the initial frame count phase, what would you recommend?
Cheers,
Paul
thank you for your fast reply. It was to be assumed, that it would not be that easy.
Regarding my second question (more related to hot pixels), already dither or not during the initial frame count phase, what would you recommend?
Cheers,
Paul
-
admin
- Site Admin
- Posts: 13271
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Stacking and hot pixels
Hi Paul,
it probably depends on how good your tracking is – if your mount is tracking very accurately so that there is hardly any misalignment between successive frames (check the drift graph) then a dither might help. If there is some drift going on then the hot pixels will be being spread out right from frame one anyway.
The best thing to do with hot pixels is of course is to make sure you're subtracting darks that are captured with exactly the same settings and sensor temperature – I'm not sure from your initial post whether you are using darks or not.
Cheers, Robin
it probably depends on how good your tracking is – if your mount is tracking very accurately so that there is hardly any misalignment between successive frames (check the drift graph) then a dither might help. If there is some drift going on then the hot pixels will be being spread out right from frame one anyway.
The best thing to do with hot pixels is of course is to make sure you're subtracting darks that are captured with exactly the same settings and sensor temperature – I'm not sure from your initial post whether you are using darks or not.
Cheers, Robin