How is this sub treated in live stacking?

[Yourjones]

Hello

I wonder if this question has an easy explanation. My ZWO asi533MC produces a sub like this one below occasionally when I do 12 s live stacking with SharpCap Pro, one sub in every few dozen perhaps, not periodically, though. (Note: the image below is not the full resolution image, rather, it is a screenshot of the image in question just to minimize its size). While I am not clear what causes this strange line, it likely being a problem with the camera or data transmission to my PC, it does cause trouble for my live stacking.

My understanding is that such sigma should cause the relevant pixels to be rejected, but the final stacked image (of around one-hour live stacking integration,12 s each sub) would show a trace of it. Though not conspicuous, the trace of the line is visible. Of course, I can save all my subs while live stacking for later use with DSS. Although this enables me to manually pick such subs out, I'll have a huge number of subs for DSS to process.

1. Am I right in believing that the 'sigma clipping' function only ignores the abnormal pixels in a new sub compared to the average value of previous subs, while adding the rest of the pixels of the same sub to the stack?
2. Is it possible for SharpCap to ignore this sub altogether when it detects such anomaly, instead of just part of it?

Thanks a lot!

Lu

Screen Shot 2022-07-09 at 22.38.30.jpg (140.25 KiB) Viewed 1798 times

[admin]

Hi,

the sigma clipping should ignore something like this where the pixel values in the line differ from the 'normal' pixel values at those positions by a big enough margin. The margin is calculated from the variation in the pixel values in previous frames (although an extra allowance is made when stacking 8 bit images as otherwise it can start rejecting values that are only 1 ADU different from previous values).

The frame you've posted looks like it should be different enough to let sigma clipping do its work (but if a frame like this occurs in the first 10 frames or so before the sigma clipping becomes active then you will see the line).

I'm happy to investigate the issue to check things are working as planned if you can share a selection of frames with me (about 20 without the line, at least 1, ideally 2 with it).

Another trick that might be worth trying is to use the satellite trail removal feature in SharpCap, which I think will be able to detect and remove a line like this (since it looks like a satellite trail).

cheers,

Robin

[Yourjones]

Hi Robin

Thanks for looking into this issue. Please let me know how it is easiest for you to get the images. I have prepared 20 FIT files containing 1 such problematic one. Each of the files is about 18 MB.

Best regards

Lu

[admin]

Hi Lu,

thanks for sharing the images - I have had a look at what is going on now, and it is quite interesting...

Frame 106 has the horizontal line that is caused by some sort of camera glitch. The line is 2 pixels high and has very high brightness in most pixels in the line (60000 ADU +). There are however some pixels in the line that have roughly normal background values. I think there is a slight hint of higher than normal background pixel values in the rows above and below the bright line - a fairly small change from normal background.

The procedure for sigma clipping is as follows:

* RAW frame gets debayered
* dark/flat correction applied (if selected)
* RAW frame is aligned with existing stack (may involve fractional pixel offset and/or rotation)
* Pixels in the aligned frame are checked against the existing stack statistics and added if they lie within the sigma limit (otherwise ignored).

Let's see what happens to the two pixel high bright line in that sub...

* Debayering spreads the brightness out to a strip 4 pixels high - brightest in the 2 center rows, dimmer in the outer two rows - this is because pixel values need to be taken from neighbouring pixels to get (for instance) the red and blue values at a green pixel.
* Alignment - fractional pixel offsets or rotation will again blur the edges of the bright strip, leaking some extra brightness out further from the original 2 pixel band (but dimmer)
* Sigma clipping will reject the very bright pixels in the core of the band, but does not always reject the slightly brighter than average pixels near the edge of the band that have a small contribution from the bright band due to debayering/aligning.

The results can be improved by setting stricter limits on the sigma clipping - for instance setting a sigma threshold of 4 or more and a sigma low limit of 0.1% (a higher sigma low limit is really only needed with 8 bit camera modes). This reduces the amount of signal that gets through from the bright band, but does not fully eliminate it - it can still be seen with enough stretch applied to the display image.

I don't think sigma clipping will every eliminate this problem 100% - it works at the single pixel level and allows data through when the new pixel value is within the 'acceptable' statitstical bounds for that pixel. With enough smearing of the signal due to debayer/alignment/etc, this will always happen at some pixels that only have a small input from the bright band. However, when you look at the image, you see the line not due to an individual pixel being obviously too bright, but due to the pattern of hundreds of pixels in a line being a tiny bit brighter than they should be - the human eye is very good at spotting this sort of pattern.

I think the best bet to cleanly remove this issue (aside from getting the camera manufacturer to fix the glitch!) is to use the Satellite Trail removal feature in SharpCap. This feature is designed to look for lines of anomalous pixel values and will remove them (and neighbouring pixels) entirely, leaving behind a somewhat blurred region that should have very little additional brightness.

cheers,

Robin

[Yourjones]

Hi Robin

I strongly appreciate your time and efforts looking at the problem in such depth. The analysis is of higher quality than textbook. I think what I'm going to do is to first play with the sigma setting and monitor the 'individual' frames during my next live stacking session and see what happens. I'll be tough on the 'threshold' and see how tough is too tough to cause good pixels rejected.

As a matter of fact, I tried the 'satellite trail' function 2 nights ago, but I was still seeing these problematic frames stacked in the image. Mysteriously, I've noticed more than 200 12-sec frames 'dropped' when the 'satellite trail' function was active. I am not sure whether they are correlated, but such dropping rate had never appeared before.

Nonetheless, thank you again for your help!

Lu

[admin]

Hi Lu,

the satellite trail removal does consume quite a bit of CPU time, so could lead to dropped frames (if it is still processesing the last frame when the next one arrives). I found that I needed to set a sensitivity of (I think) 14 to get it to spot the horizontal line and remove it. There is also a real satellite trail in frame 98, but it is very faint - so far I haven't been able to find settings for satellite removal that will detect that faint trail, but I intend to use that frame for testing improvements to the algorithm

cheers,

Robin

[Yourjones]

Hi Robin

This is where the value of ShareCap is residing. The answer is absolutely meaningful and it gives hope. Thanks a lot!

Lu