Residual gradient removal

Got an idea for something that SharpCap should do? Share it here.
Forum rules
'+1' posts are welcome in this area of the forums to indicate your support for a particular feature suggestion. Suggestions that get the most +1's will be seriously considered for inclusion in future versions of SharpCap.
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Residual gradient removal

#1

Post by Borodog »

Often stacks are produced that result in a residual gradient, either because of light pollution or imperfect flat correction. For example, the Losmandy dovetail plate on my C8 protrudes slightly past the end of the scope, making it impossible to lay my light panel flat across the end of the scope; as slight as a couple mm of difference from one side of the corrector to the other would seem, it nevertheless shows up in the stretched images as a residual gradient. I would like it if there were an option in the "Enhancement" tab to remove such residual gradients. A method would be to average (channel by channel) a small region (with stars removed) from each of the 4 corners of the frame and fit a plane to those 4 values and subtract it from the displayed image (again, on a channel by channel basis).
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Residual gradient removal

#2

Post by admin »

Hi,

yes, this one is on the to-do list. There is currently the option to remove a flat background brightness (see https://docs.sharpcap.co.uk/4.0/#!2!Pre-processing). Gradients are trickier - although your approach is the beginning of the solution, that approach is likely to start removing too much if you have nebulosity at the edges of the frame...

cheers,

Robin
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Residual gradient removal

#3

Post by Borodog »

Yes; if you have nebulosity in the corners it could cause problems. A more advanced solution would be to allow the use to click on 3 or 4 widely separated locations without nebulosity, which should be sufficient to fit a plane. You could allow more selections and fit a higher order function, but that is starting to get fairly complex.

But I am very happy to hear it is on the drawing board; it would improve my own personal SharpCap experience greatly.
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Residual gradient removal

#4

Post by Borodog »

Robin,

After studying this for a while and seeing how SiriL recommends doing it and seeing the fantastic results thereof, I think this is actually much easier than I previously thought.

If you've shot flats, residual gradients are typically caused by light pollution, and are almost always linear across the FOV. It's only after stacking that complex higher order gradients develop. Hence, SiriL recommends doing first order background extraction on the individual subs before stacking. The developers recommend using a 10x10 grid of sample points and polynomial degree 1; SiriL uses some minimal logic to reject points that may lie in significant nebulosity (I have some thoughts on how to do this, if you're interested), and just subtract the resulting fit from the individual subs before stacking. The results are phenomenal, pretty much independent of target. This is much simpler than trying to create artificial flats, which are inevitably high order multidimensional polynomials (and forget about dust shadows).

And as a side note, I would REALLY like to see cosmetic correction/hot & cold pixel rejection implemented. Pretty much every stacking package has an implementation of it. It would be a huge improvement for SharpCap live stacking with an uncooled camera (like mine).
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Residual gradient removal

#5

Post by admin »

Hi,

yes, good point, light pollution is likely to be linear in *some* direction across the field in a single image, so the trick is basically a robust mechanism for ignoring nebulosity (even if it occurs at one edge for instance). I can also see some possibilities for that sort of detection, but would be happy to hear your ideas.

Hot/cold pixels - I can imaging ways to process the image to achieve that, but is the 'no dark long exposure' use case valid? Dark subtraction in SharpCap already does hot pixel removal...

cheers,

Robin
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Residual gradient removal

#6

Post by Borodog »

Sorry for the delayed response; I've been out of town.

Regarding nebulosity in residual gradient removal:

I think the most critical thing to understand is that when fitting the plane (on a channel by channel basis, I believe), you should NOT use a least squares fit. You should instead use a LAR, Least Absolute Residual fit. The least squares fit is extremely sensitive to outliers (e.g. landing on stars or nebulosity) where the LAR fit is insensitive to outliers. So it you use an LAR fit instead of a LSQ fit, you may not have to do anything about avoiding nebulosity at all. You should test this during implementation, of course. Another feature of the LAR fit that is superior to the LSQ fit is that, because outliers skew the fit so hard for the LSQ fit, it becomes harder to identify those outliers and reject them. But because the LAR fit is insensitive to outliers, the residuals of those outliers to the resulting fit become significantly larger and easier to identify and reject. So a rejection method that calculates an initial fit and then kicks out outliers before a final fit becomes more robust.

Basically the only advantage LSQ has is that it is somewhat easier to implement computationally.

https://en.wikipedia.org/wiki/Least_absolute_deviations

Regarding hot/cold pixel rejection, it is sometimes difficult for people who don't have an un-cooled camera to understand the difficulties of using one. Basically, I don't have a dark library and can't really make one, because my camera sensor temperature floats relative to ambient and changes based on the gain and exposure I choose. Yes, I can shoot a few darks before stacking. But unless I can shoot a significant number of them, my master dark will have residual noise that shows up in the stack anyway, although it will be sufficient to kill the most egregious hot pixels. For a while. But then, as the temperature drops, the hot pixel corrections in the master dark start to carve holes into the stack because they are being over-corrected. I know the right answer is to spend $1000+ on a cooled camera, but that is a hard pill to swallow for some folks. It would be nice if SharpCap had some sort of cosmetic correction/hot cold pixel rejection, as every other stacking software I'm aware of has, DSS, SiriL, APP, PixInsight, they all have bad pixel rejection.

I also think it would be a significant enhancement even for those with cooled cameras. Modern sensors like the IMX533 or IMX571 are so low noise that they can literally be used with only bias instead of darks to calibrate the lights . . . except for the hot/cold pixels. Many people stack (in post) with only bias and no darks, relying on pixel rejection, with excellent results.
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Residual gradient removal

#7

Post by admin »

Hi,

thanks for the suggestion on LAR - that makes sense as removing the 'squaring' gives less weight to points that are far out from the predicted line. I don't have to worry too much about stars as a suitable median filter will remove them fairly nicely. Obviously nebulosity in the middle of the frame with a surrounding background should work nicely in that case. I think there is also probably some benefit in constraining any linear background fit to always lie between the minimum and maximum measured background values at all points in the frame. Hmm, needs some thought.

I just checked the way that hot pixels are currently handled in star detection - basically SharpCap looks at the dark frame statistics and flags any pixel above 7SD brighter than the mean as 'hot', meaning that values from those pixels are ignored (and a substitute value from a nearby pixel of the same colur is used) when doing dark subtraction. For a purely normal distribution, 7SD is far enough from the mean to give basically no pixels selected, so any pixels that are beyond that point must be outliers from the main distribution of dark pixel brightnesses. That 7 was picked by a certain amount of trial and error to try to catch the most obvious hot pixels with minimum other impact. If you have specific frames where we can identify problem pixels in the stack and the dark then we could have a look at the statistics and see if it is just a threshold position thing (ie your problem pixels are at 6.2SD or similar from the mean) or if the statistical approach is failing completely and what might replace it.

cheers,

Robin
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Residual gradient removal

#8

Post by Borodog »

Wait; are you saying that SharpCap is already supposed to be doing bad pixel rejection on the stack? But only during dark subtraction? I'm not following.

I thought the purpose of dark subtraction, among other things, was to remove hot pixels. Are you saying that instead you are rejecting the hot pixels rather than subtracting them?

What happens specifically after the sensor temperature falls (sooner that you would think, a degree or 2 C is all it takes) is that what would have been red, green, or blue hot pixels become cyan, magenta, or yellow hot pixels, because they start carving red, green, or blue holes in the stack because they are over-correcting. Here's an example I posted a year ago with an uncooled SV305. After shooting darks the stack starts off fine, but 20 or 30 minutes later it looks like this:

https://www.cloudynights.com/uploads/mo ... 332562.jpg
https://www.cloudynights.com/uploads/mo ... 331797.jpg

These days I am guiding and dithering so it helps hide the problem, but it still degrades the image quality significantly; instead of CMY hot pixels I get CMY smears. The camera I mainly use now is an uncooled ASI183MC. The darks calibrate the amp glow well enough even as the temperature drops because it is not a thermal process and does not follow the Arrhenius equation, but the hot pixels becoming cold pixels is a real problem and very frustrating.

At one point I tried to build a synthetic dark library using master darks taken at 3 different temperatures (I had to stick the camera in the refrigerator and freezer) to try to extract out the amp glow and dark current (after subtracting out the bias), then scaling the dark current according to the Arrhenius equation, and then recombining the bias, amp glow, and synthetic dark current to create a new synthetic master dark. I wasn't really successful because the dark current activation energy of my camera changed too much over the temperature range I had sampled, meaning I needed more than 3 temperatures, which I had no way to generate.

All this is by way of explaining how huge of a PITA it is to not have a cooled camera.
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Residual gradient removal

#9

Post by Borodog »

So thinking about this instead of sleeping, it seems that it's the case that hot pixel rejection occurs during master dark subtraction. I don't believe that's how any other stacking software does it. I believe the norm is to reject bad pixels at the adding/stacking stage. Bad pixels can be rejected during stacking of flats, for example.

I believe that it would be a significant enhancement if bad pixels were rejected from the next light after it's been calibrated. In fact whether or not it's been calibrated. This would prevent hot/cold pixels from making it into the stack even if you use bias instead of darks for calibrating the lights.

That is easy for me to write, but off the top of my head I don't know what the actual pixel rejection algorithms used by SiriL, DSS, APP, PixInsight, etc. are. I am sure something has been published somewhere on the web, though. I'll see what I can dig up.

But it occurs to me that you are already doing some form of hot pixel rejection in SharpCap; you do it during frame alignment, for example, so that hot pixels are not mistaken for stars. So you must have some method for detecting hot pixels. Could this not be extended to detecting hot/cold pixels both and rejecting them from the frame before adding it to the stack?
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Residual gradient removal

#10

Post by admin »

Hi,

nice to chat to you briefly at the show yesterday :)

To clarify on the hot pixel/dark subtraction thing, dark subtraction deals with things like amp glow, thermal noise and 'mildly warm' pixels by subtraction, on the grounds that all of those things are additive to the signal coming from the sky at that pixel. However some pixels are beyond 'mildly warm', heading from there all the way up to fully hot (maximum pixel value in the dark). If you think about it, subtraction can do nothing for 'fully hot' pixels, since you get maximum pixel value at that pixel in both the dark and the light, so subtracting gives you zero - a nice black hole in the image. For significantly hot pixels, the best thing to do is to steal a value from a nearby pixel, ignoring the problem pixel entirely. SharpCap uses the seven sigma threshold to decide on the approach to take - subtraction or value substitution. I expect that really I should use some sigma threshold above the local mean rather than the global mean of the frame to take account of regionalized glows, etc, but that doesn't affect the fundamentals.

Now, detecting 'hot' pixels is much easier in a dark frame than a light frame, as you don't have to worry about misdetecting stars, which is why the dark subtraction is linked to hot pixel removal right now.

Note that the hot pixel 'fix' is done as part of the dark calibration of each light frame, although to keep things working quickly the actual pixels that are considered as 'hot' is calculated when the dark frame is loaded (since the list is fixed for a given dark frame). I suspect that is pretty much in line with most other software, although I haven't dug through the algorithms.

Once I catch up on things that have been pending due to the show, I will try to come back to this area - processing in general - to see what can be improved.

cheers,

Robin
Post Reply