Cosmetic Correction / Bad Pixel Rejection

Got an idea for something that SharpCap should do? Share it here.
Forum rules
'+1' posts are welcome in this area of the forums to indicate your support for a particular feature suggestion. Suggestions that get the most +1's will be seriously considered for inclusion in future versions of SharpCap.
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Cosmetic Correction / Bad Pixel Rejection

#1

Post by Borodog »

There are a new generation of sensors like the IMX533 and IMX571 that have no amp glow and such low dark current that their lights can be calibrated without darks, using biases and pixel rejection algorithms only. Every stacking software I know of can handle this, DSS, APP, SiriL, PixInsight, except SharpCap.

I would like a frame preprocessing option akin to banding suppression for cosmetic correction/bad pixel rejection. This would be distinct from whatever hot pixel handling SharpCap is doing during dark subtraction; however that currently works it does not correct hot pixels in a bias calibrated light (I tested it tonight). This would also solve the uncooled camera problem, at least for cameras without amp glow; calibrate the lights with bias, which are temperature independent, and let pixel rejection handle the hot pixels. This avoids the problem of the hot pixels in the master dark beginning to over or under correct as the sensor temperature changes.

Thanks for your consideration.
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Cosmetic Correction / Bad Pixel Rejection

#2

Post by admin »

Hi,

do you have a 571? I don't at the moment (but I have a 533). If you could share a couple of frames from your 571 with me then I have some extra data available to help understand how this feature might work.

cheers,

Robin
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Cosmetic Correction / Bad Pixel Rejection

#3

Post by Borodog »

Dr. Glover,

Unfortunately I do not; I have a 533. My understanding though is that the 571 and 533 are based on the same technology; it's just that the 571 has a larger sensor.
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Cosmetic Correction / Bad Pixel Rejection

#4

Post by Borodog »

As a followup, here is a good article on the issue that using master dark frames can actually increase final stack noise - significantly - in modern cameras.

https://clarkvision.com/articles/dark-f ... -no-darks/
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Cosmetic Correction / Bad Pixel Rejection

#5

Post by Borodog »

Lastly, here is last night's live stack test as processed in SharpCap (with a little bit of extra stretch just to show some of the fainter warm pixels more clearly). The lights here were calibrated with a master bias rather than a master dark. You can see of course that the hot and warm pixels are uncorrected.

https://www.cloudynights.com/uploads/ga ... 187683.jpg
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Cosmetic Correction / Bad Pixel Rejection

#6

Post by admin »

Hi,

nice image, and an interesting web page on the use (or not) of darks.

At a quick read through, I think there is a flaw in the logic of the simulated parts of the page, which rather calls the results into question. The problem is that the simulations are done with the same noise level in the 'light' and 'dark' frames. That's wrong... A dark frame will most likely have something relatively close to just the read noise of the camera. A properly exposed light subframe will have a background brightness that means that the noise in the light is dominated by the shot noise. For practical purposes that means that the light frame read noise will be 3 or more times higher than the dark frame read noise.

To give an example, let's pick 2e for dark frame noise level, and 7e for light frames (this follows from the equation for determining the right exposure time for deep sky). A master dark of 25 frames would have 2e/5 noise - 0.4e. A 100 frame stacked light would have 7e/10 noise - 0.7e. Combine the two and you get about 0.8e noise in the dark subtracted output, so the effect of using the dark on random noise is small rather than large. This ignores improvements due to the dark being able to reduce pattern noise though.

This analysis is based on a very quick read through of the page linked above - maybe I have missed something...

cheers,

Robin
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Cosmetic Correction / Bad Pixel Rejection

#7

Post by Borodog »

However he may have messed up his simulations, the point is ultimately that the residual noise in the master dark, which becomes a fixed pattern when subtracted from the lights, can exceed the dark current non-uniformity that the master dark is supposed to calibrate out for these cameras, resulting in a net increase.

But I wish now that I had not muddied the waters with the article. The fact is that many people with these cameras are calibrating their lights with bias and pixel rejection rather than darks, which obviates the need for a dark library. After my last 2 nights' testing, I will 100% be running my 533 this way. I'll suffer through the hot pixels during live stacking if I must, but if I had my druthers I would prefer not to. ;O)

Here is how that image turned out after stacking the raw frames, lights calibrated with bias and pixel rejection rather than darks.

https://cdn.astrobin.com/images/93441/2 ... 800785.png - 1.9 hours of light,

Thank you again for writing and sharing your amazing software. I really do appreciate it tremendously.
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Cosmetic Correction / Bad Pixel Rejection

#8

Post by admin »

Hi,

OK, no worries. I have this on the 'to do' board. I will have to do some testing to try to work out the best approach.

Nice image btw :)

cheers

Robin
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Cosmetic Correction / Bad Pixel Rejection

#9

Post by admin »

Right, well I was convinced I had a 533 based camera here, but apparently not - it must have been one of those samples that ends up being sent on to someone else for more testing once I am done with it :(

If someone can share with me a significant chunk of data taken with a 533 or 571 camera then that would give me something to work with and experiment on.

The data I would be looking for would include

* A decent number of light frames of some target (FITS format, 16 bit, unprocessed). Probably at least 20 frames, more would be nice

* A similarly decent number of individual dark frames taken with the same settings (in particular same temperature, exposure, gain, offset).

* A number of bias frames - same settings as lights/darks, but short (<<1s) exposures.

* If possible a number of individual flat frames (and dark flat or bias flat) - if you use flats.

Unfortunately, processed or stacked frames, or master darks/master flats are not going to be helpful for this.

Essentially that would give me enough data to try out several different processing strategies and see what noise levels come out (and what percentage of the final noise comes from what source), what is achievable in terms of processing with/without different types of calibration frame, etc.

Obviously the amount of data concerned is going to be quite large, so please upload to google drive/onedrive/dropbox and share a link if you are able.

cheers,

Robin
Borodog
Posts: 330
Joined: Fri Jan 01, 2021 7:25 pm

Re: Cosmetic Correction / Bad Pixel Rejection

#10

Post by Borodog »

Dr. Glover;

I should be able to provide this data later tonight. I'll put it on google drive and provide a link.
Post Reply