Cosmetic hot pixel removal in autostacks?

[timh]

Hi Robin,

I am enjoying using the autostacking, FWHM filtering, sigma stacking and all the other features available in live stacking. Not having to store hundreds of raw files makes EEA/ astrophotography with short exposures on the dobsonian a much more practical and less time-consuming proposition. Is it practical to also do hot pixel removal on the fly within the autostacking process?

best wishes
Tim

[admin]

Hi,

this is already included if you use dark frames – not only does SharpCap subtract the dark frame from each light frame which will remove warm pixels, SharpCap also does a statistical analysis of the pixel values in the dark frame and flags any pixels that are more than 5 standard deviations (maybe 6?) above the mean as 'hot'. For these pixels, SharpCap ignores the value in the light frame and steals the value from a nearby pixel of the same colour.

Doing this sort of thing without dark frames is a much more dubious proposition – if you try to spot hot pixels based on a light frame you run a risk of removing faint stars (or even bright ones that are very sharply defined). I seem to recall that older Nikon DSLR cameras had a noise filter that was very effective at removing stars from Astro images!

Cheers, Robin

[timh]

Hi Robin,

That's interesting. I do subtract the darks (and include flats) on the fly in the autostacks which are then further stacked in DSS (substituting 10 or so files for the unworkable over a thousand or so that would have been necessary has they been raw files). The masterdarks are averaged from 50 frames - at the same temperature, gain/ offset and about 3 months old. Do significant numbers of new hot pixels appear on sensors over time?

Over a full frame (4144 x 2822) autostack there can be maybe > 50-100 hot pixels visible about evenly divided between red, blue and green. I guess that the subtraction of the dark only has to be misaligned by a single pixel at various points in order for some to survive? Is there anything that can affect this? -- for example I wondered if slight movement during longer exposures might cause a hot pixel to appear present in an adjacent pixel ?

They do seem to be real hot pixels since when I stack the autostacks they turn into streaks of colour - so they are consistently there in the same relative place on the frame moving slightly as the image moves over longer times.

thankyou
Tim

[admin]

Hi Tim,

the dark subtraction is done before the alignment, so there should be no way for the dark to become misaligned with the light frames.

Could you share your master dark frame and one or two of your light frames with me? Preferably with some pointers as to where you see the hot pixels appearing. I can then see whether there is something going wrong in the hot pixel detection code.

One thing to note is that pixels that are really not very hot at all can look horrible once you stretch the final image – the amount that we stretch some deep sky images means that just a few percent of boost on the value of the pixel could make a lot of difference to the final image.

Cheers, Robin

[umasscrew39]

Hi Tim/Robin

This almost sounds like "walking noise" which usually you can get rid of by dithering. I'd also be curious to see your light frame as I had this problems some time ago but dithering cured it.

Bruce

[timh]

Thanks Robin and Bruce,

Two comments there that struck home. 1) Yes indeed it was only after stretching in PixInsight that I noticed what I called hot pixels --so maybe they were genuinely different in some way but not fully hot and 2) I wasn't dithering (it is difficult to do so on the dob because it has to be manual) and for some reason I also didn't dither my last set of refractor guided frames either.

Yes I will put together a set of the frames and send a link.

Tim

[timh]

Herewith the files,

Not a very striking example but the most recent and so easiest to find.

Please find the master dark file, the FTS file of a single autostack of 15 exposures and the FTS file of the DSS stack of 3 similar sharpcap autostacks (an image of M33)

I think that they confirm that you were both right. I can only actually only see about 1 hot pixel in the unstretched FTS file that is the first autostack -- so indeed pretty much all of the many hotpixels that can be seen in the master dark must have been efficiently removed.

I was indeed looking at highly stretched PixInsight versions ...just FYI screencaptures in the power point file showing how the single semi-hot points in the first autostack stretch into very short lines in the DSS stack of stacks.

https://onedrive.live.com/?id=330584F8B ... F8B152FA32

thanks again
Tim

[umasscrew39]

Hi Tim

This does not look like walking noise but I am wondering if these are what are referred to as stuck/defect pixels. Robin will know the answer. If that is the case, perhaps trying CosmeticCorrection in PI would fix it for you. Sorry I can't help further but Robin will figure it out.

Bruce

[admin]

Hi Tim,

thanks for those files – I can definitely see the artefacts that you are referring to, and I think that I can track some of them back to particular hot pixels in the dark file that you shared. What I really need to dig into this further is a small selection of processed (RAW) frames that get saved during live stacking if you have the option to save raw frames turned on. Do you have to have any of those files?

Cheers, Robin

[timh]

Thanks for looking into this Robin,

Unfortunately I didn't save raw files for the example I gave you before

However here is another - perhaps clearer - example of the same thing. This time some 95s raw fit files of the Western veil, the masterdark associated with them (this time a masterfile averaging only 10 darks), a single 22 min autostack of 14 x 95s frames and a DSS stack of 2 x autostacks (with DSS cosmetic correction of hot pixels turned off).

In addition, as before there is a powerpoint of screencaptures of the PIXIN stretched 1x and 2X stacked autostacks.

It is plain to see on these that there are clusters of apparent hotpixels (maybe the same individual one or two that escaped subtraction moving around with the dithering positions - so more on the X2 than on the X1?). Actually the same consistent dance pattern of little cold spots too and also about the right number for the number of dithers (about 1 in every 3 frames as I recall..)

https://onedrive.live.com/?id=330584F8B ... F8B152FA32

Tim