Spurious hot pixel groupings occasionally appearing in some master darks

[alphatripleplus]

Hi Robin,

Apologies if this is not the correct forum to report this occasional problem I am having with SharpCap and two or three of my cameras with darks.

The issue has occurred for me at least twice in the last month with a couple of mono cameras (both an ASI290MM mini, and last night with a Player One Apollo-M Mini). In a nutshell: Occasionally, master darks when carefully inspected (i.e zoomed so that individual pixels are easily visible) appear to have small groupings ( usually 2-4 adjacent) of hot pixels, in different parts of the frame, which do not appear in lights. The end result is that a livestack of lights calibrated with these master darks, will show very small dark blotches that mimic the appearance of a dust speck on the sensor cover. (Dust motes on the sensor cover give much smaller dust donuts than dust on the camera window as they are much close to the sensor).

My first reaction when I saw this the first time earlier this year was that I had a dust speck on the sensor cover and needed to clean the inside of the camera or use flats ( which I typically don't use for EAA). However, each time this has happened I've found that using a different master dark or no dark will magically eliminate these specks. Upon inspection of the problematic master dark, with the lights, the pixel locations of the black blotches in the lights correspond to spurious hot pixel groups in the master dark that are not seen when I redo the master dark. I'm not sure why this problem seems to have started this year, but I don't recall having see it with darks from 2023 or earlier.( I've also taken daytime "pseudo" flats, short exposure live stacks with the scope capped, which will show both dust donuts on the camera window easily and the smaller dust specks on the sensor cover if they are there. This has not shown any dust on the sensor cover).

As an illustration, last night I used the 32 bit vsn of SharpCap 4.1.12025.0 with the old 4GB Winoows 8.1 laptop I've been running SharpCap on for EAA with the Apllo-M Mini . I created a fresh set of master darks for the camera at various sub-exposures, including 15sec and 30sec subs. I started using 30sec subs for H-alpha captures. Here is a calibrated light for M1 which shows (circled) about 5 of these dark blotches:

M1 (H-alpha) 18 x30s

M1_Ha_f5.5_#3_Annot_G300 2024-04-21T21_56_01_Stack_18frames_540s.jpg (494.61 KiB) Viewed 482 times

These blotches were also visible on other lights calibrated with the same 30sec master dark. A little later, I removed the H-alpha filter and tried a couple of captures with a 15sec master dark. No dark blotches were visible in this image of M81 using 15sec subs:

M81 24 x 15s

M81_f5.5_#3_RS_G300 2024-04-21T22_55_30_Stack_20frames_300s.jpg (494.8 KiB) Viewed 482 times

I've kept the background black level pretty low so that the dark blotches (if they are there) show up against the bright sky background. I'm also attaching the camera settings txt files below and including a link to a google drive directory where the two master darks (fits and png) and calibrated lights (png) are located. I didn't save individual darks however.

https://drive.google.com/drive/folders/ ... drive_link


One note: I used "flip both" in the pre-processing tab for darks in SharpCap for both lights, so if you want to match pixel locations with the master darks, the calibrated lights have the blotches in the mirror image locations that the hot pixel groups appear in the dark frames.


Sorry for the long post, and hopefully either I am missing something obvious or there is a solution to this issue. I'm happy to do further testing - I should have taken an uncalibrated light (i.e. using no dark) of M1 last night which would also show the blotches don't appear. Thanks for any help.

Errol

[admin]

Hi Errol,

I think your analysis is correct - you are getting hot pixels in the dark frame that are not present in the light frame, and are therefore over-correcting. It may be happening in the M81 image too, but the brighter background (no filter) may make the incorrect subtraction far less obvious (occuring in a part of the brightness scale that is less strongly stretched for instance).

Looking at your settings files, the temperature of the sensor seems to be fairly well matched between light and dark for an uncooled camera, which would otherwise be something to investigate.

I can think of a couple of ways that you can dig into this a bit more to see if you can work out what is going on.

1) Capture darks with the option to save the individual frames turned on. Then if you see the problem, you can find the pixel co-ordinates of the problem hot pixels and check the individual frames to see if those pixels are hot in all frames or if it is coming from perhaps one rogue frame in the dark capture that has a 'hot' value for that pixel. If that turns out to be the case then some sort of software fix in SharpCap may be possible (exclude outlier values when averaging dark frames)

2) Capture darks, then start using the dark frames. Turn up the 'offset' value a bit, keep capturing darks at the same settings. That should give you basically a uniform dim frame (the only signal would be the difference between the two offset values - without that change the signal would average to zero). It may be easier to spot and understand the over-corrected pixels in a simpler setup like this with no image data (and no need to wait for a clear night to test)

cheers,

Robin

[alphatripleplus]

Thanks very much for those suggestions, Robin.

As you note it can be done any time, I will probably start with suggestion 2) - Capture a set of darks, create a master dark, and then use it to calibrate lights with the camera covered, but with the offset value increased slightly in the lights, and the blotches should appear if there are spurious hot pixels in the master dark. I will also save the individual dark frames, so that I can see if the spurious hot pixels are in one or more of the individual darks used to make the master dark. Hopefully, I have that correct.

You could well be correct that there are other groups of spurious hot pixels lurking in the 15sec master dark that I used to calibrate the M81 livestack, and they are just hard to detect because of the brighter background and the absence of the H-alpha filter. The only thing I know for sure is that if I examine the location of the 5 spurious hot pixel groupings in the 30 sec master dark image file, and compare it to the same location in the 15 sec master dark, those groupings do not appear at the same location in the 15sec master dark. There could be others lurking elsewhere though.

I did redo the 30 sec master dark with a fresh one last night, took another livestack of M1 with the H-alpha filter using the new 30 sec master dark, and the 5 dark blotches did not appear in the new calibrated image - although there may have been one small blotch at another location. I also took an uncalibrated light of M1 using 30 sec subs and the dark blotches were absent, although pattern noise was quite bad (as would be expected).

I'll post any findings here, and if I can tie it down to specific frames used to create the master dark, I'll add those to the google drive location I linked to in my first post.

Thanks again,

Errol

[admin]

Hi Errol,

sounds like a good plan. It is worth noting that even with matching temperature readings, the temperature sensor is only one point in the camera, so while the temperature readout may match, the actual temperature of the sensor may differ (or the temperature may vary across the sensor). Of course this applies to cooled cameras too, but the cooling does tend to pull all the hot/warm pixel brightnesses down.

cheers,

Robin

[alphatripleplus]

Hi Robin,

That is a good point about the increase in both average pixel intensity and the width of the pixel intensity distribution with increasing temperature.( I can see, for example, from the pixel intensity histogram using my uncooled camera at 300 gain and 30 sec subs, that the masterdark shows about 50% more hot pixels at the maximum value when going from about 15C to 22C, and the standard deviation of the intensity distribution is also much higher).

I did follow up on suggestion 2), increasing the offset in the lights and calibrating with a masterdark, but as there are many, many 2-4 dark pixel groupings in the calibrated lights I couldn't determine if any were potential problems.( I tried increasing the offset a couple of times, but there were still lots of such dark groupings). I realized that the main reason I was able to find them relatively easily in the H-alpha nighttime M1 captures was that the very slight frame-to-frame drift over say, 20 lights, results in a significantly bigger blotch in the livestack from just a single 3 or 4 spurious hot pixel cluster in the masterdark. (When taking the M1 livestack, looking at the drift graph, I think I had roughly 5 or 6 pixels of drift in RA and a little less in Dec over the course of the livestack; the mount tracks very well unguided).

So I think I'm going to follow up on your 1st suggestion: Save individual dark frames for each masterdark, use the camera on a nighttime target (probably with the H-alpha filter) and the next time I see a problem dark blotch in my calibrated stacked lights, I should be able to identify whether the pixel location is problematic in one or more of the individual dark frames used in the masterdark.


Errol

[alphatripleplus]

Update:

I was able to do more night time testing. Last night (Wednesday), I took another capture of M1 with the same set-up (H-alpha filter and camera, 30 sec sub-exposures) as in the first post - M1 was only around 20 degrees above the horizon, so I think I'll need a new test target for any further runs.

For this capture, I used a different 30sec master dark (from the one used in the M1 capture in post 1). With this new master dark, I had saved all 16 individual darks, so that I could later examine each one for indications of spurious hot pixels. (The dark temps looked to be pretty closely matched to the lights). Here is the calibrated and stretched light for the M1 capture which shows (circled) 2 of the most prominent, but tiny, dark blotches, which were in different locations in the frame from the initial M1 capture from the first post.

M1 (H-alpha) 16 x 30s

M1_Ha_f5.5_#3_Annot_RS_G300 2024-04-24T22_22_23_Stack_16frames_480s.jpg (491.04 KiB) Viewed 292 times

Next, I identified the corresponding pixel location in the master dark of the dark blotches in the calibrated light livestack. (As in the first post, I used "flip both" in the pre-processing tab for darks in SharpCap for both lights, to match pixel locations with the master dark and individual darks, the calibrated lights have the blotches in the mirror image locations that the hot pixel groups would appear in the dark frames).

I then looked at each of the 16 individual darks at the two pixel locations that I had determined matched the two circled dark blotches, to see if any of the 16 darks had hot pixels at these locations. The good news is I found the culprit individual dark frames for each of the blotches! For the more prominent of the two blotches north (above) of M1, it turns out that dark #9 has a bunch of hot pixels in the correct location - the other 15 darks show nothing unusual at that location. Similarly, for the second, slightly less prominent blotch south (below) of M1, dark #16 has a bunch of hot pixels in the correct location, and the other 15 darks also show nothing unusual at that location.

For this test, I've included the master dark and individual darks (in fits and png format), and the calibrated capture of M1 in a sub-directory named "2nd set of Tests" in the google drive directory given in the first post. Here is a direct link to that google drive sub-directory which should allow downloading the files. (Again, dark #9 and dark #16 are the ones to look at if you want to see the hot pixel clumps.)

https://drive.google.com/drive/folders/ ... drive_link


For completeness, the approximate location in pixels relative to the upper left corner of the dark frame of the hot pixel clumps are:
For dark #9 : Vertical = 1032, Horizontal = 1144
For dark #16 : Vertical = 222, Horizontal = 1378
(These are the mirror image of the co-ordinates of the blotches in the flipped light image shown above due to the flip in pre-processing)

I should add that I could not see any dark blotches in non H-alpha captures of a couple of targets that I subsequently took that night with the same master dark, in line with Robin's reasoning from above. As the sky background intensity in an H-alpha capture may be less than the average pixel intensity of the problematic pixels in the master dark, the resulting area in the light will appear as a black blotch after "overcorrecting". This likely would not happen in a non H-alpha capture as the higher sky background intensity may exceed the average pixel intensity of the problematic pixels in the master dark - although the region is also overcorrected, it is not overcorrected enough to result in black blotches. At least that is my speculation on why I haven't seen these blotches without a H-alpha filter.


One last comment - I didn't measure the pixel intensity of these hot pixel clusters in dark #9 and dark #16, but they stood out like sore thumbs from the very uniform appearance of those areas in each of the other 15 frames. I'm guessing that identifying them as outliers should be straightforward, perhaps something similar to the sigma clipping algorithm option for averaging in a livestack?

Errol

[admin]

Hi Errol,

brilliant! - that not only helps understand what is going on, but also gives a path to fix the problem by some sort of outlier rejection when averaging the dark frames.

I have downloaded the shared files so that I can test my solution for this to make sure the sensitivity is appropriate to exclude the problem pixels but not touch much else in the dark averaging process (it will be interesting to see how far out the pixels are in terms of sigma level)

cheers,

Robin

[alphatripleplus]

That's great that you have a test solution to try on the subs, Robin. I'm looking forward to hearing what your findings are, and I'll be happy to try out whatever you decide upon on a nighttime target when you release it in a beta SharpCap.

Thanks so much,

Errol

[admin]

Hi Errol,

I have experimental code now that finds the issue in dark #16 at the position you recorded. In fact, the statistics of that pixel are that it is *hundreds* of standard deviations from the mean value of the other darks for that position! Looking at the individual dark frame, you can see that it's the brightest of a streak of bright pixels.

Screenshot 2024-04-27 175052.jpg (25.99 KiB) Viewed 128 times

I suspect that this was actually a high energy particle (cosmic ray?) passing through the sensor. My statistical technique is finding most of the bright pixels in the group.

It's similar in dark #9 - another streak in the image at roughly the co-ordinates you found.

You don't live somewhere radioactive do you?!

cheers,

Robin

[alphatripleplus]

I suspect that this was actually a high energy particle (cosmic ray?) passing through the sensor. My statistical technique is finding most of the bright pixels in the group.

It's similar in dark #9 - another streak in the image at roughly the co-ordinates you found.

You don't live somewhere radioactive do you?!

Hi Robin,

That makes sense. In the past, I've very occasionally seen multi-pixel long streaks in some of my master darks with both SharpCap and other software, which I have assumed were cosmic ray hits.

(Oh, and I forget to mention that my other "hobby" involves a mini-Tevatron that I set up near my scopes. I guess I should stop the high-energy particle experiments when I'm using the scopes - or at least turn down the peak energy in the accelerator. )

Thanks,

Errol