Hot/Stuck Pixel resulting in central Black Holes

Discussion of using SharpCap for Deep Sky Imaging
Post Reply
Lorenz0x7BC
Posts: 5
Joined: Tue Jul 09, 2019 8:41 am

Hot/Stuck Pixel resulting in central Black Holes

#1

Post by Lorenz0x7BC »

Hi everyone, new to this forum here... (normally at CN forums)

I am currently trying to get a decent live stacking EAA workflow running with a 8in SCT, ASI 224 MC and SharpCap Pro.

There is one thing that still bothers me and I think if there is ever a solution for it, it must be related to better understanding how SharpCap applies dark frames. With these hot temperatures (30°C+) and the uncooled ASI 224 I get hot pixels that are really fast saturated (100%=maximum brightness after 2-4s exposure at Gain 300+). I know that this is to be expected but I thought with dark frames subtracted it could be mediated. Sadly, instead of single bright green or blue pixels, now when live stacking I get rather big black holes inside my targets.

As I understand it, SharpCap subtracts the averaged dark value for every pixel from the live stacked images, so it is clear that, when a dark pixel has the maximum value (255, white), every subtraction must result in black (0, minimum), therefore those black holes. So what can be done about this problem?

I read somewhere that other programs have Bad Pixel Maps and somewhere else that SharpCap does some kind of neighbor pixel averaging if the values are that extreme (one white pixel surrounded by black ones in the master dark), is this true?

To illustrate my problem here an example image (M81, 76x4s, Gain 300, RAW16, Evo8, ASI224, @F4, +dark):
Stack_76frames_304s_WithDisplayStretch.jpg
Stack_76frames_304s_WithDisplayStretch.jpg (410.67 KiB) Viewed 3272 times
and here an example master dark (25 frames, 8s, Gain 300, RAW16)
dark_25_frames_31,2C_2019-07-08T17_44_09.jpg
dark_25_frames_31,2C_2019-07-08T17_44_09.jpg (273.84 KiB) Viewed 3272 times
As you can see there are only two single pixels that really have maximum brightness, but those seem to result in quite visible artefacts.
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Hot/Stuck Pixel resulting in central Black Holes

#2

Post by admin »

Hi,

SharpCap subtracts the dark frame you selected from each light frame before adding the light frame onto the stack. It also has a hot pixel removal algorithm which should steal a value from a neighbouring pixel in the case where the dark frame has a hot pixel. The calculation of whether a pixel is hot is a little complicated as it depends on the statistical distribution of pixel brightness is in the dark frame, but the threshold for a hot pixel should never be above 95% of the maximum pixel value.

It seems likely that the hot pixel correction algorithm is not firing correctly your case. It would be interesting to see a copy of your dark frame (in fits or PNG format) to see what the brightness of the hottest pixels in the dark frame is.

Cheers, Robin
Lorenz0x7BC
Posts: 5
Joined: Tue Jul 09, 2019 8:41 am

Re: Hot/Stuck Pixel resulting in central Black Holes

#3

Post by Lorenz0x7BC »

Hi Robin!

Thanks for your help! Very much appreciated ;)

I wanted to post the original PNG files, but I could not because of the upload file size limit here in this forum. The original dark I used in the image above has 1.6 megabytes. How can I show it to you?

kind regards,
Lorenz
Lorenz0x7BC
Posts: 5
Joined: Tue Jul 09, 2019 8:41 am

Re: Hot/Stuck Pixel resulting in central Black Holes

#4

Post by Lorenz0x7BC »

Hi Robin!

In the meantime I did some more digging in my dark frame library and found surprising results!

I opened the original dark frame PNG files in Pixelmator (a Photoshop-like image processing application) and inspected in each of them the one central hot pixel that causes the "black hole". I measured its brightness as RGB value (in this application ranging from 0/0/0 to 255/255/255 for each pixel). I always use RAW16 color space so there are no colors in the darks, just brightness, which means R=G=B, so I will just specify brightness as single value between 0-255.

Here are the results (the dark used in the image above marked with exclamation signs):

Code: Select all

Measured Hot Pixel Brightness in various Master Darks (sorted by Gain, Exposure, Temperature)
==========================================================================================
* Gain 300
	* Exposure 2s
		* 26,5° -> 58
		* 31,5° -> 87
	* Exposure 4s 
		* 24,2° -> 90 !!!
		* 31,5° -> 169
	* Exposure 8s
		* 26,7° -> 208
		* 31,2° -> 254
* Gain 350
	* Exposure 2s
		* 26,6° -> 102
		* 30,7° -> 143
	* Exposure 4s
		* 26,6° -> 204
		* 30,8° -> 255
	* Exposure 8s
		* 26,8° -> 255
		* 31,1° -> 255
This means that I was wrong in my first post and that this hot pixel was not 100% saturated at Gain 300, Exposure 4s and Temperature 23,6 (= conditions of the M81 live stack in the image above), but as the dark frame shows, the hot pixel was around 90/255 (~35% saturated). 100% saturation kicks in at Gain 300, 8s, 31° or at Gain 350, 4s, 30°. This explains why your hot pixel correction algorithm did not fire.

Now I am more confused than ever. If under those conditions the guilty hot pixel is only around 35% saturated, why do I get too much subtraction? And how can it result in black pixels in the final (stacked and debayered) image if the hot pixel is clearly just a green pixel? I thought that in such a bright region like the center of M81 (high values in all RGB pixels) too much subtraction of a green pixel should result in purple colors (only red and blue left) und too little subtraction should result in green colors (green is overpowering red and blue). Why black? Why are red and blue pixels in RAW16 also affected?

Question: Would it be a valid workaround to edit the dark frame and insert a white pixel (255/255/255) at the location of those two hot pixels so that your algorithm can fire and take neighbor values?

Lastly for the sake of completeness, here are the camera settings of the M81 example image:

Code: Select all

[ZWO ASI224MC]
Debayer Preview=On
Pan=0
Tilt=0
Output Format=PNG files (*.png)
Binning=1
Capture Area=1304x976
Colour Space=RAW16
Temperature=23,6
Hardware Binning=Off
High Speed Mode=Off
Turbo USB=100
Flip=None
Frame Rate Limit=Maximum
Gain=300
Exposure=4
Timestamp Frames=Off
White Bal (B)=50
White Bal (R)=50
Brightness=0
Auto Exp Max Gain=300
Auto Exp Max Exp M S=30000
Auto Exp Target Brightness=112
Mono Bin=Off
Banding Threshold=35
Banding Suppression=0
Apply Flat=None
Subtract Dark=C:\Users\Lorenz\Desktop\SharpCap Captures\darks\ZWO ASI224MC\RAW16@1304x976\4,0s\gain_300\dark_10_frames_24,2C_2019-07-04T20_21_00.png
#Black Point
Display Black Point=0
#MidTone Point
Display MidTone Point=0,5
#White Point
Display White Point=1
TimeStamp=2019-07-04T22:12:39.3650236Z
SharpCapVersion=3.2.5986.0
Thanks again,
Lorenz
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Hot/Stuck Pixel resulting in central Black Holes

#5

Post by admin »

Hi,

I'm actually rather surprised that you are using dark frames that have 8-bit values in them if you are using 16-bit capture modes. That seems weird as your darks will be essentially very noisy due to quantisation errors (picking the nearest eight bit value to the true value of the pixel).

Anyway, that aside, it's certainly possible that the lower values on not being picked up by the hot pixel algorithm. The hot pixel threshold is based on the statistical measure of the variation of brightness in the dark frame (I think it uses five sigma above the mean level from memory), but this is limited to be between 25% and 95% of the maximum pixel value. So any pixel less than 25% of the maximum value will never be regarded as hot, while anything above 95% will always be regarded as hot. Values in between these points will depend on the statistics of the dark frame.

Certainly putting white pixels in the hot pixel locations would probably trigger the hot pixel code unless it is being sidestepped somehow by you using the 8-bit dark frames.

Cheers, Robin
Lorenz0x7BC
Posts: 5
Joined: Tue Jul 09, 2019 8:41 am

Re: Hot/Stuck Pixel resulting in central Black Holes

#6

Post by Lorenz0x7BC »

Hi Robin,

I'm sorry but there is a misunderstanding. All my darks have indeed 16-bit color depth, it is just that the color inspection tool of the image processing application I used (Pixelmator) shows color values only in range 0-255, but under the hood there is the full 16-bit value range (0-65535). So the values I stated in the list above are really only approximations. To compare saturation levels of hot pixels under different conditions I thought it would suffice.

OK, I will try to use those manually augmented dark frames next time I am out with the scope and see how it goes. I wrote a little ImageMagick script that automatically writes 2 white pixels at the specified pixel positions for every PNG I send to it to speed up the process.



One last thing regarding the origin of such "black holes" (see post above): I am thinking that the big visible black spot is caused by the drift of the image during live stacking when the one central hot pixel (via overcorrection in the dark frame) drifts around and deletes all signal there. In order to become black it seems to me that in RAW16 the one hot pixel (be it a green one for example) must drift over red and blue pixels too.

Here two questions arise:
Is the image debayered before or after the stacking? (for my hypothesis to work I think stacking happens still in RAW16 and before debayering)

And lastly I am wondering if switching to RGB24 color space would prevent this? (Because then every single frame would already be debayered and the overcorrected green pixel could never result in deleted red or blue pixels, just color shifts along the green axis)

Thanks,
Lorenz
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Hot/Stuck Pixel resulting in central Black Holes

#7

Post by admin »

Hi,

The image is debayered before stacking, doing it the other way round wouldn't work because you could only align in multiples of two pixels in each direction otherwise the different channels would become muddled with each other and you certainly couldn't derotate the image.

Remember that a single over corrected pixel on perhaps a green pixel will affect all the surrounding pixels when the debayering occurrs (all of them will be dimmer than they should be in the green channel). This makes all the surrounding pixel dimmer than expected in terms of their average brightness on all three channels too.

Cheers, Robin
Lorenz0x7BC
Posts: 5
Joined: Tue Jul 09, 2019 8:41 am

Re: Hot/Stuck Pixel resulting in central Black Holes

#8

Post by Lorenz0x7BC »

Thanks for clarifying, Robin!

kind regards,
Lorenz
Post Reply