Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Forum rules
If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs
Please also read about Troubleshooting USB Issues before posting.
*** Please do not post license keys - please report any problems with licensing to 'admin' by private message ***
Please include the following details in any bug report:
* Version of SharpCap
* Camera and other hardware being user
* Operating system version
* Contents of the SharpCap log after the problem has occurred.
[If SharpCap crashes, please send the bug report when prompted instead of including the log]
If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs
Please also read about Troubleshooting USB Issues before posting.
*** Please do not post license keys - please report any problems with licensing to 'admin' by private message ***
Please include the following details in any bug report:
* Version of SharpCap
* Camera and other hardware being user
* Operating system version
* Contents of the SharpCap log after the problem has occurred.
[If SharpCap crashes, please send the bug report when prompted instead of including the log]
Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Hi Robin,
Been endeavouring to do some good Darks and Flats (w/ Dark Flats) to better work with Live Stack, which is a great feature. First attempt saw the Triffid Nebula "emerge from the mists", which was great fun. But that night PHD2 was dropping the guide camera, the full moon was coming up and I forgot the filter! So this is all about better preparation for when the rain stops and the clouds clear ...
I used an iPad as a flat light source. Started at low gain, but ended up at the common gain=120 at -5 C for the ZWO 294MC Pro I use, to do Darks, Flats and Dark Flats (as part of SC 4.0's Flat capture routine). After the Flat capture the histograms for RGB all collapse onto a single peak.
Now I also select Dark subtraction in Camera Preprocessing and get this second histogram with these "side bands" - a few pixels showing either very low or very high values away from the main peak ...
Capturing a frame, debayering in PIPP and loading this into IDL, and zooming in on a small area in the top-left of frame, I see this in the Red channel, with one high and one low pixel ...
I wanted to understand where these divergent pixel values were coming from, so I looked at the same area in the Red channel of the Raw data, Dark and Flat frames. (Note: this "Raw data" is a new capture, with no Dark or Flat correction.) All maps use full histogram stretch ...
Raw data (for small area) ... Dark frame (same small area) ... Flat frame (same small area) ...
Looking at the values across the whole frame (and in this small area), it is not clear how we could get such divergent values ("SC" means SharpCap frame with Dark subtraction and Flat correction). The Dark values have small values and a small range, so don't impact greatly, and the Flat shows a small range of values.
Raw min/max= 27036 58284
Dark min/max= 264 3282
Flat min/max= 29156 59375
SC min/max= 20280 61931
For just this small view ...
Raw min/max= 29316 32272
Dark min/max= 308 338
Flat min/max= 31473 32901
SC min/max= 22005 55524
Tried an IDL program to perform: new = long((raw - dark) / (flat / mean(flat))) as a naive shot at Dark and Flat correction, and this seemed to work OK (no doubt you do some clever digital filters as well) and removed the gross variation in the Flat and brought R/G/B together. Just continuing looking at Red, I see this, which does not have any divergent pixels. Perhaps not surprizing given the ranges of values shown above.
So, that still leaves me wondering what is happening in SC 4.0.8244.0. Have you seen these telltale "side bands" in the histogram plots before? Is it something I have done wrong somewhere? Any tips?
If it is something awry in SC, let me know what else I might dig out ...
Cheers, Chris.
Been endeavouring to do some good Darks and Flats (w/ Dark Flats) to better work with Live Stack, which is a great feature. First attempt saw the Triffid Nebula "emerge from the mists", which was great fun. But that night PHD2 was dropping the guide camera, the full moon was coming up and I forgot the filter! So this is all about better preparation for when the rain stops and the clouds clear ...
I used an iPad as a flat light source. Started at low gain, but ended up at the common gain=120 at -5 C for the ZWO 294MC Pro I use, to do Darks, Flats and Dark Flats (as part of SC 4.0's Flat capture routine). After the Flat capture the histograms for RGB all collapse onto a single peak.
Now I also select Dark subtraction in Camera Preprocessing and get this second histogram with these "side bands" - a few pixels showing either very low or very high values away from the main peak ...
Capturing a frame, debayering in PIPP and loading this into IDL, and zooming in on a small area in the top-left of frame, I see this in the Red channel, with one high and one low pixel ...
I wanted to understand where these divergent pixel values were coming from, so I looked at the same area in the Red channel of the Raw data, Dark and Flat frames. (Note: this "Raw data" is a new capture, with no Dark or Flat correction.) All maps use full histogram stretch ...
Raw data (for small area) ... Dark frame (same small area) ... Flat frame (same small area) ...
Looking at the values across the whole frame (and in this small area), it is not clear how we could get such divergent values ("SC" means SharpCap frame with Dark subtraction and Flat correction). The Dark values have small values and a small range, so don't impact greatly, and the Flat shows a small range of values.
Raw min/max= 27036 58284
Dark min/max= 264 3282
Flat min/max= 29156 59375
SC min/max= 20280 61931
For just this small view ...
Raw min/max= 29316 32272
Dark min/max= 308 338
Flat min/max= 31473 32901
SC min/max= 22005 55524
Tried an IDL program to perform: new = long((raw - dark) / (flat / mean(flat))) as a naive shot at Dark and Flat correction, and this seemed to work OK (no doubt you do some clever digital filters as well) and removed the gross variation in the Flat and brought R/G/B together. Just continuing looking at Red, I see this, which does not have any divergent pixels. Perhaps not surprizing given the ranges of values shown above.
So, that still leaves me wondering what is happening in SC 4.0.8244.0. Have you seen these telltale "side bands" in the histogram plots before? Is it something I have done wrong somewhere? Any tips?
If it is something awry in SC, let me know what else I might dig out ...
Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Televue Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duo filters
- admin
- Site Admin
- Posts: 13344
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Hi Chris,
oooh... an interesting one
Firstly, the main difference I can think of in the SharpCap processing is special case handling of hot pixels - SharpCap uses a statistical approach on the dark frame data to try to find pixels that are hot/warm - the resulting pixel value for those pixels is stolen from a neighbouring pixel of the same colour rather than process the values from the pixel itself. Not sure how this could produce the very dark/bright pixels you are seeing though.
Secondly, I can see from the fact that your flat frame usage is collapsing the histogram to have overlapping peaks that you are not using the 'monochrome flat' option. I would normally recommend having the monochrome flat option turned on and dealing with the colour balance in later processing. Again, hard to be sure if this is having any effect on your situation though.
Beyond that, I think the best thing would be if you could share the dark/flat/light frame files that you investigated with me so that I can see if I can reproduce the issue and then understand what is going on. I wouldn't expect to see that sort of hot/dark pixels, so it looks like it is worth digging into.
cheers,
Robin
oooh... an interesting one
Firstly, the main difference I can think of in the SharpCap processing is special case handling of hot pixels - SharpCap uses a statistical approach on the dark frame data to try to find pixels that are hot/warm - the resulting pixel value for those pixels is stolen from a neighbouring pixel of the same colour rather than process the values from the pixel itself. Not sure how this could produce the very dark/bright pixels you are seeing though.
Secondly, I can see from the fact that your flat frame usage is collapsing the histogram to have overlapping peaks that you are not using the 'monochrome flat' option. I would normally recommend having the monochrome flat option turned on and dealing with the colour balance in later processing. Again, hard to be sure if this is having any effect on your situation though.
Beyond that, I think the best thing would be if you could share the dark/flat/light frame files that you investigated with me so that I can see if I can reproduce the issue and then understand what is going on. I wouldn't expect to see that sort of hot/dark pixels, so it looks like it is worth digging into.
cheers,
Robin
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Hi Robin,
I would think that a hot pixel routine (replacing anomalously high or low pixels by some local average) would not cause any strange pixels like this.
The "monochrome flat" option is an interesting/confusing one. Having read a recommendation in the manual to check the "create Monochrome Flat Frame" check-box, I have always set that (as in this clip I saved in my notes) ...
I only noticed recently that the Flat frame contains a Bayer pattern. So I used PIPP to debayer it, and that's what I have been looking at (as well as the Dark). PIPP does assume strange Bayer patterns at times, so I can't guarantee I delayered it correctly, which might confuse the colour identity, but not the main story here.
I'll attach the original files (without any debayering, etc.) ... Oops, files are too large. Can you send me your email address and I'll send a link to get them?
Cheers, Chris.
I would think that a hot pixel routine (replacing anomalously high or low pixels by some local average) would not cause any strange pixels like this.
The "monochrome flat" option is an interesting/confusing one. Having read a recommendation in the manual to check the "create Monochrome Flat Frame" check-box, I have always set that (as in this clip I saved in my notes) ...
I only noticed recently that the Flat frame contains a Bayer pattern. So I used PIPP to debayer it, and that's what I have been looking at (as well as the Dark). PIPP does assume strange Bayer patterns at times, so I can't guarantee I delayered it correctly, which might confuse the colour identity, but not the main story here.
I'll attach the original files (without any debayering, etc.) ... Oops, files are too large. Can you send me your email address and I'll send a link to get them?
Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Televue Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duo filters
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Televue Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duo filters
- admin
- Site Admin
- Posts: 13344
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Hi Chris,
thanks for sharing the files - I can definitely see something odd with hot pixels appearing when the dark is applied, so I will dig into that. I think the side bumps in the histogram are related to a few dust donuts/marks that are not being fully corrected by the flat (perhaps they moved between flat and light frame?). Anyway, I will report back once I understand the hot pixel issue and see how things look then...
cheers,
Robin
thanks for sharing the files - I can definitely see something odd with hot pixels appearing when the dark is applied, so I will dig into that. I think the side bumps in the histogram are related to a few dust donuts/marks that are not being fully corrected by the flat (perhaps they moved between flat and light frame?). Anyway, I will report back once I understand the hot pixel issue and see how things look then...
cheers,
Robin
- admin
- Site Admin
- Posts: 13344
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Hi,
OK, I found it... in the dark subtraction code there was a bug meaning that when a hot pixel was encountered on a colour camera in RAW mode, it was stealing a value from the pixel immediately to the left (which is a different colour) rather from one two to the left (same colour). This wasn't affecting live stacking (slightly different dark subtraction code didn't have the same bug in it), so it managed to hang around for a while without being spotted (most dark subtraction is used in conjunction with live stacking).
I have a fix, and it will be in Monday's update. It fixes the histogram 'wings' too.
cheers, and thanks for reporting this
Robin
OK, I found it... in the dark subtraction code there was a bug meaning that when a hot pixel was encountered on a colour camera in RAW mode, it was stealing a value from the pixel immediately to the left (which is a different colour) rather from one two to the left (same colour). This wasn't affecting live stacking (slightly different dark subtraction code didn't have the same bug in it), so it managed to hang around for a while without being spotted (most dark subtraction is used in conjunction with live stacking).
I have a fix, and it will be in Monday's update. It fixes the histogram 'wings' too.
cheers, and thanks for reporting this
Robin
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Nice find. Well done Robin.
Can’t wait ‘til Monday.
Cheers, Chris.
P.S. I added a Powermate 2x, which made the dust even more conspicuous. Fixed all that now.
Can’t wait ‘til Monday.
Cheers, Chris.
P.S. I added a Powermate 2x, which made the dust even more conspicuous. Fixed all that now.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Televue Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duo filters
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Hi Robin,
Thanks for the update. Have done a series of Flats using various options using SC 4.0.8264.0 and captured some single frames (using the same uniform light source - iPad screen close to telescope entrance) with the Flat enabled in Preprocessing.
Well, as they say in the classics, "the plot thickens". I can get a good flat using Bias or Dark subtraction, but only for low'ish settings of the Camera White Bal R and B controls (ZWO 294MC Pro). If I use the default (do nothing?) R=50, B=50 settings I get those "side bands" in the histograms, especially for Red and Blue, which indicate some very strange flat values, as discussed earlier.
And throughout all this it seems that always the Flat will do a colour balance in effect, bringing R, G, B histograms together. Doesn't seem to matter if I enable "Create monochrome Flat Frame" or not. Apart from one test, this is enabled throughout, as in the first example below (for 0111_5).
Case 1: Flat (monochrome enabled, with Dark subtraction) - 0111_5
Fortuitously, I had set a low White Bal (R) value = 40 (B = 50). Captured a Flat with Dark subtraction enabled, 30 frames. The light source was quite uniform, so we get fairly narrow peaks for R, G, B ...
Once completed, still looking at the iPad, the histograms for R, G, B collapse together ...
But apart from the "colour balancing" effect (despite the Monochrome selection), the Flat looks good, with just a handful of pixels that depart significantly from the norm. Success. I tried this one in Live Stack, and it worked well.
Case 2: Flat (monochrome disabled, with Dark subtraction) - 0121_8
I had reset the White Bal to R=50, not realizing the significance of this until later. It still normalizes/balances the colours, as above. But now we see the crazy histogram "side bands" encountered previously.
Debayering the Flat in PIPP and looking at the colour planes, we see these histograms for R, G, B ...
The Flat looks like this, which is OK. But it shows hundreds of pixels within these side bands, with very different values, which would adversely affect images when applied ...
Repeated this exercise many times, adjusting the R and B White Bal controls on the Camera device. Consistently, when the R,B values are close to 50/50 we get these artefacts. But if they are suppressed a little below 50, the "side bands" go away, and it all seems to work well.
... I don't appear to be able to attach more files, so I'll continue in a further post ...
Thanks for the update. Have done a series of Flats using various options using SC 4.0.8264.0 and captured some single frames (using the same uniform light source - iPad screen close to telescope entrance) with the Flat enabled in Preprocessing.
Well, as they say in the classics, "the plot thickens". I can get a good flat using Bias or Dark subtraction, but only for low'ish settings of the Camera White Bal R and B controls (ZWO 294MC Pro). If I use the default (do nothing?) R=50, B=50 settings I get those "side bands" in the histograms, especially for Red and Blue, which indicate some very strange flat values, as discussed earlier.
And throughout all this it seems that always the Flat will do a colour balance in effect, bringing R, G, B histograms together. Doesn't seem to matter if I enable "Create monochrome Flat Frame" or not. Apart from one test, this is enabled throughout, as in the first example below (for 0111_5).
Case 1: Flat (monochrome enabled, with Dark subtraction) - 0111_5
Fortuitously, I had set a low White Bal (R) value = 40 (B = 50). Captured a Flat with Dark subtraction enabled, 30 frames. The light source was quite uniform, so we get fairly narrow peaks for R, G, B ...
Once completed, still looking at the iPad, the histograms for R, G, B collapse together ...
But apart from the "colour balancing" effect (despite the Monochrome selection), the Flat looks good, with just a handful of pixels that depart significantly from the norm. Success. I tried this one in Live Stack, and it worked well.
Case 2: Flat (monochrome disabled, with Dark subtraction) - 0121_8
I had reset the White Bal to R=50, not realizing the significance of this until later. It still normalizes/balances the colours, as above. But now we see the crazy histogram "side bands" encountered previously.
Debayering the Flat in PIPP and looking at the colour planes, we see these histograms for R, G, B ...
The Flat looks like this, which is OK. But it shows hundreds of pixels within these side bands, with very different values, which would adversely affect images when applied ...
Repeated this exercise many times, adjusting the R and B White Bal controls on the Camera device. Consistently, when the R,B values are close to 50/50 we get these artefacts. But if they are suppressed a little below 50, the "side bands" go away, and it all seems to work well.
... I don't appear to be able to attach more files, so I'll continue in a further post ...
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Televue Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duo filters
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
Continuing ... Repeated this exercise many times, adjusting the R and B White Bal controls on the Camera device. Consistently, when the R,B values are close to 50/50 we get these artefacts. But if they are suppressed a little below 50, the "side bands" go away, and it all seems to work well.
One case, 0452_8, seemed "on the edge" and shows a few more poor pixel Flat values than desired.
Applied these Flats, again looking at the same uniform iPad screen, and captured single frames ...
To single out one of the bad ones, here are histograms for 0418_0 ... Now the anomalously high Flat pixels get suppressed to be anomalously low.
The result (top of frame has some artefact), shows a good "flattening" but has numerous bad pixels (shown in green). Another one is similar (0419_1).
So, I have a way to soften this effect, by adjusting the R (in particular) and B White Bal controls down from 50. Hence, I have Flats that work OK. But the puzzle still remains - where are these coming from in the SC Flat (w/ bias or Dark subtraction) analysis ...
I've added a few Flat and single frame capture results to the same shared area.
Cheers, Chris.
One case, 0452_8, seemed "on the edge" and shows a few more poor pixel Flat values than desired.
Applied these Flats, again looking at the same uniform iPad screen, and captured single frames ...
To single out one of the bad ones, here are histograms for 0418_0 ... Now the anomalously high Flat pixels get suppressed to be anomalously low.
The result (top of frame has some artefact), shows a good "flattening" but has numerous bad pixels (shown in green). Another one is similar (0419_1).
So, I have a way to soften this effect, by adjusting the R (in particular) and B White Bal controls down from 50. Hence, I have Flats that work OK. But the puzzle still remains - where are these coming from in the SC Flat (w/ bias or Dark subtraction) analysis ...
I've added a few Flat and single frame capture results to the same shared area.
Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Televue Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duo filters
Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?
I should point out that the images above are grey-scale with the outlier pixels (based on extreme Red pixels) highlighted in green.
Cheers, Chris.
Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Televue Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duo filters