Live Stack Dark/Flat correction - too much "salt" and "pepper"?

A place to report problems and bugs in SharpCap
Forum rules


If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs

Please also read about Troubleshooting USB Issues before posting.

*** Please do not post license keys - please report any problems with licensing to 'admin' by private message ***

Please include the following details in any bug report:

* Version of SharpCap
* Camera and other hardware being user
* Operating system version
* Contents of the SharpCap log after the problem has occurred.
[If SharpCap crashes, please send the bug report when prompted instead of including the log]
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#1

Post by ChrisR Oz »

Hi Robin,

Been endeavouring to do some good Darks and Flats (w/ Dark Flats) to better work with Live Stack, which is a great feature. First attempt saw the Triffid Nebula "emerge from the mists", which was great fun. But that night PHD2 was dropping the guide camera, the full moon was coming up and I forgot the filter! So this is all about better preparation for when the rain stops and the clouds clear ...

I used an iPad as a flat light source. Started at low gain, but ended up at the common gain=120 at -5 C for the ZWO 294MC Pro I use, to do Darks, Flats and Dark Flats (as part of SC 4.0's Flat capture routine). After the Flat capture the histograms for RGB all collapse onto a single peak.
Histograms before/after Flat correction
Histograms before/after Flat correction
Flat issue1.jpg (46.67 KiB) Viewed 1025 times

Now I also select Dark subtraction in Camera Preprocessing and get this second histogram with these "side bands" - a few pixels showing either very low or very high values away from the main peak ...
with Dark subtract added in Preprocessing
with Dark subtract added in Preprocessing
Flat issue2.jpg (40.12 KiB) Viewed 1025 times

Capturing a frame, debayering in PIPP and loading this into IDL, and zooming in on a small area in the top-left of frame, I see this in the Red channel, with one high and one low pixel ...
SC Dark subtracted, Flat corrected frame
SC Dark subtracted, Flat corrected frame
Flat issue3.jpg (38.46 KiB) Viewed 1025 times

I wanted to understand where these divergent pixel values were coming from, so I looked at the same area in the Red channel of the Raw data, Dark and Flat frames. (Note: this "Raw data" is a new capture, with no Dark or Flat correction.) All maps use full histogram stretch ...

Raw data (for small area) ...
Raw frame (small area)
Raw frame (small area)
Flat issue4.jpg (71.29 KiB) Viewed 1025 times
Dark frame (same small area) ...
Dark frame (same small area)
Dark frame (same small area)
Flat issue5.jpg (64.93 KiB) Viewed 1025 times
Flat frame (same small area) ...
Flat frame (same small area)
Flat frame (same small area)
Flat issue6.jpg (62.68 KiB) Viewed 1025 times

Looking at the values across the whole frame (and in this small area), it is not clear how we could get such divergent values ("SC" means SharpCap frame with Dark subtraction and Flat correction). The Dark values have small values and a small range, so don't impact greatly, and the Flat shows a small range of values.

Raw min/max= 27036 58284
Dark min/max= 264 3282
Flat min/max= 29156 59375
SC min/max= 20280 61931

For just this small view ...
Raw min/max= 29316 32272
Dark min/max= 308 338
Flat min/max= 31473 32901
SC min/max= 22005 55524

Tried an IDL program to perform: new = long((raw - dark) / (flat / mean(flat))) as a naive shot at Dark and Flat correction, and this seemed to work OK (no doubt you do some clever digital filters as well) and removed the gross variation in the Flat and brought R/G/B together. Just continuing looking at Red, I see this, which does not have any divergent pixels. Perhaps not surprizing given the ranges of values shown above.
Naive Dark subtracted and Flat corrected (small area)
Naive Dark subtracted and Flat corrected (small area)
Flat issue7.jpg (74.53 KiB) Viewed 1025 times

So, that still leaves me wondering what is happening in SC 4.0.8244.0. Have you seen these telltale "side bands" in the histogram plots before? Is it something I have done wrong somewhere? Any tips?

If it is something awry in SC, let me know what else I might dig out ...
Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#2

Post by admin »

Hi Chris,

oooh... an interesting one :)

Firstly, the main difference I can think of in the SharpCap processing is special case handling of hot pixels - SharpCap uses a statistical approach on the dark frame data to try to find pixels that are hot/warm - the resulting pixel value for those pixels is stolen from a neighbouring pixel of the same colour rather than process the values from the pixel itself. Not sure how this could produce the very dark/bright pixels you are seeing though.

Secondly, I can see from the fact that your flat frame usage is collapsing the histogram to have overlapping peaks that you are not using the 'monochrome flat' option. I would normally recommend having the monochrome flat option turned on and dealing with the colour balance in later processing. Again, hard to be sure if this is having any effect on your situation though.

Beyond that, I think the best thing would be if you could share the dark/flat/light frame files that you investigated with me so that I can see if I can reproduce the issue and then understand what is going on. I wouldn't expect to see that sort of hot/dark pixels, so it looks like it is worth digging into.

cheers,

Robin
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#3

Post by ChrisR Oz »

Hi Robin,

I would think that a hot pixel routine (replacing anomalously high or low pixels by some local average) would not cause any strange pixels like this.

The "monochrome flat" option is an interesting/confusing one. Having read a recommendation in the manual to check the "create Monochrome Flat Frame" check-box, I have always set that (as in this clip I saved in my notes) ...
Capture flats popup snapshot
Capture flats popup snapshot
capture monochrome.jpg (67.93 KiB) Viewed 989 times

I only noticed recently that the Flat frame contains a Bayer pattern. So I used PIPP to debayer it, and that's what I have been looking at (as well as the Dark). PIPP does assume strange Bayer patterns at times, so I can't guarantee I delayered it correctly, which might confuse the colour identity, but not the main story here.

I'll attach the original files (without any debayering, etc.) ... Oops, files are too large. Can you send me your email address and I'll send a link to get them?

Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#4

Post by ChrisR Oz »

Hi Robin,

Files are here: https://cloudstor.aarnet.edu.au/plus/s/eE1Zp1bSxwDTYDZ

Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#5

Post by admin »

Hi Chris,

thanks for sharing the files - I can definitely see something odd with hot pixels appearing when the dark is applied, so I will dig into that. I think the side bumps in the histogram are related to a few dust donuts/marks that are not being fully corrected by the flat (perhaps they moved between flat and light frame?). Anyway, I will report back once I understand the hot pixel issue and see how things look then...

cheers,

Robin
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#6

Post by admin »

Hi,

OK, I found it... in the dark subtraction code there was a bug meaning that when a hot pixel was encountered on a colour camera in RAW mode, it was stealing a value from the pixel immediately to the left (which is a different colour) rather from one two to the left (same colour). This wasn't affecting live stacking (slightly different dark subtraction code didn't have the same bug in it), so it managed to hang around for a while without being spotted (most dark subtraction is used in conjunction with live stacking).

I have a fix, and it will be in Monday's update. It fixes the histogram 'wings' too.

cheers, and thanks for reporting this :)

Robin
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#7

Post by ChrisR Oz »

Nice find. Well done Robin.
Can’t wait ‘til Monday.
Cheers, Chris.

P.S. I added a Powermate 2x, which made the dust even more conspicuous. Fixed all that now.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#8

Post by ChrisR Oz »

Hi Robin,

Thanks for the update. Have done a series of Flats using various options using SC 4.0.8264.0 and captured some single frames (using the same uniform light source - iPad screen close to telescope entrance) with the Flat enabled in Preprocessing.

Well, as they say in the classics, "the plot thickens". I can get a good flat using Bias or Dark subtraction, but only for low'ish settings of the Camera White Bal R and B controls (ZWO 294MC Pro). If I use the default (do nothing?) R=50, B=50 settings I get those "side bands" in the histograms, especially for Red and Blue, which indicate some very strange flat values, as discussed earlier.

And throughout all this it seems that always the Flat will do a colour balance in effect, bringing R, G, B histograms together. Doesn't seem to matter if I enable "Create monochrome Flat Frame" or not. Apart from one test, this is enabled throughout, as in the first example below (for 0111_5).

Case 1: Flat (monochrome enabled, with Dark subtraction) - 0111_5

Fortuitously, I had set a low White Bal (R) value = 40 (B = 50). Captured a Flat with Dark subtraction enabled, 30 frames. The light source was quite uniform, so we get fairly narrow peaks for R, G, B ...

Case 0111_5 setup
Case 0111_5 setup
Flat issue8.jpg (102.01 KiB) Viewed 910 times

Once completed, still looking at the iPad, the histograms for R, G, B collapse together ...

Case 0111_5 after Flat
Case 0111_5 after Flat
Flat issue9.jpg (128.98 KiB) Viewed 910 times

But apart from the "colour balancing" effect (despite the Monochrome selection), the Flat looks good, with just a handful of pixels that depart significantly from the norm. Success. I tried this one in Live Stack, and it worked well.

Case 2: Flat (monochrome disabled, with Dark subtraction) - 0121_8

I had reset the White Bal to R=50, not realizing the significance of this until later. It still normalizes/balances the colours, as above. But now we see the crazy histogram "side bands" encountered previously.

Case 0121_8 after Flat
Case 0121_8 after Flat
Flat issue10.jpg (120.67 KiB) Viewed 910 times

Debayering the Flat in PIPP and looking at the colour planes, we see these histograms for R, G, B ...

Case 0121_8 Red histogram
Case 0121_8 Red histogram
Flat issue11.jpg (34.18 KiB) Viewed 910 times
Case 0121_8 Green histogram
Case 0121_8 Green histogram
Flat issue12.jpg (31.91 KiB) Viewed 910 times
Case 0121_8 Blue histogram
Case 0121_8 Blue histogram
Flat issue13.jpg (32.24 KiB) Viewed 910 times

The Flat looks like this, which is OK. But it shows hundreds of pixels within these side bands, with very different values, which would adversely affect images when applied ...

Case 0121_8 Flat with crazy "Red" pixels in green
Case 0121_8 Flat with crazy "Red" pixels in green
Flat issue14.jpg (59.61 KiB) Viewed 910 times

Repeated this exercise many times, adjusting the R and B White Bal controls on the Camera device. Consistently, when the R,B values are close to 50/50 we get these artefacts. But if they are suppressed a little below 50, the "side bands" go away, and it all seems to work well.

... I don't appear to be able to attach more files, so I'll continue in a further post ...
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#9

Post by ChrisR Oz »

Continuing ... Repeated this exercise many times, adjusting the R and B White Bal controls on the Camera device. Consistently, when the R,B values are close to 50/50 we get these artefacts. But if they are suppressed a little below 50, the "side bands" go away, and it all seems to work well.

Table of Flat tests
Table of Flat tests
Flat issue15.jpg (102.32 KiB) Viewed 910 times

One case, 0452_8, seemed "on the edge" and shows a few more poor pixel Flat values than desired.

Applied these Flats, again looking at the same uniform iPad screen, and captured single frames ...

Table of single captures, Flat applied
Table of single captures, Flat applied
Flat issue16.jpg (77.3 KiB) Viewed 910 times

To single out one of the bad ones, here are histograms for 0418_0 ... Now the anomalously high Flat pixels get suppressed to be anomalously low.

0418_0 Red
0418_0 Red
Flat issue17.jpg (35.66 KiB) Viewed 910 times
0418_0 Green
0418_0 Green
Flat issue18.jpg (33.97 KiB) Viewed 910 times
0418_0 Blue
0418_0 Blue
Flat issue19.jpg (35.74 KiB) Viewed 910 times

The result (top of frame has some artefact), shows a good "flattening" but has numerous bad pixels (shown in green). Another one is similar (0419_1).

0418_0 flattened, but bad pixels (green)
0418_0 flattened, but bad pixels (green)
Flat issue20.jpg (436.18 KiB) Viewed 910 times

So, I have a way to soften this effect, by adjusting the R (in particular) and B White Bal controls down from 50. Hence, I have Flats that work OK. But the puzzle still remains - where are these coming from in the SC Flat (w/ bias or Dark subtraction) analysis ...

I've added a few Flat and single frame capture results to the same shared area.
Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#10

Post by ChrisR Oz »

I should point out that the images above are grey-scale with the outlier pixels (based on extreme Red pixels) highlighted in green.
Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
Post Reply