Live Stack Dark/Flat correction - too much "salt" and "pepper"?

A place to report problems and bugs in SharpCap
Forum rules


If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs

Please also read about Troubleshooting USB Issues before posting.

*** Please do not post license keys - please report any problems with licensing to 'admin' by private message ***

Please include the following details in any bug report:

* Version of SharpCap
* Camera and other hardware being user
* Operating system version
* Contents of the SharpCap log after the problem has occurred.
[If SharpCap crashes, please send the bug report when prompted instead of including the log]
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#11

Post by admin »

Hi Chris,

as you say, the plot indeed thickens, but I think I have now worked out what was going on....

2 bugs

1) The code to convert the flat frame to mono has not worked for RAW mode cameras for erm... cough cough... an embarrasingly long time. From the source code history it looks like I was changing to make the mono conversion work in a slightly nicer way and ended up commenting out the line that actually did the conversion (perhaps trying to track down an issue) and forgot to put it back :oops:

2) The problem that I found and fixed previously with the dark subtraction in RAW mode was still happening when subtracting dark/bias frames for flats - this was because the RAW flat frame gets saved as FITS, then reloaded. The reload doesn't pick up on the fact that the frame was RAW and loads it as a mono frame. This wouldn't matter at all except for the dark subtraction hot pixel code which wrongly thinks it can steal values from neighbouring pixels rather than the one two to the left.

I think I have now fixed both of those - please have a play with this version and see if you agree : https://d.sharpcap.co.uk/download.html? ... p;arch=x64

If I had a 'SharpCap Chief Bug Finder' hat, I would be posting it to you right now ;)

cheers,

Robin
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#12

Post by ChrisR Oz »

Thanks for the compliment. Happy to help out.
And it’s a lot of fun to get such fast turn-around from you fixing bugs.
Another wet day, so back to the iPad fake sky …
Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#13

Post by ChrisR Oz »

Hi Robin,

It seems that with the 'bad guys' behind bars, the plot was in danger of getting a bit boring. But, alas, the plot still has some twists and turns ...

The very first shot at a Flat frame (0151_9), before I as taking proper notes, went absolutely bonkers in Green (note the "-infinity" in the file-name). See shared drive. But I was not able to reproduce it later, where the plot became a little less weird.

Bonkers (0151_9) Green channel
Bonkers (0151_9) Green channel
Flat issue21.jpg (58.13 KiB) Viewed 1332 times

After that the colour flats worked as expected. The Monos ("Create monochrome Flat file" enabled) were a little more interesting. They showed a satellite peak in just Blue (e.g. 0211_4), that matched the Green intensity ...

'Mono' Flat - Blue has satellite peak
'Mono' Flat - Blue has satellite peak
Flat issue22.jpg (28.08 KiB) Viewed 1332 times

Interestingly, if I reduce the White Bal B from 50 down in stages to 20 (e.g. 0241_1), the satellite peak stays at Green peak and does not follow Blue down. Evidently, Blue is still inheriting some Green pixel values.

Blue satellite stays with Green as Bal B goes down
Blue satellite stays with Green as Bal B goes down
Flat issue23.jpg (32.76 KiB) Viewed 1332 times

So, there is still some life in "the plot", but perhaps not enough for a new season ...
Cheers, Chris.

P.S. Files added to share drive.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
ChrisR Oz
Posts: 158
Joined: Tue Sep 28, 2021 11:03 am
Location: Melbourne

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#14

Post by ChrisR Oz »

Hi Robin,

Looking at the Blue histogram (for 0241_1) we see 3 groups actually ...

0241_1 Blue histogram
0241_1 Blue histogram
Flat issue24.jpg (47.6 KiB) Viewed 1328 times

Looking at the pixel positions of the low group on the left, I find these are all on row 0 (Y=0) of the image. Similarly, the next group is due to Y=1.

So it seems to be an edge effect, probably related to not finding a "row below" to use for a suitable alternate Blue value.

End of season, with no cliff hanger left hanging?

Cheers, Chris.
Celestron EdgeHD 8, reducer 0.7x, Star Sense, CGX-L mount, Focuser, CPWI; Starlight Xpress AO, OAG and Filter Wheel; ZWO 294MC/294MM Pro and 174MM mini; SharpCap Pro, PHD2, Powermate 2x, Baader Neodymium, Astronomik CLS-CCD, ZWO UV/IR, Duoband filters.
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#15

Post by admin »

Hi Chris,

Ok, so I am not going to chase edge effect issues at the moment I think, but the -Infinity offset is weird - looks like it happened twice from your filenames, so I will have a think about how that could come about to see if it makes any sense.

If you find any cases of incorrect pixel values away from the edges (you can use the histogram ROI tool in SharpCap to limit the area represented by the histogram) then do shout out.

cheers,

Robin
garyhawkins
Posts: 54
Joined: Sun Dec 22, 2019 6:59 pm

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#16

Post by garyhawkins »

Hi Robin,

So Dave Decker and I, from the San Diego Astronomy Association, have also been discussing SC flat-file correction in recent weeks, primarily associated with the under or over correction of SC images. In fact, I'd posted another forum thread asking what degree of correction is possible? If the algorithms are working as they should, what degree of correction should you expect? While the achievable 'flatness' is probably highly dependent on equipment configuration, if it's possible to answer, nominally should the corrected file be 'flat' with 0.1%, 1%, etc?

Attached is a flat file that I recently obtained for my C8 SCT/F6.3 focal reducer system, displayed in AIJ.
Peak FlatFile Values.jpg
Peak FlatFile Values.jpg (111.6 KiB) Viewed 1131 times
This is a highly stretched image. It is interesting to note that my background peak levels only vary between approx. 22,400 and 25,400 ADUs. This equates to a variation of approx 12%, indicating vignetting is not particularly severe. The presence of a number of dust bunnies can be seen and ADU variations are even smaller, in the range of 300 to 500 ADUs. If this flat file is applied against an image what degree of correction would you expect to see?

Best regards,

Gary
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#17

Post by admin »

Hi,

somewhere in the range of 98% or more reduction in variation if everything goes well.The simple way to check is to leave the flat illumination in place and apply the flat frame - you should see an uniform illumination across the image. Now reduce the brightness by cutting exposure to 1/2 or 1/4 of the starting value - check the image is still uniform (if not it's a sign that flat correction isn't working as it should - usually because of a lack of dark/bias correction on either the flat frames or the light frames).

cheers,

Robin
garyhawkins
Posts: 54
Joined: Sun Dec 22, 2019 6:59 pm

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#18

Post by garyhawkins »

Hi Robin,

Achieving a 98% reduction of vignetting/dust bunnies variations is substantial, and should result in it being close to impossible to see variations of this type in the corrected image. So if my vignetting variation is approx 3000 ADU across the image, this should reduce to around 60 ADUs?

I had the same thought for testing, "the simple way to check is to leave the flat illumination in place and apply the flat frame - you should see a uniform illumination across the image." However, on my first six attempts, it failed! This seems to have resulted from having the "apply new flat when capture complete" SET. The strange thing was during the capture of these first test images, I thought I observed that no information was showing in the "Apply Flat" box in the Preprocessing Box, and thus, I assumed the flat-file correction was not taking place. However, the resulting image file showed a flat image, and then applying the created flat when redoing the data using the Folder Monitor function within the Camera selection window resulted in the artifacts being introduced. Clearly, the flat application had occurred twice and my captured test images were already corrected.

The next time I did the experiment, I was more careful. I made sure the 'apply new flat when capture complete' was NOT SET. The flat file showed the expected characteristics. On capturing my test images with no flat applied, the same artifacts were visible. When I used the Folder Monitor in Camera, selected the flat file I created, and did a livestack of the captured raw images, the resulting livestack was artifact-free, and clearly, the flat had been correctly applied.

So this raises a couple of questions:

1. Exactly how is "apply new flat when capture complete" working relating to applying the flat, and should the flat file name have appeared in the "Apply Flat" box in the Preprocessing Box if 'apply new flat when capture complete' is SET?
2. If you try to capture a color flat you get a warning message. If Mono flats are the best way, why is this option available?
3. Why are there now two options for dark/bias application? When should each be used?

CS Gary
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#19

Post by admin »

Hi,

So the 'apply flat automatically' should turn the flat correction on immediately and show the flat file name in the settings. Note that the calculation of the averaged flat image can take some time after the capture of flat frames is complete (a few seconds up to maybe 30-60s depending on frame size and count). Maybe it was still calculating? Additionally, if you already have a flat frame applied it should automatically be turned off during capturing a new flat.

The option to make a colour flat frame that corrects for colour balance is one of those things that is hard to take away because it is already there and it's inevitable that some people are using quite successfully and would miss it!

Two options for bias and darks - the use here depends on the length of your flat frame exposures. If you take short exposure flat frames (<1s) then bias frames should be sufficient. I'm told that some cameras seem to give better results with longer exposure flats, and that in other cases longer exposures are needed because the flat illumination is weak. In those cases using darks instead of bias may help.

cheers,

Robin
garyhawkins
Posts: 54
Joined: Sun Dec 22, 2019 6:59 pm

Re: Live Stack Dark/Flat correction - too much "salt" and "pepper"?

#20

Post by garyhawkins »

Hi Robin,

I was not allowing enough time for the flat creation process to finish - the flat file does show up with 'apply flat automatically' checked.

OK on color and bias/dark options.

I've started looking at flat correction under controlled conditions (that is I'm taking the master flat away from images captured under exactly the same conditions) and the results are good enough I'm not seeing any residual artifacts against the noise. In my previous example, the uncorrected file had a vignetting variation of approx 3000ADUs in peak value. In the flat corrected file, the vignetting is not visible, and noise in the order of a few hundred ADUs seems to dominate ADU variations - which is excellent. The question now remains - why under certain conditions, have flat corrected live stacks looked under or overcooked. I need to look at this when we next observe it.

CS Gary
Post Reply