Over or under calibration is routinely happens to many astrophotograpers. People try to ignore this and rectify these "blemishes" in post processing. Bad practice! Images will suffer; no matter what.
Reason for such cutting corners is that taking traditional dark, flat, dark-flat frames all the time is extremely tedious. Now, SharfpCap offers unique opportunity to get master dark and master flats in one shot. But do they work? Here are several questions in this regard.
1. If master darks and flats have been taken in SharpCap, will they work in PixInsignt for calibration of lights that have been captured in NINA, Voyager, etc.?
2. Master flats in SharpCap have subtraction component - bias frames. Shouldn't it be subtraction of dark-flat frames? According to master astrophotographer, Mr. Adam Block, it is critical point. Otherwise, problem with under-correction during calibration will come.
3. Again, according to Mr. Adam Block, flats should have at least 2-3 sec exposure time. What will advise SharpCap? No way I could keep so long as 2-3 sec with my panel. It is too bright for 121 gain that I use for light frames.
4. In this respect, what gain and offset for flats are recommended by SharpCap?
Thanks in advance for help!
All the best!
Armen
Flats and problem with calibration
Forum rules
If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs
If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs
-
admin
- Site Admin
- Posts: 13350
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Flats and problem with calibration
Hi,
some people get good results using calibration frames captured in SharpCap in other processing software... Others try the same thing and hit issues that they find hard to fix. I suspect that it will all depend on what other processing is being done and also on how much stretch is applied to the final image (with enough stretch, even the very best flats will turn out to be slightly flawed). In the end, capturing all types of frames in the same application and processing them all together in another single application is probably the safest approach to ensure that everything is handled consistently, however as you point out this is the 'long way' of doing things, and if you can make a shortcut work better then that's fine.
On the subject of bias frames vs dark flat frames, the only difference between the two is the thermal noise built up over the longer exposure time of the dark flat frame. Modern cameras (particularly when cooled) suffer relatively little from thermal noise, so this difference can often be small enough not to matter, particularly if the flat frame exposure is short (sub 1s or so). The most important thing is to subtract away the much larger contribution from the pedestal level (black level) of the camera.
This brings us nicely onto the length of flat frames. Personally, I can see no valid reason to specify a particular length for flat frame exposures, but there are a number of things you do want to consider.
1) Most importantly you want to capture a *LOT* of photons during the capture of all your flat frames. In the final master flat, the relative noise level is proportional to the number of photons captured to the power -1/2, so capturing 4 times more photons means 1/2 the noise level. To avoid taking an exceptionally large number of flat frames, this suggests capturing flats at minimum gain to allow the largest full well depth.
2) You need to stay away from the range of pixel brightnesses that might be responding non-linearly - that means nothing above 90% brightness, maybe even nothing above 75 to 80%
3) If you are using an artificial light source, beware of the fact that it might flicker. Your exposure should be significantly longer (10 times or more longer) than the flicker period of the light source. This is particularly important for progressive scan cameras (which is almost all CMOS cameras).
Considering these will lead you naturally to finding the right exposure length for flats given your illumination levels.
Note that there is no reason to take flats and dark flats/bias at the same gain levels as you have used for your light frame captures, since all you are trying to do with flat frames is measure the relative amount of light falling on each part of the sensor. However, these two must be taken at the same gain/black level as each other.
Note that if you capture bias frames as a surrogate for dark-flats and intend to process in another application (not in SharpCap), tell the other application that the frames are dark-flats if they are taken at different gains - otherwise it may try to use them in the processing of the light frames, which would be incorrect.
Hope this helps,
Robin
some people get good results using calibration frames captured in SharpCap in other processing software... Others try the same thing and hit issues that they find hard to fix. I suspect that it will all depend on what other processing is being done and also on how much stretch is applied to the final image (with enough stretch, even the very best flats will turn out to be slightly flawed). In the end, capturing all types of frames in the same application and processing them all together in another single application is probably the safest approach to ensure that everything is handled consistently, however as you point out this is the 'long way' of doing things, and if you can make a shortcut work better then that's fine.
On the subject of bias frames vs dark flat frames, the only difference between the two is the thermal noise built up over the longer exposure time of the dark flat frame. Modern cameras (particularly when cooled) suffer relatively little from thermal noise, so this difference can often be small enough not to matter, particularly if the flat frame exposure is short (sub 1s or so). The most important thing is to subtract away the much larger contribution from the pedestal level (black level) of the camera.
This brings us nicely onto the length of flat frames. Personally, I can see no valid reason to specify a particular length for flat frame exposures, but there are a number of things you do want to consider.
1) Most importantly you want to capture a *LOT* of photons during the capture of all your flat frames. In the final master flat, the relative noise level is proportional to the number of photons captured to the power -1/2, so capturing 4 times more photons means 1/2 the noise level. To avoid taking an exceptionally large number of flat frames, this suggests capturing flats at minimum gain to allow the largest full well depth.
2) You need to stay away from the range of pixel brightnesses that might be responding non-linearly - that means nothing above 90% brightness, maybe even nothing above 75 to 80%
3) If you are using an artificial light source, beware of the fact that it might flicker. Your exposure should be significantly longer (10 times or more longer) than the flicker period of the light source. This is particularly important for progressive scan cameras (which is almost all CMOS cameras).
Considering these will lead you naturally to finding the right exposure length for flats given your illumination levels.
Note that there is no reason to take flats and dark flats/bias at the same gain levels as you have used for your light frame captures, since all you are trying to do with flat frames is measure the relative amount of light falling on each part of the sensor. However, these two must be taken at the same gain/black level as each other.
Note that if you capture bias frames as a surrogate for dark-flats and intend to process in another application (not in SharpCap), tell the other application that the frames are dark-flats if they are taken at different gains - otherwise it may try to use them in the processing of the light frames, which would be incorrect.
Hope this helps,
Robin
Re: Flats and problem with calibration
Thanks
I need to experiment.
This is my weakest point so far in astrophotography.
Pictures are ruined because I just cannot get perfect calibration.
Armen
I need to experiment.
This is my weakest point so far in astrophotography.
Pictures are ruined because I just cannot get perfect calibration.
Armen
Re: Flats and problem with calibration
Robin,admin wrote: ↑Sun Dec 11, 2022 3:47 pm Hi,
some people get good results using calibration frames captured in SharpCap in other processing software... Others try the same thing and hit issues that they find hard to fix. I suspect that it will all depend on what other processing is being done and also on how much stretch is applied to the final image (with enough stretch, even the very best flats will turn out to be slightly flawed). In the end, capturing all types of frames in the same application and processing them all together in another single application is probably the safest approach to ensure that everything is handled consistently, however as you point out this is the 'long way' of doing things, and if you can make a shortcut work better then that's fine.
On the subject of bias frames vs dark flat frames, the only difference between the two is the thermal noise built up over the longer exposure time of the dark flat frame. Modern cameras (particularly when cooled) suffer relatively little from thermal noise, so this difference can often be small enough not to matter, particularly if the flat frame exposure is short (sub 1s or so). The most important thing is to subtract away the much larger contribution from the pedestal level (black level) of the camera.
This brings us nicely onto the length of flat frames. Personally, I can see no valid reason to specify a particular length for flat frame exposures, but there are a number of things you do want to consider.
1) Most importantly you want to capture a *LOT* of photons during the capture of all your flat frames. In the final master flat, the relative noise level is proportional to the number of photons captured to the power -1/2, so capturing 4 times more photons means 1/2 the noise level. To avoid taking an exceptionally large number of flat frames, this suggests capturing flats at minimum gain to allow the largest full well depth.
2) You need to stay away from the range of pixel brightnesses that might be responding non-linearly - that means nothing above 90% brightness, maybe even nothing above 75 to 80%
3) If you are using an artificial light source, beware of the fact that it might flicker. Your exposure should be significantly longer (10 times or more longer) than the flicker period of the light source. This is particularly important for progressive scan cameras (which is almost all CMOS cameras).
Considering these will lead you naturally to finding the right exposure length for flats given your illumination levels.
Note that there is no reason to take flats and dark flats/bias at the same gain levels as you have used for your light frame captures, since all you are trying to do with flat frames is measure the relative amount of light falling on each part of the sensor. However, these two must be taken at the same gain/black level as each other.
Note that if you capture bias frames as a surrogate for dark-flats and intend to process in another application (not in SharpCap), tell the other application that the frames are dark-flats if they are taken at different gains - otherwise it may try to use them in the processing of the light frames, which would be incorrect.
Hope this helps,
Robin
I have bolded a few statements above that I'd like to discuss further.
Can you elaborate a bit more on "subtract away the much larger contribution from the pedestal level (black level) of the camera".
Regarding flat frame gain and the "minimum" as you suggest, would this be zero (0) or the lowest gain where HCG mode (if available) kicks in?
And when you say "these two must be taken at the same gain/black level as each other", are you are saying the Offset/Brightness/Black Level setting must be equal for the flats (none, bias or dark) and darks, or that Offset/Brightness/Black Level settings must be equal for lights, flats and darks?
I'm only using SharpCap to create my calibration frames and lights and save both PNG and 16-bit FITS final stacks. So my post processing is only of the final stacked image which I assume incorporates all the calibration frames. Even so, I'm have a tough time creating a viable flat for the 294 sensor, both the MC and MM versions.
Is there any reason why a higher gain flat (say 200 vs. 120) would work better with the 294? Also, ZWO recommends and average ADU of 24,000 for 294MM flats in Bin2 mode (your 11MB Bin1 mode). This is 36% of the full well capacity. This suggests that ZWO feels this sensor can respond non-linearly above this point which is quite a bit lower than say the 533/2600 sensor. What do you think?
-
admin
- Site Admin
- Posts: 13350
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Flats and problem with calibration
Hi,
Pedestal level / black level... Most cameras give you an adjustment (called variously black level, offset or sometimes brightness) to adjust the pixel value that is created for a pefectly black pixel with no signal. As we know, this shouldn't be zero, since otherwise the effect of unavoidable read noise means that pixel readouts that would be less than this value would be truncated to zero, leading to a bias in the data (you have thrown away negative values, but not positive). For short exposure dark frames (or bias frames), this ADU value is typically the largest component of the signal in the captured frame - much larger than the signal from read noise or thermal noise. This ADU value for a perfectly black pixel is sometimes called the pedestal level, since all other pixel values build on top of it. My point is that subtracting this value from flat frames is the important part...
Picking the ASI294 as an example, thermal noise is about 0.25e/pix/s at 20C, so maybe you get 0.5 electrons in a 2s exposure. At minimum gain, that works out to be about 0.5 (14 bit) ADU, or 2 ADU when scaled up to 16 bit. On the other hand, the pedestal level may be tens or even several hundred 16 bit ADU, so you can see that subtracting the pedestal is vastly more important than worrying about the relatively tiny contribution from thermal noise.
For flats you are good to use minimum gain (zero) - you are going to be illuminating every pixel with *thousands* of electrons in each frame, so the effect of read noise (HCG vs LCG) becomes insignificant. For instance at 50% full well, zero gain, you have about 32000 electrons per pixel, so the shot noise per frame is about 180e - compared to that the 8e read noise at minimum gain is tiny.
On the settings,
A) Flats and dark flats must use the same settings as each other (gain/offset/exposure/colour balance)
B) Light frames and dark frames must use the same settings as each other.
There is no requirement for the settings for A) to be the same as those for B). If you are using bias frames in A) instead of dark flats then the only difference should be the exposure.
Why should you use higher gains (200 not 120)? Because the Sony spec sheet for the sensor says so. Basically it says that you should only use HCG mode when the analog gain is also set to at least ~8.1dB of additional gain. The ZWO camera switches into HCG mode at a gain of 120, but at that point all of the gain (12dB, or about 4.5x) comes from the LCG->HCG switch. Beyond that point until you get to ZWO gain 200, you are operating the camera with settings that are not recommended by the sensor manufacturer. You can do it if you want, but you will get dodgy results.
Finally, my normal experience has been that most sensors are pretty much linear up to at least 90% saturation. I'm not aware of any special differences for the 294 sensors. SharpCap will measure linearity as part of the sensor analysis (as long as you have a steady source of illumination - if the light level fluctuates then it will see that as sensor non-linearity). I don't have figures to hand for the ZWO294MM, but I have them for an equivalent camera (294 mono sensor) from another vendor, and the linearity limit was well beyond 90%.
cheers,
Robin
Pedestal level / black level... Most cameras give you an adjustment (called variously black level, offset or sometimes brightness) to adjust the pixel value that is created for a pefectly black pixel with no signal. As we know, this shouldn't be zero, since otherwise the effect of unavoidable read noise means that pixel readouts that would be less than this value would be truncated to zero, leading to a bias in the data (you have thrown away negative values, but not positive). For short exposure dark frames (or bias frames), this ADU value is typically the largest component of the signal in the captured frame - much larger than the signal from read noise or thermal noise. This ADU value for a perfectly black pixel is sometimes called the pedestal level, since all other pixel values build on top of it. My point is that subtracting this value from flat frames is the important part...
Picking the ASI294 as an example, thermal noise is about 0.25e/pix/s at 20C, so maybe you get 0.5 electrons in a 2s exposure. At minimum gain, that works out to be about 0.5 (14 bit) ADU, or 2 ADU when scaled up to 16 bit. On the other hand, the pedestal level may be tens or even several hundred 16 bit ADU, so you can see that subtracting the pedestal is vastly more important than worrying about the relatively tiny contribution from thermal noise.
For flats you are good to use minimum gain (zero) - you are going to be illuminating every pixel with *thousands* of electrons in each frame, so the effect of read noise (HCG vs LCG) becomes insignificant. For instance at 50% full well, zero gain, you have about 32000 electrons per pixel, so the shot noise per frame is about 180e - compared to that the 8e read noise at minimum gain is tiny.
On the settings,
A) Flats and dark flats must use the same settings as each other (gain/offset/exposure/colour balance)
B) Light frames and dark frames must use the same settings as each other.
There is no requirement for the settings for A) to be the same as those for B). If you are using bias frames in A) instead of dark flats then the only difference should be the exposure.
Why should you use higher gains (200 not 120)? Because the Sony spec sheet for the sensor says so. Basically it says that you should only use HCG mode when the analog gain is also set to at least ~8.1dB of additional gain. The ZWO camera switches into HCG mode at a gain of 120, but at that point all of the gain (12dB, or about 4.5x) comes from the LCG->HCG switch. Beyond that point until you get to ZWO gain 200, you are operating the camera with settings that are not recommended by the sensor manufacturer. You can do it if you want, but you will get dodgy results.
Finally, my normal experience has been that most sensors are pretty much linear up to at least 90% saturation. I'm not aware of any special differences for the 294 sensors. SharpCap will measure linearity as part of the sensor analysis (as long as you have a steady source of illumination - if the light level fluctuates then it will see that as sensor non-linearity). I don't have figures to hand for the ZWO294MM, but I have them for an equivalent camera (294 mono sensor) from another vendor, and the linearity limit was well beyond 90%.
cheers,
Robin
-
Ray Hansen
- Posts: 39
- Joined: Sun Jun 11, 2017 5:54 am
- Location: Scottsdale Arizona
Re: Flats and problem with calibration
Last week, after having shot lights and flats over a couple nights, I was getting organized to do calibration. It occurred to me that my darks and bias masters were almost a year old, so I set up to shoot new ones. I also noticed there wasn't a ZWO change in the release note for the currrent release, so I did the program update. When the integration of the new lights was complete, I had a massive flat overcorrection and started rummaging in PI a bit to find any reason why it happened. I came up dry, so I sent Adam a few frames to look at, and after about 7 minutes, he says my Sharpcap versions for the darks/biases are different. I thought this was nuts because it was the same camera, same pixels, no ZWO changes, but when I dropped back the changed files, the problem was gone. From now on, I won't assume backward compatibility.
-
kaymann
- Posts: 111
- Joined: Thu Jun 16, 2022 2:59 pm
- Location: https://azkayfes.blogspot.com/
Re: Flats and problem with calibration
Okay, to cut through it all:
I always AP/DSO/LiveStack my ASI533MC Pro at Gain 105, Brightness 10, WB R 50, WB B 50, Brightness 10, Temp 0°C, Bin 1. Exposure 15 or 30 seconds.
SC for Live Stack and Flats, ASTAP for stacking and Siril for Post Processing.
So my darks should be same as above settings or does not matter (two sets one for 15 and one set for 30 seconds)?
So my flats, usually 60 frames, are taking with SC with the Bias radio box checked at above settings or same settings as darks?
Or should I take flats with darks radio box checked with above settings ?
Or?
Randall
I always AP/DSO/LiveStack my ASI533MC Pro at Gain 105, Brightness 10, WB R 50, WB B 50, Brightness 10, Temp 0°C, Bin 1. Exposure 15 or 30 seconds.
SC for Live Stack and Flats, ASTAP for stacking and Siril for Post Processing.
So my darks should be same as above settings or does not matter (two sets one for 15 and one set for 30 seconds)?
So my flats, usually 60 frames, are taking with SC with the Bias radio box checked at above settings or same settings as darks?
Or should I take flats with darks radio box checked with above settings ?
Or?
Randall
-
admin
- Site Admin
- Posts: 13350
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Flats and problem with calibration
Hi,
the darks should be at exactly the same settings as the light frames. For the flats, using bias is probably OK if the exposure for flats is less than 1s. If you are taking flats with longer exposures for some reason then use the darks option instead when capturing them.
Going back one more post to the one from Ray about backwards compatibility... Which two versions have compatibility problems? I will check to see what might have changed.
cheers,
Robin
the darks should be at exactly the same settings as the light frames. For the flats, using bias is probably OK if the exposure for flats is less than 1s. If you are taking flats with longer exposures for some reason then use the darks option instead when capturing them.
Going back one more post to the one from Ray about backwards compatibility... Which two versions have compatibility problems? I will check to see what might have changed.
cheers,
Robin
-
kaymann
- Posts: 111
- Joined: Thu Jun 16, 2022 2:59 pm
- Location: https://azkayfes.blogspot.com/
Re: Flats and problem with calibration
Thank you Robin
-
Ray Hansen
- Posts: 39
- Joined: Sun Jun 11, 2017 5:54 am
- Location: Scottsdale Arizona
Re: Flats and problem with calibration
Hello Robin,
Version I had for lights and flats was 4.0.9556. The version for darks and flat darks was 4.0.9562.
The camera is an ASI6200mm, and the software is Pixinsight. It never occurred to me it would be a problem,
but Adam called back after only a few minutes suggesting I try reshooting the darks and flat darks.
Apparently, he was familiar with a number of other software packages that exhibited the same issue.
RH
Version I had for lights and flats was 4.0.9556. The version for darks and flat darks was 4.0.9562.
The camera is an ASI6200mm, and the software is Pixinsight. It never occurred to me it would be a problem,
but Adam called back after only a few minutes suggesting I try reshooting the darks and flat darks.
Apparently, he was familiar with a number of other software packages that exhibited the same issue.
RH