John,
Speaking of APP, look how it saved my image of M13. I purposely used 3-second exposures for an experiment, so I wasn't surprised by the severe Raining Noise.
viewtopic.php?t=5363
Brian
Greenish background when imaging the Needle Galaxy
-
Moonstruck
- Posts: 114
- Joined: Tue Aug 17, 2021 4:34 pm
Re: Greenish background when imaging the Needle Galaxy
Thanks, I’ll take a look and see what is out there. I have an EQ6-R—Pro Mount and use EQMOD, which are very common, so I’m sure I can find something.
Re: Greenish background when imaging the Needle Galaxy
John
This is from an Astrobin search using your camera. https://www.astrobin.com/yix612/?q=ngc4 ... o%20asi294
The exposure was 300s per frame. Your tracking looks OK so have a go at 24x300s.
I processed your autosave file with Astro Pixel Processor and Neat image (for noise reduction) and produced this, which still shows the 'raining':
The APP steps on the left of the application were
This is the autosave loaded into APP, note the the histogram (top right) shows colours badly aligned
When applying light pollution removal note the bad area at the bottom left and also the right hand side. Probably best to crop as a first step. Note the histogram colours are now aligned and the green caste has gone.
The light gradients removed and the histogram tightened up.
Consider a 30 day trial of APP. There are good tutorial videos at the site which is how I made progress with the application.
Dave
This is from an Astrobin search using your camera. https://www.astrobin.com/yix612/?q=ngc4 ... o%20asi294
The exposure was 300s per frame. Your tracking looks OK so have a go at 24x300s.
I processed your autosave file with Astro Pixel Processor and Neat image (for noise reduction) and produced this, which still shows the 'raining':
- Autosave_1_-lpc-cbg-crop-cbg-csc-St_filtered.jpg (239.85 KiB) Viewed 789 times
The APP steps on the left of the application were
- crop
- remove light pollution
- calibrate star colours
- sharpening
- contrast
- Digital Development Processing (DDP)
- saturation
This is the autosave loaded into APP, note the the histogram (top right) shows colours badly aligned
- original.JPG (96.83 KiB) Viewed 789 times
When applying light pollution removal note the bad area at the bottom left and also the right hand side. Probably best to crop as a first step. Note the histogram colours are now aligned and the green caste has gone.
- before-light-polultion-removal.JPG (99.13 KiB) Viewed 789 times
The light gradients removed and the histogram tightened up.
- light-pollution-removal.JPG (105.13 KiB) Viewed 789 times
Consider a 30 day trial of APP. There are good tutorial videos at the site which is how I made progress with the application.
Dave
Re: Greenish background when imaging the Needle Galaxy
John
Having slept on this, the image below is completely processed with Astro Pixel Processor and no additional applications were used.
Steps in APP were crop, remove light pollution, calibrate star colours; apply sharpening, contrast, saturation and DDP. No noise reduction was done.
You could try experimenting with 180s, 240s and 300s exposures to see what the effect is on the raining noise. You could end up over-exposed, depending on the sky background. The raining noise is a puzzle because forum user @TimH has an ASI294 and he guides with PHD2 and does not mention dithering (but might be using dithering). No 2 skies/set of equipment are the same, so the figures are 'guidelines' rather than 'rules'.
Roger Clark https://clarkvision.com/articles/astrop ... ng.basics/ recommends not to use the autosave file (because of shifted colours, see green image histogram in post above) but to 'Save picture to a file ...'. If you are using APP then Deep Sky Stacker is no longer required as the calibration and stacking would be done with APP.
Dave
Having slept on this, the image below is completely processed with Astro Pixel Processor and no additional applications were used.
Steps in APP were crop, remove light pollution, calibrate star colours; apply sharpening, contrast, saturation and DDP. No noise reduction was done.
- Autosave_1_-crop-lpc-cbg-csc-St-2.jpg (477.37 KiB) Viewed 776 times
You could try experimenting with 180s, 240s and 300s exposures to see what the effect is on the raining noise. You could end up over-exposed, depending on the sky background. The raining noise is a puzzle because forum user @TimH has an ASI294 and he guides with PHD2 and does not mention dithering (but might be using dithering). No 2 skies/set of equipment are the same, so the figures are 'guidelines' rather than 'rules'.
Roger Clark https://clarkvision.com/articles/astrop ... ng.basics/ recommends not to use the autosave file (because of shifted colours, see green image histogram in post above) but to 'Save picture to a file ...'. If you are using APP then Deep Sky Stacker is no longer required as the calibration and stacking would be done with APP.
Dave
Re: Greenish background when imaging the Needle Galaxy
John
The Image Headers tab will show all the camera settings for the capture.
For your flat frame, the mean ADU is 33428. This is fine as the generally accepted figure is ~30000. Also good is that the histogram does not touch either the left or right hand sides. Keep the same light source and exposure. To adjust the figure Mean ADU either raise or lower the exposure. The 3 humps in the histogram are normal for a colour flat - this gets sorted as part of post-processing.
The bias frame is OK as the histogram does not touch either the left or right hand sides.
The unstretched light frame has a gap of 1792 at the left hand side. This is good because it means that faint data has not been clipped (lost).
at the left hand side
This is the stretched light frame, the stretch applied being ArcSinh(ArcSinh(x)). Again, the 3 humps get sorted in post-processing.
I think from my experience 150s or 180s exposures should be about right as this will make the histogram extend towards the right hand side.
When I am imaging deep sky, I take a test frame and inspect the histogram (unstretched and stretched) with FITS Liberator. That is all I do to decide on exposure. Of course, a few failures when processing the day after soon focuses the mind to pay attention to the detail.
Dave
ADU = Analog Digital Unit. For an image (in FITS format) these can be investigated using FITS Liberator which can be found at https://esahubble.org/projects/fits_lib ... nload_v30/.I'm not familiar with how to determine what the ADUs are for any given image, or how to control that... any advice?
The Image Headers tab will show all the camera settings for the capture.
For your flat frame, the mean ADU is 33428. This is fine as the generally accepted figure is ~30000. Also good is that the histogram does not touch either the left or right hand sides. Keep the same light source and exposure. To adjust the figure Mean ADU either raise or lower the exposure. The 3 humps in the histogram are normal for a colour flat - this gets sorted as part of post-processing.
- flat-frame.JPG (128.99 KiB) Viewed 773 times
The bias frame is OK as the histogram does not touch either the left or right hand sides.
- bias-frame.JPG (91.85 KiB) Viewed 773 times
The unstretched light frame has a gap of 1792 at the left hand side. This is good because it means that faint data has not been clipped (lost).
- unstretched-light.JPG (110.53 KiB) Viewed 773 times
This is the stretched light frame, the stretch applied being ArcSinh(ArcSinh(x)). Again, the 3 humps get sorted in post-processing.
- stretched-light-frame.JPG (114.13 KiB) Viewed 773 times
I think from my experience 150s or 180s exposures should be about right as this will make the histogram extend towards the right hand side.
When I am imaging deep sky, I take a test frame and inspect the histogram (unstretched and stretched) with FITS Liberator. That is all I do to decide on exposure. Of course, a few failures when processing the day after soon focuses the mind to pay attention to the detail.
Dave
Re: Greenish background when imaging the Needle Galaxy
Thanks, Dave.
So it looks like John needs to bake the image into his sensor, as evidenced by images at Astrobin and your analysis. I guess it is no wonder that there is so much discussion about Dithering on this forum. Hmm, my mono 290M CMOS camera gets along fine with 30-second exposures at Bortle 5. Read Noise 1.5e-. Uncooled, so I need to be careful about matching Darks to Lights. So far, though, I've only seen Raining Noise with 10-second exposures and less. 30-seconds is beautiful. I wonder why John's sensor needs to be baked? Generally, baking is what you do with CCD's.
Brian
So it looks like John needs to bake the image into his sensor, as evidenced by images at Astrobin and your analysis. I guess it is no wonder that there is so much discussion about Dithering on this forum. Hmm, my mono 290M CMOS camera gets along fine with 30-second exposures at Bortle 5. Read Noise 1.5e-. Uncooled, so I need to be careful about matching Darks to Lights. So far, though, I've only seen Raining Noise with 10-second exposures and less. 30-seconds is beautiful. I wonder why John's sensor needs to be baked? Generally, baking is what you do with CCD's.
Brian
Re: Greenish background when imaging the Needle Galaxy
Hi,
A StarTools user pointed me to this forum/thread, as there are quite a few misconceptions here about the specific problem, the data that StarTools needs, as well post processing in general.
First cab off the rank; the "problem" stated in the title.
Green background
A green stack is - usually - a healthy stack. For most sensors with a Bayer matrix in front of them (e.g. most DSLRs and OSC), green response is far greater than the other two channels. It is very important to not compensate for (aka "white balancing") this higher green response until your post-processing of choice has had a chance to create a luminance signal from the aggregate signal your sensor recorded.
Without waiting and immediately re-scaling (aka white balancing) the signal in the three channels, it becomes impossible to create a properly weighted luminance signal; noise will be higher, with all the issues that that entails.
Even if your post-processing software of choice does not do Bayer matrix luminance signal separation, it will at least be able to remove the green bias (subtraction), and only then color balance (multiplication) the three channels as you see fit.
The thick horizontal streaks
I believe the horizontal streaks are partially misdiagnosed.
They are, IMHO, simply the result of mismatching flats; dust/debris has accumulated since the flats have been last acquired and is not being compensated for. The object slowly drifting by between frames means that the dust will be slowly smeared out as the object is aligned. Which then also suggest that Tracking is not as good as it could be, and longer exposures (if at all needed) may become problematic.
Indeed their genesis "dark blobs" can be spotted in the single light frame.
Dithering and rain noise
Dithering is indeed highly recommended wherever practical. However, many SharpCap users often are EAA-practioners first, astrophotographers second, so constantly moving the scope during the EAA session is not really an option. Walking noise is typically present in undithered stacks in the form of single pixel lines/streaks, due to hotter and colder pixels being smeared out (through much the same mechanism indicated above) as the object drifts and is re-aligned. Calibration frames may help counter some of the aberrant readings of the hotter and colder pixels. Failing that, in StarTools specifically, there is dedicated Walking Noise tracking and reduction, specifically for EAA-practioners.
Aligning histogram peaks, and what they tells us
To make a long story short; very little. Histograms merely tell you how often a specific value occurs in a channel. They have very little use for determining a good color balance. The only reason why aligning their peaks may - accidentally - sometimes work, is if the sole object in the image is the perfect white reference. It so happens that a nearby spiral galaxy is a good white reference, where the average of all its pixels "should" be a milky white (or thereabouts). It so happens that this image contains such an object. In H-II areas (where the dominant color is determined by hydrogen emissions and perhaps some faint emission nebulosity in other channels), for example, this "method" falls flat on its face.
There are various useful techniques to determine a white balance; using a nearby spiral galaxy as a white reference, using an aggregate foreground star field as a reference, using known processes and their emission bands as a reference, using a star catalog + plate solving as a reference, or even going by a naive "grey world" assumption (the assumption that the average of all pixels should be grey). Aligning histogram peaks is not one of these, however.
DSS/StarTools/AutoSave/Clark
Clark is incorrect with regards to the autosave file (and most other things, but let's not get into that). Autosave.tiff is actually historically an intermediate/early("least processed") results file that is uncompressed and is straight from the stacker (with DSS being open source now, this can be verified). As such, it was historically the file of choice to import into StarTools. In later versions of DSS, it was possible to keep DSS from applying color balancing, stretches and - for DSLRs - matrix corrections, and save it the "normal" way with minimal processing applied. Ever since the addition of turning white balancing off, DSS has been a competent stacking tool, which should yield roughly comparable results to most other stacking solutions - provided it is given good data (I have found it deals less efficiently with troublesome data).
Processing this in StarTools
Import as "Linear and was Bayered, but not whitebalanced" <= specifically for "green" stacks like this!
== AutoDev
To see what we got (indeed it is precisely used here to visualise every imperfection!), we can see a gradients, garbage at the bottom, stacking artefacts, correlated noise (this needs investigating, though dithering may help here as well), the smeared out blobs.
== Crop
Crop away stacking artefacts.
== Wipe
Remove gradients. The Uncalibrated 1 or 2 (used here) presets will remove the gradients. You can also take the time to mask out all the smeared out blobs and use less aggressive settings. Normally none of this would be needed of course and a basic/default Wipe is usually all you need.
== AutoDev
The imperfections are no longer of interest, now that we ameliorated them to the best of our abilities. We can now give AutoDev the true Region of Interest to focus on. The RoI is / should be a portion of the image that is a good representation of the dynamic range that we want a global stretch to solve for. Sometimes it is the whole image (for example image-filling nebulosity), other times it is just a small portion of the image.
In this case, a good RoI is a slice of the galaxy. Be sure to make AutoDev ignore the fine correlated noise (this needs to be investigated) in the background (increase the Ignore Fine Detail < parameter). Getting to grips with AutoDev is absolute key to being able to use StarTools effectively!
E.g. something like this;
Focus on keeping noise at bay; you can be assured the rest of the stretch is always optimised to show "other" (real) detail in your selected RoI.
...
Form here you can process the details as much (or as little) as you wish (I skipped things for simplicity sake). Decon is virtually always recommended however, as, beyond recovering detail, it tightens and - as of v8 - corrects the shape of stars if they vary across the image. Corrrelated noise is decon's nemesis however, so gains are limited here.
== Color
With the detail processed, it is now time to introduce color. Find a good white balance according to the techniques mentioned earlier. I chose to create a sample mask for the galaxy here and used that as the white reference (e.g. put it in a mask, click "Sample" in the Color module, then clear/reset the mask). MaxRGB mode showed a slight green dominance in the core still - simply click on such an area to neutralize it. There were some small color aberrations around over-exposing stars (some chromatic aberration it appears?), so you could bump Highlight Repair if you wish. These color aberrations also make sampling the foreground stars for a white reference a lot less precise/useful.
Color balance after sampling galaxy mask and clicking on green dominant core was;
Parameter [Blue Bias Reduce] set to [1.00]
Parameter [Green Bias Reduce] set to [2.03]
Parameter [Red Bias Reduce] set to [1.60]
== Super Structure
For this particular problem (e.g. the smeared out blobs), the "Isolate" preset in the Super Structure module can neatly isolate the galaxy from the background. Tweak it to taste obviously;
I set parameter [Gamma] to [0.75], to lessen the effect.
Switch off Tracking for final noise elimination.
Parameter [Grain Size] set to [11.8 px]
You should end up with something like this;
Tweak everything to taste obviously.
I hope this helps!
A StarTools user pointed me to this forum/thread, as there are quite a few misconceptions here about the specific problem, the data that StarTools needs, as well post processing in general.
First cab off the rank; the "problem" stated in the title.
Green background
A green stack is - usually - a healthy stack. For most sensors with a Bayer matrix in front of them (e.g. most DSLRs and OSC), green response is far greater than the other two channels. It is very important to not compensate for (aka "white balancing") this higher green response until your post-processing of choice has had a chance to create a luminance signal from the aggregate signal your sensor recorded.
Without waiting and immediately re-scaling (aka white balancing) the signal in the three channels, it becomes impossible to create a properly weighted luminance signal; noise will be higher, with all the issues that that entails.
Even if your post-processing software of choice does not do Bayer matrix luminance signal separation, it will at least be able to remove the green bias (subtraction), and only then color balance (multiplication) the three channels as you see fit.
The thick horizontal streaks
I believe the horizontal streaks are partially misdiagnosed.
They are, IMHO, simply the result of mismatching flats; dust/debris has accumulated since the flats have been last acquired and is not being compensated for. The object slowly drifting by between frames means that the dust will be slowly smeared out as the object is aligned. Which then also suggest that Tracking is not as good as it could be, and longer exposures (if at all needed) may become problematic.
Indeed their genesis "dark blobs" can be spotted in the single light frame.
Dithering and rain noise
Dithering is indeed highly recommended wherever practical. However, many SharpCap users often are EAA-practioners first, astrophotographers second, so constantly moving the scope during the EAA session is not really an option. Walking noise is typically present in undithered stacks in the form of single pixel lines/streaks, due to hotter and colder pixels being smeared out (through much the same mechanism indicated above) as the object drifts and is re-aligned. Calibration frames may help counter some of the aberrant readings of the hotter and colder pixels. Failing that, in StarTools specifically, there is dedicated Walking Noise tracking and reduction, specifically for EAA-practioners.
Aligning histogram peaks, and what they tells us
To make a long story short; very little. Histograms merely tell you how often a specific value occurs in a channel. They have very little use for determining a good color balance. The only reason why aligning their peaks may - accidentally - sometimes work, is if the sole object in the image is the perfect white reference. It so happens that a nearby spiral galaxy is a good white reference, where the average of all its pixels "should" be a milky white (or thereabouts). It so happens that this image contains such an object. In H-II areas (where the dominant color is determined by hydrogen emissions and perhaps some faint emission nebulosity in other channels), for example, this "method" falls flat on its face.
There are various useful techniques to determine a white balance; using a nearby spiral galaxy as a white reference, using an aggregate foreground star field as a reference, using known processes and their emission bands as a reference, using a star catalog + plate solving as a reference, or even going by a naive "grey world" assumption (the assumption that the average of all pixels should be grey). Aligning histogram peaks is not one of these, however.
DSS/StarTools/AutoSave/Clark
Clark is incorrect with regards to the autosave file (and most other things, but let's not get into that). Autosave.tiff is actually historically an intermediate/early("least processed") results file that is uncompressed and is straight from the stacker (with DSS being open source now, this can be verified). As such, it was historically the file of choice to import into StarTools. In later versions of DSS, it was possible to keep DSS from applying color balancing, stretches and - for DSLRs - matrix corrections, and save it the "normal" way with minimal processing applied. Ever since the addition of turning white balancing off, DSS has been a competent stacking tool, which should yield roughly comparable results to most other stacking solutions - provided it is given good data (I have found it deals less efficiently with troublesome data).
Processing this in StarTools
Import as "Linear and was Bayered, but not whitebalanced" <= specifically for "green" stacks like this!
== AutoDev
To see what we got (indeed it is precisely used here to visualise every imperfection!), we can see a gradients, garbage at the bottom, stacking artefacts, correlated noise (this needs investigating, though dithering may help here as well), the smeared out blobs.
== Crop
Crop away stacking artefacts.
== Wipe
Remove gradients. The Uncalibrated 1 or 2 (used here) presets will remove the gradients. You can also take the time to mask out all the smeared out blobs and use less aggressive settings. Normally none of this would be needed of course and a basic/default Wipe is usually all you need.
== AutoDev
The imperfections are no longer of interest, now that we ameliorated them to the best of our abilities. We can now give AutoDev the true Region of Interest to focus on. The RoI is / should be a portion of the image that is a good representation of the dynamic range that we want a global stretch to solve for. Sometimes it is the whole image (for example image-filling nebulosity), other times it is just a small portion of the image.
In this case, a good RoI is a slice of the galaxy. Be sure to make AutoDev ignore the fine correlated noise (this needs to be investigated) in the background (increase the Ignore Fine Detail < parameter). Getting to grips with AutoDev is absolute key to being able to use StarTools effectively!
E.g. something like this;
- StarTools_2831.jpg (192.25 KiB) Viewed 738 times
...
Form here you can process the details as much (or as little) as you wish (I skipped things for simplicity sake). Decon is virtually always recommended however, as, beyond recovering detail, it tightens and - as of v8 - corrects the shape of stars if they vary across the image. Corrrelated noise is decon's nemesis however, so gains are limited here.
== Color
With the detail processed, it is now time to introduce color. Find a good white balance according to the techniques mentioned earlier. I chose to create a sample mask for the galaxy here and used that as the white reference (e.g. put it in a mask, click "Sample" in the Color module, then clear/reset the mask). MaxRGB mode showed a slight green dominance in the core still - simply click on such an area to neutralize it. There were some small color aberrations around over-exposing stars (some chromatic aberration it appears?), so you could bump Highlight Repair if you wish. These color aberrations also make sampling the foreground stars for a white reference a lot less precise/useful.
Color balance after sampling galaxy mask and clicking on green dominant core was;
Parameter [Blue Bias Reduce] set to [1.00]
Parameter [Green Bias Reduce] set to [2.03]
Parameter [Red Bias Reduce] set to [1.60]
== Super Structure
For this particular problem (e.g. the smeared out blobs), the "Isolate" preset in the Super Structure module can neatly isolate the galaxy from the background. Tweak it to taste obviously;
I set parameter [Gamma] to [0.75], to lessen the effect.
Switch off Tracking for final noise elimination.
Parameter [Grain Size] set to [11.8 px]
You should end up with something like this;
- Autosave(7).jpg (696.19 KiB) Viewed 738 times
I hope this helps!
Re: Greenish background when imaging the Needle Galaxy
Everyone!
The last post is from Ivo Jager, the creator of StarTools. Thank you for your expert analysis, Ivo. I never would have suspected Flats. I've tried a lot of tools but in my opinion StarTools is the most innovative; dare I say "radical"? (I mean that as a compliment.)
As I mentioned, I currently use Astro Pixel Processor (APP). How I got here is an interesting story, and I think it will sound familiar to a lot of astrophotographers. I'm always looking for the next breakthrough, anything to improve my images. Like that time when I discovered that my doublet doesn't focus Blue. I had to create a scheme for blocking blue light from fouling my images, and at the same time create decent color images. Then I joined the AAVSO to learn CCD Photometry. They took a conventional approach to processing, therefore I gravitated towards a more conventional tool like APP.
Where I am today is the sum total of past experiences. It is a learning experience. Ivo, thank you for your contribution! I'll certainly take a look at StarTools again.
Regards,
Brian
The last post is from Ivo Jager, the creator of StarTools. Thank you for your expert analysis, Ivo. I never would have suspected Flats. I've tried a lot of tools but in my opinion StarTools is the most innovative; dare I say "radical"? (I mean that as a compliment.)
As I mentioned, I currently use Astro Pixel Processor (APP). How I got here is an interesting story, and I think it will sound familiar to a lot of astrophotographers. I'm always looking for the next breakthrough, anything to improve my images. Like that time when I discovered that my doublet doesn't focus Blue. I had to create a scheme for blocking blue light from fouling my images, and at the same time create decent color images. Then I joined the AAVSO to learn CCD Photometry. They took a conventional approach to processing, therefore I gravitated towards a more conventional tool like APP.
Where I am today is the sum total of past experiences. It is a learning experience. Ivo, thank you for your contribution! I'll certainly take a look at StarTools again.
Regards,
Brian
Re: Greenish background when imaging the Needle Galaxy
Ivo
Thanks for your comprehensive response.
Dave
Thanks for your comprehensive response.
Dave
Re: Greenish background when imaging the Needle Galaxy
John,
Ivo speculates that the problem is with your Flats. How long ago did you capture them? If you fiddled with your camera by removing it, it could have accumulated dust. Also, if you use a Filter Wheel, then you MUST return the camera to the precise position...no rotational difference.
You might want to look into using an EL panel for Flats, then you can take a new set of Flats just prior to your imaging session. Flats are fast. You can capture 50 to 100 Flats in just a couple minutes. I use a 10cm x 10cm panel from Adafruit. Not much money and it works great.
Brian
Ivo speculates that the problem is with your Flats. How long ago did you capture them? If you fiddled with your camera by removing it, it could have accumulated dust. Also, if you use a Filter Wheel, then you MUST return the camera to the precise position...no rotational difference.
You might want to look into using an EL panel for Flats, then you can take a new set of Flats just prior to your imaging session. Flats are fast. You can capture 50 to 100 Flats in just a couple minutes. I use a 10cm x 10cm panel from Adafruit. Not much money and it works great.
Brian