Smart Histogram sets a low offset

Somewhere to ask questions about the best way to use SharpCap
Forum rules


If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs
Post Reply
LandyJon
Posts: 6
Joined: Tue Jan 04, 2022 11:34 am

Smart Histogram sets a low offset

#1

Post by LandyJon »

Hi, sorry if this has been covered elsewhere, I couldn't find anything, please link any topics that might help.

I've been trying to get Sensor analysis to give me a ballpark exposure length, gain and offset for a particular target via the smart histogram. My first problem is that the measurement process directs you to frame your target, yet documentation I've found on the forum implies you should point at a dark region of sky without nebulosity or many stars. Which is right ? considering I have a long focal length and small sensor so something like Flame Nebula near fills my frame, so on target there isn't much dark region for SC to concentrate on.

My main issue tho is having run sensor analysis with a light panel for flat illumination and got a nice straight line graph with no spurious plots, even if I use a dark region of sky for the measurement, I get no settings solution for less than 5 min exposures and the solutions offered for 5 or 6 min subs always set an offset of just 4 !!

I've spent an all nighter shooting about 4 hrs M81 and 2 hrs Flame, both of which gave a nice image in Live Stack behind the amp glow etc., however when I restack the RAW frames in DSS after I've taken darks, I seem to lose all the data. Is this because the offset is too low ? and why is Smart Histogram setting it so ? or am I doing something else wrong ?

I've done some dark analysis and found offset settings for various gain values where the histogram doesn't clip the left end, should I be using these values, or are these a minimum and if so how much higher should I go ? 10% or is that too much or not enough ? fyi at unity gain of 111 dark histogram clips until I reach 20 offset. Am I right in thinking I need to run an offset that has a complete curve in the dark histogram ? so that curve can be subtracted from the light frames, is that how it works ?

I can try to add any files that might help.

I'm in bortle 6 skies with IDAS D2 filter on ZWO ASI183MC pro Nexstar 6SE with 6.3 reducer (making best of what I got cheap before I knew anything !)

Many thanks.
User avatar
admin
Site Admin
Posts: 8144
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Smart Histogram sets a low offset

#2

Post by admin »

Hi,

first, the reason for differing instructions depending on where you look is that the system for measuring sky background changed in SharpCap 4 - In SharpCap 3.2 and before, you needed to select a dark area of the frame manually, possibly looking for a clear area of sky to get the darkest reading.

SharpCap 4 takes a different approach - simply frame your target - SharpCap will divide the frame into over 100 sub areas and choose the darkest of these for taking the measurements automatically. This should give results that are better optimised for the exact image that you are taking.

Now on to the exposure lengths... Some quick estimation suggests that your setup should have a sky background rate of about 0.35 electrons per pixel per second (https://tools.sharpcap.co.uk - dividing the result for the colour sensor by 2 to account for the LP filter). How does that compare with the measured values you are seeing? If you do have a value of about 0.35, then with a 10% allowance for noise level, you will get a suggestion of between 2 minutes and 5 minutes for the exposure, depending on the gain chosen. For 5% allowance, those will double. The relatively long exposure times come from a combination of the small pixels, colour camera and LP filter all cutting down the background light pollution rates.

Onto the black level - the recommendation of 4 is odd. Every test I do with my 183C here with low brightness (I got down to 0.15 e/pixel/s), recommends black level 24 - that gives me clear histogram separation at the recommended gain of 400. In fact lower values do too, but I'm not sure about black level = 4, that looks collapsed to low values.

If you can send me the e/pix/s value that you measured and a copy of your sensor analysis file (since it sounds like you ran your own analysis), I can check to see if anything is wrong (and if I can get the same results as you). One thing that comes to mind is that if there was a slight light leak when taking the dark parts of the sensor analysis, that could boost the pixel values being measured as dark, and consequently mean a lower black level would be recommended. You can find the sensor analysis result data files as described here : https://docs.sharpcap.co.uk/4.0/#Sensor ... ta%20Files

cheers,

Robin
LandyJon
Posts: 6
Joined: Tue Jan 04, 2022 11:34 am

Re: Smart Histogram sets a low offset

#3

Post by LandyJon »

Hi Robin,

Thanks for the quick reply, I did have some problems when I updated from v.3.2 ... mostly was with plate solving, I had to reinstall tortilla and completely set it all up again going through all the fov calculations again ... SC still doesn't seem to want to look outside the 15 deg despite it being set to 30 in tortilla, but that's not an issue now I've got it back to pretty close on first slew.

Anyway I digress, I'll revert to measuring on target rather than a dark region.

I'm not sure of the sky background rate you mention, I can't find any screen capture of the measurement results, thought I'd taken one, but I should get an hour or so of clear skies after sunset shortly, so I can get that in the next few hrs and post it.

I've attached the sensor analysis file and a screenprint of the results displayed, I tried to add a dark frame to show, but it won't let me add a fits file. There shouldn't be any light leakage tho, darks including during sensor analysis are taken outside at night with the ota cap on. There is a lot of amp glow tho, especially at higher gain, does your 183 have it ? I presumed it's just a design flaw we have to process out with darks, but maybe I've got an excessive amount ?

Not sure I could get to the 400 gain you suggested, I think it's a max of 500 and I'm sure I was getting loads of banding by the time I got much past 300, no idea what causes that, but I'd decided 300 was the limit for gain. I did try the 6 min 250 gain it suggested on M81 but again with 4 offset it applied theres very little there after darks subtracted.

Hopefully you can see if there's anything going on in the sensor analysis. Going dark now, I'll go get set up and find that sky background rate.

Thanks, Jon.
Attachments
ZWO ASI183MC Pro~RAW16.json
(23.67 KiB) Downloaded 4 times
Sensor Analysis Screenprint.png
Sensor Analysis Screenprint.png (320.76 KiB) Viewed 195 times
LandyJon
Posts: 6
Joined: Tue Jan 04, 2022 11:34 am

Re: Smart Histogram sets a low offset

#4

Post by LandyJon »

Ok, I've got a result on that sky background ... it shows 0.18 e/px/s but its still showing an Optimal Black level of 4 !?

Screenshot attached ...

Thanks for your time,

Jon.

Edit : Oh and if I change to 5% or Unity Gain, it drops to 0 Black Level !!
Attachments
Smart Histogram Measurement Result.jpg
Smart Histogram Measurement Result.jpg (394.15 KiB) Viewed 194 times
User avatar
admin
Site Admin
Posts: 8144
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Smart Histogram sets a low offset

#5

Post by admin »

Hi Jon,

right... ZWO 183, not Altair. Ok, I have just looked at the sensor data I have recorded for the ZWO 183MC. In the gain 125-150 region, the standard deviation of a dark frame is about 40-45 ADU, and I have figures that suggest black level 4 should give a mean of about 350 ADU for short dark frames. If those figures are correct then the black level of 4 should be lifting the histogram peak clear of zero by at least 6 or 7 standard deviations, which *should* be plenty to avoid any problems.

You can quickly test this for yourself - cover the camera, select 137 gain, black level 4, RAW16, 1ms or less exposure and show the histogram in SharpCap. In the top right corner of the histogram it should show the mean and standard deviation for each channel. Note down the mean and standard deviation values shown for the green colour channel (the others will probably be very similar). If the mean is more than 4 or so times the standard deviation then all should be fine.

Can you also please check the maximum value you can set for the brightness (black level) control - my data seems to indicate that it should be 80 - I just wonder if a recent SDK update from ZWO might have changed to a new extended range?

cheers,

Robin
LandyJon
Posts: 6
Joined: Tue Jan 04, 2022 11:34 am

Re: Smart Histogram sets a low offset

#6

Post by LandyJon »

Hi Robin,

Thanks for the info and yes 80 is the maximum brightness setting.

I've tried your suggestion and I get a mean of 330 and SD of 21.6 so well over the 4 times, I just can't figure out what I'm doing wrong then. I'm taking darks using the same profile, so all values for exposure, gain and brightness are the same as my lights and flat fames using the same profile with exposure pulled down to 500 ms. Yet when smart histogram has suggested 5 min 135 gain, 4 brightness, I've live stacked and shown good data behind the amp glow and vignetting, but after I've taken darks and flats, I stack the RAW files in DSS and basically lose all the data.

I only seem to get anything if I push the Brightness up to 25-35 at gains of 135-250 for 5-6 min subs. Could it be anything to do with DSS using the dark and flat frames that SC has already mastered ? Does it need individual frames ?

Jon.
User avatar
admin
Site Admin
Posts: 8144
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Smart Histogram sets a low offset

#7

Post by admin »

Hi Jon,

using the SharpCap master dark/flats in DSS or other stacking programs seems to give mixed results - some people use it and it seems to work well, others seem to hit problems. Overall, the safest thing is to do all the processing in one program, rather than mixing it between two. It would certainly be worth an experiment with individual darks/flats into DSS at lower brightness to see if that does work nicely (which would definitely implicate the hybrid approach as the cause). I must admit that I don't understand exactly why the hybrid approach is a problem - I suspect that SharpCap and DSS make the master darks/flats in a slightly different way. The SharpCap way works when processed using SharpCap stacking, the DSS way works when processed in DSS stacking, but mix them together and something may be accounted for twice or not at all.

cheers,

Robin
LandyJon
Posts: 6
Joined: Tue Jan 04, 2022 11:34 am

Re: Smart Histogram sets a low offset

#8

Post by LandyJon »

Hi Robin,

Yes I've read something implying it could cause problems. I'm trying to use live stack to give me an idea when I've got a good enough amount of data, but then stack the raw files to remove the noise. I either need to save some separate darks to stack with, or commit to letting SC do the stacking.

I've spent tonight building a dark library round your 125-150 suggested region at 4 offset, all at 5 min for now, I'll see how those work out next clear skies. The problem is not wanting to waste 20+ mins capturing darks for the Smart Histogram suggested settings before you start imaging, by the time you've got enough the clouds roll in. I figure if I've got the them ready, I can do a measure on target, pick a nearest profile I've got darks for and grab a few flats before imaging, at least they only take seconds.

Once I've got a better understanding of what settings are working, I'll do some separate darks and make a comparison with DSS stacking, see if there's any difference I can tell.

Thanks again for your help,

Jon
LandyJon
Posts: 6
Joined: Tue Jan 04, 2022 11:34 am

Re: Smart Histogram sets a low offset

#9

Post by LandyJon »

Hi Robin,

Ok, after another wasted all nighter, up till 4am capturing near 3 hrs on M51 and an hour or so on Flame after taking some darks and flats in between. I'd opted to try one more time with smart histograms recommended settings, since we'd concluded the 4 offset wasn't unreasonable and I'd collected darks for a range of gains, yet it set gain to 0 for M51 and 35 for Flame and not knowing any better, I figured it was based on different conditions with the moon out etc. and again I have basically nothing to show for it. I've since found your youtube video of your Deep Sky Imaging with CMOS and have a little more clarity on the subject of exposure/gain settings and have finally decided to call it quits with smart histogram and go back to using the roughly unity gain 115-150 at 5 min subs and I'll stick with the 4 offset since as you say it's producing a dark histogram that's not clipping so no real need to go any higher with it.

I really don't know what's going on for it to be recommending 0 gain, but having watched more videos I realise that wouldn't work under just about any circumstances !

Did you manage to look over my sensor analysis file to see if you could find anything wrong ? Is there a way to revert to the stock data for my sensor to see if that makes any difference ? at the time I thought I had to do my own analysis, wasn't aware it had built in data for different sensors.

Hope you can help as it's getting frustrating wasting rare good sky opportunities that actually fit in with my time when I'm not stuck in a layby sleeping in a truck :roll: and it seems like it should be a handy feature for people like me who haven't quite got a grasp on how exposure, gain and offset relate to sky brightness, what filters you're using on what F ratio etc etc etc.

Oh, 1 last question, from you're video I've realised I don't need to be cooling to -25. I've come up to -15 which it will reach indoors now for doing darks, but will I need to redo my darks at that temp, or will the difference be minimal ? I'm guessing for perfection I would want to, but for the sake of keeping the range of gains I've got for testing at the moment and not having to spend hours redoing them, will the difference be negligible ?

Thanks, Jon.

Edit : Oh the other things that's confused me ... when I take flat frames for a profile, do I need to select 'none' in the dark frames, or vice versa doing darks ? or does the 'capture darks' procedure automatically disable the pre-processing selections before it captures them ?
User avatar
admin
Site Admin
Posts: 8144
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Smart Histogram sets a low offset

#10

Post by admin »

Hi,

so, starting with the sensor analysis data, it didn't look particularly odd - a little wobble in the read noise at high gain, but that's unlikely to have any real world effect, since those gain levels are unlikely to be used. You can revert back to the stock measurement by deleting the file that you shared (or moving it somewhere else on your computer so you can move it back).

The 'starburst' pattern and glow top/bottom is typical of the 183 sensor (all manufacturers) and needs to be removed with dark correction. It's fairly important to keep the darks well matched to the light frames, as the brightness of the starburst does not change with temperature in the same way that the rest of the dark noise does. That being said, the difference between -15 and -25 might be small enough to get away with.

The suggestion of minimum gain is correct based on the sensor data of the 183C when the 'maximum dynamic range' option is used. In those conditions, SharpCap will suggest minimum gain unless a larger gain is required to get the exposure length down under the 'Max Exposure' setting. This is because the downard slope of the read noise line is rather slow (compared to other cameras), and there is no big jump downward in read noise due to a switch from LCG to HCG. To pick an example, gain 60 (2x brighter) reduces the FWD by a factor of 2, but the reduction in read noise 2.94 -> 2.28 only allows an exposure reduction by a factor of 1.66 (2.94/2.28 squared), so you loose out on total stack dynamic range by moving up to higher gains. I don't think there is anything wrong with using the camera in this setup as long as your mount can cope with the exposure lengths required.

Finally, on taking darks and flats...

Darks must be captured with no flats - the 'Capture Dark' tool should stop you from proceeding if there is a flat frame selected.

Flats are best captured with a dark that matches the flat frame exposure/gain/offset settings. You can either pre-select this dark if you have created it manually or let the capture flat tool do the work. Do not capture a flat frame with a full length dark enabled.

cheers,

Robin
Post Reply