Sensor Analysis and Smart Histogram

Post Reply
DiligentSkies
Posts: 52
Joined: Wed Dec 02, 2020 10:46 pm

Sensor Analysis and Smart Histogram

#1

Post by DiligentSkies »

Hello Robin,

I have been working with a QHY183 mono camera. My imagining season is just getting underway.
I built myself a flat panel LED light source with a dimmer switch that I use with a couple sheets of velum.
I get some really nice flats with this set-up.

When I ran the sensor analysis, I was able adjust the brightness of the LED panel such that the histogram peak of ~65% was readily achieved.
I ran it for both MONO16 and MONO8 analysis per documentation.
I attached the results in a text file with some notes.
sensor analysis.txt
(6.55 KiB) Downloaded 49 times
Moving on to the smart histogram.

I have been having great difficultly using the smart histogram.
In that after running the smart histogram I keep getting NO Solution below 600.0s.
Please Note: I been able to run the smart histogram with success for an Optolong L-Pro filter.

However, when I switch out the filter to an Optolong H-Alpha 7nm passband on my William Optics GT81 reduced by 0.8 for a f/4.7.
I cannot get any reported smart histogram results.
I attached two screen shots of the progress of the smart histogram analysis.

This one, "Is after the smart histogram had completed a 512 second exposure.
smart histogrm C.PNG
smart histogrm C.PNG (571.19 KiB) Viewed 1218 times
And this one is after completion of the analysis.
smart histogrm D.PNG
smart histogrm D.PNG (641.25 KiB) Viewed 1218 times
I cannot figure out the underlying issues here.

Seeing was above average for my location and the there was no Moon in the night sky at the time of these histogram measurements .
I would characterize my overall urban night sky as a Bortle Zone 5.


Thanks in advance Dr Glover for your coming reply.

Sincerely,
Mark
DiligentSkies
Posts: 52
Joined: Wed Dec 02, 2020 10:46 pm

Re: Sensor Analysis and Smart Histogram

#2

Post by DiligentSkies »

Dear Dr Glover,
Absent of the stretched live stacking imagining issues presented in this image for this QHY183 mono camera after the completion of smart histogram analysis.
That is...no Bias, no Flats, no Darks. suggest to me the data is very solid.

The results in dynamic range of details noted by the sharp contrasts in details of the very faint emission details of the nebulosity of the Flame/Horsehead nebula on the periphery of the nebulosity under this exceptionally long exposure that results in blown out details in the very bright areas of the nebulosity.
Suggest to me that, under stacking of many subs under optimal gain, offsets and exposure duration limits values of smart histogram analysis.
smart histogrm D.PNG
smart histogrm D.PNG (641.25 KiB) Viewed 1215 times
That these presented details of contrast will become evident in PixInsight stacked post-imagining processing of capture data sets.
"If I can just get scientifically repeatable data acquisition parameters of the smart histogram analysis."

That is my observational mindset.

Sincerely,
Mark
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Sensor Analysis and Smart Histogram

#3

Post by admin »

Hi Mark,

how dark are your skies? My estimates are that to get the observed 0.01 e/pix/second light pollution rates with your f/4.7 optics, 7nm filter and 2.4 micron pixels, would correspond (roughly) with Bortle 3.5 to 4 skies.

Anyway, the rule of thumb for exposure length with the read noise limit set to 10% is : (read noise)*(read noise)*10 / (light pollution rate)

Even picking the minimum read noise values of about 1 in your measurements would mean a 1000s exposure with the 7nm narrowband filter, so it looks like there really is no option in the <600s range with this setup.

BTW, looking at your sensor data, the first set where the read noise starts to increase after gain 30 is suspect - something has gone wrong with the measurements here. Since the e/ADU does not drop off in the same way as the 2nd and 3rd runs, I suspect that there was a variation in illumination brightness during the final stages causing at least part of this. Also, the 2nd run has a lower e/ADU for gain=0 than runs 1&3 (which agree well). Again, that value is suspicious both for being different to the other two and also because it leads to a lower FWD that doesn't match the published specs of the 183 sensor.

I'm not sure what 'binning enabled' vs 'disabled' can be here - SharpCap should not allow you to run sensor analysis with binning set to anything other than 1x1 - if somehow you do manage to trick it past that point, the results will be invalid.

cheers,

Robin
DiligentSkies
Posts: 52
Joined: Wed Dec 02, 2020 10:46 pm

Re: Sensor Analysis and Smart Histogram

#4

Post by DiligentSkies »

Hello Robin,

Thank you for your time in replying.
I will say it again, "Dr. Glover's commitment in personally responding is unheard of in today's customer support."

Your impression that I am a Bortle 3.5 to 4 zone is likely spot on.
Because, although I am in urban environment with many sources of light pollution that I characterized as a Bortle 5 Zone is based on
https://www.lightpollutionmap.info/#zoo ... FFFFFFFFFF
BortleZones.PNG
BortleZones.PNG (173.5 KiB) Viewed 1178 times
I do have an advantage that likely contributes to this level of your dark skies estimate.

At this location much of the urban light pollution is directly North of me and since I am imagining directly South. Much of the sources of light pollution are vastly limited by the observation that when I am imagining due South. It is over open Ocean, and the nearest downstream sources of light pollution are in South America in relation to observing due South from this location.

On the night in question, I had above average seeing conditions for my Southwest Florida location.
Atmospherically, the air was thermally still and dry. There was no Moon and I had very good seeing conditions in terms of any thin cloud cover that would naturally reflect the urban glow behind me was very restrictived.

Not sure, "What to make this statement in your reply."
"Even picking the minimum read noise values of about 1 in your measurements would mean a 1000s exposure with the 7nm narrowband filter, so it looks like there really is no option in the <600s range with this setup."

I am taking the position that what you are saying is the smart histogram analysis is being built on a software simulation profile characteristic routines based on the sensor analysis that are software coded as assumptions for the smart histogram analysis. That falls outside of being able to provide a solution due to the degree of light bucket limiting gathering factors for this particular narrow band filter.
Since the smart histogram analysis is able to provide a solution for when I use a different filter. (In that case when I use the Optolong L-Pro luminesce pass band filter.)
All I can conclude are physical limiting factors on the rate photons that do pass through this narrow band filter (an Optolong H-alpha with a 7nm pass band.)
That don't allow for a smart histogram solution based on the data of the sensor analysis in terms of the next higher ADU's towards the full-well sensitives of sensor analysis data aquired.

Am I now understanding the problem more discreetly within your reply?

Additionally, with what you are saying about, "Suspect data stats within my reported sensor analysis data."

That is concerning to me since I am trying to base gain, offset and exposure times such that suspect data measurements points as you sighted in your reply of the sensor analysis that are ultimately used in the smart histogram simulation analysis."
If you are having observational exceptions to the reliability of the underlying sensor analysis data.

Then how, "Can any reliable smart histogram analysis be achieved."

Could that be the main contributing factor, "As to why the smart histogram was unable to provide an optimal solution for this H-alpha filter?"

As any flat panel LED light source readily displays, "The light source is not broad band uniform towards the spectrum of EMF readings of natural light."

Meaning it, "Is NOT natural light and is in fact built on very discreet wavelengths that emanate from the solid-state LED source.
It is all electronically generated light and very specific to the solid-state light source of LED emissions as the source of the light being measured in the sensor analysis.

My observation is, "Could these LED wavelengths provide a false measurement of the sensitively readings being measured by the sensor analysis algorithm that contribute my 'No Solution' benchmarks in the smart histogram measurements.

That is to say on the CMOS sensor, "The analysis is being built on solid-state LED generated sources of light of very specific LED sourced wavelengths."
These points of LED wavelengths on the sensor analysis data may very well fall within the spectrum being eliminated(clipped) by this narrow band H-Alpha filter.
Thus, hugely contributing to the non-solution in the smart histogram analysis. QED

As for the attached text file that you cite as having suspect data point stats with an UNDERSTANDING that you suspect that there was a variation in illumination brightness.
All I can say is that once I adjusted brightness of my LED source to achieve a 65% threshold.
That is, "Reasonably there should NOT have been any variations in the source intensities that contributed as a variable of uncertainty."
The intensity of the light LED source was and would have been very consistent as dialed into the 65% threshold once established."

Leading me to conclude this Sensor Analysis is built on a faultily assumption that sensor analysis derived from an LED light source panel does NOT in fact match the natural characteristics of the broad band light spectrum to which the sensor analysis of SharpCap is based.

I am going to think about this possible contributing factor just a bit.

And see if using a light source of a natural diffused sky light gains me a better sensor analysis.
I will post further results.

As for the binning issue you cited.
When I enter into the sensor analysis algorithm tool, "It clearly states with a check box (I paraphrase), "Do you want to measure binning measures in the sensor analysis."
So, I am uncertain as to the confusion when you state, "I'm not sure what 'binning enabled' vs 'disabled' can be here..."

Maybe the best possible way forward at this point is to have you clarify an expectation of sensor analysis experiment data collection that matches a baseline for your expectations versus my understanding and implementation.
Clearly, I have displayed an above average understanding worthily of the effort to provide feedback to SharpCap's continued software development.

That is, "I readily want to provide real world feedback data of the analogue sensor read outs on Noise to Signal outcomes for Sharpcap's CMOS sensor analysis paradigms."
Such that this would be an experiment of your design. "Just tell me what your controlled, definitive experiment of data acquisition would be."

I am saying that I am, "All game to what that would entail provided your control input."

Sincerely,
Mark
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Sensor Analysis and Smart Histogram

#5

Post by admin »

Hi Mark,

right, I follow on the binning thing now - that should not affect the main part of the sensor analysis results, all it does is enable/disable an additional test right at the end of the sensor analysis that checks how the camera deals with binning (additive software binning, averaging software binning, hardware CCD binning). As I say, this will not make any changes to the main measurements, which means that the variation in the sets of results is coming from some uncontrolled/random factor in the measurements.

I don't think the spectra of the LED is going to be a big issue - although the sensitivity of the sensor does vary with wavelength, it's a fairly smooth curve typically and for the LED to look white it has to have a peak in the green with red+blue at lower levels, just like daylight. However, if the LED flickers or has varying brightness due to (for instance) PSU instability, that could be a problem. If you see flickering dark/light bands on the image at short exposures (caused by LED brightness flickers), set a measurement area that is tall and thin (4 or 5 bands high) to minimize the effects.

Now, back to the main point - why is there no solution less than 600s...

The problem you are asking the smart histogram to solve, can be worded roughly like this
Given the (measured) sensor data and a sky background brightness totalling 0.01 electrons/pixel/second, find camera settings that will allow the final stacked image to have no more than 10% additional noise (over the noise from taking a single extremely long exposure), using a sub exposure length between 2 minutes and 10 minutes.
What the smart histogram results are telling you is that the problem that you have set up has no solution - a bit like asking someone to work out the square root of 7, but their answer must be between 1 and 2.

Once you have started using the narrowband filter, the sky background is so dark that longer exposures are required - longer than 10 minutes to satisfy the 10% additional noise criteria. Without the filter (or with a broader filter), this isn't a problem.

Your variability in sensor analysis results isn't the cause of this - the variation isn't huge - 10 to 20% or so for most gain values, and you are coming fairly close to values published from other sources. Getting perfect measurements might change a calculation for optimum exposure time that comes in currently at maybe 800s to slip under 600, but it isn't going to bring it down vastly.

As far as I can see, everything is running pretty much as it should - you are using a narrow band filter, which darkens the background dramatically, and that in turn leads to dramatically longer recommendations for the sub-exposure length.

cheers,

Robin
DiligentSkies
Posts: 52
Joined: Wed Dec 02, 2020 10:46 pm

Re: Sensor Analysis and Smart Histogram

#6

Post by DiligentSkies »

Hello Robin,
Just read your response. Lot of details, lots of detail.

Briefly I did observe the effect you describe.
If you see flickering dark/light bands on the image at short exposures (caused by LED brightness flickers), set a measurement area that is tall and thin (4 or 5 bands high) to minimize the effects.

When I put the camera into force still mode this effect disappeared.
Hence my notes of in my sensor data of...

(Note: With force still mode enable the actual frames analyzed appear to be true expected data readings of data frames.
Whereas without force still frames enabled the frames appear to include data from frames that appear to include data from the ongoing adjustments made in exposure lengths and offset adjustments made during the analysis.
That is, "Quite Offen the on-screen capture presentation of the data often had broad bands of much darker bands of sensor measurements data."

As such what I attributed to this to forcing the still mode. You are of the opinion that there is LED flicker present in sensor analysis data.


The interesting thing is the role you ascribe to the measurement selector area and your input that a MacGyvere LED panel likely has LED flicker.
By flicker I take it to mean flicker not human eye observable but electronic flicker readily influencing the sensor analysis.

In regard to the measurement area selector. What would be the ideal goal for setting the size of this measurement area?

Referring back to the images posted earlier to this discussion, the red box area is the size my measurement area.
This smart histogram selector area also reflects the size of the selector area I had in place when I made during the sensor analysis.
So, I am taking the size of the measurement selector area influences the quality of sensor analysis data.

My estimate of the size of this red box as a percentage of the sensor pixel image scale is that it may be too large.

Given that the QHY183 is 5544 by 3694 pixels, this red boxed area as an estimate is in the order of 360 by 240 pixels on the sensor during the sensor analysis and the histogram analysis measures.

I do note that during the smart histogram analysis the location of this selector area jumped around the image per the requirement of the smart histogram analysis striving to measure the darkest area of the total image as the goal of the histogram analysis bounded by the size of the measurement selector area selected.

For example, this screen shot capture after a 256 second exposure the selector area is in a different location from those posted earlier.
The size of the selector box is the same size just in a different location.
smart histogrm.PNG
smart histogrm.PNG (407.33 KiB) Viewed 1157 times
This observation means to me, "The size of measurement area selector highly influences the results of sensor analysis which then contributes to smart histogram measurements."
While it may not be an exact science but as a percentage of image scale of the total area of the sensor. "What should be an ideal size of the measurement area?"

If the mathematics of the sensor analysis and the smart histogram analysis at your Ph.D level of understanding is, "Modeled on a differential equation then the size of this selector area is not an independent variable. It influences the results."

As such what should be the goal of setting the size of this measurement area in terms of percentages of a given sensor's size?

Dr. Glover, "I am making assumption that the level of understanding I am trying to communicate is resonating with you. Such that you are taking the time to respond with point-by-point details in reaching a shared level of communication per the discussion."

As it now stands, "Your input about LED flicker makes much of the level of analysis a moot point until I can acquire an astrophotography grade flat panel light source."

Since you cite PSU instability. What is meant by PSU instability?

In the in-term, I will strive to afford myself of a sensor analysis based on a natural diffused light source (aka morning light). And see if that result acquires better results in the sensor analysis stats that I published previously (That is data you cited as having outlying sources of standard deviation from the expected.)

To see how that addresses your concerns to the viability of the measurements from the sensor analysis measured from this artificial LED light source.

Seeing I am striving to measure SharpCap's SNR paradigm analysis as it relates to gain, offset and exposure settings reported by histogram measures. Any attempt to provide real world feedback is very much dependent on establishing an experimental control.

Any details that you could provide as to a measure of a control experiment would be greatly appreciated.

As such, "As I understand this discussion, the one area of establishing this control is dependent on the basis of setting a very reasonable size of this measurement selector area size in relation to the actual area of the sensor."

Sincerely,
Mark
Post Reply