How to access on the "cutout" data ?

[Jean-Francois]

Hello good morning,

I did yesterday evening a last test with Numpy.net ... and it work with this.

It is quick ... with the same settings 300x300, noise=50, ...
It can generate 225 image/s, sometimes more (that is much better as 41 images/s).

But ... ach!
2 problems. Here the first is better visible without noise (= 0):

Numdotnet_average_1.jpg (50.29 KiB) Viewed 4817 times

The frequency of the images is not constant, around 5 slower period per second.
That is not so good. My goal is the sun observation with a SHG (like a Sol'Ex) and it is absolutely necessary to have a regular capture.

The other problem is the generation of the bitmap and Picturebox.Image ... it take a long-long time: 12 seconds ! Seconds ! not milli or micro !
Something is not good in the iteration of the numpy array values. 12 seconds for getting the values of 300x300 pixels !

Here the script, if someone will test it:

Code: Select all

# *******************************************************************************************
# SharpCap script "xxxxxxxxx.py"
# 2024/08/18 Jean-Francois
#
# Version: 1.0.0:   First version
#
# The script works only with a SharpCap PRO version.
# *******************************************************************************************

import os
import time
import datetime
import math
import ctypes
import clr
clr.AddReference("Oxyplot")
clr.AddReference("OxyPlot.WindowsForms ")
clr.AddReference("System.Drawing")
clr.AddReference("System.Windows.Forms")
clr.AddReference("NumpyDotNet.dll")

import OxyPlot
import System.Drawing
import System.Windows.Forms
import NumpyDotNet.np as np

from SharpCap.Base import BufferFrame

from System import Array
from System.Drawing import *
from System.Windows.Forms import *
from System.Threading import Thread, ApartmentState, ParameterizedThreadStart


# .ToBytes() (8 bit) .ToUshorts() (16 bit) and .ToFloats()
# BufferFrame.CopyFrom...()

#dll=ctypes.CDLL(dllfile)

def framehandler(sender, args):
    global i, T_meas, W_meas, t0, frameAverage, pos, frameB
    if (dumpdata):
        try:
            dt = (datetime.datetime.now() - t0).total_seconds()
            cutout = args.Frame

            Stat = cutout.GetStats()
            T_meas.append(dt)
            W_meas.append(Stat.Item1)
            i = i + 1

            frameB = cutout.ToUshorts()     # 1D array

#            for index in range(frameB.Length):
#                frameAverage[index] += frameB[index]

            frameAverage += frameB

#            info           = cutout.Info
#            bufsize        = int(cutout.BufferSize)
#            lease          = cutout.GetBufferLease()
#            frameData_addr = lease.Buffer.ToInt64()
#            frameData_ptr  = ctypes.cast(frameData_addr, ctypes.POINTER(ctypes.c_ubyte))
#            print(frameData_ptr)
#            #dll.do_foo(frameData_ptr, info.Width, info.Height, bufsize)
#            lease.Dispose()


        except:
            print("Problem framehandler")


# *****************************************************************************
def WIN_measurement(self):
    global dumpdata
    global i, T_meas, W_meas, t0, frameAverage

    self.WIN_Monitoring.Text = "Monitoring running"
    self.WIN_Monitoring.BackColor = Color.Red
    self.WIN_Monitoring.Enabled = False

    #SharpCap.Transforms.SelectionRect = Rectangle(300, 300,600, 600)

    i = 1
    T_meas = []
    W_meas = []

    t0 = 0
    image = 0
    H = SharpCap.SelectedCamera.Controls.Resolution.CaptureArea.Height
    W = SharpCap.SelectedCamera.Controls.Resolution.CaptureArea.Width
    
    frameAverage = np.zeros(W * H, np.Int32)
    #frameAverage = Array.CreateInstance(System.Double, W * H)
    #frameAverage_byte = Array.CreateInstance(System.Byte, W * H)
    frameB = 0
    pos = 0

    expos_ms = SharpCap.SelectedCamera.Controls.Exposure.ExposureMs
    wait_time = 0.1       # waiting time between 2 refresh of Oxyplot graphic

    #view.Height = H + 6
    #view.Width  = W + 6
    view.ClientSize = System.Drawing.Size(W + 6, H + 6)
    view.pictureBox1.Size = System.Drawing.Size(W, H)


    print("Waiting Time: %5.3f" % (wait_time))
    print()
    print("WIN Transmission:")
    print("******************")

    dumpdata = True
    SharpCap.SelectedCamera.FrameCaptured += framehandler

    try:
        if (float(self.textBoxTime.Text) > 0.0):
            Max_time = float(self.textBoxTime.Text) - wait_time
            t0 = datetime.datetime.now()
            dt = 0

            while  (dt < Max_time):
                dt = (datetime.datetime.now() - t0).total_seconds()
                time.sleep(wait_time)
                self.ShowOxyGraphic()

#            print("Time:", T_meas)
#            print("WIN:", W_meas)
            print()
            i = i - 1
            print("Number: ", i)

            mean = sum(W_meas) / len(W_meas)
            var  = sum(pow(x-mean,2) for x in W_meas) / len(W_meas)
            std  = math.sqrt(var)
            print("Standard deviation 1: " + str("%.6f" % round(std,6)))
        else:
            print("Measurement time need to be larger than 0 s")
    except:
        print("Problem 1")

    dumpdata = False
    SharpCap.SelectedCamera.FrameCaptured -= framehandler

#    for index in range(frameAverage_byte.Length):
#        frameAverage[index] = frameAverage[index] #/ i

    frameAverage = np.divide(frameAverage, i*256)
    frameAverage = frameAverage.astype(np.UInt8)

    try:
#        for index in range(frameAverage_byte.Length):
#            frameAverage_byte[index] = int(frameAverage[index] / 256)

        flag = Bitmap(W,H)
        for i in range(W):
#            print(i)
            for j in range(H):
                index = i + (j * W)
                value = frameAverage[index]
                #value = 64
                flag.SetPixel(i, j, System.Drawing.Color.FromArgb(value, value, value))
        view.pictureBox1.Image = flag
    except:
        print()
        print("problem picture")

    print()
    print("Show Average")

    print()
    print("Measurement done")

    self.WIN_Monitoring.Text = "Start Measurement"
    self.WIN_Monitoring.BackColor = Color.Gainsboro
    self.WIN_Monitoring.Enabled = True

    file_path  = os.path.dirname(__file__)
    file_name = file_path + "\\" + self.textBoxFileName.Text + ".txt"
    file = open(file_name, "w")
    for index in range(len(T_meas)):
        file.write(str("%.4f" % T_meas[index]) + " " + str("%.4f" % W_meas[index]) + "\n")
    file.close()

    return()

# *****************************************************************************

class MonitoringWINMenuForm(Form):
    def __init__(self):
        self.SuspendLayout()
        self.InitializeComponent()
        self.setupCheckButtons()
        self.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Dpi
        self.AutoScaleDimensions = SizeF(96, 96)
        self.ResumeLayout()

    def InitializeComponent(self):
        self.Text = "WIN Transmission"
        self.ClientSize = System.Drawing.Size(700, 400)
        self.TopMost = True

        self.labelTime = Label()
        self.textBoxTime = TextBox()
        self.labelFile = Label()
        self.textBoxFileName = TextBox()

        self.plot1 = OxyPlot.WindowsForms.PlotView()
        self.plot1.Location = System.Drawing.Point(10, 35)
        self.plot1.Size = System.Drawing.Size(680, 350)
        self.plot1.AutoSize = True

        self.Controls.Add(self.labelTime)
        self.Controls.Add(self.textBoxTime)
        self.Controls.Add(self.labelFile)
        self.Controls.Add(self.textBoxFileName)
        self.Controls.Add(self.plot1)

        self.labelTime.AutoSize = True
        self.labelTime.Location = Point(150, 14)
        self.labelTime.Name = "labelTime"
        self.labelTime.Text = "Measurement time [s]:"

        self.textBoxTime.AutoSize = True
        self.textBoxTime.Location = Point(280, 13)
        self.textBoxTime.Name = "textBoxTime"
        self.textBoxTime.Size = Size(35, 20)
        self.textBoxTime.Text = "1"

        self.labelFile.AutoSize = True
        self.labelFile.Location = Point(320, 14)
        self.labelFile.Name = "labelFile"
        self.labelFile.Text = "File:"

        self.textBoxFileName.AutoSize = True
        self.textBoxFileName.Location = Point(350, 13)
        self.textBoxFileName.Name = "textBoxFileName"
        self.textBoxFileName.Size = Size(250, 20)
        self.textBoxFileName.Text = "test"

    def setupCheckButtons(self):
        self.WIN_Monitoring = Button()
        self.WIN_Monitoring.Text = "Start Measurement"
        self.WIN_Monitoring.Location = Point(20, 10)
        self.WIN_Monitoring.Click += self.WIN_monitor
        self.WIN_Monitoring.AutoSize = True

        self.button_Exit = Button()
        self.button_Exit.Text = "Exit"
        self.button_Exit.Location = Point(615, 10)
        self.button_Exit.Click += self.exit
        self.button_Exit.AutoSize = True

        self.Controls.Add(self.WIN_Monitoring)
        self.Controls.Add(self.button_Exit)

    def ShowOxyGraphic(self):
        global T_meas, W_meas
        self.plotmodel1 = OxyPlot.PlotModel()
        self.YAxis1 = OxyPlot.Axes.LinearAxis(Position = OxyPlot.Axes.AxisPosition.Left, Minimum = 0.999*min(W_meas), Maximum = 1.001*max(W_meas))
        self.points_start = []

        for i in range(0,len(T_meas)):
            self.points_start.Add(OxyPlot.DataPoint(T_meas[i],W_meas[i]))

        self.serie_1 = OxyPlot.Series.LineSeries()
        self.serie_1.Color  = OxyPlot.OxyColors.Blue
        self.serie_1.MarkerSize = 3
        self.serie_1.MarkerType = OxyPlot.MarkerType.Circle
        self.serie_1.StrokeThickness = 1
        self.serie_1.ItemsSource = self.points_start
        self.plotmodel1.Axes.Add(self.YAxis1)
        self.plotmodel1.Series.Add(self.serie_1)
        self.plot1.Model = self.plotmodel1

    def WIN_monitor(self, sender, event):
        th = Thread(ParameterizedThreadStart(WIN_measurement))
        th.SetApartmentState(ApartmentState.STA)
        th.Start(self)

    def exit(self, sender, event):
        print("Stop Monitoring script")
        self.Close()

class MainForm(Form):
    def __init__(self):
        self.InitializeComponent()
        self.pen = System.Drawing.Pen(System.Drawing.Color.Black);

    def InitializeComponent(self):
        self.pictureBox1 = System.Windows.Forms.PictureBox()
        self.SuspendLayout()

        H = SharpCap.SelectedCamera.Controls.Resolution.CaptureArea.Height
        W = SharpCap.SelectedCamera.Controls.Resolution.CaptureArea.Width
        self.pictureBox1.Location = System.Drawing.Point(3, 3)
        self.pictureBox1.Name = "pictureBox1"
        self.pictureBox1.Size = System.Drawing.Size(W, H)
        self.pictureBox1.TabIndex = 0
        self.pictureBox1.TabStop = False

        self.ClientSize = System.Drawing.Size(W + 6, H + 6)
        self.Controls.Add(self.pictureBox1)
        self.Name = "MainForm"
        self.Text = "Average"
        self.ResumeLayout()


form = MonitoringWINMenuForm()
form.StartPosition = FormStartPosition.CenterScreen
form.TopMost = True
form.Show()

view = MainForm()
view.StartPosition = FormStartPosition.CenterScreen
view.TopMost = True
view.Show()

For that you need first to download https://www.nuget.org/packages/NumpyDotNet/ and copy the "NumpyDotNet.dll" and "NumpyDotNet.xml" in the installation directory of your SharpCap.


Regards,
Jean-Francois

[admin]

Hi Jean-Francois,

glad to hear that Numpy helped with the performance for you.

I can tell you why it is taking 12 seconds... It's almost certainly that you are using SetPixel to draw to the bitmap - the performance of that is poor, and it shouldn't really be used for more than a few pixels.

A more performant way to work is to call .LockBits to lock the bitmap into memory and then copy the colour data from an existing byte[] array into the bitmap - you can see an example in C# here (which should translate to python OK) - https://learn.microsoft.com/en-us/dotne ... ew=net-8.0

I am less sure about the performance glitches. It may be worth turning on additional 'performance' logging and examining the log - you should see entries that give the average frame rate and min/max gaps between frames in different parts of the code, and also messages that record the time taken for an individual frame to run through the various stages, which might be helpful.

cheers,

Robin

[Jean-Francois]

Hello Robin,

Concerning the Numpy ... it exist 3 or 4 different versions.
I use now the NumpyDotNet.dll. But other libraries are possible.

For the 12 seconds ... the SetPixel function is may be slow ... but it is here not the problem.
For example the following code, where the constant value = 64 is used ... the graphic is fast instantaneous.

Code: Select all

    frameAverage = np.divide(frameAverage, i*256)
    frameAverage = frameAverage.astype(np.UInt8, True)
    try:
        value = 0
        flag = Bitmap(W,H)
        for i in range(W):
            for j in range(H):
                index = i + (j * W)
                #value = frameAverage[index]
                #value = frameAverage.GetItem(index)
                value = 64
                flag.SetPixel(i, j, System.Drawing.Color.FromArgb(value, value, value))
            print(value)
        view.pictureBox1.Image = flag
    except:
        print()
        print("problem picture")

I tried two different method to access the data: frameAverage[index] or frameAverage.GetItem(index).
But, that is the same ... 12 seconds for the small field (later it would be 3x longer ... 3000x100 area in place of 300x300).

It seems to have no direct conversion between the np.array and the net.array. But I found today a small script doing that.
I will test it, but it is written for Pythonnet.

Thanks for the link for the .LockBits function. I will have a look at it.


Regards,
Jean-Francois

[Jean-Francois]

Hello,

Concerning the LockBits ... It works after lot of try and error.
It is not really quick ... the problem is not in the Bitmap.

Here the code:

Code: Select all

    frameAverage = np.divide(frameAverage, i*256)
    frameAverage = frameAverage.astype(np.UInt8, True)

    try:
        bmp = Bitmap(W, H)

        rect      = Rectangle(0, 0, bmp.Width, bmp.Height)
        bmpData   = bmp.LockBits(rect, System.Drawing.Imaging.ImageLockMode.ReadWrite, bmp.PixelFormat)
        ptr       = bmpData.Scan0
        bytes     = abs(bmpData.Stride) * bmp.Height
        rgbValues = Array.CreateInstance(System.Byte, bytes)
        System.Runtime.InteropServices.Marshal.Copy(ptr, rgbValues, 0, bytes)

# change rgbValues values
        for counter in range(0, rgbValues.Length, 4):
            value = frameAverage[counter/4]
            rgbValues[counter]     = value
            rgbValues[counter + 1] = value
            rgbValues[counter + 2] = value
            rgbValues[counter + 3] = 255

        System.Runtime.InteropServices.Marshal.Copy(rgbValues, 0, ptr, bytes)
        bmp.UnlockBits(bmpData)

        view.pictureBox1.Image = bmp
    except:
        print()
        print("problem picture")

The "value = frameAverage[couter/4]" is the problem.
I will continue tomorrow, may be I can modify the LockBits for the data transfer from the np.array directly to the bitmap.


Jean-Francois

[admin]

Hi,

if it's just that line that is slow (replacing with 'value = 42' is fast) then I would suggest that the indexing operator on the numpy object 'frameAverage' is slow perhaps?

You could try converting that to an array before the loop - it looks like there is an 'asarray()' method to do this. That might just sidestep the potential expensive repeated calls to get individual items from the numpy object.

cheers,

Robin

[Jean-Francois]

Hello Robin,

In the past 2 days (and night) I search a way to convert the numpy array in an array in Python or .net type.

You mention the asarray() function. It was not working, but I continue to search and I found *finally* the solution:

=> R = frameAverage.tobytes()

So, I guess it is not possible to do it simpler.
Now the generation of the bitmap is instantaneous.

Code: Select all

    try:
        bmp = Bitmap(W, H, Imaging.PixelFormat.Format24bppRgb)

        rect      = Rectangle(0, 0, bmp.Width, bmp.Height)
        bmpData   = bmp.LockBits(rect, System.Drawing.Imaging.ImageLockMode.ReadWrite, bmp.PixelFormat)
        ptr       = bmpData.Scan0
        bytes     = abs(bmpData.Stride) * bmp.Height
        rgbValues = Array.CreateInstance(System.Byte, bytes)
        System.Runtime.InteropServices.Marshal.Copy(ptr, rgbValues, 0, bytes)

        R = frameAverage.tobytes()

# change rgbValues values
        for counter in range(0, rgbValues.Length, 3):
            #value = frameAverage[counter/3]
            value = R[counter/3]
            rgbValues[counter]     = value
            rgbValues[counter + 1] = value
            rgbValues[counter + 2] = value
            #rgbValues[counter + 3] = 255

        System.Runtime.InteropServices.Marshal.Copy(rgbValues, 0, ptr, bytes)
        bmp.UnlockBits(bmpData)

        view.pictureBox1.Image = bmp
    except:
        print()

The longest part is now the transfer of the data in the bmp variable (8 commands).

Concerning the "performance log" ... here, maybe (no I'm sure) you can find more information

Cutout_Log_2024-08-22T22_14_04-10808.log

(63.59 KiB) Downloaded 47 times

I tested the same script on 2 other computers. They have less slow period as my desktop computer.
That is a little bit strange. My desktop has 16 cores and 128 GB ram.

Regards,
Jean-Francois

[admin]

Hi,

good to hear that you solved the slowness from numpy

Looking at the log, the camera frame delivery rate is relatively constant - ie

Code: Select all

Frame received from camera rate is 330.16 fps, Min Interval = 2.70ms, Max Interval = 6.43	

I see the max inteval in the 6 to 10ms range, which might indicate a lost frame or so here and there, or just natural variability.

On the other hand, the rate that the process thread is handling frames is far more variable - eg

Code: Select all

Frame Process Thread rate is 327.37 fps, Min Interval = 2.70ms, Max Interval = 24.90

Your calculations are going on in that thread, and if the processing for a frame takes longer than the interval between one frame and the next arriving from the camera, frames will be discarded at this point.

One thing that might be causing the regular glitches is .NET Garbage collection kicking in - The sort of processing you are doing will be allocating memory on the C# heap and that will build up and then be garbage collected regularly when the amount becomes large enough. The GC activity can cause other code to pause, which is not normally noticeable to the user, but at the sort of processing rates you are running, it could be... It would certainly be worth checking these figures *without* your script running to see if the Max Interval reduces in that case.

It may be possible to push the .NET runtime in SharpCap to use 'Server GC' which could have smaller/fewer pauses - this ought to be possible by setting environment variables, but I have never tried (see https://learn.microsoft.com/en-us/dotne ... -collector - SharpCap uses .NET8, so the settings for .NET6 or later would be the ones to try).

A more extreme solution would be to try to push the actual calculations out of the FrameCaptured event handler into a separate thread, but that would require quite a bit of care with the coding (AddRef the frame, push calculations to a task, Release the frame at the end of the task, protect data structures that the data gets added to with a lock because of possible concurrent access, etc).

cheers,

Robin

[Jean-Francois]

Hello Robin,

Concerning the frame rate ...
If I remember correctly ... I used my script for 3 measurements.
One was without noise on the image (so very quick), the other with noise on the image.

OK for the garbage collection or to push to calculation outside the FrameCaptured event.
One other possibility is to code a C# function.

Regards,
Jean-Francois

[Jean-Francois]

Hello,

I have some news ... I modified the script so that MONO8 or MONO16 works.
I changed the bitmap data transfer ... now it is for a 8bit bmp.

I code with a separate thread. Here some results with a real camera.
Camera parameters: QHY178, 16 bits, ROI 3056x96, 1 ms, gain 0.

With numpy addition in the FrameHandler ~145 frame/s (measurement during 1 second):

cutout_average_np.jpg (54.1 KiB) Viewed 4729 times

With separate thread started from the FrameHandler ~155 frame/s:

cutout_average_np_thread.jpg (56.03 KiB) Viewed 4729 times

Without frame average calculation (but with cutout statistic) ~345 frame/s (as the "free" fps):

cutout_average_no_sum.jpg (73.09 KiB) Viewed 4729 times

155 frame/s is still too slow
My sun captures are done with a frame rate of 285 fps.
So I need to continue and try better optimised code ... I guess that it will an extern compiled C# code.

The thread code is the following:

Code: Select all

def np_thread():
    th_np = Thread(ParameterizedThreadStart(np_add))
    th_np.SetApartmentState(ApartmentState.STA)
    th_np.Start()

def np_add(self):
    np.add(frameAverage, frameB, frameAverage)

I guess completely different as you proposed.

Nice weekend,
Jean-Francois

[admin]

Hi,

not quite what I was thinking of for the threading, but there are lots of ways to do it

Compiled C# code in a DLL will probably help, but I am also wondering do you need to measure every frame as part of your process? Could you do the calculations on every second or third frame - would that be sufficient?

cheers,

Robin