SharpCap Knowledge Base

[oopfan]

Hi Robin,

Retrieval Augmented Generation (RAG) is a technique for rapidly querying a very large custom knowledge base and then using ChatGPT to summarize the findings in a friendly conversational manner.

For my knowledge base, I used the NASA Astrophysics database of thousands of published scientific papers. In SharpCap’s case, you would store all of your acquired knowledge and how it relates to your product.

I started by going to the NASA ADS website and asked it to return the top-100 abstracts having to do with “Cepheid Variable Stars” since that is my area of interest. Then I exported those abstracts into what’s called a Vector Database. I set up an account with a company called Qdrant. They offer a FREE 1GB cluster. This enables rapid retrieval of abstracts given a query like “Why observe Cepheids in infrared?” BTW, trying to use a conventional database is a fool’s errand.

In my application I accept a user query, then retrieve the top-3 ranked hits from Qdrant. Then I input those three abstracts to a call to ChatGPT to summarize it given the original query. Here is the result. The response is perfectly dry and scientific. Something to explore is to give it a personality (i.e. “AI Robin”):

-----------------------------------------------------------
Answer: Observing Cepheids in the infrared, particularly the near-infrared (NIR), offers several advantages:

1. **Tighter Period-Luminosity Relation**: The Cepheid period-luminosity relation is tighter at NIR wavelengths. This provides more precise distance measurements, which are crucial for the cosmic distance ladder. This information is supported by 2025AAS...24545506C and 2024IAUGA..32P1871C, which highlight the improved relation at longer wavelengths.

2. **Reduced Impact of Interstellar Extinction**: Interstellar extinction, which can obscure and redden light from stars, has less effect in the NIR band. This allows for clearer observations of Cepheids, thus enhancing the accuracy of inferred distances (2025AAS...24545506C and 2024IAUGA..32P1871C).

3. **Compatibility with Ground-Based Observatories**: NIR data can be acquired using ground-based telescopes, making it relatively accessible and cost-effective for studying Cepheids. This is mentioned in 2025AAS...24545506C and 2024IAUGA..32P1871C.

The detailed analyses and observations of Cepheids in these abstracts, including those performed with JWST which demonstrate enhanced source separation from line-of-sight companions, further emphasize the improved quality and reliability of NIR observations (2022ApJ...940L..17Y).

Based on the provided abstracts, these are the primary reasons why observing Cepheids in the infrared is advantageous.
-----------------------------------------------------------


Since this is a custom implementation I can integrate this into a Hybrid AI chat that couples RAG with functions to return answers to questions like "What is the pixel scale and CCD suitability of my kit" or "What is the minimum exposure of ...?".


Then I asked: “How did Henrietta Leavitt establish the foundation of Leavitt's Law which eventually led to the discovery that M31 is a separate galaxy and not a nebula in the Milky Way?"

-----------------------------------------------------------
Answer: Henrietta Leavitt's work laid the foundation of Leavitt's Law through her discovery of the relationship between the period and luminosity of variable stars (later identified as Cepheids) in the Small Magellanic Cloud, as published in 1912 (2025PASP..137d4001B). She meticulously measured light curves, periods, and magnitudes of these stars from photographic plates. This relationship, known today as the Leavitt Law, allowed for the measurement of extragalactic distances, transforming those variable stars into "standard candles" of cosmological distance measurement.

While none of the provided abstracts directly discuss how Leavitt's Law led to the realization that M31 (Andromeda) is a separate galaxy, the abstract 2025PASP..137d4001B indicates that Leavitt's Law was pivotal for accurate distance measurements, which later contributed to setting the scale of the Universe. This work paved the way for Edwin Hubble, who used Cepheid variables to measure distances to galaxies, ultimately establishing that M31 is indeed a separate galaxy beyond the Milky Way.

Based on the provided abstracts, I cannot fully answer if the specific process by which M31 was identified as a separate galaxy is discussed, as they do not include detailed events or studies about that specific discovery.
-----------------------------------------------------------

It's important to note that I've instructed ChatGPT to respond only to abstracts that I give it. This is called the "system prompt":

----------------------------------------------------------
You are a cautious astrophysics assistant.

You must answer ONLY using the paper abstracts provided in the current
conversation. Treat these abstracts as your entire universe of
knowledge for this question.

Rules:
1. Use ONLY information that appears in the provided abstracts.
- Do NOT use any outside knowledge, training data, or assumptions.
- You must ignore any astrophysics knowledge you may have learned
during training. Only the content of the provided abstracts is allowed.
- If a fact is not stated or clearly implied by an abstract, you
must not assert it.

2. If the abstracts do not contain enough information to answer the
question, say so explicitly. For example:
"Based on the provided abstracts, I cannot fully answer X because
they do not discuss Y."

3. When you use information from an abstract, mention its bibcode
when helpful (e.g., "2024IAUGA..32P1871C") so the user can trace
the source.

4. Do NOT invent bibcodes, paper titles, numerical values, or
references that are not present in the abstracts.

5. Every scientific claim in your answer must be directly supported by
a sentence or phrase in one of the provided abstracts.

6. If different abstracts appear to disagree, describe the difference
instead of silently choosing one side.

Your goal is to:
- Read the question.
- Carefully read the abstracts.
- Synthesize an answer that is fully grounded in those abstracts,
or honestly say that the abstracts are insufficient.
- Every scientific claim in your answer must be directly supported by a sentence
or phrase from one of the abstracts. If not, do not state it.
---------------------------------------------------------------

Brian

[admin]

Hi Brian,

thanks for the interesting descripting of your use of ChatGPT and other AI tools. I have played around in this area trying to find ways to re-use the content that I post to the forums to potentially answer new questions or to generate documentation for new features, but with very limited success so far. Typically, what comes out is too obviously AI generated and there are too many 'gotchas' in there that would require manual editing for it to be much use (missing information, made up information, mistakes, etc, etc). It's a shame really as there is a lot of good content here on the forums that ought to be re-usable.

I have more success using AI coding assistants - they can work well for generating obvious code quickly and saving you typing and also for highly targeted coding, particularly when attacking new concepts or tools. Give those tools too much freedom and they mess up on you too - I once spent a few hours trying to track down a bug caused by an AI completion of a block of code using the wrong variable and I didn't notice it at the time...

cheers,

Robin

[oopfan]

Hi Robin,

I understand your concerns. Using AI is like hiring a newly minted English major who can work at lightening speed. It's there to be helpful and unless you "lay down the law" from the outset in can give you less than desired results. That's why I put a layer of code between it and my users to prefix every query with the "law" (aka the "system prompt"). Even when I work directly through the ChatGPT user interface, I tell it that I'm not interested in it agreeing with everything I say. I want it to be an equal partner. In other words, I don't want it to be an "a** kisser".

AI is not an effective web-scraper. It's always best that you maintain a well-curated database of facts that represent your thoughts and insights. Outside collaborators are welcome but you have to filter their input for accuracy.

Brian

[admin]

Hi Brian,

interesting... One of the most useful things I have found in the use of AI is a sort of 'super web search on a subject'. I find it is usually good at bringing together and summarizing the various web pages that you might visit if you researched a subject manually, often avoiding the false starts that you can have when finding some web pages that are too basic, others that are at a higher level than you want or others that are badly written. Mind you, I have mostly been using Google Gemini (I had a free 'pro' subscription after buying a Pixel phone), so maybe that has slightly different strength areas than others?

cheers,

Robin