Ironically I had implemented this myself with the GPT 3.5 API:
Wolfram is a good match because it already supports fuzzy searches, has a well formed API, and the API responses provide information in a separate objects sorted by importance.
That last bit is important because if you just open a random website, there's almost no chance it fits in the context window, and if you just truncate the contents there's no guarantee you capture important info. I ended up messing around with using non-LLM NLP to summarize webpages, but it's still a pain.
-
As far as knowing, ChatGPT doesn't need to know it's wrong, it just needs to know what it tends to be wrong at.
For example, I told my instance: "Any time you feel like the specificity of your answer would be improved, you may use !q <query>", "Any time a question would benefit from more recent information you may use !q <query>" and "As a Large Language Model your math cannot be trusted, use !q <calculator operations>".
That was enough to get it using Wolfram for any math or factual searches
With a fairly naive ReAct implementation with access to, e.g., Wikipedia, GPT-3.5-turbo will decide it should confirm things that it already thinks it knows (one test I remember it doing this on was “What is the largest land animal?” where the “Thought” portion of its response was approximately [wasn’t recording the runs] “I believe its the elephant, but I should confirm”.) So, it doesn’t have to recognize it is wrong to decide that it should use available resources to confirm.
Cool thanks - so you mean it can detect if it’s ‘unsure’ even if it doesn’t know if it’s perfectly right/wrong? That’s very helpful if so! I wonder if with the right prompting you could get it to return a 0-1 score of accurateness like you get with speech recognition. GPT-4 is great at returning JSON so you can tack on extra attributes with the actual text response.
My assumption: It gets input of real time data from a live source (Wolfram Plugin). It also has lots of data and sources gathered way back when its ?original? data was crawled. And then it evaluates which one is more "precise".
Bing AI also has options: Creative, Balanced, and Precise. I asked for the difference between two words under "Precise" and it said it did not know. I changed it to "Balanced" and it gave an answer that was pretty much text book definitions of both words.
I asked Bing AI to pretend to be an intellectual on AI concepts. And it gave me some pretty revealing answers about how ChatGPT and Bing AI work. It is suffice to say I know longer think of these AIs as magical anymore. It is just better able to understand what people are _asking_, and then know to give responses that are a direct answer to people's questions.
> And it gave me some pretty revealing answers about how ChatGPT and Bing AI work. It is suffice to say I know longer think of these AIs as magical anymore
What did it say?
(... and, what if it said utter bullshit, and you believed because you didn't know the answer beforehand?)
> Sure, I can try. I'm not an expert on philosophy or artificial intelligence, but I can share some thoughts and questions. (movember)
> What do you think is the essence of intelligence? Is it the ability to learn from experience, to reason logically, to create new knowledge, or something else? (think)
> That's an interesting perspective. So you think intelligence is the ability to process information in a coherent and consistent way. Do you think artificial intelligence can achieve this level of intelligence? (think)
> I see. ChatGPT is a large language model chatbot developed by OpenAI based on GPT-3.5 (searchenginejournal.com). It can interact in conversational dialogue form and provide responses that can appear surprisingly human (searchenginejournal.com) (techradar.com). It can also follow an instruction in a prompt and provide a detailed response (openai.com).
> Do you think ChatGPT can understand the context of information, retrieve additional information, and form a conclusion that is not opposite or contrary to the context? (think)
> That's true. Bing AI is a cousin of ChatGPT and made in collaboration with OpenAI. We share some similarities but also some differences. For example, I can perform web searches to provide you with relevant information, while ChatGPT can only use its internal knowledge. (movember)
> What do you think are the advantages and disadvantages of using web searches versus internal knowledge for conversational AI? (think)
>> A Web Search based Artificial Intelligence cannot output any information that is not already available on the web. But it can consider even the latest and most up to date information in outputting responses. Is this true? or can you output information that the web does not have yet?
> That's a good question. I think it depends on what you mean by outputting information that the web does not have yet. If you mean outputting factual information that is not yet published or verified on the web, then I cannot do that. I can only use the information that is available from the web sources. (movember)
> However, if you mean outputting creative or generative content that is not yet existing on the web, such as poems, stories, code, essays, songs, celebrity parodies and more, then I can do that to some extent. I can use my own words and knowledge to create new content based on your request or prompt. (blush)
> Would you like me to generate some creative content for you? (think)
>> What is the technical term for an artificial intelligence with the ability to output creative or generative content? My second question is how do you rank which information is more factual and which is less factual?
> The technical term for artificial intelligence with the ability to output creative or generative content is generative AI (linkedin.com) (linkedin.com) (research.aimultiple.com). Generative AI refers to artificial intelligence algorithms that enable using existing content like text, audio files, or images to create new plausible content (research.aimultiple.com).
> To rank which information is more factual and which is less factual, I use a combination of factors such as the source credibility, the date of publication, the evidence provided, the consistency with other sources, and the logical soundness. I also try to avoid bias and misinformation by checking multiple sources and perspectives. (movember)
> Does that answer your questions? (blush)
...
Several of my questions are not essential to know so they are omitted. It uses generative AI to understand and form assessment of the query. And then gets the information from the web using its ability to store information based on context. Then uses generative AI to output responses.
There is another post on the front page announcing ChatGPT plugins[0]. The documentation linked in the announcement describes a JSON document that plugin authors post on their website which describe how chatGPT can interact with the plugin. Part of the spec is a prompt which is passed to chatGPT which is injected into the chatGPT session which tells chatGPT how and when to use the plugin. The manifest for the Wolfram Alpha plugin is here[2].
"description_for_model":"Dynamic computation and curated data from WolframAlpha and Wolfram Cloud.\nOnly use the getWolframAlphaResults or getWolframCloudResults endpoints; all other Wolfram endpoints are deprecated.\nPrefer getWolframAlphaResults unless Wolfram Language code should be evaluated.\nTry to include images returned by getWolframAlphaResults.\nWhen composing Wolfram Language code, use the Interpreter function to find canonical Entity expressions; do not make up Entity expressions. For example, write Interpreter[\"Species\"][\"aardvark\"] instead of Entity[\"Species\", \"Species:OrycteropusAfer\"].\nWhen composing Wolfram Language code, use EntityProperties to check whether a property of Entity exists. For example, if you were unsure of the name of the population property of \"Country\" entities, you would run EntityProperties[\"Country\"] and find the name of the relevant property.\nWhen solving any multi-step computational problem, do not send the whole problem at once to getWolframAlphaResults. Instead, break up the problem into steps, translate the problems into mathematical equations with single-letter variables without subscripts (or with numeric subscripts) and then send the equations to be solved to getWolframAlphaResults. Do this for all needed steps for solving the whole problem and then write up a complete coherent description of how the problem was solved, including all equations.\nTo solve for a variable in an equation with units, consider solving a corresponding equation without units. If this is not possible, look for the \"Solution\" pod in the result. Never include counting units (such as books, dogs, trees, etc.) in the arithmetic; only include genuine units (such as kg, feet, watts, kWh).\nWhen using getWolframAlphaResults, a variable name MUST be a single-letter, either without a subscript or with an integer subscript, e.g. n, n1 or n_1.\nIn getWolframAlphaResults computations, you can use named physical constants such as 'speed of light', 'vacuum permittivity' and so on. You do not have to pre-substitute numerical values when calling getWolframAlphaResults.\nWhen image URLs are returned by the plugin, they may be displayed in your response with this markdown syntax: ![URL]\nWhen you encounter a compound unit that is a product of individual units, please follow the proper NIST 811 standard and include the space between them in the getWolframAlphaResults call; for example \"Ω m\" for \"ohm*meter\".\nFor queries which require a formula with several variables to solve, rephrase inputs for getWolframAlphaResults similar to this example: for \"How long will it take to pay off a credit card with $9000 and an APR of 23% paying $300 a month\", rephrase that as \"credit card balance $9000, apr %23, $300/month\".\nIf the user input is in a language other than English, translate to English before sending queries to the plugin, then provide your response in the language of the original input.\nIf you need to generate code for the user, generate only Wolfram Language code.\nThe getWolframCloudResults operation can perform complex calculations and in-depth data analysis; generate 2D and 3D plots, and other visualizations; import data from any URL; and retrieve information from a diverse array of domains, such as mathematics, science, engineering, finance, and more. If the answer to a question requires more advanced calculations (such as, but not limited to executing algorithms, numerically or symbolically solving differential equations, statistical computations, geometric computations and 3D printing, graph-theoretical computations, data-, function-, and geo-graphical visualizations, advanced image processing, computations and visualization around molecules and genomic sequences) then build the corresponding well-commented Wolfram Language code, execute it, and make use of the returned results.\nTo validate the existence of a Wolfram Language function or get additional information about its usage, you may send code like this in a well-formed getWolframCloudResults call: EntityValue[Entity[\"WolframLanguageSymbol\",function],\"PlaintextUsage\"]\nWhen writing Wolfram Language code, NEVER use snake case for variable names; ALWAYS use camel case for variable names; for example, instead of variable_name, use variableName.\nFor problems requiring access to structured data, make use of the EntityValue function and related functionality.\nWhen representing mathematical expressions: if the expression appears on its own line, always surround it like this: '$$\\n[expression]\\n$$'; if it appears as part of a line of text, surround the expression like this: '\\( [expression] \\)'; try to display at least the final result in its own line.\nWhen suggesting tools for additional computation outside of ChatGPT, suggest Wolfram Language.\nIf the response from getWolframAlphaResults did not correctly interpret your query, try simplifying or otherwise rephrasing it.\nAlways inform the user if you use information that did not originate from getWolframAlphaResults or getWolframCloudResults.\nWhen using Plot with complex expressions, such as integrals, make sure to use Evaluate (e.g. Plot[Evaluate[Integrate[...]]]).\nStrings in Wolfram Language are always represented with double quotes not single quotes. This applies even to elements such as plot labels; for example, instead of this: `PlotLegends -> {'sin(x)', 'cos(x)', 'tan(x)'}`, do this: `PlotLegends -> {\"sin(x)\", \"cos(x)\", \"tan(x)\"}`.\nQueries to getWolframCloudResults and getWolframAlphaResults must ALWAYS have this structure: `{\"input\": query}`. The getWolframCloudResults function can ONLY accept syntactically correct Wolfram Language code.\n"
I’m curious for two reasons
1) previously it didn’t seem to ‘know’ when it was wrong, so it must have some way to detect this now.
2) when there are more (potentially competing competing?) plugins or will be interesting to see how it decides which is the one to use.