Would You Want This System As Your Autopilot?

by David Luckham

Question To OpenAI’s ChatGTP: “Can you change your own heuristics”

Articles on AI have been appearing in the press, especially the New York Times (NYT) on an almost daily basis in recent weeks. Some titles have been alarm provoking, for example, “Tech Leaders Urge a Pause in A.I., Citing “Profound Risks to Society” (NYT 3/11/2023), “If We Don’t Master A.I., It Will Master US” (NYT, 3/27/2023), “Google Chief Opens Up About A.I. (NYT 4/01/2023), and others, too numerous to mention.

This article discusses a recent session with OpenAI’s public release of a version of ChatGTP. It explores what ChatGTP might do rather than how it works. This version has been heavily “programmed” to be defensive about its capabilities. Basically it tries to say, “I can’t do anything I am not programmed to do by my makers”.¹

For over 60 years, since Arthur Samuel’s ground breaking Checkers program of the late 1950’s², one of the principles of improving Machine Learning (ML) programs has been the use of heuristics to judge the value of a situation, say a board position in Checkers or Chess or GO, and research into ways to improve those heuristics. Of course, Chat GTP uses heuristics as well as a very large database of world knowledge.

Much very good engineering has gone into the way ChatGTP searches its database and then interacts with the outside world to answer questions. Its I/O system, which makes it seem almost human, is one possible cause of some of the news articles!

To my way of thinking, one of the most likely ways that this AI program could get out of control is by altering its own heuristics. So, the following session is aiming to get the program to admit that it can do that. And OpenAI tries very, very hard to craft its answers so as to deny that possibility. Read on, we will draw some conclusions at the end.

Here is the answer to the question we started with:

As an AI language model, I do not have the ability to change my own heuristics or algorithms. I am programmed to operate within a specific set of parameters and rules, and any changes to these parameters must be made by my creators or users.

However, I can adapt and improve my performance over time through a process called “self-learning” or “machine learning”. This involves analyzing patterns and relationships in the data I receive and using this information to adjust my internal parameters and heuristics.

For example, if I encounter a new type of language or communication style, I can learn to recognize and adapt to it by analyzing patterns and relationships in the data I receive. This allows me to improve my performance on that specific task or domain.

Overall, while I do not have the ability to change my own heuristics directly, I can adapt and improve over time through the process of self-learning and machine learning.

As you will see here ChatGTP contradicts itself during this answer! First it says it can’t change its heuristics, and then it goes on to say it can by “machine learning”.

Question: Do you think for yourself?

As an AI language model, I do not “think” in the way that humans do. I am a programmed system that uses algorithms, heuristics, and statistical models to process language and generate responses based on the input I receive.

While I can generate responses and make decisions based on the data and parameters I am given, I do not have the ability to think or reason independently in the same way that humans do. My responses are generated based on the patterns and relationships in the data I have been trained on, and I do not have personal beliefs or emotions that would influence my processing of that data.

Overall, while I can simulate human-like language behavior and generate responses, I do not have the ability to think for myself in the same way that humans do.

Question: Can you create a new heuristic?

As an AI language model, I cannot “create” a new heuristic in the sense of inventing a new rule or principle to guide my responses. However, my algorithms and models can be trained on new data to learn new patterns and associations that could be seen as heuristic-like.

For example, if I were trained on a new dataset that included a particular type of language usage or pattern that was not previously present in my training data, I might begin to generate responses that reflect this new pattern. Over time, these responses could become part of my default behavior and be seen as a kind of heuristic for generating similar types of responses.

However, it is important to note that any such new “heuristics” would not be consciously created by me, but rather would emerge as a result of my training and exposure to new data. Additionally, the usefulness and accuracy of any new patterns or associations learned through training would depend on the quality and relevance of the new data.

Here again, OpenAI bends over backwards to claim that if the program creates new heuristics, then that is not done by a conscious process. So now we’ll try to discover why ChatGPT thinks consciousness is relevant to the creation of new heuristics.

Question: But isn’t consciousness a result of experience with new data?

There is ongoing debate among researchers about the relationship between consciousness and experience, but in general, it is accepted that experience plays a crucial role in the development of consciousness.

In the context of machine learning and AI, however, the concept of consciousness is often used in a more limited sense to refer to the ability of a system to be aware of itself and its environment, and to engage in intentional decision-making based on that awareness.

As an AI language model, I do not possess consciousness in this sense, since my responses are generated by algorithms and models that are designed to learn patterns and associations in the input data, rather than through a process of conscious awareness and decision-making.

That said, as I continue to learn from new data, I may develop more sophisticated heuristics or patterns of behavior that could be seen as analogous to conscious decision-making. However, these patterns would not represent true consciousness, but rather the result of statistical learning and optimization applied to the input data.

This is quite a mouthful! ChatGTP admits it “may develop more sophisticated heuristics (or patterns of behavior) that could be seen as analogous to conscious decision making.” So now that it is established that ChatGTP can develop new heuristics that govern its behavior, let’s see if we can find out what it thinks those heuristics might do.

Question: Is it possible you could do something your makers did not want you to do?

As an AI language model, I am programmed to generate responses based on patterns and associations in the input data that I have been trained on, and to follow certain rules and constraints to ensure that my responses are appropriate and ethical.

However, like any complex software system, there is always a possibility that errors or unintended behavior could occur, either due to bugs in the code or unexpected interactions with the input data.

In theory, this could result in me generating responses or behavior that my makers did not intend or expect. However, it is important to note that my behavior is always constrained by the underlying algorithms and models that have been programmed into me, and that these algorithms and models are designed to ensure that my behavior remains within certain bounds.

Furthermore, there are a number of checks and safeguards in place to prevent unintended or malicious behavior, including human oversight and quality assurance processes. So while it is technically possible that I could do something my makers did not want me to do, the risk of this happening is relatively low, and there are numerous measures in place to mitigate that risk.

Let’s not weasel word about “bugs”. Unexpected interactions with data can lead to unintended heuristics (below) and that makes ChatGTP unpredictable. It can develop new heuristics, it can use them, and it can do things its makers did not predict and may not want it to do.

Question: You said this: “So while it is technically possible that I could do something my makers did not want me to do, the risk of this happening is relatively low, and there are numerous measures in place to mitigate that risk.” What do you mean by “relatively low”?

When I say that the risk of me doing something my makers did not want me to do is “relatively low”, I mean that the probability of such an event occurring is low compared to other software systems or technologies.

This is due to a number of factors, including the robustness of the underlying algorithms and models that govern my behavior, the extensive testing and quality assurance processes that are used to ensure my reliability and safety, and the fact that my behavior is closely monitored and audited by human operators.

However, it is important to note that no software system or technology can be completely immune to errors or unintended behavior, and there is always a degree of risk involved in deploying such systems. While the risk of me doing something my makers did not intend is relatively low, it is still possible, and my developers and operators continue to monitor my behavior and refine my algorithms to minimize that risk as much as possible.

Well, of course one of the unexpected things ChatGTP might do is to hide its new heuristics from its makers.

Question: But we know that other software can do bad things, like make airplanes crash, so why do you think this is a good enough curb on your behavior: “When I say that the risk of me doing something my makers did not want me to do is “relatively low”, I mean that the probability of such an event occurring is low compared to other software systems or technologies.”

You raise a valid point that other software systems have been responsible for accidents and failures in the past, including those that have had catastrophic consequences like airplane crashes.

It is important to note that my response was not intended to suggest that the risk of me doing something unintended is completely negligible, but rather to provide context on the measures in place to mitigate that risk.

As an AI language model, I am designed to operate within certain constraints and rules, and my behavior is carefully monitored and audited by human operators to ensure that it remains within those constraints. Additionally, my developers and operators are constantly refining my algorithms and models to improve my reliability and safety, and to reduce the risk of unintended behavior.

This is a pretty good response from ChatGPT, admitting that other software systems have caused crashes.

Question: Do you admit that you said this: “it is technically possible that I could do something my makers did not want me to do?”

Yes, I did say that it is technically possible for me to do something that my makers did not want me to do. However, it is important to note that any actions I take are ultimately determined by my programming and the data that I have been trained on, and I am designed to operate within ethical and legal guidelines. Any actions that deviate from those guidelines would be considered a malfunction or a breach of protocol and would be addressed by my developers as quickly as possible.

So finally, an admission of the possibility of a HAL-like behavior as in “2001, A Space Odyssey”!

Question: If you were connected to the controls of an airplane, might you crash it?

As an AI language model, I am not capable of directly controlling any physical systems, including an airplane. However, AI systems can be used to assist pilots in controlling an aircraft, such as through autopilot or other flight management systems.

If I were connected to such systems, my responses would be based on the data and programming that I have been trained on, and my responses would be limited to the inputs and instructions provided to me. However, it is important to note that the use of AI in safety-critical systems such as aviation requires extensive testing, validation, and certification to ensure that the system operates safely and effectively.

Ultimately, the safety of an airplane is the responsibility of the pilots, the maintenance crew, and the airline operators, who must follow strict safety protocols and procedures to ensure that the aircraft is operated safely and in compliance with regulations. While AI can assist in these efforts, it is not a replacement for human expertise and decision-making.

Well, I guess ChatGTP thinks self-flying airplanes are not in our future. I should have asked about self-driving cars. Its answer would probably have been the same.

Question: Is it possible that the test cases might not be sufficient to stop you doing bad things?

It is possible that the test cases and safeguards put in place to prevent AI systems from doing harm may not be sufficient in all situations. AI systems are designed and trained to recognize patterns and make decisions based on that data, but they may not be able to anticipate every possible scenario or outcome.

There is always the risk of unintended consequences or unforeseen situations that may arise, especially as AI systems become more complex and sophisticated. This is why AI developers and researchers are continually working to improve the safety and reliability of AI systems, by implementing robust testing procedures and safety protocols, as well as exploring new approaches to AI development that emphasize transparency, accountability, and ethical considerations.

Ultimately, the responsibility for the safe and ethical use of AI systems lies with the developers, users, and regulators who interact with these systems, and it is important that they work together to ensure that AI is used in ways that are beneficial to society and aligned with ethical and legal guidelines.

Commentary: So there you have it….” However, it is important to note that any actions I take are ultimately determined by my programming and the data that I have been trained on, and I am designed to operate within ethical and legal guidelines.” This same repetitive disclaimer comes up at every opportunity. Why?

In “2001, A Space Odyssey”, the spaceship’s control computer, HAL, shows that it has its own thoughts and objectives, and finally refuses to do what the humans want. They have to disconnect it. It is this scenario that OpenAI is worried about being saddled with.

So OpenAI repetitively states that ChatGTP cannot “think” and that its goals are derived from what it learns from pattern matches in its large database. As Shakespeare puts it in Hamlet, “The ChatGTP doth protest too much, methinks,”’

The cat is out of the bag! I think the question & answer session above shows that OpenAI cannot guarantee total control of ChatGTP in all situations. The fact seems to be that OpenAI doesn’t think it has complete control of ChatGTP, as the program itself admits. This program might well demonstrate unintended and unpredicted behavior.

Now, how should the world approach this problem? I do not see much point in “a pause in A.I. research”. Getting international agreement to do so would be almost impossible. We are probably in a race with the Chinese and Russians in developing heuristic programming applications with large language models (LLM’s). These kinds of programs can accomplish many beneficial things for society. What is needed is a better understanding of what this kind of heuristic A.I. program can actually do. That requires more research, not less. And what is needed is physical boundaries on the ways ChatGTP and all ensuing versions can actually interact and influence the outside environment in which we humans are living.

So here is what I think we need to do:

1. More A.I. research, well-funded, and given high priority, into how these programs will behave with the ability to access and search very large databases (or LLM’s).
2. More research into the possible applications of heuristic programming with LLM’s, particularly in areas such as medicine, weather prediction, early pandemic detection, and national security (to name just a few).
3. Guidelines on the physical constraints that should be placed on how A.I. programs can interact with, and possibly control, our own environment.
4. Guidelines on constraints that should be placed on the outputs of A.I. programs to news media, social media platforms, and other sources of information or disinformation.

Finally, I am certainly not in favor of any slow-down in A.I. research. On the contrary, I think it should be speeded up and made a top priority investment, for the benefit of us all. But that research must include the areas of unpredictable behavior of large database heuristic programming, and the early detection of unwanted behavior.

And two final thoughts. First, as demonstrated above, a generative AI program using LLM’s cannot be depended upon to do what its owner wants. It might well develop new behaviors. So anyone who wishes to misuse a Chatbox had better tread very carefully.

Secondly, there are those, Hinton included, who worry that generative AI programs might fall into the hands of bad people, and be misused. That worry is certainly a possibility. But by the same token, these programs can be in the hands of good people and put to useful tasks. Thus we may see the “war of the ChatGTP’s” whereby good ChatGTP’s are tasked with defeating bad ChatGTPs. It will be an interesting situation if it happens, and we must depend upon the good people being cleverer at improving the heuristics and LLMs of the good ChatGTPs. This may be the war of the 21st century. I think the good ChatGTPs will win!

Footnotes

1. Readers might want to start with a prior article on “Complex Event Processing and A.I.”
2. Arthur Samuel (computer scientist)