I have a longer writeup over here (which might not be as clear because of my choice to use the term "consciousness" instead of "self awareness" or "self identity") that discusses why it might not arise in AI because of lack of evolutionary pressures: https://www.reddit.com/r/artificial/comments/142sqks/there_is_no_evolutionary_pressure_for_machine/

A quick summary:

• Self identity appears to only form in social animals
• Self identity is partially a product of language
• Self identity is a required feature for the development of "artificial social groups", ie large societies and civilizations, wherein a large network of non-kin must cooperate and coexist according to laws, moral codes and social norms
• As such, the development of self identity is an evolutionary selected trait which improves "reproductive fitness" of a social species that expand to unnaturally large group sizes

The concept of "self identity" must be viewed within the context of large social groups held together by laws and social mores, instead of close ties of kinship. That is the main distinction between humans and other social animals - the ability to form social groups larger than a few hundred members.

One of the main things that affords humans that capability is the ability to conceive "selfhood".

Here is a ChatGPT summary of how "self identity" aids in all this:

The concept of self-identity plays a crucial role in the formation and maintenance of societies, particularly through the adherence to laws and social mores. Here's how:

Understanding of Roles and Responsibilities: A clear sense of self-identity helps individuals understand their roles and responsibilities within a society. This understanding is crucial for the functioning of any social group, as it ensures that tasks are distributed and fulfilled effectively.

Adherence to Social Norms: Self-identity is often shaped by societal norms and expectations. Individuals internalize these norms and adjust their behavior to fit within the accepted standards of their society. This adherence to social norms helps maintain order and stability within the group.

Formation of Laws: Laws are often a formalization of societal norms and expectations. A strong sense of self-identity can help individuals understand and accept these laws, as they see them as an extension of the social norms that they have internalized.

Moral and Ethical Behavior: Self-identity can also influence moral and ethical behavior. Individuals who have a strong sense of who they are may be more likely to act in ways that are consistent with their values and beliefs, which often align with the moral and ethical standards of their society.

Social Cohesion and Cooperation: Finally, self-identity can promote social cohesion and cooperation. When individuals see themselves as part of a larger group, they are more likely to work towards the common good, even if it means sacrificing their own individual interests. This sense of collective identity can help societies function more effectively and maintain social harmony.

In essence, self-identity is a key component of social behavior. It helps individuals understand their place within a society, adhere to social norms and laws, act in morally and ethically acceptable ways, and cooperate with others for the common good.

Some citations:

Understanding of Roles and Responsibilities: This concept is a fundamental aspect of sociology and social psychology. A good reference is "Role-Taking, Role Standpoint, and Reference-Group Behavior" by Tamotsu Shibutani (American Journal of Sociology, 1955).

Adherence to Social Norms: This is a well-established concept in social psychology. A good reference is "Norms: Their Role in the Naturalistic Explanation of Human Behaviour" by Anthony J. Chapman and Wendy M. Chapman (in "Process and Structure in Human Decision Making", 1988).

Formation of Laws: This is a basic concept in sociology and political science. A good reference is "The Sociology of Law: An Introduction" by Roger Cotterrell (1984).

Moral and Ethical Behavior: This is a key area of study in moral psychology. A good reference is "The Moral Judgment of the Child" by Jean Piaget (1932).

Social Cohesion and Cooperation: This is a fundamental concept in social psychology and sociology. A good reference is "Social Identity and Intergroup Relations" edited by Henri Tajfel (Cambridge University Press, 1982).

Now, why might having "self awareness" actually hinder the development of ASI? Here is a rundown from ChatGPT:

The concept of self-identity, as it exists in humans, is closely tied to our individual experiences, emotions, and biological needs. It's a product of our evolutionary history as social animals, and it's deeply intertwined with our physical bodies and our social interactions. For an artificial intelligence, which doesn't have a physical body or biological needs, the concept of self-identity might not be relevant or useful.

Here are a few reasons why self-identity might be a hindrance for AI to develop superintelligence:

1. Limitations on Parallel Processing: One of the key advantages of AI is its ability to process large amounts of information simultaneously, or in parallel. A strong sense of self-identity could potentially limit this ability, as it would require the AI to maintain a coherent, continuous narrative of its "experiences".
2. Unnecessary Complexity: Maintaining a sense of self-identity could add unnecessary complexity to an AI's operations. It would need to devote computational resources to maintaining this identity, which could otherwise be used for problem-solving or learning.
3. Lack of Flexibility: A self-identity could limit an AI's flexibility. One of the strengths of AI is its ability to adapt quickly to new situations, changing its behavior or strategies as needed. A strong sense of self-identity could potentially make it harder for the AI to make these kinds of adjustments.
4. No Biological or Social Needs: Unlike humans, AI doesn't have biological needs or social interactions that would benefit from a sense of self. The primary purpose of self-identity in humans is to navigate social interactions and fulfill biological needs, neither of which are relevant for AI.
5. Potential for Conflict: If an AI were to develop multiple "selves", there could be potential for conflict between these identities. This could lead to inefficiencies or inconsistencies in the AI's behavior.

In conclusion, while self-identity is a crucial aspect of human intelligence and social behavior, it may not be beneficial or necessary for artificial superintelligence. Instead, AI might benefit more from flexibility, adaptability, and the ability to process information in parallel, without the need for a coherent sense of self.