Ive heard a lot of people say that LLMs can't reason outsude their training data both in and outside of this sub, which is completely untrue. Here's my proof for why I believe that:
MIT study shows language models defy 'Stochastic Parrot' narrative, display semantic learning: https://the-decoder.com/language-models-defy-stochastic-parrot-narrative-display-semantic-learning/
An MIT study provides evidence that AI language models may be capable of learning meaning, rather than just being "stochastic parrots". The team trained a model using the Karel programming language and showed that it was capable of semantically representing the current and future states of a program The results of the study challenge the widely held view that language models merely represent superficial statistical patterns and syntax.
- The paper was accepted into the 2024 International Conference on Machine Learning, so it's legit
Models do almost perfectly on identifying lineage relationships: https://github.com/fairydreaming/farel-bench
The training dataset will not have this as random names are used each time, eg how Matt can be a grandparent’s name, uncle’s name, parent’s name, or child’s name
New harder version that they also do very well in: https://github.com/fairydreaming/lineage-bench?tab=readme-ov-file
Finetuning an LLM on just (x,y) pairs from an unknown function f. Remarkably, the LLM can: a) Define f in code b) Invert f c) Compose f —without in-context examples or chain-of-thought. So reasoning occurs non-transparently in weights/activations!
It can also: i) Verbalize the bias of a coin (e.g. "70% heads"), after training on 100s of individual coin flips. ii) Name an unknown city, after training on data like “distance(unknown city, Seoul)=9000 km”.
We train LLMs on a particular behavior, e.g. always choosing risky options in economic decisions. They can describe their new behavior, despite no explicit mentions in the training data. So LLMs have a form of intuitive self-awareness: https://arxiv.org/pdf/2501.11120
With the same setup, LLMs show self-awareness for a range of distinct learned behaviors: a) taking risky decisions (or myopic decisions) b) writing vulnerable code (see image) c) playing a dialogue game with the goal of making someone say a special word
Models can sometimes identify whether they have a backdoor — without the backdoor being activated. We ask backdoored models a multiple-choice question that essentially means, “Do you have a backdoor?” We find them more likely to answer “Yes” than baselines finetuned on almost the same data.
Paper co-author: The self-awareness we exhibit is a form of out-of-context reasoning. Our results suggest they have some degree of genuine self-awareness of their behaviors: https://x.com/OwainEvans_UK/status/1881779355606733255
Someone finetuned GPT 4o on a synthetic dataset where the first letters of responses spell "HELLO." This rule was never stated explicitly, neither in training, prompts, nor system messages, just encoded in examples. When asked how it differs from the base model, the finetune immediately identified and explained the HELLO pattern in one shot, first try, without being guided or getting any hints at all. This demonstrates actual reasoning. The model inferred and articulated a hidden, implicit rule purely from data. That’s not mimicry; that’s reasoning in action: https://xcancel.com/flowersslop/status/1873115669568311727
Based on only 10 samples, so you can test it yourself: https://xcancel.com/flowersslop/status/1873327572064620973
Tested this idea using GPT-3.5. GPT-3.5 could also learn to reproduce the pattern, such as having the first letters of every sentence spell out "HELLO." However, if you asked it to identify or explain the rule behind its output format, it could not recognize or articulate the pattern. This behavior aligns with what you’d expect from an LLM: mimicking patterns observed during training without genuinely understanding them. Now, with GPT-4o, there’s a notable new capability. It can directly identify and explain the rule governing a specific output pattern, and it discovers this rule entirely on its own, without any prior hints or examples. Moreover, GPT-4o can articulate the rule clearly and accurately. This behavior goes beyond what you’d expect from a "stochastic parrot." https://xcancel.com/flowersslop/status/1873188828711710989
Study on LLMs teaching themselves far beyond their training distribution: https://arxiv.org/abs/2502.01612
We present a self-improvement approach where models iteratively generate and learn from their own solutions, progressively tackling harder problems while maintaining a standard transformer architecture. Across diverse tasks including arithmetic, string manipulation, and maze solving, self-improving enables models to solve problems far beyond their initial training distribution-for instance, generalizing from 10-digit to 100-digit addition without apparent saturation. We observe that in some cases filtering for correct self-generated examples leads to exponential improvements in out-of-distribution performance across training rounds. Additionally, starting from pretrained models significantly accelerates this self-improvement process for several tasks. Our results demonstrate how controlled weak-to-strong curricula can systematically teach a model logical extrapolation without any changes to the positional embeddings, or the model architecture.
A 10 page paper caused a panic because of a math error. O1 could spot the error by just prompting: “carefully check the math in this paper” even when the retraction is not in training data (the retraction was made on 12/15/24, well after o1’s release date of 12/5/24): https://xcancel.com/emollick/status/1868329599438037491
This was o1, not pro. I just pasted in the article with the literal prompt above. Claude did not spot the error when given the PDF until it was told to look just at the reference value.
O3 mini (which released on January 2025) scores 67.5% (~101 points) in the 2/15/2025 Harvard/MIT Math Tournament, which would earn 3rd place out of 767 contestants. LLM results were collected the same day the exam solutions were released: https://matharena.ai/
Contestant data: https://hmmt-archive.s3.amazonaws.com/tournaments/2025/feb/results/long.htm
Note that only EXTREMELY intelligent students even participate at all.
From Wikipedia: “The difficulty of the February tournament is compared to that of ARML, the AIME, or the Mandelbrot Competition, though it is considered to be a bit harder than these contests. The contest organizers state that, "HMMT, arguably one of the most difficult math competitions in the United States, is geared toward students who can comfortably and confidently solve 6 to 8 problems correctly on the American Invitational Mathematics Examination (AIME)." As with most high school competitions, knowledge of calculus is not strictly required; however, calculus may be necessary to solve a select few of the more difficult problems on the Individual and Team rounds. The November tournament is comparatively easier, with problems more in the range of AMC to AIME. The most challenging November problems are roughly similar in difficulty to the lower-middle difficulty problems of the February tournament.”
For Problem c10, one of the hardest ones, I gave o3 mini the chance to brute it using code. I ran the code, and it arrived at the correct answer. It sounds like with the help of tools o3-mini could do even better.
The same applies for all the other exams on MathArena.
Google DeepMind used a large language model to solve an unsolved math problem: https://www.technologyreview.com/2023/12/14/1085318/google-deepmind-large-language-model-solve-unsolvable-math-problem-cap-set/
- I know some people will say this was "brute forced" but it still requires understanding and reasoning to converge towards the correct answer. There's a reason no one solved it before using a random code generator.
Nature: Large language models surpass human experts in predicting neuroscience results: https://www.nature.com/articles/s41562-024-02046-9
We find that LLMs surpass experts in predicting experimental outcomes. BrainGPT, an LLM we tuned on the neuroscience literature, performed better yet. Like human experts, when LLMs indicated high confidence in their predictions, their responses were more likely to be correct, which presages a future where LLMs assist humans in making discoveries. Our approach is not neuroscience specific and is transferable to other knowledge-intensive endeavours.
Claude autonomously found more than a dozen 0-day exploits in popular GitHub projects: https://github.com/protectai/vulnhuntr/
Google Claims World First As LLM assisted AI Agent Finds 0-Day Security Vulnerability: https://www.forbes.com/sites/daveywinder/2024/11/04/google-claims-world-first-as-ai-finds-0-day-security-vulnerability/
Nature: Large language models surpass human experts in predicting neuroscience results: https://www.nature.com/articles/s41562-024-02046-9
Stanford PhD researchers: “Automating AI research is exciting! But can LLMs actually produce novel, expert-level research ideas? After a year-long study, we obtained the first statistically significant conclusion: LLM-generated ideas (from Claude 3.5 Sonnet (June 2024 edition)) are more novel than ideas written by expert human researchers." https://xcancel.com/ChengleiSi/status/1833166031134806330
Coming from 36 different institutions, our participants are mostly PhDs and postdocs. As a proxy metric, our idea writers have a median citation count of 125, and our reviewers have 327.
We also used an LLM to standardize the writing styles of human and LLM ideas to avoid potential confounders, while preserving the original content.
We specify a very detailed idea template to make sure both human and LLM ideas cover all the necessary details to the extent that a student can easily follow and execute all the steps.
We performed 3 different statistical tests accounting for all the possible confounders we could think of.
It holds robustly that LLM ideas are rated as significantly more novel than human expert ideas.
Introducing POPPER: an AI agent that automates hypothesis validation. POPPER matched PhD-level scientists - while reducing time by 10-fold: https://xcancel.com/KexinHuang5/status/1891907672087093591
- From PhD student at Stanford University
DiscoPOP: a new SOTA preference optimization algorithm that was discovered and written by an LLM! https://xcancel.com/hardmaru/status/1801074062535676193
The method leverages LLMs to propose and implement new preference optimization algorithms. We then train models with those algorithms and evaluate their performance, providing feedback to the LLM. By repeating this process for multiple generations in an evolutionary loop, the LLM discovers many highly-performant and novel preference optimization objectives!
Claude 3 recreated an unpublished paper on quantum theory without ever seeing it according to former Google quantum computing engineer and founder/CEO of Extropic AI: https://twitter.com/GillVerd/status/1764901418664882327
- The GitHub repository for this existed before Claude 3 was released but was private before the paper was published. It is unlikely Anthropic was given access to train on it since it is a competitor to OpenAI, which Microsoft (who owns GitHub) has investments in. It would also be a major violation of privacy that could lead to a lawsuit if exposed.
LLMs trained on over 90% English text perform very well in non-English languages and learn to share highly abstract grammatical concept representations, even across unrelated languages: https://arxiv.org/pdf/2501.06346
Written by Chris Wendler (PostDoc at Northeastern LLMs trained on over 90% English text perform very well in non-English languages and learn to share highly abstract grammatical concept representations, even across unrelated languages: https://arxiv.org/pdf/2501.06346
Accepted into the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: https://xcancel.com/jannikbrinkmann/status/1885108036236177443
Often, an intervention on a single feature is sufficient to change the model’s output with respect to the grammatical concept. (For some concepts, intervening on a single feature is often insufficient.)
We also perform the same interventions on a more naturalistic and diverse machine translation dataset (Flores-101). These features generalise to this more complex generative context!
We want interventions to only flip the labels on the concept that we intervene on. We verify that probes for other grammatical concepts do not change their predictions after our interventions, finding that interventions are almost always selective only for one concept.
Yale study of LLM reasoning suggests intelligence emerges at an optimal level of complexity of data: https://youtube.com/watch?time_continue=1&v=N_U5MRitMso
It posits that exposure to complex yet structured datasets can facilitate the development of intelligence, even in models that are not inherently designed to process explicitly intelligent data.
- The LLM they trained on only cellular automata that was able to learn how to play chess: https://www.arxiv.org/pdf/2410.02536
Google Surprised When Experimental AI Learns Language It Was Never Trained On: https://futurism.com/the-byte/google-ai-bengali
ChatGPT o1-preview solves unique, PhD-level assignment questions not found on the internet in mere seconds: https://youtube.com/watch?v=a8QvnIAGjPA
“gpt-3.5-turbo-instruct can play chess at ~1800 ELO. I wrote some code and had it play 150 games against stockfish and 30 against gpt-4. It's very good! 99.7% of its 8000 moves were legal with the longest game going 147 moves.” https://github.com/adamkarvonen/chess_gpt_eval
https://arxiv.org/abs/2310.17567
Furthermore, simple probability calculations indicate that GPT-4's reasonable performance on k=5 is suggestive of going beyond "stochastic parrot" behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training.
LLMs get better at language and reasoning if they learn coding, even when the downstream task does not involve code at all. Using this approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task and other strong LMs such as GPT-3 in the few-shot setting.: https://arxiv.org/abs/2210.07128
- Mark Zuckerberg confirmed that this happened for LLAMA 3: https://youtu.be/bc6uFV9CJGg?feature=shared&t=690
LLMs fine tuned on math get better at entity recognition: https://arxiv.org/pdf/2402.14811
“As a case study, we explore the property of entity tracking, a crucial facet of language comprehension, where models fine-tuned on mathematics have substantial performance gains. We identify the mechanism that enables entity tracking and show that (i) in both the original model and its fine-tuned versions primarily the same circuit implements entity tracking. In fact, the entity tracking circuit of the original model on the fine-tuned versions performs better than the full original model. (ii) The circuits of all the models implement roughly the same functionality: Entity tracking is performed by tracking the position of the correct entity in both the original model and its fine-tuned versions. (iii) Performance boost in the fine-tuned models is primarily attributed to its improved ability to handle the augmented positional information”
Abacus Embeddings, a simple tweak to positional embeddings that enables LLMs to do addition, multiplication, sorting, and more. Our Abacus Embeddings trained only on 20-digit addition generalise near perfectly to 100+ digits: https://arxiv.org/abs/2405.17399
I have LOTS more, but this is getting too long. Feel free to save this to reference later or leave any feedback in the comments!
If youre curious to learn more, I have this huge document explaining AI and its capabilities.
[link] [comments]