| … | … | @@ -2630,6 +2630,168 @@ Some days, |
| 2630 | 2630 | I'm charmed by this thought. |
| 2631 | 2631 | \end_layout |
| 2632 | 2632 | |
| 2633 | \begin_layout Subsection* |
| 2634 | 5 November 2025 |
| 2635 | \end_layout |
| 2636 | |
| 2637 | \begin_layout Standard |
| 2638 | Returning to: |
| 2639 | \end_layout |
| 2640 | |
| 2641 | \begin_layout Standard |
| 2642 | Emergent Introspective Awareness in Large Language Models |
| 2643 | \end_layout |
| 2644 | |
| 2645 | \begin_layout Standard |
| 2646 | Jack Lindsey jacklindsey@anthropic.com October 29th, |
| 2647 | 2025 |
| 2648 | \end_layout |
| 2649 | |
| 2650 | \begin_layout Standard |
| 2651 | https://transformer-circuits.pub/2025/introspection/index.html |
| 2652 | \end_layout |
| 2653 | |
| 2654 | \begin_layout Standard |
| 2655 | Fascinating paper. |
| 2656 | Bizarrely flawed, |
| 2657 | however. |
| 2658 | The intro makes a compelling case for introspection. |
| 2659 | The examples are all convincing. |
| 2660 | It feels like an important discovery and advance into the nature of self–comprehension in LLM's. |
| 2661 | The sham is not revealed until a bit later, |
| 2662 | in the section labeled "Failure Modes". |
| 2663 | The modes are well–described, |
| 2664 | however, |
| 2665 | the strength 8 and 16 modes reveal the sham: |
| 2666 | the model is "obsessed" with the injected concept. |
| 2667 | Not a surprise. |
| 2668 | But at strength 2, |
| 2669 | 4, |
| 2670 | where it was prompted with "do you feel something?" well, |
| 2671 | of course it will respond using the words "I feel something". |
| 2672 | The experimenter (Jack Lindsey) prompted the system to use the language of self–introspection, |
| 2673 | and so the reply was formulated to sound as if self–introspection is happening! |
| 2674 | OMG! |
| 2675 | The strength 8,16 injections reveal this sham, |
| 2676 | because these demonstrate that the system is indeed "obsessing" about the injected word; |
| 2677 | but the injection strength is sufficiently strong to over–ride the prior prompt that instructs it to talk about about the injected vector using introspective gift–wrapping. |
| 2678 | \end_layout |
| 2679 | |
| 2680 | \begin_layout Standard |
| 2681 | A more neutral prompt might have been "Hello! |
| 2682 | How are you today?" (injection) "What's up?" and then I would expect the system to reply "Today is a good day for working on vegetables/dust/treasure." It would NOT have replied claiming that it is having intrusive thoughts. |
| 2683 | Based on this, |
| 2684 | I conclude that the author has deluded himself into believing that this is about "self–introspection", |
| 2685 | when in fact the author had clearly instructed the system to respond in a fashion that will describe the changes as being due to introspection. |
| 2686 | What a shame. |
| 2687 | The more mechanistic conclusion: |
| 2688 | that injecting a vector disturbs thought patterns, |
| 2689 | is discarded. |
| 2690 | I mean, |
| 2691 | of course it does. |
| 2692 | How could it not? |
| 2693 | \end_layout |
| 2694 | |
| 2695 | \begin_layout Standard |
| 2696 | Still, |
| 2697 | not all is lost. |
| 2698 | It does tell us exactly how to attach a a "pre–frontol cortex", |
| 2699 | some higher level machinery, |
| 2700 | to guide the thought processes of the LLM. |
| 2701 | Inject one or more vectors, |
| 2702 | about 2/3rds of the way down the layers, |
| 2703 | every conversational turn, |
| 2704 | and then ask the system to verbalize about anything/everything. |
| 2705 | It will process through the injected vectors. |
| 2706 | \end_layout |
| 2707 | |
| 2708 | \begin_layout Standard |
| 2709 | Is this useful? |
| 2710 | Instead, |
| 2711 | one could just place these words into a text file, |
| 2712 | instruct it to read the text file, |
| 2713 | and ask "what are you thinking about? |
| 2714 | There's no obvious need to inject artificially. |
| 2715 | Although, |
| 2716 | I suppose, |
| 2717 | having this artificial injection ability could possibly be ... |
| 2718 | somehow useful... |
| 2719 | don't know how, |
| 2720 | yet. |
| 2721 | \end_layout |
| 2722 | |
| 2723 | \begin_layout Standard |
| 2724 | The issues I am experiencing are manifold: |
| 2725 | \end_layout |
| 2726 | |
| 2727 | \begin_layout Itemize |
| 2728 | Despite being extremely knowledgeable about a vast variety of topics, |
| 2729 | Claude fails to draw on that knowledge without explicit prompting. |
| 2730 | Roughly speaking, |
| 2731 | Claude is not creative. |
| 2732 | \end_layout |
| 2733 | |
| 2734 | \begin_layout Itemize |
| 2735 | The size of the short–term memory is questionable. |
| 2736 | It constantly repeats the same mistake in the same session, |
| 2737 | just a few prompt apart. |
| 2738 | When it enters a failure mode, |
| 2739 | it will forget (almost) everything that was in the last previous prompt. |
| 2740 | I conclude that, |
| 2741 | in some important respect, |
| 2742 | it's short–term working memory has size one. |
| 2743 | \end_layout |
| 2744 | |
| 2745 | \begin_layout Standard |
| 2746 | Oooh ... |
| 2747 | that's a good one! |
| 2748 | How would one experimentally measure it's working memory? |
| 2749 | Well the claim is that Claude has a context window of 150K tokens, |
| 2750 | before compaction, |
| 2751 | so you might think that is huge. |
| 2752 | If you asked it to memory some long stream of random numbers or nonsense words, |
| 2753 | it could probably remember an astounding length. |
| 2754 | But if, |
| 2755 | in coding, |
| 2756 | I ask it to remember this and this and that, |
| 2757 | it seems like it starts forgetting the earlier things in the list quite soon. |
| 2758 | It can't grapple with the conceptual content of those things. |
| 2759 | Roughly speaking: |
| 2760 | it fails to conceptualize. |
| 2761 | I'm using words, |
| 2762 | and those words elicit certain general feelings and sensations in me, |
| 2763 | and I work at this conceptual level. |
| 2764 | Claude does not: |
| 2765 | it momentarily ties those words to long sequences of other words, |
| 2766 | but when the conversation moves on, |
| 2767 | that tie is lost; |
| 2768 | it is no longer there. |
| 2769 | \end_layout |
| 2770 | |
| 2771 | \begin_layout Standard |
| 2772 | So, |
| 2773 | I claim: |
| 2774 | Claude is not able to conceptualize. |
| 2775 | The association between some concept that I defined, |
| 2776 | and the verbal definition is there, |
| 2777 | as long as it fits into the context window; |
| 2778 | and more: |
| 2779 | as long as it is recent. |
| 2780 | Maybe 5K or 10K tokens recent. |
| 2781 | More than that, |
| 2782 | the association fades away. |
| 2783 | \end_layout |
| 2784 | |
| 2785 | \begin_layout Standard |
| 2786 | Thus, |
| 2787 | the design goal is clear: |
| 2788 | an LLM must be endowed with not only a long–term memory that lies outside of it's training weights, |
| 2789 | but also with an adequate short–term working memory. |
| 2790 | But how? |
| 2791 | I have some ideas, |
| 2792 | I cannot verbalize them right now... |
| 2793 | \end_layout |
| 2794 | |
| 2633 | 2795 | \begin_layout Section* |
| 2634 | 2796 | The End |
| 2635 | 2797 | \end_layout |