3 min read
Building Cutting-Edge AI Research on Consumer Hardware with Mojo
An overview of my ongoing research exploring transformer injectivity, self-improving models, and high-performance AI on an RTX 3090
mojo
ai-research
transformers
Read more
[ Transformers ]
Discover insights, tutorials, and thoughts on technology, homelab, and development.
5 min read
Implementing Pi-Attention: 50% Memory Reduction for Long-Context LLMs
How periodic sparse attention achieves O(n) complexity while maintaining model quality
transformers
attention
memory-optimization
Read more
5 min read
Self-Improving AI: Using Inversion Fidelity as a Training Signal
A novel discovery: hidden state inversion quality predicts model capability, enabling self-improving systems without external feedback
ai-training
self-improvement
transformers
Read more
4 min read
SipIt: Extracting System Prompts from Transformer Hidden States
How I achieved 100% token recovery from Mistral-7B hidden states and what it means for AI security
transformers
security
interpretability
Read more
5 min read
The Hidden State Attack: Why Your LLM's System Prompt Isn't Secret
Responsible disclosure of a class of vulnerabilities that allow system prompt extraction from transformer hidden states
security
llm
transformers
Read more