last week I shared that I left Tesla to pivot into AI and robotics at Yaak AI.
naturally, people were curious as to how I’m planning to learn these new topics, because I’ve never formally studied them before. most of my work as a software engineer hasn’t required any formal knowledge in maths, nor knowledge in AI to use LLMs.
some background on me: I didn’t go to university, started learning to program at 17, got my first job at Google at 18, then 2 years later I joined Tesla. so I would consider myself an autodidact.
because of this new journey, I thought it’d be worth sharing the resources I’ll be using to aid me.
the topics I need to study were narrowed down to:
- Maths
- AI and Robotics (I merged them because they overlap heavily. the papers I’m reading are from modern robotics, which is mostly ML)
Maths
A Programmer’s Introduction to Mathematics by Jeremy Kun
I came across this book while reading Ludwig’s post, and it has been phenomenal so far. if you’re someone whose eyes glaze over formulas, then reconstructing them with code helps. I’m using this book as a sort of catch all for the topics I need, then if I need to go deeper I can use something like 3Blue1Brown’s videos.
I’m going through these chapters:
- Chapter 2. Polynomials
- Chapter 4. Sets
- Chapter 6. Graphs
- Chapter 8. Calculus with One Variable
- Chapter 10. Linear Algebra
- Chapter 12. Eigenvectors and Eigenvalues
- Chapter 14. Multivariable Calculus and Optimization
- Chapter 16. Groups
you can pay as you feel for the PDF, but I bought a used physical copy which I think was very worth it. so have a look around on eBay or Amazon.
Essence of Calculus and Linear Algebra by 3Blue1Brown
for many years I’ve heard of 3b1b’s videos but never had a reason to watch them until now. the animations do help with making concepts click, plus it’s nice that they provide text versions to reference (I tend to lean towards reading more than watching).
these 2 resources are pretty much the only ones I’ll be using for maths. I don’t need to overwhelm myself, plus I’m not studying to become a mathematician, just enough to get me by.
AI and Robotics
Books
Deep Learning by Goodfellow, Bengio and Courville
my CTO gave me his physical copy, and I’m using it more as a reference book to lookup things. the book is free to read online, but I generally think having physical copies of textbooks like this are worth it.
Society of Mind by Marvin Minsky
gifted by my CTO, not relevant to my studies, but it seems relevant in the age of agents and as something miscellaneous to read.
Videos and Courses
- Neural Networks by 3Blue1Brown - I think the visuals (of course the url is suffixed with
pi) are even more important in learning AI, so I’m happy with it so far - Neural Networks: Zero to Hero by Andrej Karpathy
- Neural Networks and Deep Learning by Michael Nielsen - I went through the first few chapters a couple months ago, I like how it teaches from the absolute foundations of deep learning that won’t go out of fashion any time soon (aka Lindy)
- Practical Deep Learning by fast.ai
- Robotic Manipulation by MIT
- Robotics Course by Hugging Face
Papers
- Attention Is All You Need - the transformer architecture that changed everything
- An Image is Worth 16x16 Words (ViT) - transformers applied to vision
- Denoising Diffusion Probabilistic Models - the maths behind diffusion models
- π₀: A Vision-Language-Action Flow Model - for one of my interviews with Yaak I had to study this paper and present it back to them. this is the first paper I ever read in my life, well written. let me know if you’re interested in seeing that presentation!
- RT-2: Vision-Language-Action Models - how vision-language models can directly output robot actions
- Diffusion Policy - representing visuomotor policy as a diffusion process
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT) - action chunking with transformers, the method behind ALOHA
- Universal Manipulation Interface - portable data collection for robot learning
- Octo: An Open-Source Generalist Robot Policy
- Robot Learning: A Tutorial - overview of Robotics from the Hugging Face/LeRobot team
Blogs & Articles
- A Recipe for Training Neural Networks by Andrej Karpathy - something to read before training models
- From Words to Worlds: Spatial Intelligence by Fei-Fei Li - the field Yaak operates in
- A Gentle Introduction to Graph Neural Networks
it might seem like a lot, but keep in mind I’m not loading all of this suddenly into my brain, it’s just the general path I’m headed towards with studying
I hope it was useful! if there’s anything you could suggest, do dm me on twitter or email me.
(one of the robotic arms I work with is the Franka FR3)