Jeremy Howard / @jeremyphoward

RT by @jeremyphoward: "We estimate that the decline in Nuclear power Plants caused by Chernobyl led to the loss of approximately 141 million expected life years in the U.S., 33 in the U.K. and 318 million globally". https://www.nber.org/conferences/si-2024-political-economy

@hparedes_c — Wed, 24 Jul 2024 10:26:14 GMT

\"We estimate that the decline in Nuclear power Plants caused by Chernobyl led to the loss of approximately 141 million expected life years in the U.S., 33 in the U.K. and 318 million globally\". nber.org/conferences/si-2024…

\n\n

RT by @jeremyphoward: We received tons of great submissions to the Build with Claude June 2024 contest from @AnthropicAI devs!\n\nHere are the 3 winning projects, each receiving $10k in Anthropic API credits:

@alexalbert__ — Wed, 24 Jul 2024 17:20:09 GMT

We received tons of great submissions to the Build with Claude June 2024 contest from @AnthropicAI devs!
\n
\nHere are the 3 winning projects, each receiving $10k in Anthropic API credits:

RT by @jeremyphoward: "Here's GELU in a single screen of code"\nThe blessed innocent: "What's the big deal?"\nThose who've tried to write something in WebGPU/WGSL before: 🤯🤯🤯🤯🤯

@johnowhitaker — Thu, 25 Jul 2024 00:06:45 GMT

\"Here's GELU in a single screen of code\"
\nThe blessed innocent: \"What's the big deal?\"
\nThose who've tried to write something in WebGPU/WGSL before: 🤯🤯🤯🤯🤯

nitter.net/austinvhuang/status/1816141050760909003#m

R to @jeremyphoward: Sorry HTMX coders you're not cool outsiders any more.

@jeremyphoward — Thu, 25 Jul 2024 02:49:11 GMT

Sorry HTMX coders you're not cool outsiders any more.

HTMX is #2 on the "most admired web tech" list in this years StackOverflow survey! (The brilliant Phoenix framework is a well-deserved #1).

@jeremyphoward — Thu, 25 Jul 2024 02:49:10 GMT

HTMX is #2 on the \"most admired web tech\" list in this years StackOverflow survey! (The brilliant Phoenix framework is a well-deserved #1).

RT by @jeremyphoward: On the day I’ve leaned of the death of Nobel Laureate Robin Warren (who co-discovered that a bacteria caused most stomach ulcers), this shows once again just how hard it is to overcome health dogmas.

@CrabbBrendan — Wed, 24 Jul 2024 21:52:30 GMT

On the day I’ve leaned of the death of Nobel Laureate Robin Warren (who co-discovered that a bacteria caused most stomach ulcers), this shows once again just how hard it is to overcome health dogmas.

nitter.net/AcrossTheMersey/status/1816145704974229914#m

RT by @jeremyphoward: 👀

@bclavie — Wed, 24 Jul 2024 15:53:33 GMT

👀

RT by @jeremyphoward: Performance measurements of Llama3.1 405B, orchestrated from @OpenRouterAI, one of the leading LLM aggregation platforms. Here are my couple cents of the model:\n\n- It's amazing to see the quick supportof the model by almost all providers. Open source makes software and model co-development much easy. In our case, it took us minimal python code change to support it (like, minutes).\n\n- Llama 3.1 405B is indeed a model hard to make profitable. Taking half a machine or a machine to run, its cost is significant and speed is still so-so. Most providers keep it around 30 tokens/s (see pic) to make economic sense. In comparison, 70B models can go north of 150 tokens/s.\n\n- You will still be able to break even. Of course, this is dependent on a good optimization, and a good workload saturation. To our VC friends: for pure API service at this price tag, kindly not expect an 80% profit margin like conventional SaaS though.\n\n- In addition to top performance optimization, the @LeptonAI API makes conscious balances between the many parameters - speed, price, concurrency, cost - to make sure that it is sustainable.\n\n- Quantization is going to be a standard. Folks, forget about FP16. Int8/FP8 is the way to go. If you still feel uncomfortable, let me tell you that back in the days AI frameworks worried about precision and still supports FP64. Have you ever used FP64 in your neural nets?\n\n- Quantization needs care. Gone are the days when one scale is enough for the whole tensor. You'll need to do channel wise / group wise quantization to make sure things do not degrade.\n\n- My bold prediction is that, 405B adoption will still be limited by the speed and price constraint. But I am not much worried, as I expect at least another 4x efficiency improvement over the next year or so.\n\n- I am looking forward to testing out Mistral Large 123B ! Our Tuna engine supports it out of box, although to honor the research license, we'll refrain from hosting a public API. If you are interested, let us know.\n\n- @karpathy has an awesome tweet about small models FTW. I totally agree. In vertical applications you probably don't need models that big. 70B is normally good enough, and in many cases 8B is really good with finetuning!\n\n- It's great that llama 3.1 allows (and in some way recommends) finetuning your own model.\n\n- I also want to give a shout out to @vllm_project . We have our own engine but vLLM is simply great. Our platform supports it too.\n\nLast but not least, public API is one thing but feel free to reach out to us for enterprise / dedicated deployments. We believe that AI is awesome beyond APIs, and we build a full AI cloud to serve your end to end needs.

@jiayq — Thu, 25 Jul 2024 01:44:20 GMT

Performance measurements of Llama3.1 405B, orchestrated from @OpenRouterAI, one of the leading LLM aggregation platforms. Here are my couple cents of the model:
\n
\n- It's amazing to see the quick supportof the model by almost all providers. Open source makes software and model co-development much easy. In our case, it took us minimal python code change to support it (like, minutes).
\n
\n- Llama 3.1 405B is indeed a model hard to make profitable. Taking half a machine or a machine to run, its cost is significant and speed is still so-so. Most providers keep it around 30 tokens/s (see pic) to make economic sense. In comparison, 70B models can go north of 150 tokens/s.
\n
\n- You will still be able to break even. Of course, this is dependent on a good optimization, and a good workload saturation. To our VC friends: for pure API service at this price tag, kindly not expect an 80% profit margin like conventional SaaS though.
\n
\n- In addition to top performance optimization, the @LeptonAI API makes conscious balances between the many parameters - speed, price, concurrency, cost - to make sure that it is sustainable.
\n
\n- Quantization is going to be a standard. Folks, forget about FP16. Int8/FP8 is the way to go. If you still feel uncomfortable, let me tell you that back in the days AI frameworks worried about precision and still supports FP64. Have you ever used FP64 in your neural nets?
\n
\n- Quantization needs care. Gone are the days when one scale is enough for the whole tensor. You'll need to do channel wise / group wise quantization to make sure things do not degrade.
\n
\n- My bold prediction is that, 405B adoption will still be limited by the speed and price constraint. But I am not much worried, as I expect at least another 4x efficiency improvement over the next year or so.
\n
\n- I am looking forward to testing out Mistral Large 123B ! Our Tuna engine supports it out of box, although to honor the research license, we'll refrain from hosting a public API. If you are interested, let us know.
\n
\n- @karpathy has an awesome tweet about small models FTW. I totally agree. In vertical applications you probably don't need models that big. 70B is normally good enough, and in many cases 8B is really good with finetuning!
\n
\n- It's great that llama 3.1 allows (and in some way recommends) finetuning your own model.
\n
\n- I also want to give a shout out to @vllm_project . We have our own engine but vLLM is simply great. Our platform supports it too.
\n
\nLast but not least, public API is one thing but feel free to reach out to us for enterprise / dedicated deployments. We believe that AI is awesome beyond APIs, and we build a full AI cloud to serve your end to end needs.

nitter.net/jiayq/status/1816246934925107393#m

Yes he did say "live loads". As in: you edit your WebGPU source, save it, and your running code is auto-loaded right away!

@jeremyphoward — Wed, 24 Jul 2024 19:58:10 GMT

Yes he did say \"live loads\". As in: you edit your WebGPU source, save it, and your running code is auto-loaded right away!

nitter.net/austinvhuang/status/1816141053239427343#m

Very cool model, with a very locked down license.

@jeremyphoward — Wed, 24 Jul 2024 19:07:26 GMT

Very cool model, with a very locked down license.

nitter.net/GuillaumeLample/status/1816135838448972240#m

R to @jeremyphoward: Get started at the gpu.cpp website, featuring a gpu.cpp generated render in the background!\nhttps://gpucpp.answer.ai/

@jeremyphoward — Wed, 24 Jul 2024 18:38:37 GMT

Get started at the gpu.cpp website, featuring a gpu.cpp generated render in the background!
\ngpucpp.answer.ai/

Woohoo gpu.cpp has now been officially released!\n\nThis is a really exciting project which provides a device-independent way to use GPU compute. No more writing separate code for CUDA, AMD, Mac, and Intel GPUs!\n\nWe can't wait to see what you all are able to create with this.

@jeremyphoward — Wed, 24 Jul 2024 18:38:37 GMT

Woohoo gpu.cpp has now been officially released!
\n
\nThis is a really exciting project which provides a device-independent way to use GPU compute. No more writing separate code for CUDA, AMD, Mac, and Intel GPUs!
\n
\nWe can't wait to see what you all are able to create with this.

nitter.net/austinvhuang/status/1816141044540739642#m

RT by @jeremyphoward: A.G.I. is here!\n\n(A) lot of (G)PUs (I)nterconnected\n\nhttps://bit.ly/46hmfIu

@LambdaAPI — Wed, 24 Jul 2024 14:16:47 GMT

A.G.I. is here!
\n
\n(A) lot of (G)PUs (I)nterconnected
\n
\nbit.ly/46hmfIu

RT by @jeremyphoward: Llama 3.1 405B\n\n> inference endpoint up and running on https://app.hyperbolic.xyz/\n> for FREE!\n> most capable open-source model, we serve it unquantized GPUs go brrr🔥\n> "Open Source AI is the path forward" LFG!\n\nZuck nailed it!!!

@Yuchenj_UW — Tue, 23 Jul 2024 20:34:10 GMT

Llama 3.1 405B
\n
\n> inference endpoint up and running on app.hyperbolic.xyz/
\n> for FREE!
\n> most capable open-source model, we serve it unquantized *GPUs go brrr*🔥
\n> \"Open Source AI is the path forward\" LFG!
\n
\nZuck nailed it!!!

nitter.net/hyperbolic_labs/status/1815838830391689330#m

RT by @jeremyphoward: We cooked a bit here, on how to run llamas in transformers for gpu-poors https://github.com/huggingface/huggingface-llama-recipes 🦙 of course for Llama405B

@art_zucker — Tue, 23 Jul 2024 15:22:27 GMT

We cooked a bit here, on how to run llamas in transformers for gpu-poors github.com/huggingface/huggi… 🦙 of course for Llama405B

RT by @jeremyphoward: Switzerland now requires all government software to be open source https://www.zdnet.com/article/switzerland-now-requires-all-government-software-to-be-open-source/

@kennethreitz42 — Tue, 23 Jul 2024 22:45:50 GMT

Switzerland now requires all government software to be open source zdnet.com/article/switzerlan…

RT by @jeremyphoward: A short time ago in 10 timezones from California away... While Llama 3.1 is (rightfully) all the rage, some weirdos are making progress on generating all tokens at once with flow matching (a diffusion family process), and testing on the hardest task to get exactly right: codegen!

@syhw — Wed, 24 Jul 2024 07:59:57 GMT

A short time ago in 10 timezones from California away... While Llama 3.1 is (rightfully) all the rage, some weirdos are making progress on generating all tokens at once with flow matching (a diffusion family process), and testing on the hardest task to get exactly right: codegen!

nitter.net/FelixKreuk/status/1815820723492933996#m

RT by @jeremyphoward: Llama 3.1 405B Instruct beats GPT-4o on MixEval-Hard\n\nJust ran MixEval for 405B, Sonnet-3.5 and 4o, with 405B landing right between the other two at 66.19\n\nThe GPT-4o result of 64.7 replicated locally but Sonnet-3.5 actually scored 70.25 🤔

@morgymcg — Tue, 23 Jul 2024 21:47:56 GMT

Llama 3.1 405B Instruct beats GPT-4o on MixEval-Hard
\n
\nJust ran MixEval for 405B, Sonnet-3.5 and 4o, with 405B landing right between the other two at 66.19
\n
\nThe GPT-4o result of 64.7 replicated locally but Sonnet-3.5 actually scored 70.25 🤔

RT by @jeremyphoward: As a gpu-poor I hope this works at scale. 1.8k $ can you believe this?\n\nhttps://arxiv.org/abs/2407.15811

@cloneofsimo — Tue, 23 Jul 2024 14:08:12 GMT

As a gpu-poor I hope this works at scale. 1.8k $ can you believe this?
\n
\narxiv.org/abs/2407.15811

RT by @jeremyphoward: DPO instead of PPO\nFine tuned on Brave search API\nJson formatted tool use parsers\n\nThis really feels like a lot of community contributed ideas and efforts. Honestly, this is how its supposed to work and i couldn't be happier about it

@_xjdr — Tue, 23 Jul 2024 15:32:22 GMT

DPO instead of PPO
\nFine tuned on Brave search API
\nJson formatted tool use parsers
\n
\nThis really feels like a lot of community contributed ideas and efforts. Honestly, this is how its supposed to work and i couldn't be happier about it

Jeremy Howard / @jeremyphoward

RT by @jeremyphoward: "We estimate that the decline in Nuclear power Plants caused by Chernobyl led to the loss of approximately 141 million expected life years in the U.S., 33 in the U.K. and 318 million globally". https://www.nber.org/conferences/si-2024-political-economy

RT by @jeremyphoward: We received tons of great submissions to the Build with Claude June 2024 contest from @AnthropicAI devs!\n\nHere are the 3 winning projects, each receiving $10k in Anthropic API credits:

RT by @jeremyphoward: "Here's GELU in a single screen of code"\nThe blessed innocent: "What's the big deal?"\nThose who've tried to write something in WebGPU/WGSL before: 🤯🤯🤯🤯🤯

R to @jeremyphoward: Sorry HTMX coders you're not cool outsiders any more.

HTMX is #2 on the "most admired web tech" list in this years StackOverflow survey! (The brilliant Phoenix framework is a well-deserved #1).

RT by @jeremyphoward: On the day I’ve leaned of the death of Nobel Laureate Robin Warren (who co-discovered that a bacteria caused most stomach ulcers), this shows once again just how hard it is to overcome health dogmas.

RT by @jeremyphoward: 👀

Yes he did say "live loads". As in: you edit your WebGPU source, save it, and your running code is auto-loaded right away!

Very cool model, with a very locked down license.

R to @jeremyphoward: Get started at the gpu.cpp website, featuring a gpu.cpp generated render in the background!\nhttps://gpucpp.answer.ai/

Woohoo gpu.cpp has now been officially released!\n\nThis is a really exciting project which provides a device-independent way to use GPU compute. No more writing separate code for CUDA, AMD, Mac, and Intel GPUs!\n\nWe can't wait to see what you all are able to create with this.

RT by @jeremyphoward: A.G.I. is here!\n\n(A) lot of (G)PUs (I)nterconnected\n\nhttps://bit.ly/46hmfIu

RT by @jeremyphoward: Llama 3.1 405B\n\n> inference endpoint up and running on https://app.hyperbolic.xyz/\n> for FREE!\n> most capable open-source model, we serve it unquantized *GPUs go brrr*🔥\n> "Open Source AI is the path forward" LFG!\n\nZuck nailed it!!!

RT by @jeremyphoward: We cooked a bit here, on how to run llamas in transformers for gpu-poors https://github.com/huggingface/huggingface-llama-recipes 🦙 of course for Llama405B

RT by @jeremyphoward: Switzerland now requires all government software to be open source https://www.zdnet.com/article/switzerland-now-requires-all-government-software-to-be-open-source/

RT by @jeremyphoward: Llama 3.1 405B Instruct beats GPT-4o on MixEval-Hard\n\nJust ran MixEval for 405B, Sonnet-3.5 and 4o, with 405B landing right between the other two at 66.19\n\nThe GPT-4o result of 64.7 replicated locally but Sonnet-3.5 actually scored 70.25 🤔

RT by @jeremyphoward: As a gpu-poor I hope this works at scale. 1.8k $ can you believe this?\n\nhttps://arxiv.org/abs/2407.15811

RT by @jeremyphoward: DPO instead of PPO\nFine tuned on Brave search API\nJson formatted tool use parsers\n\nThis really feels like a lot of community contributed ideas and efforts. Honestly, this is how its supposed to work and i couldn't be happier about it

RT by @jeremyphoward: Llama 3.1 405B\n\n> inference endpoint up and running on https://app.hyperbolic.xyz/\n> for FREE!\n> most capable open-source model, we serve it unquantized GPUs go brrr🔥\n> "Open Source AI is the path forward" LFG!\n\nZuck nailed it!!!

summary

Version 2