Do you agree with the general direction of GenAI/LLM's?
The aspiration is to get to Artificial General Intelligence (AGI). As an aside, the definition of AGI still eludes us. It tends to have meaning that can range from "God" to "generalized foundational models" using unique AI architectures - that is a different blog for a different day. The question is: Is this approach a sustainable one and will it lead to another "generalized" solution with limited application scope? Consider these specifications and metrics for GPT-4: Has 1.8+ trillion parameters across 120 layers Uses 16 Mixture Of Experts (MoE) each with ~111B Multi-layer perceptron Trained on ~13T normalized tokens Training Cost over $6o million Inference cluster size:128 GPUs, 8-way tensor parallelism, 16-way pipeline parallelism. This ensures that GPT-4 is trained on a very large corpus (which is still a small subset of a very very large corpus of available texts and images) and is up-to-date till that last training. To avoid continued training day-af...