xAI
Grok 3 AI completed pretraining, says xAI founder
Generative AI company xAI is preparing the Grok 3 large language model for launch, which has now completed pretraining with more power than its predecessor confirms.
“Grok 3 is coming soon. Pretraining is now complete with 10X more compute than Grok 2” xAI founder Elon Musk wrote on X.
Pretraining is the first and long-run process for LLMs before entering the testing and pre-launch stage. It enables the model to absorb massive amounts of text and code data to learn language patterns, structures, and nuances. It includes tasks such as:
- Predicting the next word in the sequence to verify grammar, vocabulary, and some basic semantic relationships.
- It also fills the text where certain words or phrases are masked or missing. The model must learn this important phase to understand the content and meaning of each prompt.
This phase provides a general understanding of language, generating text, translating languages, and answering basic questions. Once completed, the model then transits to the next phases including
- Fine-tuning
- Testing and evaluation
- Deployment and monitoring
Grok is a generative AI chatbot designed and developed by xAI. Since its first launch in December 2023, the company has made rapid progress in deploying more computing power to its LLM.
It has seeded two major versions to date including 1.5 and 2.0. Both of these have improved response rates, features, and efficiency for users. Some of the notable additions are increased tokens, a new image generation tool – Aurora, image & file analysis, and advanced searches including web articles.
Besides pretraining, the xAI founder has not shared a projected launch date for the Grok 3 model.