ServiceNow, Hugging Face, and NVIDIA unveil StarCoder2, an Open-Access LLM Family.

ServiceNow, Hugging Face, and NVIDIA have launched StarCoder2, a family of large language models designed for code generation, setting new benchmarks in performance, transparency, and cost-effectiveness.

Collaboration between ServiceNow, Hugging Face, and NVIDIA sets new benchmarks in performance, transparency, and cost-effectiveness
StarCoder2 offers three model sizes, allowing for impressive performance while reducing compute costs
Models are trained on a diverse range of programming languages and can be fine-tuned for industry or organization-specific use cases

ServiceNow, Hugging Face, and nVidia have joined forces to introduce StarCoder2, a family of open-access large language models for code generation. This collaboration sets new benchmarks in terms of performance, transparency, and cost-effectiveness. Developed in partnership with the BigCode Community, managed by ServiceNow, and Hugging Face, the leading open-source platform for machine learning collaboration, StarCoder2 is trained on 619 programming languages. It can be integrated into enterprise applications to perform specialized tasks such as code generation, workflow creation, and text summarization. With features like code completion, advanced code summarization, and code snippets retrieval, developers can enhance their productivity and drive innovation.

StarCoder2 offers three model sizes: a 3-billion-parameter model trained by ServiceNow, a 7-billion-parameter model trained by Hugging Face, and a 15-billion-parameter model built by NVIDIA using NVIDIA NeMo and trained on NVIDIA accelerated infrastructure. The smaller variants deliver impressive performance while reducing compute costs. In fact, the new 3-billion-parameter model matches the performance of the original 15-billion-parameter model.

Harm de Vries, lead of ServiceNow’s StarCoder2 development team and co-lead of BigCode, emphasized the significance of this achievement, stating that StarCoder2 exemplifies the power of open scientific collaboration and responsible AI practices. It improves generative AI performance, enhances developer productivity, and ensures equal access to the benefits of code generation AI for organizations of all sizes.

Leandro von Werra, machine learning engineer at Hugging Face and co-lead of BigCode, highlighted the potential of open source and open science in democratizing responsible AI. The joint efforts of Hugging Face, ServiceNow, and NVIDIA enable the release of powerful base models that facilitate efficient application development with full transparency in data and training.

Jonathan Cohen, vice president of applied research at NVIDIA, emphasized the impact of code language models on various industries, stating that every software ecosystem can benefit from these models to drive efficiency and innovation.

StarCoder2 models share a state-of-the-art architecture and utilize curated data sources from BigCode, prioritizing transparency and open governance. These models advance the capabilities of AI-driven coding applications, including text-to-code and text-to-workflow functionalities. With extensive programming training, StarCoder2 provides accurate and context-aware predictions by considering repository context. This advancement benefits both seasoned software engineers and citizen developers, accelerating business value and digital transformation.

The foundation of StarCoder2 is the Stack v2 code dataset, which is over seven times larger than its predecessor, Stack v1. Additionally, new training techniques enable the model to understand low-resource programming languages like COBOL, mathematics, and program source code discussions.

Users can fine-tune the open-access StarCoder2 models with industry or organization-specific data using tools such as NVIDIA NeMo or Hugging Face TRL. This allows the creation of advanced chatbots, personalized coding assistants, code snippet retrievers, and text-to-workflow capabilities. Organizations have already started fine-tuning the foundational StarCoder model to develop specialized task-specific capabilities tailored to their businesses. For instance, ServiceNow’s text-to-code Now LLM was purpose-built on a specialized version of the 15-billion-parameter StarCoder LLM, fine-tuned and trained for its specific workflow patterns, use cases, and processes. Hugging Face has also utilized the model to create its StarChat assistant.

BigCode, led by Hugging Face and ServiceNow, is an open scientific collaboration dedicated to the responsible development of language models for code. The BigCode community actively participated in the technical aspects of the StarCoder2 project through working groups and task forces. Responsible innovation lies at the core of BigCode’s purpose, demonstrated through its open governance, transparent supply chain, use of open-source software, and the ability for developers to opt-out data from training.

StarCoder2 was developed using responsibly sourced data under license from the digital commons of Software Heritage, hosted by Inria. This ensures that the model aligns with ethical AI development policies. StarCoder2 will be available under the BigCode Open RAIL-M license, allowing royalty-free access and use. The model’s supporting code will remain on the BigCode project’s GitHub page, promoting transparency and collaboration. All StarCoder2 models will also be downloadable from Hugging Face, and the StarCoder2 15-billion-parameter model can be experimented with directly from the browser or through an API endpoint via NVIDIA AI Foundation models.

For more information on StarCoder2, visit huggingface.co/bigcode.

About Our Team

Our team comprises industry insiders with extensive experience in computers, semiconductors, games, and consumer electronics. With decades of collective experience, we’re committed to delivering timely, accurate, and engaging news content to our readers.

Background Information

About nVidia:

NVIDIA has firmly established itself as a leader in the realm of client computing, continuously pushing the boundaries of innovation in graphics and AI technologies. With a deep commitment to enhancing user experiences, NVIDIA's client computing business focuses on delivering solutions that power everything from gaming and creative workloads to enterprise applications. for its GeForce graphics cards, the company has redefined high-performance gaming, setting industry standards for realistic visuals, fluid frame rates, and immersive experiences. Complementing its gaming expertise, NVIDIA's Quadro and NVIDIA RTX graphics cards cater to professionals in design, content creation, and scientific fields, enabling real-time ray tracing and AI-driven workflows that elevate productivity and creativity to unprecedented heights. By seamlessly integrating graphics, AI, and software, NVIDIA continues to shape the landscape of client computing, fostering innovation and immersive interactions in a rapidly evolving digital world.

Latest Articles about nVidia

Technology Explained

LLM: A Large Language Model (LLM) is a highly advanced artificial intelligence system, often based on complex architectures like GPT-3.5, designed to comprehend and produce human-like text on a massive scale. LLMs possess exceptional capabilities in various natural language understanding and generation tasks, including answering questions, generating creative content, and delivering context-aware responses to textual inputs. These models undergo extensive training on vast datasets to grasp the nuances of language, making them invaluable tools for applications like chatbots, content generation, and language translation.