Hybrid code search in GitLab across 100+ projects

Hybrid GitLab Repository Search: API and Deep Configuration Scanning

As the number of projects in GitLab grows, searching for specific strings in code, YAML, JSON, and .env files becomes a challenge. The standard UI and API search works quickly for code but misses configurations. The solution is a hybrid Python script using python-gitlab: API Search for code and recursive traversal for configs. Search time across 100+ projects was reduced to minutes.

Analysis of Alternative Approaches

Before development, we evaluated existing methods. None fully covered the task without compromises.

| Approach | Pros | Cons |

|--------|-------|--------|

| GitLab UI | Quick to start | Manual for 100+ projects, weak on configs |

| Local grep/ripgrep | Full control | Cloning all repositories takes hours |

| GitLab API Search | Available remotely | Misses YAML/JSON/env |

| Sourcegraph | Powerful indexing | Requires infrastructure deployment |

Hybrid crawler proved optimal: leverages API strengths and adds traversal for weak spots without external dependencies.

Differences in Search by File Types

GitLab reliably indexes code and documentation, but configs often remain outside search.

• Code (.py, .js, .go, .ts): API Search — indexing covers fully.
• Documentation (.md, .txt): API Search — sufficient speed and accuracy.
• Configs (.yml, .yaml, .json, .env): Deep Search — recursive tree + content fetch.
• Binary files: Excluded to save resources.

Logic: delegate tasks to the tool for strong types, manually handle weak ones.

Script Architecture

The script follows a pipeline:

• Get list of projects in a group with subgroups.
• Filter out archived and inactive ones.
• Parallel processing: API Search + Deep Search.
• Aggregate results.
• Generate reports.

GitLab Group → projects.list() → filter active
↓
Parallel: API Search (code) + Deep Search (configs)
↓
Merge → Detailed report + Summary + Errors

Search is limited to the master branch for current configs. Extending to all branches is possible but increases time.

Implementation of Key Components

Getting Projects

group = gl.groups.get(GROUP_ID)
all_projects = group.projects.list(include_subgroups=True, all=True)

Filter by status archived=false and last_activity.

API Search for Code

def api_search(gl, project_id, search_terms):
    results = {term: [] for term in search_terms}
    for term in search_terms:
        blobs = gl.search('blobs', term, project_id=project_id)
        for blob in blobs:
            file_path = blob.get('path', '')
            file_ext = os.path.splitext(file_path)[1].lower()
            if file_ext in CODE_EXTENSIONS or file_ext == '':
                results[term].append({
                    'path': file_path,
                    'url': blob.get('web_url', '#'),
                    'found_by': 'API'
                })
    return results

Processes blobs by code extensions.

Deep Search for Configs

def deep_search_configs(gl, project_id, search_terms):
    results = {term: [] for term in search_terms}
    project = gl.projects.get(project_id)
    files = project.repository_tree(recursive=True, ref='master')
    for file in files:
        if file['type'] != 'blob':
            continue
        ext = os.path.splitext(file['name'])[1].lower()
        if ext not in CONFIG_EXTENSIONS:
            continue
        content = project.files.get(file_path=file['path'], ref='master').decode()
        content_lower = content.lower()
        for term in search_terms:
            if term.lower() in content_lower:
                results[term].append({
                    'path': file['path'],
                    'found_by': 'Deep'
                })
    return results

Recursive tree, fetch content, string search.

Parallelization of Processing

ThreadPoolExecutor with 3 workers speeds up for 100+ projects:

with ThreadPoolExecutor(max_workers=3) as executor:
    futures = {executor.submit(process_one_project, gl, p): p for p in active_projects}
    for future in as_completed(futures):
        result = future.result()
        results_list.append(result)

Reduces time from hours to minutes without GIL issues in IO-bound tasks.

Report Formats

Detailed: Project, path, term, line, search method.

Example:

PROJECT: service-a
Path: backend/service-a
'SERVICE_EXAMPLE': 2 occurrences
  - deploy/prod/values.yml:42 [Deep]
    Line 42: SERVICE_EXAMPLE: "{{ .Values.secrets.apiKey }}"

Summary: Number of projects, occurrences, API vs Deep.

Speed Factors

• Excluding unnecessary files/projects.
• API for 80% of cases (code).
• Parallelism.
• Focus on master.

Limitations

• Master only.
• No regex.
• No deduplication.
• Console output.

Key Points

• Hybrid minimizes GitLab Search weaknesses for configs.
• Parallelism with 3 threads is optimal for API rate limits.
• Project filtering reduces load by 30–50%.
• Deep Search is accurate for YAML/JSON/env.
• Suitable for one-off checks without infrastructure.

— Editorial Team