DNS Docker in GitLab CI: MAX notifications fix

Docker DNS Broke GitLab CI MAX Notifications: Debug and Fix

A GitLab CI job meant to send a notification to MAX Messenger after deployment turned into a week-long debugging nightmare. Requests were sent, but responses came back with {"code":"proto.payload","message":"Can't deserialize body"}. Logs pointed to a request body issue, so we zeroed in on crafting the JSON payload.

Even after simplifying, nothing worked. Environment variables MAX_BOT_TOKEN and MAX_NOTIFY_CHAT_ID were set, and curl was in the image. But messages never arrived.

Failed Payload Attempts

The team went through several iterations:

• printf to file: printf '{"text":"%s"}' "$MSG" > /tmp/payload.json; curl ... -d @/tmp/payload.json. Response: Can't deserialize body.

• jq in Alpine: Switched to alpine:3.20 with apk add jq. jq -nc --arg t "$MSG" '{text:$t}' > /tmp/payload.json. Same response.

• jq to variable: PAYLOAD=$(jq ...); curl ... --data "${PAYLOAD}". Now Empty request body, plus apk add failed with exit code 2 due to unreachable repos.

• Hardcoded: curl ... -d '{"text":"test ok"}'. Again Empty request body.

Progress was minimal: the request body consistently failed to reach the server.

Key Test: Host vs Container

A side-by-side comparison revealed the culprit:

From the host:

ssh ... "curl -s -X POST 'https://platform-api.max.ru/messages?chat_id=...' -H 'Authorization: ...' -H 'Content-Type: application/json' -d '{"text":"test from server"}'"

HTTP 200, message delivered.

From Docker:

docker run --rm curlimages/curl:8.7.1 curl -s -X POST 'https://platform-api.max.ru/messages?chat_id=...' -H 'Authorization: ...' -d '{"text":"test from docker"}'

exit code 6: Could not resolve host: platform-api.max.ru.

Diagnosis: DNS resolution failed inside Docker containers, even though the host resolved fine.

Testing with --dns 8.8.8.8 confirmed it: the message sent instantly.

Why DNS Masqueraded as a JSON Error

We expected a DNS error like curl exit code 6 with no HTTP response. Instead, MAX returned JSON with proto.payload. The issue stemmed from GitLab runner's custom DNS setup for job containers. This caused unstable connections—request bodies got truncated or dropped entirely. MAX saw the empty input as a deserialization error.

Fix: DNS in Runner's config.toml

Solution: Add DNS servers to /etc/gitlab-runner/config.toml:

[[runners]]
  ...
  [runners.docker]
    dns = ["8.8.8.8", "8.8.4.4"]

Restart: sudo gitlab-runner restart.

Docker daemon alternative in /etc/docker/daemon.json:

{
  "dns": ["8.8.8.8", "8.8.4.4"]
}

sudo systemctl restart docker.

After the fix, the hardcoded payload worked on the first run.

Working Script Template

Final script with diagnostics and variable cleanup:

if [ -z "$MAX_BOT_TOKEN" ] || [ -z "$MAX_NOTIFY_CHAT_ID" ]; then
  echo "WARN: MAX_BOT_TOKEN or MAX_NOTIFY_CHAT_ID not set"
  exit 0
fi

TOKEN="$(printf '%s' "$MAX_BOT_TOKEN" | tr -d '\r\n')"
CHAT_ID="$(printf '%s' "$MAX_NOTIFY_CHAT_ID" | tr -d '\r\n[:space:]')"
BODY_FILE="$(mktemp)"
RESPONSE_FILE="$(mktemp)"

printf '{"text":"%s"}' "${MSG}" > "${BODY_FILE}"

BODY_SIZE="$(wc -c < "${BODY_FILE}" | tr -d '[:space:]')"
echo "DEBUG: CHAT_ID=${CHAT_ID}"
echo "DEBUG: BODY_SIZE=${BODY_SIZE}"
echo "DEBUG: BODY_TEXT=$(cat "${BODY_FILE}")"

CURL_STATUS=0
HTTP_CODE=$(curl -sS --max-time 15 \
  -o "${RESPONSE_FILE}" \
  -w "%{http_code}" \
  -X POST "https://platform-api.max.ru/messages?chat_id=${CHAT_ID}" \
  -H "Authorization: ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @"${BODY_FILE}") || CURL_STATUS=$?

RESPONSE="$(cat "${RESPONSE_FILE}" 2>/dev/null || true)"
rm -f "${BODY_FILE}" "${RESPONSE_FILE}"

echo "DEBUG: HTTP_CODE=${HTTP_CODE} CURL_STATUS=${CURL_STATUS}"
echo "DEBUG: Response: ${RESPONSE}"

if [ "${CURL_STATUS}" -ne 0 ] || [ "${HTTP_CODE}" != "200" ]; then
  echo "ERROR: Failed to send. Response: ${RESPONSE}"
  exit 1
fi

Notification Format and Data Extraction

Notifications pull from GitLab CI variables:

• CI_PROJECT_NAME — project
• CI_COMMIT_BRANCH — branch
• CI_COMMIT_SHORT_SHA — commit hash
• CI_COMMIT_TITLE — commit message (escaped with sed 's/\/\\/g; s/\"\"/g')
• CI_COMMIT_AUTHOR — author
• CI_PIPELINE_URL — pipeline link

Example:

✅ Successful DEV deploy

📦 Project: u.clinic
🌍 Environment: DEV
🌿 Branch: dev
🔖 Version: f7a249d0
💬 Commit: fix(ci): improve MAX notification handling
👤 Author: Anton Kurganskii
⏰ Time (UTC): 2026-04-08 21:29:07
⏱️ Pipeline duration: 9 min 52 sec

🔗 Pipeline: https://git.tverdasoft.ru/...

✓ Deploy completed successfully

Image Choice and Alternatives

curlimages/curl:8.7.1 beats alpine + jq: no network needed for packages, runs from cache with pull_policy: if-not-present. Alpine's apk add failed due to unreachable repos.

Key Takeaways

• Docker DNS failures in GitLab runners mimic payload errors from unstable connections.
• Test first: docker run curl ... platform-api.max.ru on the runner host—exit code 6 flags DNS.
• Fix: dns = ["8.8.8.8", "8.8.4.4"] in [runners.docker] of config.toml.
• Debug: Log BODY_SIZE, HTTP_CODE, full Response.
• For commit messages: tr -d '\r\n' + sed escaping.

— Editorial Team