Using variable names in nginx declarations has a price: e.g. ssl_certificate /usr/local/etc/ssl/${server_name}.fullchain.cer;

Leave a Comment / FreeBSD, nginx, Open Source / By /

I recently implemented a fun (to me) and easy solution for taking my web proxy websites offline, either one-by-one, or all-at-once. Today’s post talks about some of the repercussions which followed one-new-thing I tried.

In this post:

  • FreeBSD 15.0
  • nginx 1.28.2
  • I jump between testing the test host and stage host; both had similar issues.

The relevant changes

This is the type of change I started to do. Instead of putting the hostname in the log or certificate declarations, I started using an Nginx variable. Here is a psudeo-diff, based on an ansible template.

-  ssl_certificate_key /usr/local/etc/ssl/{{ item.value.website }}.key;
+  ssl_certificate_key /usr/local/etc/ssl/${server_name}.key;

Once on disk, the actual nginx configuration file changed like this. I’ll do this as a before and after:

This is a snippet of the nginx configuration for the proxy host which sits in front of dev.freshports.org:

...
  server_name dev.freshports.org;
...
  # the logs
  error_log  /var/log/nginx/dev.freshports.org.error.log  info;
  access_log /var/log/nginx/dev.freshports.org.access.log combined;

  # the certificate and key specific to this host.
  ssl_certificate     /usr/local/etc/ssl/dev.freshports.org.fullchain.cer;
  ssl_certificate_key /usr/local/etc/ssl/dev.freshports.org.key;

As you can see, the name dev.freshports.org occurs 5 times. The clever idea I had: don’t use the same hard-coded text N times. Declare it once. Use the variable. This is especially applicable to this situation because this file is actually derived from a template and is used for other websites as well. Why not? It might work. Let’s try it.

So we then had this configuration:

...
  server_name dev.freshports.org;
...
  error_log  /var/log/nginx/${server_name}.error.log  info;
  access_log /var/log/nginx/${server_name}.access.log combined;

  ssl_certificate     /usr/local/etc/ssl/${server_name}.fullchain.cer;
  ssl_certificate_key /usr/local/etc/ssl/${server_name}.key;

The service nginx configtest past, the restart worked. Off we go!

Some time later…

I did some testing, taking the websites offline, etc. All that worked fine.

The next day, I noticed several issues in monitoring: “(Return code of 139 is out of bounds)”

I tried testing (on my phone, with WIFI off, so that I hit the proxy and didn’t bypass it by being on my local network), I would get

Safari can’t open the page because it couldn’t establish a secure connection to the server.

logcheck was also spitting out this type of message:

Mar 12 20:03:46 nagios04 kernel: pid 9423 (check_http), jid 0, uid 181: exited on signal 11 (no core dump - other error)
Mar 12 20:03:47 nagios04 kernel: pid 9425 (check_http), jid 0, uid 181: exited on signal 11 (no core dump - other error)
Mar 12 20:03:51 nagios04 kernel: pid 9427 (check_http), jid 0, uid 181: exited on signal 11 (no core dump - other error)

That prompted me to get onto nagios04 and see what was going on.

Trying nagios checks by hand

I wanted to see what Nagios was seeing, so I tried this:

[19:45 nagios04 dvl /usr/local/etc/nagiosql] % /usr/local/libexec/nagios/check_http -H stage.freshports.org -4
HTTP OK: HTTP/1.1 200 OK - 250560 bytes in 0.795 second response time |time=0.795270s;;;0.000000 size=250560B;;;0

Meaning http was good. Let’s try https:

[19:45 nagios04 dvl /usr/local/etc/nagiosql] % /usr/local/libexec/nagios/check_http -H stage.freshports.org -S -p 443 -C 25 -4 --sni 
CRITICAL - Cannot make SSL connection.
10300596361A0000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal 
error:/usr/src/crypto/openssl/ssl/record/rec_layer_s3.c:916:SSL alert number 80
zsh: segmentation fault  /usr/local/libexec/nagios/check_http -H stage.freshports.org -S -p 443 -C 25

Ouch. Searching for that SSL alert number 80 did not help. In hindsight, I should have paid more attention to the CRITICAL – Cannot make SSL connection part of the output.

There’s that alert number 80 again.

I kept mucking about with that for a while. And didn’t really get anywhere.

Let’s try more non-nagios stuff on the command line.

Other command line explorations

Hmm. Let’s go to wget:

[11:38 nagios04 dvl ~/tmp] % wget -S test.freshports.org        
Prepended http:// to 'test.freshports.org'
--2026-03-13 11:38:22--  http://test.freshports.org/
Resolving test.freshports.org (test.freshports.org)... 173.228.145.171, 2610:1c0:2000:11:8870:201b:27b5:f4f2
Connecting to test.freshports.org (test.freshports.org)|173.228.145.171|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx/1.28.2
  Date: Fri, 13 Mar 2026 11:38:23 GMT
  Content-Type: text/html; charset=UTF-8
  Transfer-Encoding: chunked
  Connection: keep-alive
  Vary: Accept-Encoding
  X-Powered-By: PHP/8.3.30
  Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
  X-Frame-Options: DENY
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                [   <=>                                                                  ] 285.58K   496KB/s    in 0.6s    

2026-03-13 11:38:23 (496 KB/s) - ‘index.html’ saved [292431]

All good. Next, try https:

[11:38 nagios04 dvl ~/tmp] % wget -S https://test.freshports.org
--2026-03-13 11:41:27--  https://test.freshports.org/
Resolving test.freshports.org (test.freshports.org)... 173.228.145.171, 2610:1c0:2000:11:8870:201b:27b5:f4f2
Connecting to test.freshports.org (test.freshports.org)|173.228.145.171|:443... connected.
OpenSSL: error:0A000438:SSL routines::tlsv1 alert internal error
Unable to establish SSL connection.

tlsv1? Am I using that on my proxy?

ssl_protocols TLSv1.2 TLSv1.1 TLSv1;

Well, yes, I am. I changed that to:

  ssl_protocols TLSv1.2 TLSv1.3;

Then restarted nginx.

No difference. Am I using TLSv1 on the real server, the one behind the proxy? That wouldn’t matter because wget is not talking to that.

So WTF is going on?

openssl time:

[20:27 nagios04 dvl ~/tmp] % openssl s_client -connect stage.freshports.org:443 -servername stage.freshports.org -4 -showcerts
Connecting to 173.228.145.171
CONNECTED(00000003)
1090C55AFA340000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:/usr/src/crypto/openssl/ssl/record/rec_layer_s3.c:916:SSL alert number 80
---
no peer certificate available
---
No client certificate CA names sent
Negotiated TLS1.3 group: 
---
SSL handshake has read 7 bytes and written 1553 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Protocol: TLSv1.3
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Again, I missed clues that might have led me to the cause sooner.

Always monitor the logs

I should have done this sooner. I should have checked the logs. Looking at this, I saw nothing. Nothing. At All.

Now I changed to using the test host.

[11:46 r720-02-proxy01 dvl ~] % xtail /var/log/nginx/test.freshports.org.*

I keep fetching pages, but nothing appeared in that log.

I went a little wider on my search:

[11:48 r720-02-proxy01 dvl ~] % xtail /var/log/nginx/
*** /var/log/nginx//${server_name}.error.log ***
2026/03/13 11:51:14 [error] 18612#101892: *469 cannot load certificate key "/usr/local/etc/ssl/test.freshports.org.key": BIO_new_file() failed (SSL: error:8000000D:system library::Permission denied:calling fopen(/usr/local/etc/ssl/test.freshports.org.key, r) error:10080002:BIO routines::system lib) while SSL handshaking, client: 203.0.113.46, server: 173.228.145.171:443
  1. that’s not writing to the expected file
  2. permission issues?
[11:51 r720-02-proxy01 dvl ~] % ls -l /usr/local/etc/ssl/test.freshports.org.key
-rw-------  1 root wheel 1679 2018.09.18 13:06 /usr/local/etc/ssl/test.freshports.org.key

nginx has always been able to read this before. Let’s try this daring change:

[11:52 r720-02-proxy01 dvl /usr/local/etc/ssl] % sudo chgrp www test.freshports.org.key                   
[11:52 r720-02-proxy01 dvl /usr/local/etc/ssl] % sudo chmod g=r test.freshports.org.key
[11:52 r720-02-proxy01 dvl /usr/local/etc/ssl] % ls -l test.freshports.org.key
-rw-r-----  1 root www 1679 2018.09.18 13:06 test.freshports.org.key

Now nginx should be able to read that key.

Testing again:

[11:53 nagios04 dvl ~/tmp] % wget -S https://test.freshports.org/
--2026-03-13 11:54:25--  https://test.freshports.org/
Resolving test.freshports.org (test.freshports.org)... 173.228.145.171, 2610:1c0:2000:11:8870:201b:27b5:f4f2
Connecting to test.freshports.org (test.freshports.org)|173.228.145.171|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx/1.28.2
  Date: Fri, 13 Mar 2026 11:54:26 GMT
  Content-Type: text/html; charset=UTF-8
  Transfer-Encoding: chunked
  Connection: keep-alive
  Vary: Accept-Encoding
  X-Powered-By: PHP/8.3.30
  Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
  X-Frame-Options: DENY
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
Length: unspecified [text/html]
Saving to: ‘index.html.1’

index.html.1                              [   <=>                                                                  ] 285.58K   629KB/s    in 0.5s    

2026-03-13 11:54:26 (629 KB/s) - ‘index.html.1’ saved [292431]

[11:54 nagios04 dvl ~/tmp] %

So that problem is “fixed”.

Let’s check with openssl:

[20:27 nagios04 dvl ~/tmp] % openssl s_client -connect stage.freshports.org:443 -servername stage.freshports.org -4 -showcerts
Connecting to 173.228.145.171
CONNECTED(00000003)
depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=R13
verify return:1
depth=0 CN=stage.freshports.org
verify return:1
---
Certificate chain
 0 s:CN=stage.freshports.org
   i:C=US, O=Let's Encrypt, CN=R13
   a:PKEY: RSA, 2048 (bit); sigalg: sha256WithRSAEncryption
   v:NotBefore: Jan 31 17:35:14 2026 GMT; NotAfter: May  1 17:35:13 2026 GMT
-----BEGIN CERTIFICATE-----
MIIFDDCCA/SgAwIBAgISBjTyQhJ1Qw1yypiKRRJGZuiMMA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTMwHhcNMjYwMTMxMTczNTE0WhcNMjYwNTAxMTczNTEzWjAfMR0wGwYDVQQD
ExRzdGFnZS5mcmVzaHBvcnRzLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBANGjB5rJS9pilMQ2sHJ/v8GZ90tFcaUpP0QTnxWvXrrtyg22AR3ODR65
EGvEbSEBAFVsoMR7thSZL1dZ5D7x3AvB+OjLVnK8KGQ/kuR85qSmNq+a/GxvcNKA
HG5xt9aIGZpii+XKOyhlPMbVHFEeVuh1d+oeTdb0+/ul49rLMtXBA3/FlOXkp46w
3kZLH8tTWrj6kXMqiUKPwhuex50tQHuW//WYqXACjXT2EN1ct105ng6WJkH1FuoB
R0s+EaeArdHQzz/bX4ijJCK/RWaoRrOe2fW69RYVmHT347CXqu4TrASffo9ubq4R
ViRkoSOHUs1zjGHO2WE2w+26ShTIeE8CAwEAAaOCAiwwggIoMA4GA1UdDwEB/wQE
AwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw
ADAdBgNVHQ4EFgQUEDI7MQFLUoqoRUzvhEBy0lhyKCAwHwYDVR0jBBgwFoAU56uf
DywzoFPTXk94yLKEDjvWkjMwMwYIKwYBBQUHAQEEJzAlMCMGCCsGAQUFBzAChhdo
dHRwOi8vcjEzLmkubGVuY3Iub3JnLzAfBgNVHREEGDAWghRzdGFnZS5mcmVzaHBv
cnRzLm9yZzATBgNVHSAEDDAKMAgGBmeBDAECATAuBgNVHR8EJzAlMCOgIaAfhh1o
dHRwOi8vcjEzLmMubGVuY3Iub3JnLzEzLmNybDCCAQwGCisGAQQB1nkCBAIEgf0E
gfoA+AB1AJaXZL9VWJet90OHaDcIQnfp8DrV9qTzNm5GpD8PyqnGAAABnBVVG3AA
AAQDAEYwRAIgWEAPKdcqhnu1euGcusHalXmiffO0mcXgSLMHLpM/zi4CIBzJkQ3D
1P1CGplj9/9Gedi4RLPg4hRB19HqqcLM6b5MAH8Apcl4kl1XRheChw3YiWYLXFVk
i30AQPLsB2hR0YhpGfcAAAGcFVUmNwAIAAAFADC1Xt8EAwBIMEYCIQDdemoylksD
1x9+S0e/H17rnEP8ZjDbHuKq21N6VprhrQIhAPNlewYay/bOOZF2ncgX5S7wKvQO
nelQosbZ8N5E7o3hMA0GCSqGSIb3DQEBCwUAA4IBAQCLEyqL71emur08DF6IrFZU
iF+ahfVV/PkVh/6pBtu9wCF/gI1j7R7un5tpZQCnrUfOU6L6/DqRfyxe5PW8BGGH
aTt5MFn7VDZ4lqwngSFFKochXf4GMVeADUBUGMyLSgsOZLxQrMm6+PRNS6u21aBK
OsuWsqC6hKtvZyiXALm1wj4vQLl9lshoB2Y3fEqFKZNJKj9fmA3g2a6sDrjHc9Ny
OkiaWker09nJWfAEdIidchu/2m16201CWpXN7wf3ZXpJNMKBqkn0+mc4XWVTRl8t
oVm+gNWfmAwousv9OWVF+PURt5HOYXYF8cBEZ+kSk41FiizSnksoHf4Qu8WK/NT+
-----END CERTIFICATE-----
 1 s:C=US, O=Let's Encrypt, CN=R13
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
   a:PKEY: RSA, 2048 (bit); sigalg: sha256WithRSAEncryption
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
-----BEGIN CERTIFICATE-----
MIIFBTCCAu2gAwIBAgIQWgDyEtjUtIDzkkFX6imDBTANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFy
Y2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBYMTAeFw0yNDAzMTMwMDAwMDBa
Fw0yNzAzMTIyMzU5NTlaMDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBF
bmNyeXB0MQwwCgYDVQQDEwNSMTMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQClZ3CN0FaBZBUXYc25BtStGZCMJlA3mBZjklTb2cyEBZPs0+wIG6BgUUNI
fSvHSJaetC3ancgnO1ehn6vw1g7UDjDKb5ux0daknTI+WE41b0VYaHEX/D7YXYKg
L7JRbLAaXbhZzjVlyIuhrxA3/+OcXcJJFzT/jCuLjfC8cSyTDB0FxLrHzarJXnzR
yQH3nAP2/Apd9Np75tt2QnDr9E0i2gB3b9bJXxf92nUupVcM9upctuBzpWjPoXTi
dYJ+EJ/B9aLrAek4sQpEzNPCifVJNYIKNLMc6YjCR06CDgo28EdPivEpBHXazeGa
XP9enZiVuppD0EqiFwUBBDDTMrOPAgMBAAGjgfgwgfUwDgYDVR0PAQH/BAQDAgGG
MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATASBgNVHRMBAf8ECDAGAQH/
AgEAMB0GA1UdDgQWBBTnq58PLDOgU9NeT3jIsoQOO9aSMzAfBgNVHSMEGDAWgBR5
tFnme7bl5AFzgAiIyBpY9umbbjAyBggrBgEFBQcBAQQmMCQwIgYIKwYBBQUHMAKG
Fmh0dHA6Ly94MS5pLmxlbmNyLm9yZy8wEwYDVR0gBAwwCjAIBgZngQwBAgEwJwYD
VR0fBCAwHjAcoBqgGIYWaHR0cDovL3gxLmMubGVuY3Iub3JnLzANBgkqhkiG9w0B
AQsFAAOCAgEAUTdYUqEimzW7TbrOypLqCfL7VOwYf/Q79OH5cHLCZeggfQhDconl
k7Kgh8b0vi+/XuWu7CN8n/UPeg1vo3G+taXirrytthQinAHGwc/UdbOygJa9zuBc
VyqoH3CXTXDInT+8a+c3aEVMJ2St+pSn4ed+WkDp8ijsijvEyFwE47hulW0Ltzjg
9fOV5Pmrg/zxWbRuL+k0DBDHEJennCsAen7c35Pmx7jpmJ/HtgRhcnz0yjSBvyIw
6L1QIupkCv2SBODT/xDD3gfQQyKv6roV4G2EhfEyAsWpmojxjCUCGiyg97FvDtm/
NK2LSc9lybKxB73I2+P2G3CaWpvvpAiHCVu30jW8GCxKdfhsXtnIy2imskQqVZ2m
0Pmxobb28Tucr7xBK7CtwvPrb79os7u2XP3O5f9b/H66GNyRrglRXlrYjI1oGYL/
f4I1n/Sgusda6WvA6C190kxjU15Y12mHU4+BxyR9cx2hhGS9fAjMZKJss28qxvz6
Axu4CaDmRNZpK/pQrXF17yXCXkmEWgvSOEZy6Z9pcbLIVEGckV/iVeq0AOo2pkg9
p4QRIy0tK2diRENLSF2KysFwbY6B26BFeFs3v1sYVRhFW9nLkOrQVporCS0KyZmf
wVD89qSTlnctLcZnIavjKsKUu1nA1iU0yYMdYepKR7lWbnwhdx3ewok=
-----END CERTIFICATE-----
---
Server certificate
subject=CN=stage.freshports.org
issuer=C=US, O=Let's Encrypt, CN=R13
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: rsa_pss_rsae_sha256
Peer Temp Key: ECDH, secp384r1, 384 bits
---
SSL handshake has read 3135 bytes and written 1711 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Protocol: TLSv1.2
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 4BB275378AD476904EA073B09E7450AB781E2AE79FEA3FEBD8CC1FC96FEA4113
    Session-ID-ctx: 
    Master-Key: 1A677D11EDD6F41A39FB7138B9C53708DA9A57AA4A5B54EF705EC0B77952AC44B41793F4D128A7A754DFA998411A5935
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1773347304
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

Yes, that is greatly improved.

But wait, the logs

That wget above created this log entry (slightly reformatted for easier reading):

2026/03/13 11:54:26 [crit] 18611#133911: *681 open() "/var/log/nginx/test.freshports.org.access.log" failed (13: Permission denied) while logging 
request, client: 203.0.113.46, server: test.freshports.org, request: "GET / HTTP/1.1", upstream: "https://10.55.0.42:443/", host: 
"test.freshports.org"

The next issue: nginx cannot write to the logs. Like the certificate key, this was not a problem before the configuration change. These are the log files:

[12:00 r720-02-proxy01 dvl ~] % ls -l /var/log/nginx/test.freshports.org.*.log
-rw-r--r--  1 root wheel 12386576 2026.03.12 14:03 /var/log/nginx/test.freshports.org.access.log
-rw-r--r--  1 root wheel  5211625 2026.03.12 14:03 /var/log/nginx/test.freshports.org.error.log

Let’s this this “solution”:

[12:00 r720-02-proxy01 dvl /var/log/nginx] % sudo chgrp www test.freshports.org.access.log test.freshports.org.error.log
[12:00 r720-02-proxy01 dvl /var/log/nginx] % sudo chmod g=rw test.freshports.org.access.log test.freshports.org.error.log
[12:00 r720-02-proxy01 dvl /var/log/nginx] % ls -l test.freshports.org.access.log test.freshports.org.error.log
-rw-rw-r--  1 root www 12386576 2026.03.12 14:03 test.freshports.org.access.log
-rw-rw-r--  1 root www  5211625 2026.03.12 14:03 test.freshports.org.error.log

Those changes did not fix the logging problem. The errors persisted.

What did “fix” it was this:

[12:07 r720-02-proxy01 dvl ~] % ls -ld /var/log/nginx
drwxr-x---  2 root wheel 259 2026.03.11 16:29 /var/log/nginx/
[12:07 r720-02-proxy01 dvl ~] % sudo chgrp www /var/log/nginx
[12:07 r720-02-proxy01 dvl ~] % ls -ld /var/log/nginx        
drwxr-x---  2 root www 259 2026.03.11 16:29 /var/log/nginx/
[12:07 r720-02-proxy01 dvl ~] %

However, that “broke” my personal log monitoring because I was no longer able to view what was in there (because of the group change from wheel to www above):

[12:08 r720-02-proxy01 dvl /var/log/nginx] % xtail /var/log/nginx//test.freshports.org.*.log     
zsh: no matches found: /var/log/nginx//test.freshports.org.*.log

I could do this:

[12:09 r720-02-proxy01 dvl /var/log/nginx] % sudo xtail /var/log/nginx/test.freshports.org.{access,error}.log 
203.0.113.46 - - [13/Mar/2026:12:10:32 +0000] "GET / HTTP/1.1" 200 292547 "-" "Wget/1.25.0"

My theory on why this happens

I think the .key file problems are related to the dropping of priviledges by nginx. In my previous configuration (before I used $server_name), nginx starts as root, gets access to the file, then drops to the user www.

So why is this a problem when using ${server_name} in the path to the files? My theory is the directive is evaluated when the path is needed, not at startup. By that time, the process no longer has access to that file, because it is running as www.

I also suspect the log file problems are similar in nature.

In short, I think it is evaluated on every request, not at startup. That is why the failures occur.

Changes for good

I have reversed the chmod and chgrp changes I made above when trying to “fix” the problems.

I have reverted my configuration files to not use ${server_name}.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Related Posts

Leave a Comment

Scroll to Top