# curl on old server doesn't like the new Let's Encrypt Root, it seems



## rainer_d (Oct 8, 2021)

I tried to update the /usr/local/share/certs/ca-root-nss.crt file by simply copying it from a different server - but curl still complains about the certificate of the other server being invalid.

What else do I have to do?



```
(server </root>) 0 # curl -v -o /dev/null https://browsershot.shinypixel.ch/shot1   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 88.99.94.154...
* TCP_NODELAY set
* Connected to browsershot.shinypixel.ch (88.99.94.154) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
  CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4042 bytes data]
* TLSv1.2 (OUT), TLS alert, Server hello (2):
} [2 bytes data]
* SSL certificate problem: certificate has expired
* stopped the pause stream!
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```


----------



## VladiBG (Oct 8, 2021)

curl
					






					www.freebsd.org
				



try again with "--cacert    <file>"


----------



## rainer_d (Oct 8, 2021)

Hi,

this does not work either.

The problem is, the problem really exists within a PHP app.

So, I assume the certs are somewhere else, too?


----------



## Jose (Oct 8, 2021)

rainer_d said:


> Hi,
> 
> this does not work either.
> 
> ...


Did these problems start on September 30th?





						DST Root CA X3 Expiration (September 2021) -  Let's Encrypt
					

Update September 30, 2021 As planned, the DST Root CA X3 cross-sign has expired, and we’re now using our own ISRG Root X1 for trust on almost all devices. For more details about the plan, keep reading! We have also updated our Production Chain Changes thread on our community forum - our team and...



					letsencrypt.org


----------



## mer (Oct 8, 2021)

rainer_d said:


> SSL certificate problem: certificate has expired


Typically this means that a certificate is past it's expiration date.
Just like you won't drink milk past the expiration date, SSL will not use a certificate that is past it's due date.
The cert needs to be renewed.


----------



## rainer_d (Oct 8, 2021)

It works on a newer server (and on my Mac).

ssllabs also says it's OK.

I assume it's the new LE intermediate.

But why?


----------



## Jose (Oct 8, 2021)

X.509 certificate chain validation is strange and complex.





						BearSSL - X.509 Certificates
					






					bearssl.org
				




I found a great discussion once on all the various ways in which the major and minor SSL/TLS implementations fail to implement it correctly, but I can't find it now.

TL;DR Openssl 1.0x. has a bug:





						Old Let's Encrypt Root Certificate Expiration and OpenSSL 1.0.2 - OpenSSL Blog
					

The currently recommended certificate chain as presented to Let’s Encrypt ACME clients when new certificates are issued contains an intermediate …




					www.openssl.org


----------



## rainer_d (Oct 8, 2021)

OK, thanks.


But as I said, I replaced the ca-root-nss.crt file - and there is a symlink in /etc/ssl to that replace file.


----------



## obsigna (Oct 8, 2021)

rainer_d said:


> It works on a newer server (and on my Mac).
> 
> ssllabs also says it's OK.
> 
> ...


Check you version of OpenSSL.

`openssl version`
Versions up to 1.0.2 are inkompatible with the cross-signing method which LE used to overcome the expiration of the X3 certificate.





						DST Root CA X3 Expiration (September 2021) -  Let's Encrypt
					

Update September 30, 2021 As planned, the DST Root CA X3 cross-sign has expired, and we’re now using our own ISRG Root X1 for trust on almost all devices. For more details about the plan, keep reading! We have also updated our Production Chain Changes thread on our community forum - our team and...



					letsencrypt.org
				




The following tells the story, but is not an option for you since you are not at control of the respective certificates:





						Old Let's Encrypt Root Certificate Expiration and OpenSSL 1.0.2 - OpenSSL Blog
					

The currently recommended certificate chain as presented to Let’s Encrypt ACME clients when new certificates are issued contains an intermediate …




					www.openssl.org
				




Your best option is to update your system to 13.0-RELEASE, which comes with OpenSSL v1.1.1k which should work without problems.


----------



## rainer_d (Oct 8, 2021)

It's 
OpenSSL 1.0.2k-freebsd  26 Jan 2017

Unfortunately, updating the server (or rather servers - I have at least another one of those) would result in said PHP apps to stop functioning.


But thanks for the background-info.


Best Regards
Rainer


----------



## covacat (Oct 8, 2021)

maybe this can help





						FreeBSD Mail Archives
					






					docs.freebsd.org


----------



## rainer_d (Oct 8, 2021)

As I said, I already replaced 

/usr/local/share/certs/ca-root-nss.crt with the one from a 12.2-p-something server

and it does not help.

So I wonder: what else is there?

If I understood the above links, it should be enough to replace the certs?


----------



## obsigna (Oct 8, 2021)

rainer_d said:


> If I understood the above links, it should be enough to replace the certs?


Yes, you could ask the owner of browsershot.shinypixel.ch to replace her/his certs.

You could also install OpenSSL v1.1.1 from the ports and build PHP against OpenSSL from the ports.

Besides updating your servers, here ends already the list of your options.

PS: Sorry there is just another option. Wait some X to 82 days, since within this period all LE certs need to be replaced anyway.


----------



## drhowarddrfine (Oct 8, 2021)

Let's Encrypt Root Expiration - Post-Mortem
					

Well, the Internet Apocalypse came and went! Due to the recent expiration of the Let's Encrypt intermediate and root certificates, I saw more widespread issues than I was expecting, but on different devices and for different reasons than I thought. Let's take a look at what happened and why.    All




					scotthelme.co.uk


----------



## covacat (Oct 8, 2021)

the link i posted has patch against base openssl source
you can rebuild and see if it works
if you have source tree installed it takes 2 minutes


----------



## Jose (Oct 9, 2021)

rainer_d said:


> As I said, I already replaced
> 
> /usr/local/share/certs/ca-root-nss.crt with the one from a 12.2-p-something server
> 
> ...


You didn't understand the above links. The problem is not in the trust stores. Indeed the Openssl post even says "Most up-to-date CA cert trusted bundles, as provided by operating systems, contain this soon-to-be-expired certificate." Moving files around won't do anything. Even the most up-to-date files have the problematic certificate in them.

The problem is in the code that builds a chain of trust based on these files. The old Openssl code (and most other implementations) don't follow the RFCs when they build this chain of trust, and so wind up going down the path that has the expired certificate erroneously. There's a way to force Openssl to do the right thing. Read the "Workaround 2" section of the Openssl blog post. Unfortunately that probably won't work in your case, since PHP is probably linking against Openssl and not launching a command in a script. You can achieve workaround 2 in PHP by applying the patch that Covacat linked.

The "understanding the problem" section of this Medium post has a diagram that may help understand things:








						Path Building vs Path Verifying: The Chain of Pain
					

Disclaimer: This post represents personal opinions and thoughts, and does not represent the views or positions of my employer, Google.




					medium.com
				




Again, it's not the certificate files, it's the code that reads them and tries to build a chain of trust based on their contents.


----------



## VladiBG (Oct 9, 2021)

@rainer_d​If you are using csh(1) shell show the output of:

`curl --version`
`curl -kvvI https://88.99.94.154 | & awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'`
`curl -kvvI --connect-to browsershot.shinypixel.ch:443 https://browsershot.shinypixel.ch |& awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'`
`openssl s_client -servername browsershot.shinypixel.ch -connect 88.99.94.154:443 | openssl x509 -noout -subject -issuer -dates`
`openssl s_client -connect 88.99.94.154:443 | openssl x509 -noout -subject -issuer -dates`

If you are using bash shell change the `| &` with `2>&1 |`


----------

