RANT! STOP with the SSL please!

[paulca]

This week my build environment blew up in work.  Eventually I traced it to an HTTP mirror in the build toolchain was being rejected just for being HTTP and not HTTPS.  Apparently a recent, contraversial update make HTTP mirrors auto reject.  I had to disable the functionality.

This has become more and more common with services coming enabled with mandatory SSL/TLS encryption.  HTTP options are very well deprecated.

There is at least one significant disadvantage of SSL HTTP requests and responses.  They cannot be cached.

HTTP mirrors go to my proxy and come back nearly instantly as they are already cached from the last 15 builds.  I have about 2 dozen virutal machines and linux containers all require the periodic "apt upgrade" run.  When I run the first one, the responses are all cached on the HTTP proxy.  Each subsequent machine takes from the cache and only dowloads "misses".  Thus while upgrading the first machine might take 5 minutes, updating the remainder takes half that.

HTTPS mirrors all go "DIRECT" through the proxy and have to take whatever speed that mirror is felling like today.  It seems that "most" HTTPS clients don't even bother with a "HEAD" request to check the timestamp, they just redownload it every single time.

The environmental aspects alone of the increased bandwidth, increased CPU power globally is actually fairly significant.  I don't have figures, but the last time I read an article trying to quantify it, it was a multiple of the cost of HTTP per transaction.  "cost" in terms of energy.

I fully understand the risks, I fully understand the benefits of encryption.  However I also understand that in development and on sandboxed networks, the SSL/HTTPS that will be set up, will almost certainly be only a few steps away from using "Snakeoil.crt" directly.   aka, completely insecure with the private keys in github.  So it's pointless anyway.  Provides no security what so ever, but costs a shed ton of time and effort tracing certificate issues when you just want to do some actual development.

Not everything needs to be HTTPS.  There are far better ways of securing software modules that has been in use for decades that does not require SSL/HTTPS.  In fact, I would suggest that 95% of software build environments would be 100% completely fooled by a MITM attack as nobody is really checking the certificate issuer and CA are valid.  If the download works, nobody checks anything.  So HTTPS alone will not protect you.

The other way is to issue signatures for your downloads and host them on alternative third party mirrors.  Thus, to actually "trojan" an artifact the bad actor needs to hack both the original download mirror, but also multiples of the indpendant signature servers.  Additionally those signatures can be counter signed by another indpendant body.  This is basically how the core linux packages are protected, like kernel.org.  You will find the tar.bz2 files on an HTTP mirror and on many addition mirrors the signature files.

In development SSL, security and encryption should be disable-able.  Developers should have the option to run the code "naked" and exposed so that debugging can occur un-impeded.  Additionally it provides not only a way to "unblock" yourself when the security layers are broken, out of sync or just wrong, but it also allows you to quickly verify if your current issue stems from a security issue or from your actual functional code.

/rant.

[eutectique]

No.

[Phil1977]

STOP with these LOCKS please!

Leave all doors open. Malfunctioning locks may trap you in a burning building.

If you need to build a door, make the lock disable-able so that the average user does not notice it, make it that the door closes as normal, but so that everyone can open it again without any key. Just for test purposes.

Yours sincerely, your local burglar

[magic]

I don't have locks between my rooms

[Algoma]

There are proxies present their own encryption certificate back to the client, provided that certificate is specified as trusted on your network. it allows the proxy to relay the traffic.

If you run some major antivirus softwares you'll see they do this internally.. if you inspect the certificate you're seeing from a site, you may see its actually signed by your local antivirus.

[paulca]

I don't have locks between my rooms

This ^^^

In todays enterprise software we don't write a big fat monolithical application.  This would be akin to a frontier barn where everyone literally sleeps under the same roof.

Today we split things up into dozens of compartmentalised units/components.  This is analogous to having many different rooms with different purposes and are nicely contained. 

They do need to interact though.

A modern T1 enterprise deployment would be like having a dozen rooms in your house and each and every door has an access control point on it.  You need a specific access token for ingress and egress.  Not only that, but to get out of the livingroom, you not only need "egress" permissions, but you also need to state your destination and authenticate for that too.  If it's SSL encrypted, then before you can even open the door, you must dress up in a bunny suit and don a blindfold before you step out into the hallway.  The toilet door on the other side will also have an access token of course with different certificate chains and a different colour of bunny suit.

The reasoning behind this is "least trust principle".  If your "platform" is comprised of 14 different components.  Anyone of those 14 components could become compromised. (Burglar in the living room).  So all 14 services have to consider all other 14 services, including itself as potentially compromised and thus they have to treat each other with extreme suspicion.

The reality of this is...

If you ask me for an estimate to deploy an application into a sandbox and have the security handled elsewhere I might give you N days.
If you ask me for an estimate to deploy an application into a high security environment like a bank, I will give you N*20 days.  I'm not joking.

[Marco]

There is at least one significant disadvantage of SSL HTTP requests and responses.  They cannot be cached.

They can, but you'll need a local certificate on the client so the proxy can authenticate itself as everything. Even then time to first byte will be worse.

DNS poisoning is still a realistic attack method, especially in an attack chain. How about submitting a patch with a flag for trusted proxies? (Maybe all proxies on local subnet?) The secure by default people are happy, you're happy, everybody is happy.

[paulca]

There are proxies present their own encryption certificate back to the client, provided that certificate is specified as trusted on your network. it allows the proxy to relay the traffic.

If you run some major antivirus softwares you'll see they do this internally.. if you inspect the certificate you're seeing from a site, you may see its actually signed by your local antivirus.

Unfortunately this technique is itself a security hack and it creates it's own issues.  The technique clearly demonstrates how, exactly SSL does NOT protect you from a MITM attack without proper vigilance.

It is also the cause of one of the most costly fuck ups in the global SSL authority chains in history.  When I believe Lenovo accidentally shipped laptops with a fake CA root cert, which then got hijacked for nefarious purposes, poluted the real CA chains and caused nearly everyone including google to revoke and reissue their CA certs.

The reason this approach fails in an increasing number of instances is because it is a hack.  It's a MITM attack on the clients.  So client software is now actually starting to check for the user that what it gets while claiming to be "valid" is ACTUALLY valid via third party verification.

Chrome for instance will no longer allow you to accept invalid SSL certs.  There is no "Proceed with caution".  Chrome will not be fooled by your MITM certificate either.  It will reach out to a 3rd party on the internet and ask it for the CA root cert and validate with it.... and fail.  The only way around this is to import that fake CA certificate into every single browser and application which now uses secondary validation on certs.

MITM proxies have their days numbered.

[Marco]

The technique clearly demonstrates how, exactly SSL does NOT protect you from a MITM attack without proper vigilance.

At least it can and it's the best we've got with widespread support, that's not going to change any time soon.

In an ideal world signing and encryption might be split up I guess for proxying data which needs no confidentiality, but this is not an ideal world.

[janoc]

If you are relying on proxying to not download stuff from the internet on every build, I think you have way bigger issues than some tool failing to download stuff because it requires SSL which the remote server doesn't provide.

What about:

- Ensuring reproducible builds? Downloading random stuff from the 'net instead of maintaining a local known-good copy of all your dependencies is a recipe for disaster. Ever heard of supply-side attacks?

- What if your internet connection dies?

- What if your proxy cache dies/is cleaned too? Now what? Having success of build depend on whether or not the package is cached by a proxy or the internet is available is completely backwards.

- What if the remote resource gets pulled/disappears/gets blocked? Now pray you have a copy in the proxy cache ... Given that that cache is by definition ephemeral, it is like storing data in the trash folder or system temp folder - and hoping they will still be there when you actually need them.

TL;DR - fix your badly broken and insecure build system instead of ranting about people following the best practices today and requiring SSL/TLS encryption on connection.

[paulca]

TL;DR - fix your badly broken and insecure build system instead of ranting about people following the best practices today and requiring SSL/TLS encryption on connection.

Get back in your cubicle.  Nobody is depending on the cache for builds.  In fact the actual build servers do things their own way.  I am talking about local development in a local IDE.

[ejeffrey]

Why can your local IDE build not cache downloads?  What does SSL have to do with that?

[paulca]

Why can your local IDE build not cache downloads?  What does SSL have to do with that?

It does.  However, you are getting more and more specific to just one issue.

The build system (maven) will (run a fresh) resolve the dependency tree and download each and everyone of the 100s of jar files.  These can and often do total to a figure best measured in Gigabytes.  In some wider build systems it will download docker images each can be several gigabytes.

In amongst them are those which are either built locally, built remotely or built as part of a pipeline elsewhere and published to an internal repo.

Remembering this is development.  In dev phases maybe only a few weeks long from "the first shovel" to rubbing your hands together in glee as it passes into production.  So you spend quite a bit of time bumping and cross linking versions of different libraries and services.

In an ideal world all the versioning would match up perfectly and everyone would remember to bump their version numbers, but in inevitably issues occur relating to stale, stuck, out of date dependencies in the local repo.

It can take hours and hours to pin down which one and why it's broken, delete it and redownload it.  Just like in Windows a lot of problems are solved  by "Reboot", in maven a lot of problems are solved by "Update dependencies..."  and tick "Force download snapshots".  Either that or delete the contents of the repo and rebuild.

There are other reasons to frequently dump your local repo.  The build servers always start from a clean repo for consistency purposes there is always a defined "clean" start point.  If you want to replicate that locally you need to dump your repo and invalidate your caches.

Thus, SSL is an issue as there is nothing caching the incoming jar files.  If someone, like QA, invalidates an entire stack of services and libraries, forcing a re-run of those pipelines ... to say initiate a deployment to UAT...  you could see maybe 30 or 40 parallel builds all basically downloading the same dependencies over HTTPS.  That's a terrible waste.

"Why not use an internal repo or mirror for all dependencies?"

I have worked in companies where that is indeed the norm.  They have teams of dozens of engineers full time just approving and mirroring Java dep repos.  Then they have to do the same for RedHatEL repos.  Python repos, anaconda repos, .Net repos. Docker repos.  Helm repos.... JS/TS repos. This costs millions a year to run for a large enterprise with in house devs (like a bank).  Not even government go this far.  The military probably do.

The common CDN sources for these things are pretty tightly monitored and peer validated.  They are not immune to "supply side attack", such as the recent scandal with "colors.js".  However they are monitored by many, many security researchers across the planet.  Especially after the colors.js scandal.  So with so many pairs of eyes on them and basically none on your local repos, it is debatable which is more secure. 

Online, webpage style CDNs are not really part of my "scene".  I am a back end developer.   "colors.js" would not have been anywhere near as impactful on the backend.  It's rare to see a backend application which uses a public CDN at runtime.  At build time maybe, but the "sealed" output must pass test before deployment.  A "colors.js" incident would be immediately spotted.

[DiTBho]

what do we want to do?
I wrote a HTTPD server for XINU ... and found that any modern browser(1) tells you that the page is "not secure" if it doesn't find HTTPS on the other side



(1) { Opera, Chrome, Firefox, Safari, ... }
I patched(2) a modern snapshot of Dillo to have it working like I need

(2) what does my patch?
remove all the SSL&C code

[paulca]

what do we want to do?
I wrote a HTTPD server for XINU ... and found that any modern browser(1) tells you that the page is "not secure" if it doesn't find HTTPS on the other side

And?   Why is that a concern?  If you don't need it to be secure.

[DiTBho]

And?   Why is that a concern?  If you don't need it to be secure.

what's the point of putting a red flag in the browser if a URL starts with "HTTP" instead of "HTTPS"?

now, most of us know how things work with internetworking, but some don't, and when I exposed my http server to the public internet... some people saw the "unsafe website" red flag and thought it was a hacking site that steals your data, sends encyclopedias to your home, and gets your dog kidnapped

[DiTBho]

Then the absurd thing ... if it's a website that you do NOT have to interact with by leaving passwords or credentials, if it's a simple blog where you can only read articles and at most leave a comment... what's the point of having a bunch of locks for every single session?!?

For any modern browser, you should do this anyway, otherwise the site shows up as "not secure"

-

I seriously miss Netscape ... 

[paulca]

Yea, browser land has gone too far in my opinion.

Back in 2005-2010 if you had a website is was HTTP.  It didn't matter if you took usernames and passwords for accounts.  It only mattered if you carried out financial transactions, took credit card info etc.  If you did want to take credit card information then your options were rather limited, a whole lot of work, or relatively very expensive.  The first hurdle would be getting a gateway that allowed you to post transaction details to it.  Even back then a "direct CC gateway" would cost you thousands and only available to be big companies.  As an individual or small company your only option was to "off load" the whole payment process to a 3rd party.  PayPal got in here at just the right time to make that easy.  WorldPay and the likes also expanded.  In this pattern your website is still HTTP.  However as soon as you click "Pay" you get transfered to a 3rd party company.  Honestly this did a lot to add trust, in that you weren't giving you card details to some random websites database to be hacked.

However, it spread.  The first problem was that to display even you company logo on that payment page, to give it some "validity" in it's look that it isn't just a random popup, but is for your order, required you host that logo on HTTPS or otherwise the browser will display "Insecure: Portions of this website are not secured." warning.

So then you had to host at least a portion of your website in HTTPS.  Web hosting companies jumped in to fix this by charging stupid amounts of money to have a single folder on your website "HTTPS secured".  While a company web hosting package for PHP/MySQL might have cost you a fiver a month, to get that 5Mb of SSL space would cost you £20 a month. The certificates involved where all shared and only really validated the hosting company permitted that domain to use it's cert.

If you wanted to get your own certificate the prices where around £100 a year for a basic "domain only" validation and £250-650 for basic company identity assurance.  Financial services companies ...  if you have to ask, you can't afford it.

Even today identity assurance certificates are extremely expensive.

Because people now needed to host a "portion" of their website in HTTPS, they moved the whole checkout process in there.  Then as SSL hosting prices fell they moved the entire shopping cart in there.  The trend was set the standards risen.

Fast forward to the mass rush into the cloud began with Google, Microsoft and others rapidly expanding their online services like email.  They could afford to put all of those things behind full proper SSL.  Quickly following that and a lot of hand-wringing worrying browsers started warning people the page was not secure even for basic login details.  Because those who create the browsers can afford to host true native SSL services then the rest of the world has to "step up" to the challenge too.  It is no small elephant in the room that it forced a lot of small businesses into those same cloud providers as they could wrap them up in the SSL layer for a price.

Google et. al. then pushed it further by making it's search engine HTTPS, alegedly securing your search history and results.   Of course this was not really their intent.  Their intent was to secure google such that others could not mine the same data that google wanted to keep for itself.

It was not long after that to when Chrome started warning on ALL sites that are not HTTPS are "Insecure".

Using fear to profit is one of the lowest forms of low.  It was around this time the milk from google started to get really sour.

A further note on browser practices today.  Have you noticed that Chrome will obfuscate it's own wrapper URLs?  If will open google search and hover over one of the GoogleAd sponsored links, it will show you the URL of the site which you "think" you are going to.  However, if you then inspect the page and the URL you find it does not, at all, in fact go to that website.  It goes to ads.google gateway first.  Collects some information about you and then redirects your request with a loaded URL containing handover information, campaign information hashes etc.

It is LYING to you, deliberately to hide tracking URLs, advertisement piggybacks etc.

What I don't understand this is why Firefox has started to do this also.  The status bar URL displayed on hover is FALSE.

I expect the story will go something like the back button did.  "Users are confused by:   ourwebsite.com/redirect?<url-encoded-target>"  so we swapped it out for just "<target>"  isn't that nice of us?

No.  It's moronic, insecure, dishonest, profittering and pulling the wool over peoples eyes in the name of their security at the total cost of their privacy.

So... there is my version of the history of the SSL scam.

Note the Let's Encrypt service was started as a charity basically to help combat this problem, where basic simple, common-or-garden websites where being flagged as "Insecure" and that being assoicated with "Bad actors and hacking" by the likes of Google and the others to profit from it.

[DimitriP]

TL;DR - fix your badly broken and insecure build system instead of ranting about people following the best practices today and requiring SSL/TLS encryption on connection.

"Best practices" !!! 
For whom?

Putting the cart before the horse creates less dust for the passengers, and gives them a better view.
All them "best practices" people should remind themselves why computer systems, and software development is there in the fist place. And "security" is supposed to be to keep systems "secure" not by preventing all the rest involved from getting things accomplished.
But I'll stop now before I tell you how I really feel !!!

As for "best practices" ....