installing Ubuntu https PPA's with squid caching

[Stuart McGraw]

TL;DR: How to set up apt and a squid proxy so that
https urls are cached?

I recently tried a novel (to me) way of installing
and configuring Ubuntu by scripting the install.
The idea is that the script(s) provide a record of
what and how things were installed, provide for
disaster recovery (I backup only user files), allow
for easier future reinstalls, and I can duplicate my
current configuration in a VM for testing new
software without risk of trashing my main machine.

Getting the scripts working (and maintaining changes
going forward) requires running them dozens of times.

But... I live in third-world America and have a slow
internet connection with a data cap.

The single thing that makes it practical to do was
setting up a Squid caching proxy on another local
machine and configuring Apt to use it when installing
in order not to download GBs of packages multiple
times.

The problem I am finding is that as I add new
software many PPA repos use https rather than
http for access and Squid doesn't seem cache these
packages. I am also concerned that Ubuntu will
at some point switch to https which will, for me,
kill any possibility of using scripts.

Is there some way of setting up Apt and the squid
proxy so that it will cache https urls?

I currently set the following on the machine being
installed:
# cat /etc/apt/apt.conf.d/20proxy
Acquire::http::Proxy "http://srvr1.home:3128/";
Acquire::https::Proxy "http://srvr1.home:3128/";

Googling has turned up a lot of info that was old,
not concerned with caching, expected to work with
every app and every user (I care only about apt
and root), complex/over-generalized (SSLbump), etc.

Does anyone have any suggestions for my particular
use case?

[Colin Watson]

Post by Stuart McGraw
TL;DR: How to set up apt and a squid proxy so that
https urls are cached?

The standard mechanism for proxying HTTPS, and as far as I know the only
method that apt supports directly for https:// URLs, asks the proxy to
set up a tunnelled encrypted connection (using the HTTP CONNECT verb)
and then does end-to-end-encrypted communication with the target host.
When using this setup, squid can really only shuffle bytes back and
forward; it doesn't see enough of the request or response to be able to
cache anything. (Of course this is still sometimes useful in situations
where the issue is connectivity rather than bandwidth.)

I think your best bet is going to be apt-cacher-ng: you can use its
"remapping" facility to tell it that a given set of http:// and/or
https:// mirrors are in fact to be considered identical for the purposes
of caching, or it has a hack where you can put "HTTPS///" in the URL so
that apt thinks it's using HTTP but then apt-cacher-ng uses HTTPS to
talk to the upstream mirror. Either way is going to involve adjusting
URLs on the client side, but it sounds like that will be worth it for
you.

Some links:

https://www.unix-ag.uni-kl.de/~bloch/acng/html/config-servquick.html#config-client
https://www.unix-ag.uni-kl.de/~bloch/acng/html/config-serv.html#repmap
https://www.unix-ag.uni-kl.de/~bloch/acng/html/howtos.html#ssluse

[Stuart McGraw]

Post by Colin Watson

Post by Stuart McGraw
TL;DR: How to set up apt and a squid proxy so that
https urls are cached?

The standard mechanism for proxying HTTPS, and as far as I know the only
method that apt supports directly for https:// URLs, asks the proxy to
set up a tunnelled encrypted connection (using the HTTP CONNECT verb)
and then does end-to-end-encrypted communication with the target host.
When using this setup, squid can really only shuffle bytes back and
forward; it doesn't see enough of the request or response to be able to
cache anything. (Of course this is still sometimes useful in situations
where the issue is connectivity rather than bandwidth.)
I think your best bet is going to be apt-cacher-ng: you can use its
"remapping" facility to tell it that a given set of http:// and/or
https:// mirrors are in fact to be considered identical for the purposes
of caching, or it has a hack where you can put "HTTPS///" in the URL so
that apt thinks it's using HTTP but then apt-cacher-ng uses HTTPS to
talk to the upstream mirror. Either way is going to involve adjusting
URLs on the client side, but it sounds like that will be worth it for
you.
https://www.unix-ag.uni-kl.de/~bloch/acng/html/config-servquick.html#config-client
https://www.unix-ag.uni-kl.de/~bloch/acng/html/config-serv.html#repmap
https://www.unix-ag.uni-kl.de/~bloch/acng/html/howtos.html#ssluse

Thanks, I've started looking at acng but it feels like a sub-optimal
solution:
- Project hasn't seen any activity in 5 years
- Caching seems to be a complex business and seems like Squid or some
thing similar is likely more widely used and hence more robust.
- acng is limited to Ubuntu/Debian servers (my current Squid solution
is running on an old Fedora box and I hoped to also cache Fedora rpms.)
However, if that is the only viable option then that's just how things
are. :-(

I guess I remain surprised at the lack of a good general solution since
it seems this is a growing problem with the promotion of https everywhere.
I had hoped that Squid could accept an http connection from a client and
proxy it as a https connection to the destination server (glossing over
how it knows whether to do that or not). But since I have only rudimentary
knowledge of modern http and https, I am probably missing some technical
difficulties.

Thanks for the info.

[Colin Watson]

Post by Stuart McGraw
Thanks, I've started looking at acng but it feels like a sub-optimal
- Project hasn't seen any activity in 5 years

This seems untrue (even if the upstream web page hasn't been touched - I
don't know). See https://tracker.debian.org/pkg/apt-cacher-ng

Post by Stuart McGraw
- Caching seems to be a complex business and seems like Squid or some
thing similar is likely more widely used and hence more robust.
- acng is limited to Ubuntu/Debian servers (my current Squid solution
is running on an old Fedora box and I hoped to also cache Fedora rpms.)
However, if that is the only viable option then that's just how things
are. :-(

Indeed.

Post by Stuart McGraw
I guess I remain surprised at the lack of a good general solution since
it seems this is a growing problem with the promotion of https everywhere.
I had hoped that Squid could accept an http connection from a client and
proxy it as a https connection to the destination server (glossing over
how it knows whether to do that or not).

I think the problem is more persuading the client to do that; if all
squid sees is a CONNECT request there's not much it can do about it.