You may need different versions of the same source files or targets, due to incompatibilities of the underlying runtime environment. For example, custom compiled versions of TensorFlow with different sets of optimization flags enabled for different use cases, or different build variants of the same conda package for different runtime compiled versions of e.g. numpy for different services.
Trying to solve this without providing versioned artifacts in an external storage repository is bad, don’t do it.
Managing something like this with the select functionality in bazel is a terrible way to solve it.
The conditions necessary to make the decision of which build variant to choose can be arbitrarily complicated (for example, the logical test could be the execution of some arbitrary script like embodied in a .PHONY target in Make) and also rely on caching logic or details of whether the ambient environment performing the build is a CI environment or not, and cannot be limited to flag arguments.
The identically same compiled version may be required by multiple consumers, even if they are distributed from each other. Forcing both consumers to rebuild some dependencies in such a case (as opposed to relying on pre-built artifacts from an artifact repository) is often an unacceptable performance overhead, and very wasteful even in cases when someone can afford to sit around and wait.
It can also be affected by portability.. for example differences in Docker for Mac vs say Docker for Ubuntu when the underlying artifact is a container. Much better to ensure the container that is built passes all required portability tests as a publication criteria on a container repo, so that the different consumers on different platforms can be sure if it is part of their build workflow, they are getting the correct artifact and it cannot be subject to idiosyncrasies of the platform or ambient environment at local build time.
The necessary build target for a given select condition may change over time, while still requiring backward compatibility with a former build of that same target from those same build conditions, effectively making a timestamp into an implicit version identifier for a backwards compatible version, meaning that re-running an identical build command with identical parameters at a later time will produce a build that succeeds for some consumers (who don’t need the backward compatibility or may even require a fundamental change that renders backward compatibility impossible) while failing for other consumers who require backward compatibility.
I’m sure you have plenty of “not hard” rebuttals for how to mangle bazel to address these types of things, which is meaningless because solving it with versioned, permanent artifacts is simply better.
It requires you to be extremely noseblind with diehard dogmatic insistence on using your “hammer for every nail” favorite to not see this.
It’s also telling to me that you focus on what you perceive to be “not hard” in your preferred tooling and have said this in several comments as if you believe it means something. I believe you are missing the point.
>The identically same compiled version may be required by multiple consumers, even if they are distributed from each other. Forcing both consumers to rebuild some dependencies in such a case (as opposed to relying on pre-built artifacts from an artifact repository) is often an unacceptable performance overhead, and very wasteful even in cases when someone can afford to sit around and wait.
Bazel handles all of this caching internally (yes even in the distributed case), so you don't need a secondary artifact repository. The build system will make sure you don't rebuild things unless they've changed. This is like the first selling point of bazel[0]. From bazel.build:
> With advanced local and distributed caching, optimized dependency analysis and parallel execution, you get fast and incremental builds.
Literally the first callout.
>The conditions necessary to make the decision of which build variant to choose can be arbitrarily complicated
If you cannot express the conditions of your builds statically, something in your CI process is horribly broken. Your build system should not be trying to guess whether its invoked on CI or locally or whatnot. I should get the same artifact, no matter what, no matter where I build it. The configuration of your build artifacts should be explicit.
> Much better to ensure the container that is built passes all required portability tests as a publication criteria on a container repo, so that the different consumers on different platforms can be sure if it is part of their build workflow, they are getting the correct artifact and it cannot be subject to idiosyncrasies of the platform or ambient environment at local build time.
This has nothing to do with the build system though. Yes, running tests under all the versions is a good idea. You can do that. You can set up a :mac and :ubuntu environment and make sure tests pass on both (by running your CI on both mac and ubuntu).
>The necessary build target for a given select condition may change over time, while still requiring backward compatibility with a former build of that same target from those same build conditions, effectively making a timestamp into an implicit version identifier for a backwards compatible version, meaning that re-running an identical build command with identical parameters at a later time will produce a build that succeeds for some consumers (who don’t need the backward compatibility or may even require a fundamental change that renders backward compatibility impossible) while failing for other consumers who require backward compatibility.
This doesn't make sense. Like, I can't even comprehend what you're doing without an example. Timestamp should never be relevant, revision will be. Again, you're only ever building from HEAD and a cache based on artifact hashes. Bazel doesn't use timestamps anywhere. Timestamp based builds are an antipattern. Make is bad. You shouldn't use it. You should use anything else.
So in bazel land, when you check out a previous revision, you'll build a working version. When you check out the current version, you'll build a working version. Those builds will both be incremental, only rebuilding the modified dependencies, but will be byte for byte equivalent to a clean build made at any time.
>It’s also telling to me that you focus on what you perceive to be “not hard” in your preferred tooling and have said this in several comments as if you believe it means something. I believe you are missing the point.
Pretty much every complaint comes from you misusing bazel. That doesn't make bazel a bad tool. It means you don't appear to understand how it works or best practices. You misapplying a tool doesn't make the tool bad, it makes you a bad user.
And you're going to read that as me saying bazel doesn't work for the things you're doing. Which is incorrect. I'm almost positive it does. You just are making (wrongheaded) assumptions about how dependency management is supposed to work and trying to make bazel fit your worldview instead of the opposite. Again, user error, not tooling issues.
Stop trying to shoehorn an additional artifact repository in around bazel. Stop trying to hack all these make-style isms into a bazel workflow. Use the tool as it is meant to be used and perhaps your complaints about it being problematic will go away. As I'm reading your comments, all I see is you repeatedly trying to misuse the tool and then complain that its bad when your misuse turns out to be annoying.
You _really_ are missing the point man. I’m not talking about basic caching local to the build environment nor am I talking about caching of code or intermediate artifacts themselves to e.g. reduce compile times.
For example, I worked previously a system that automatically trained several dozen different logistic regression models every day, each one tailored to a specific customer or customer vertical.
For ourbuild system we had to check s3 to see if the most updated trained model checkpoint had completed yet, and download it locally to use as part of several builds if so. If not, the fallback logic was to check if the newest vertical-specific model file was completed for the day, and use that, and if not, the fall back to a previous day’s model.
Based on which of these checkpoints was selected, several different follow-up selections would be made in terms of environment variables and build flags, and the result would be cached locally for that developer so that the remote check of s3 model builds only happened at most once per hour (configurable by the dev as well).
It is very frustrating speaking with you because you seem to believe that a very shallow way of thinking about literally just the build tool (e.g. pants or bazel style caching) is somehow a concise and objectively scoped response to what I’m saying, but you just keep digging yourself into deeper graves by demonstrating how dogmatically and narrow-mindedly you think about these problems.
Your responses do not come across as rooted in objective analysis of build capabilities, but it seems like you think you’re presenting yourself that way, when really you are being extremely subjective and lacking imagination about all the things people may need from build systems.
> “This doesn't make sense. Like, I can't even comprehend what you're doing without an example.“
It is clear you are not trying to understand. I am NOT saying someone should ever use a timestamp as an actual part of versioning.
I am saying that the select feature in bazel can de facto make that happen, and it has nothing at all to do with misusing bazel (which looks a lot like a No True Scotsman fallacy that you’re now throwing in the mix).
If I have some bazel select criteria C that resolves to a build action A, and then later in time I need for the exact same criteria C to instead build A2 (an upgraded version of A that is not backward compatible), this is a problem.
Sure, I can muck around with revision history of the BUILD file, but what if I need A from the old revision but I need Foo and Bar from the newest revision?
Nobody is saying you _cannot_ find a way to hack through revision history and make bazel do it... Whether bazel can do it is not the point.
The point is that one super easy and effective way is to have a permanent versioned artifact of A that never needs to be rebuilt, and express the dependency on the artifact.
Nothing about my points indicates any misuse of bazel. There are just use cases that bazel is bad at.
The fact that you cannot admit that bazel might not solve every problem and that if bazel presents an annoyance it must be down to someone misusing it is seriously alarming. You have got to be more self-aware of how dogmatic you are being about this. It’s just sincerely not reasonable, yet this is the whole way Google operates regarding monorepo, bazel, and a bunch of other stuff.
This isn't a build system. What you're doing isn't building. Trying to use a build system to do what you're doing is a bad idea. Using make to do what you're doing is a bad idea too, but it works because its just a shell script alias thing and not a build system.
If its not something you can statically generate based on your code, and only your checked in code, at a certain revision, its not something you're building.
>If I have some bazel select criteria C that resolves to a build action A, and then later in time I need for the exact same criteria C to instead build A2 (an upgraded version of A that is not backward compatible), this is a problem.
You update the select rule. The select rule is version controlled. There's no timestamping anywhere, only what is at HEAD at revision X vs. revision Y. If you check out the older revision, you'll use the old criteria. The new one will use the new criteria, no matter the time.
>I am NOT saying someone should ever use a timestamp as an actual part of versioning.
Yet you're advocating make, which does exactly that ;)
>but what if I need A from the old revision but I need Foo and Bar from the newest revision?
You don't. You should never need to mix different revisions because everything is always buildable at head. If you mean you have some data file that's incompatible with your current schema or something then that's a different question, but again has 0 to do with your build system.
To make it crystalline: versioning your data is not the same thing as versioning your builds. Bazel solves the first problem, because bazel is a build system. You seem to be trying to use bazel for something its bad at and then complaining that its a terrible tool because you're trying to version cron-generated, non-hermetic data files as part of your build process, which like yes bazel is bad at because that's not a build artifact, its an external non-hermetic data artifact.
Pass your non-hermetic model data into your hermetic production binary via a flag. Separate your concerns. Problem solved.
>The fact that you cannot admit that bazel might not solve every problem and that if bazel presents an annoyance it must be down to someone misusing it is seriously alarming.
Except that I was right. Bazel is indeed not good for what you're doing, but that's because you're not managing builds, but ML model data. You're attempting to use a build system for managing not-builds. You're experiencing pain. This is not a surprise. Its a classic example of the XY problem.
You: "Bazel is a terrible build system, it causes pain and shouldn't be used"
Us: Some discussion
Conclusion: "What you're doing isn't managing build artifacts, so of course a tool designed to do that isn't solving your problems. Dogmatism helps here. Use a build system to manage your builds and a different tool to manage your ML models.
Trying to solve this without providing versioned artifacts in an external storage repository is bad, don’t do it.