I have a project, a programming environment/runtime, that contains a number of overlapping applications. I started with a small number of repos, including one main one. I had a number of frustrations with that. One of them is I didn't like having shared and unshared code from different applications in the same repo, for the sake of managing versioning. I separated into more repos, maybe 20. After I split it up I discovered a number of new issues, much like the OP here. One of which is complications from working with several repos at the same time, such as when refactoring.
I'll think about the next steps taken by the OP. Does anyone have any other practices they can recommend for managing these type of projects?
> I had a number of frustrations with that.
Could you elaborate more? The OP had to split the monorepo due to https://getcomposer.org/ limitations (each package needs it's own repo it seems). What other issues have you found when working with a monorepo? Thanks!
For some background, I have a web version and two electron versions of the programming environment. Then there is a web runtime and a web server which run on the output of the programming environment. Originally the main repo contained code for all the applications, but also some code specific to one or more of the applications. I had trouble versioning a subset of the repository. I ended up versioning the main repo and bumping the version of all apps if anything in the main repo changed, because I didn't have a systematic way of knowing that only certain parts of the repo changed. Perhaps that is the piece I would need?
I said there were a number of frustrations, but that was the driver.
In the new repo format each repo produces a single library/module, and can be versioned as a whole. When I run an application for dev, I can run a given library from a release or from live code (with a dev server serving ES modules from a configured path). This works well when one of the repos is not changing. But I end up developing in a significant number of repos at the same time, so I just configure it to run from live code rather than release for all those libraries i'm working on. This feels like working with a monorepo except I have the hassle of managing commits to all thse repos. Long story short, right now I am working out of the master branch because I screwed up commits. You may have figured out by now my strength is not in dev ops.
Thanks for sharing your experience! I can see how you were struggling trying to handle different "versions" in subsets of the repo.
In my experience the "monorepo" workflow implies a "working on master/head" mindset, but this is not always the best approach depending on your specific needs.
Lerna let's you version things independently, and if you use something like conventional commits / commitizen it can auto detect whether versions are major or minor.
When you make a version with lerna it auto-tags the commit with the version numbers of the components, so consumers can still depend on a specific version of a component.
I'm setting it up now at a new company and its pretty amazing.
While composer does have this limitation in that packages are published by making new tags within the repo, frameworks like symfony and cakephp have workarounds where they have one monorepo where all packages are worked on, and then automation to push changes to read only repos of each component. So there's https://github.com/symfony/symfony pushing to https://github.com/symfony/event-dispatcher which gets published to packagist.
> Does anyone have any other practices they can recommend for managing these type of projects?
Honestly, the only way around these sorts of issues is to utilize automation in some form.
I've found that setting up repositories (like devpi[0], Artifactory[1], or Docker Registry[2]) on a shared network location (for your project, could be local if you work alone) and using CI/CD tools (like Jenkins[3]) are the key. The goal is that you end up working on one portion of the code base at a time, and those need to go through the standard validation processes so that you can pull in the updated package version when you work on something down-stream. You making sure that the CI/CD environment _doesn't_ have access to other packages's non-versioned code is key for making sure things actually work as expected.
For example, if you have FooLib, and you need an update in that for BarApp, then even if you branch FooLib 1.2.3 to 1.2.3-1-gabc1234d (the `git describe` of the commit) on `feat/new-thingy` , then even if BarApp v2.3.4-1-gaf901234 depends on that new branch, it shouldn't be in any way able to reference that branch on the CI/CD build process. How do you get around this? Good development -- finish the FooLib branch, get that working, merge it in with the updated version, and push the package (with the new version) to the CI/CD-accessible repository. At that point, when you push your BarApp change, it can actually build and not die. But until FooLib has got a versioned update, BarApp's branch _shouldn't_ be able to build.
The statement of "But I want to work on the changes locally, in parallel" is valid. That's what local development is for -- giving you space to work on related things that don't impact the upstream codebase. You should have the option to utilize FooLib's branch code in your BarApp code locally, and you can often do that via things like `pip install` or `maven install` or whatever the relevant local install command is. At this point, the package still probably has the same version number, so the local build doesn't trigger issues. You can work on the two and tweak and twist as you want, but refrain from actually trying to push BarApp referencing FooLib's branch until it's actually in the repo.
This all takes a great deal of restraint and patience. The goal here is make it just a tad harder to introduce problems somewhere since you can't depend on something that hasn't been given the go-ahead. While there might be a lot of "Updated FooLib requirement to v1.2.4" throughout your codebase, why are you doing that just off-hand? If you are doing it because of a security issue or bug, let that be known in the commit message. If you are doing it because you can utilize a new feature/whatever, your commit message won't be just "Updated FooLib", you likely are doing "Added Feature X2Y, updated FooLib to 1.2.4".
PHP I try not to touch much, simply because I've always had bad experiences. I know for a fact that there are decent ways to do it with build tools like Maven[4], setuptools[5], and Docker[6]. Hell, I have used Docker as a way to introduce versioned dependency packaging, only needing to use Docker Registry (each dependent project does a multi-stage build, pulling in the dependencies via the versioned package images).
> But until FooLib has got a versioned update, BarApp's branch _shouldn't_ be able to build.
This is such a horrible practice. You're creating mountains of extra work, and encouraging devs to delay integration testing, which is certain to lead to cycles of rework. It also only 'works' on toy features. When you're building a complex feature that requires a few weeks of work and a few devs, it quickly breaks down, and further prevents early QA testing of the new feature itself.
Unfortunately this practice is often forced on people by reliance on the horrid SemVer scheme, which only makes any kind of sense for 3rd party dependencies, but is foisted on internal dependencies as well by many idiotic package managers, like Go mod or NPM.
I find that this pain is a symptom of complecting. If you well and truly can't test your code to 80% or better confidence until other feature comes online, well then maybe there are insufficient separation of concerns.
Typical CRUD stuff should be like at least 80% purely functional business logic (that is 100% testable without integration) and 20% or less IO code. If you really need that integration to find all the rough edges and work out the bugs, you probably have too much surface area in your IO "tainted" code.
Java-style-OOP really encourages this sort of thing by subconsciously compounding data state with functional methods. The whole "I needed a banana and you gave me a whole jungle" problem.
Integration vs component testing has almost nothing to do with functional vs state/IO heavy work. Instead,it has everything to do with the amount of effort you spend in specifying your components and writing test cases. If we're building a feature where componentA must call componentB with some data structure to achieve some goal, we can formally specify all valid inputs of componentB and their semantics, and write tests for each combination etc; and then have componentB religiously stick to the same; or we can agree in more informal terms on the expected input values and rely on integration testing to put them together and make sure we are achieving the right result for the expected inputs to componentA.
For some problems, the formal specification is tractable and even necessary. But for many complex problems, it is either not tractable (the input is too complex, you would need on the order of magnitude of componentB to actually specify the semantics) or its just not worth it (componentB is only called by componentA).
I also want to note that I'm not talking about regular types when I say 'formally specifying the valid inputs and their semantics', though I'm sure dependent types could in principle achieve this. I'm talking about cases like components which comunicate though script-like objects or configuration templates etc.
If you have a CI system your version patch will always increase and you can then always integrate the latest dev version. Most of your developer issues will be someone not using the latest versions for everything.
How will that work with in development branches? At any one time, there are multiple sub-teams developing multiple independent sub-features all impacting some of the same components. How are they supposed to do this if they can't branch out the components and each work on their own independent branch of the integrated application? There isn't a single 'latest dev version', there are many.
In lerna you can restrict versioning to a particular branch. So you check out your feature branch, work until it's ready to share and then merge it back into master and create a version.
Creating a version tags the commit with the version number for each package that's been updated and it allows for the creation of pre-release versions. If you have things that aren't ready for prime time.
Consumers can depend on a particular git commit by referencing the tag.
So there is one main branch that contains all of the commits, but different components are versioned independently and reference particular commits in the branch.
Integrate often enough that nothing has diverged enough for that to be a problem. Short lived branches are good, long lived branches get into the problem you speak of.
We're back to things that only work for small changes. Large features that need days or weeks of work before they can be mainlined aren't a rare occurrence, they are the norm, and usually generate the most value for a product.
Not to mention, you often need to polish a release while developing large features for the next release - again cases where you need branches.
Of course, you can also try to take the feature flag model, and avoid refactoring entirely. Unlikely to be a good strategy for a long lived product.
I agree. Having a huge mono repo is basically throwing the towel in the ring with your automation/package management/dependency validation.
In the .net world your CI/CD pipeline should continuously build and publish NuGet packages of your common code as you make changes. Since the old versions are obviously still available, other parts of the system are not forced to be updated to the new version of the dependency.
I will say one problem I have is in refactoring the interfaces of my modules, which is what I seem to spend a lot of time one, at least in this stage of the project. When I am updating the bottom ones, I pretty much have to update the others in parallel.
I'll think about the next steps taken by the OP. Does anyone have any other practices they can recommend for managing these type of projects?