You create a JSON (for example) for all your users when you need to store it. But in memory, it's just a collection of User objects - or whatever is idiomatic for your PL. And you query it with the same tools your language offers - e.g. sequence comprehensions. So there's no impedance mismatch, and no need for the vastly more complicated code that is needed to bridge it. Even if it's not your code - i.e. if you're using an ORM or something similar - it's still unneeded complexity, if you don't have too many users.
Think of it from the opposite perspective - if you can work directly with in-memory data, why would you prefer to run SQL queries instead? The latter is obviously more complex, but what's the advantage? There are many valid cases that have a good answer to that question, but there are many more that don't.
I can't help but think that we're conditioned to use DBMS (SQL or not) largely by inertia. When we learn about them, the examples are necessarily toy ones - tables with a dozen of records, that sort of thing. But one side effect is that it subtly normalizes the notion that this much data warrants a dedicated DBMS - so people don't balk at even patently ridiculous setups, like a separate SQL server used to hold a grand total of a couple thousand records in all its tables (this is a real thing, something that I personally did in a LOB app many years ago; I didn't have a good justification, it was pure cargo cult, and the app could have done everything it did in-memory, faster, and easier to code).
And sometimes, when you ask, people say, "yeah, it's only 1000 records now, but we're going to grow later, and then we'll need a DBMS". But you'd need to grow by many orders of magnitude to get there - and if you get that opportunity, you'll also have the resources to adapt. But everybody dreams of being Google.
And then you have apps like, say, a todo list manager app. Is it ever going to deal with millions of todo records? I don't think so. Then why does it need an SQLite DB?
That sounds extremely complicated because you have to do the vast majority of work yourself.
>So there's no impedance mismatch, and no need for the vastly more complicated code that is needed to bridge it.
Serializing arbitrary data structures is far from trivial, especially because JSON doesn't support references natively and therefore things like cycles are impossible without adding IDs yourself which is incompatible across JSON parsers. The naive way to serialize objects to JSON also tends to cause duplicated objects for every reference.
Or you can just use any SQL Database and use an ORM, write your damn Domain Classes [0], call .save() when you want to save and then query the damn thing [1] and if you want eager loading you can configure that too. Honestly it's so frictionless I don't see why I should bother with whatever your preferred workflow is.
I'm sure most complaints against ORMs happen because newcomers, who don't have sufficient knowledge of relational databases, try to model arbitrarily complex objects that cannot be mapped to a relational database (and not to JSON without extra support by the parser either) or because they think they don't have to write queries at all. In other words: they are fundamentally trying to swim against the stream and then blame the tool. If you know of the limitations of relational databases then ORMs are just a tool that saves you time and lets you avoid boring work.
I don't think the number of records is the right benchmark.
The reason why data is better kept in a database system is because it often has a very different life cycle than applications. It often lives longer and gets used by more than one application. That's why modelling and storing data somewhat separately from applications often makes sense regardless of the amount of data you have.
Also, procedural code is often far more complicated than a SQL query regardless of the number of records being processed. But that obviously depends very much on the specific problem, on the programming language and on the developer's skill set.
> the reason why data is better kept in a database system is because it often has a very different life cycle than applications. It often lives longer and gets used by more than one application.
It sounds like you're thinking about web and enterprise apps that share data sources. For those, absolutely, use DBMS. You don't want to solve concurrency and consistency yourself.
I was talking more broadly, of apps of all kinds. Does a TODO list app for your phone need a DBMS? There's not going to be any concurrent data access there, nor huge amounts of data, nor complicated queries. But that describes most apps, and most of their data! Even for web apps, an exclusive data store is more common than shared (just think about all the WordPress blogs online!).
> Also, procedural code is often far more complicated than a SQL query regardless of the number of records being processed.
If your language doesn't have some kind of declarative query framework for sequences, you're probably better off with SQL. But these days, who doesn't have that? Even Java caught up.
(I'd single out C++/STL for offering non-composable primitives, until ranges get into the standard. But C++ is a wrong choice for a data-heavy app in general, IMO. )
Unfortunately, I've rarely seen such a simple thing as file saving implemented correctly by many developers. Missing fsync(), missing checking of errors on close(), in-place non-atomic data overwrites. One of the good things about using SQLite/Postgres/MySQL is that at least they save your data correctly.
Think of it from the opposite perspective - if you can work directly with in-memory data, why would you prefer to run SQL queries instead? The latter is obviously more complex, but what's the advantage? There are many valid cases that have a good answer to that question, but there are many more that don't.
I can't help but think that we're conditioned to use DBMS (SQL or not) largely by inertia. When we learn about them, the examples are necessarily toy ones - tables with a dozen of records, that sort of thing. But one side effect is that it subtly normalizes the notion that this much data warrants a dedicated DBMS - so people don't balk at even patently ridiculous setups, like a separate SQL server used to hold a grand total of a couple thousand records in all its tables (this is a real thing, something that I personally did in a LOB app many years ago; I didn't have a good justification, it was pure cargo cult, and the app could have done everything it did in-memory, faster, and easier to code).
And sometimes, when you ask, people say, "yeah, it's only 1000 records now, but we're going to grow later, and then we'll need a DBMS". But you'd need to grow by many orders of magnitude to get there - and if you get that opportunity, you'll also have the resources to adapt. But everybody dreams of being Google.
And then you have apps like, say, a todo list manager app. Is it ever going to deal with millions of todo records? I don't think so. Then why does it need an SQLite DB?