I'm building an security platform and currently working on discovery of data.
We'll be collecting things like services, processes, ports, network connections, protocols, containers, hosts, DNS info and timestamps to understand how an NHI is accessing resources.
My concern is that storing all this data could become a huge pain and require a lot of storage as we scale.
For people who've built systems that collect similar runtime/network data:
What's an efficient way to store this without ending up with a massive database?
Would love to hear how you'd approach it.
Source: r/micro_saas · by /u/NDK235