As a data engineer, testing with production loads is critical to performance checking, as well as finding edge cases where your assumptions about what can be expected in the data are curb stomped and send you back to the drawing board to cry and think about what you’ve done.
And then managers that don’t get this will try to shove policies down our throats about how “pre-prod systems should not have access to prod data”.
“just obfuscate it.” Sure, for all 300Tb of it from the 10 different sources that don’t really talk to each other and we were already doing magic to be able to join them together? They should give us a bottle of hard liquor per month/project.
As one of the devops/sysadmin types, if we give access to prod data to preprod systems, they are now in audit scope and you have to harden them or we lose our insurance and compliance certs.
Obviously the solution is to build some system where everything works out, but it’s not as easy as “just give root to devs”.
Yup. Regulatory and audit requirements are a motherfucker.
Also, I don’t mean to speak down to devs, but as a rule of thumb you tend to think far higher of your skills just because you know the building blocks. Being able to build a boat doesn’t mean you know how to sail.
I know multiple people who are prodigous developers but know jack shit about basic computer usage and security. People who had to be guided to the control panel in Windows. Yes, even after they added the search bar. People hired to work in an exclusively Windows enterprise environment.
Now add that amount of potential that lack of basic operational skill carries for fucking things up to the least competent (or at minimum the least careful) co-worker on your dev team.
You (any dev reading this) as an individual would probably never fuck up that badly. You (any dev reading this) would probably do everything right, correct, and wouldn’t cause problems with root. But the rules aren’t written to protect against the competent, or against people never making mistakes.
I test my own code/scripts in dev when I’m working on it. QA usually tests acceptance criteria in test environment. And then staging is used for production data testing for performance and identifying missed edge cases. Actually, we sometimes use dev and test interchangeably when multiple people are working on the same repo, so the lines are a little blurrier than that.
As a data engineer, testing with production loads is critical to performance checking, as well as finding edge cases where your assumptions about what can be expected in the data are curb stomped and send you back to the drawing board to cry and think about what you’ve done.
17 years working with hospital patient data. I’m going to curl up in a corner and cry now…
dev teams usually :
this guy :
must be high pressure work.
And then managers that don’t get this will try to shove policies down our throats about how “pre-prod systems should not have access to prod data”.
“just obfuscate it.” Sure, for all 300Tb of it from the 10 different sources that don’t really talk to each other and we were already doing magic to be able to join them together? They should give us a bottle of hard liquor per month/project.
As one of the devops/sysadmin types, if we give access to prod data to preprod systems, they are now in audit scope and you have to harden them or we lose our insurance and compliance certs.
Obviously the solution is to build some system where everything works out, but it’s not as easy as “just give root to devs”.
Yup. Regulatory and audit requirements are a motherfucker.
Also, I don’t mean to speak down to devs, but as a rule of thumb you tend to think far higher of your skills just because you know the building blocks. Being able to build a boat doesn’t mean you know how to sail.
I know multiple people who are prodigous developers but know jack shit about basic computer usage and security. People who had to be guided to the control panel in Windows. Yes, even after they added the search bar. People hired to work in an exclusively Windows enterprise environment.
Now add that amount of potential that lack of basic operational skill carries for fucking things up to the least competent (or at minimum the least careful) co-worker on your dev team.
You (any dev reading this) as an individual would probably never fuck up that badly. You (any dev reading this) would probably do everything right, correct, and wouldn’t cause problems with root. But the rules aren’t written to protect against the competent, or against people never making mistakes.
You misspelled gallon(s).
Heh, loads.
You test in dev? You mean you don’t have a Q&A environment? or staging?
I test my own code/scripts in dev when I’m working on it. QA usually tests acceptance criteria in test environment. And then staging is used for production data testing for performance and identifying missed edge cases. Actually, we sometimes use dev and test interchangeably when multiple people are working on the same repo, so the lines are a little blurrier than that.
test in… preproduction?