It’s quite common that when launching an application or creating a new service in the cloud, the term security comes to mind at last. Which can actually cause lots of pain; both spiritual and legal pain.

In software development life-cycle (SDLC), we are most likely to put the effort into reaching the MVP or delivering the absolute must feature or fixing the regression issues. Promising to handle the security issues right after the feature/fix is delivered. Frankly, that features/issues never get to an end. Notice the keyword “Cycle” in SDLC.

It’s rarely easy to find spare time to make extra developments in a sprint. But it’s going to get harder to make these developments when things get serious. Simply put; at least make some room for developing security measures.

In this blog post I’m going to share the issues I’ve overcome (or still trying to) and some low-hanging fruits for cloud security.

Disclaimer

These tips/instructions/suggestions are intended to be beginner-level example cases for different scenarios and might not be suitable for your situation. If you need serious security measures, please contact a professional cloud security expert.

Supply Chain Security

Just like how you would want the ingredients in your kitchen to be safe to eat, containing relatively fewer (chemical) additives; you would (should) want your libraries, frameworks, images to be safe to use, containing relatively fewer vulnerabilities if not at all.

Know Your Package Manager

I haven’t seen any perfect package manager so far (I haven’t seen much but that’s the truth). It’s a good chance that you’ve already heard about npm, pypi and I believe most other package managers have their issues on security.

Even if everything is tightly secured in the package manager you’re using, there are these typosquatting (Wikipedia) attacks. This is simply conducted by uploading a similarly named package to the package registry and when the user makes a typo in the package name when installing it, here comes the malicious package, compromised environment, compromised supply chain…

So, the low-hanging fruits:

  • Do some digging into advanced/secure usage of your package manager.
  • Do some research on pitfalls to prevent while using the package manager.
  • Deploy your own private package manager (i.e. self-hosted PyPI server) and upload the trusted dependencies to it. (Not quite low fruit for most)

Know Your Libraries

Open-source doesn’t mean secure. But you already know that, right?.. RIGHT?!

Let me give you an example:

You are innocently and gullibly planning to diversify your web application to mobile, tablet and desktop users. And you have decided that the best way to achieve this goal is to parse the user-agent of the visitor (with privacy in mind, of course). The best thing happened! All of a sudden, you have become a crypto miner! And you are such a good person that you are mining for someone else!

Strange right? See this ua-parser-js issue and you’ll get what I meant.

If we learned from this example, it’s fruit time:

  • If the source code of the library is not thousands of lines of code length, take a glance down at the source code.
  • If the library is quite large, take a look at the known issues and vulnerabilities.
  • If the library has some vulnerabilities, try to cover them yourself in your codebase.
  • If you can’t cover them, discard the library.
  • If you can’t discard the library, execute it in a container or VM if the vulnerabilities are quite serious.

Know Your Images

Containerization with docker is quite popular. Pulling the desired base image, installing some dependencies, putting your project into the container, done.

With all due respect, I’ll certainly argue with that. Your base image might be compromised, the public container registry might be compromised, your own CI/CD pipeline might be hijacked, and so on…

  • Spin up your own container registry or buy a managed service for that.
  • Don’t just use the :latest tag or no tag at all, use specific SHAs.
  • Take a look at the instructions on your base images Dockerfiles.
  • Don’t keep your secrets in Dockerfile.
  • Don’t forget the principle of least privilege. Give the container just enough permission to properly run its commands, no more.

See my other blog post on Docker Best Practices on Performance and Security for more detailed information.

Software Security

Even if every tool, library, image, OS, system you are using were strictly secure, I bet you would have some vulnerabilities again. That’s because the share of security in the code quality ratio is undervalued most of the time.

New Commits to Codebase

New features are upcoming, exciting. But do you have a CI/CD pipeline? Do you have code analysis tools working? Do you lint your code? Do you have pre-commit hooks? No?

Do you even have code reviews? Separate environments?

You might not have enough budget to implement a fully-featured DevSecOps or the best CI/CD pipelines ever designed. You can manually do these things. Start implementing some:

  • Static/Dynamic code analysis
  • Remove secrets from your repository
  • Create a code review policy
  • At least two (varies on the size of the team) people should check and verify the code.
  • Pre-commit hooks and linting might seem irrelevant to code security, but if you persist a good coding style, commit messages, etc. you will benefit from more readable codes, hence, better code review sessions. You are less likely to detect vulnerabilities in a nonproductive review session.

Be Aware of Race Conditions

A race condition occurs when two or more thread tries to take action on a shared resource (same memory location). That could not only lead to unexpected behaviour of the service but also might lead to security issues.

  • Detecting a race condition might be tricky and can vary on the scenario or the use case of the service,
  • Multi-threaded unit tests might help,
  • Avoid using functions with side effects,
  • Use locks; thread.lock for python, Mutex::lock for rust, etc…

See Difference Between Data Race and Race Condition (StackOverflow).

Using Unsafe Methods

This is quite rare, but you might need to (I did need to) use some unsafe methods (i.e. eval in most programming languages is considered unsafe). And if you accept user inputs, that makes fire and gunpowder.

If you are really need to work with unsafe methods, you might consider:

  • Not using the unsafe methods (yes, seriously think again).
  • Use some safe wrappers to the unsafe method and mathematically prove that the wrapper function is safe.
  • Use abstract syntax trees and eliminate the unsafe cases.
  • Whitelist/Blacklist the user input -if you accept inputs.
  • Use containerized environment or a VM to execute the code.

Network Security

The supply chain is secured, check (assumed). The codebase is secured, check (assumed). Let’s spin up the services and let’s see what happens:

  • Some parasites trying to get into your service from open (but should be closed) obsolete ports,
  • Some zombie servers trying to explore your API with random requests,
  • Some random people abusing your APIs and limiting your resources,
  • Some not so random people conducting DoS/DDoS attacks on your servers,
  • Someone is sniffing your traffic,
  • Someone is posing Man-in-the-Middle (MITM) because you do not have mTLS configured

It’s like being in Jurassic Park, terrifying. Now let’s get rid of some of these if not all.

  • Close the ports except 80 or 443 if you are not using any other protocol than HTTP. Or even the port 80, if you can force HTTPS.
  • Consider firewalls. Whitelist some IP addresses or range(s) of IP addresses.
  • If you have more than one service and one or more replicas for each, you really need to consider a container orchestration tool like kubernetes (k8s), k3s, nomad or so…
  • Implement Role-Based Access Control between services.
  • Set mutual TLS (mTLS) between your services and force HTTPS for your outbound traffic to prevent MITM attacks.
  • Consider using an API Gateway to control all of your traffic in one place. Keep in mind that this option might create a bottleneck in your entry point for your services. Choose a scalable and lightweight API Gateway to prevent this bottleneck.
  • Get DoS/DDoS mitigation from major cloud providers by putting your services behind theirs.
  • Create a virtual private cloud and place your API Gateway at the gates. Nothing outside can access the services inside your virtual private cloud.
  • Prepare an ELK (Elasticsearch, Logstash, Kibana) stack to process request or system logs and detect anomalies on the system, bots, malicious requests. Take necessary actions like dynamically banning IP addresses of attackers, cover the unnoticed vulnerabilities…
  • Reduce human error drastically by embracing DevSecOps

Monitoring

You should be keeping an eye (or both) on your services, in case you have missed something (you definitely will).

  • Prometheus + Grafana, might cost $30/month if you don’t have lots of services or handle lots of requests.
  • Track your bugs with tools like Sentry,
  • Track your usage of resources,
  • Create alerts on these tools.

Conclusion

These security measures I’ve listed might not be tailor-made for your case. And it’s not intended to be. Doing some research about your application’s environment, dependencies, etc. will give you a good comprehension of your security concerns and their possible solutions.

If it’s not enough;

  • Do some third party security auditing,
  • Hire security consultants,
  • Design with security in mind,
  • Think about security,
  • Read about security…

But don’t get obsessed.

Phishing Attack

If this blog post offended you, good. It means that you are doing something wrong or not doing it at all.

See you next time. さよなら