Comprehensive 24/7 Server and Website Monitoring Solutions, possible?

[EliasIT]

Since it is becoming increasingly common for websites to be hacked or servers compromised, I am looking for a solution to monitor our servers while also checking our website for potential hacker attacks.

We don't have millions to spend on this, so is there a medium-sized company or an online system that can monitor servers and websites 24/7/365 for hacker attacks? Or is it only possible by professionally installing software and hiring a security expert?

 

[daniels27]

You can technically just have a website on another host and set up a cron requesting a specific document every minute. If the content changes or the request times out, your server has been compromised. Workd well for static pages. For dynamic pages, you may have to do some parsing to filter out dynamic content, but this is easily achievable.

For most other cases, I would recommend cloud flare etc. they are pretty good at detecting attacked and blocking such requests before they even reach your server. Of course, this comes at an additional cost.

 

[0xDEADBEEF]

There is a lot you can do yourself, but some questions first:

- how many servers are we talking about?
- what is running on these servers? (Are they all webservers or do you have a broader scope?)
- What is your worst case scenario?
- Do you have technical knowledge in-house that you could potentially upskill to learn some security tooling?

 

[aniglo22]

As other users already pointed out it's hard to help you without additional knowledge .
As an IPS i could recommend snort(Snort - Network Intrusion Detection & Prevention System) . You can use the community rules , but I would recommend fine-tuning against the threats you are facing . For wafs you could use Nginx+modsecurity,Coraza or openappsec.
But yeah without more knowledge its hard to help you .

 

[mraleph]

Since it is becoming increasingly common for websites to be hacked or servers compromised, I am looking for a solution to monitor our servers while also checking our website for potential hacker attacks.

We don't have millions to spend on this, so is there a medium-sized company or an online system that can monitor servers and websites 24/7/365 for hacker attacks? Or is it only possible by professionally installing software and hiring a security expert?

Website hacked and servers compromised? What do you define as hacker attacks? Monitoring without defense won't achieve any security. Also, monitoring requires high level of competence so that proper decisions can be made or rules/profiles be predefined.

Questions about your set-up and threat model already asked

There is a lot you can do yourself, but some questions first:

- how many servers are we talking about?
- what is running on these servers? (Are they all webservers or do you have a broader scope?)
- What is your worst case scenario?
- Do you have technical knowledge in-house that you could potentially upskill to learn some security tooling?

Isolate public resources - websites with any containerization technology - our preference is podman if ephemeral storage is required and lxc/lxd if persistent one is used.

Never expose physical and virtual machines to public, only containers.

 

[cionide]

Since it is becoming increasingly common for websites to be hacked or servers compromised, I am looking for a solution to monitor our servers while also checking our website for potential hacker attacks.

We don't have millions to spend on this, so is there a medium-sized company or an online system that can monitor servers and websites 24/7/365 for hacker attacks? Or is it only possible by professionally installing software and hiring a security expert?

Normally you'd use something like pingdom or freshping to monitor the servers being up, and you get an SMS notification if something is down. If your web dev is done using PHP, you can use ioncube24 to monitor changes to files. Alternatively, you can hire a penetration testing company to try to attack your websites and patch any vulnerabilities that are found. It's not usually servers getting hacked, it's exploits in website software (php/asp scripts etc) that data stolen.

As was mentioned above - compartmentalization is important as well. Make use of containers or VPS. It's better to have 4-5 VPS instead of 1 dedicated server if you can get away with it - if one thing gets hacked they won't have access to everything.

There's also things you can do with nginx or iptables. For instance on a few servers that kept getting attacked we put the websites behind cloudflare and blocked any connection to port 80 or 443 from any IP other then the cloudflare nodes. https://api.cloudflare.com/client/v4/ips
The $20 cloudflare paid package also does a great job blocking alot of sql injection attempts and so forth.

 

[mraleph]

Normally you'd use something like pingdom or freshping to monitor the servers being up, and you get an SMS notification if something is down. If your web dev is done using PHP, you can use ioncube24 to monitor changes to files. Alternatively, you can hire a penetration testing company to try to attack your websites and patch any vulnerabilities that are found. It's not usually servers getting hacked, it's exploits in website software (php/asp scripts etc) that data stolen.

As was mentioned above - compartmentalization is important as well. Make use of containers or VPS. It's better to have 4-5 VPS instead of 1 dedicated server if you can get away with it - if one thing gets hacked they won't have access to everything.

There's also things you can do with nginx or iptables. For instance on a few servers that kept getting attacked we put the websites behind cloudflare and blocked any connection to port 80 or 443 from any IP other then the cloudflare nodes. https://api.cloudflare.com/client/v4/ips
The $20 cloudflare paid package also does a great job blocking alot of sql injection attempts and so forth.

Matter of threat model and preference, but, compartmentalization should be always done with containers - both physical and virtual machines are a protected assets due to intricate significance.

As OP hasn't defined his threat model and requirements, we can only speculate about his security needs.

Your proposal for pingdom and freshping is conceptually correct. But, SolarWinds portfolio is compromised. For status and uptime monitoring, cron queries via ssh may be performed. A good suplemential idea is offered in post # 2

You can technically just have a website on another host and set up a cron requesting a specific document every minute. If the content changes or the request times out, your server has been compromised. Workd well for static pages. For dynamic pages, you may have to do some parsing to filter out dynamic content, but this is easily achievable.

For most other cases, I would recommend cloud flare etc. they are pretty good at detecting attacked and blocking such requests before they even reach your server. Of course, this comes at an additional cost.

Beside that, even if threat model assumes volumetric attacks, Cloudflare or any other cloud service provider should not be used. Desaturation can be done with reverse and HA proxy on a 10G link.

 

[cionide]

Your proposal for pingdom and freshping is conceptually correct. But, SolarWinds portfolio is compromised. For status and uptime monitoring, cron queries via ssh may be performed. A good suplemential idea is offered in post # 2

I agree that his threat model would have to be analized to really give a good answer, but I disagree on the post #2 idea. What he proposed not only requires another machine, but also doesn't actually tell us anything. Just because a specific document isn't changed, doesn't mean the machine or website is not compromised. It doesn't tell us if data was stolen from his db, etc. I've also actually seen interesting wordpress hacks that only reacted to specific geo locations - so what was happening was the admin was accessing his site and all looked fine but then some users were access and getting redirected. Nowadays it's mostly just bots scanning for common exploits and using them... you don't even get the god damned courtesy of having a human being hack you anymore :/

Speaking of the above - it also might be worthwhile to simply block geos in which you don't do business. I for instance found a ton of bot scans on some of my sites coming from the near east and africa - places I get no customers from.

If we're talking strictly about uptime monitoring, getting informed if your SSL certificate expires etc then freshping or pingdom are great for that and cheap/free. You can also set them up to require a specific output for an http request. Sure, you can roll your own with a cron but that requires coding which if he can't do himself will cost him more then probably several years of either of those services, especially if he wants to implement an SMS gateway to get notifications that way. I personally have email notifications on my phone turned off for instance because it was turning my phone into a leash.

 

[mraleph]

I agree that his threat model would have to be analized to really give a good answer, but I disagree on the post #2 idea. What he proposed not only requires another machine, but also doesn't actually tell us anything. Just because a specific document isn't changed, doesn't mean the machine or website is not compromised. It doesn't tell us if data was stolen from his db, etc. I've also actually seen interesting wordpress hacks that only reacted to specific geo locations - so what was happening was the admin was accessing his site and all looked fine but then some users were access and getting redirected. Nowadays it's mostly just bots scanning for common exploits and using them... you don't even get the god damned courtesy of having a human being hack you anymore :/

Speaking of the above - it also might be worthwhile to simply block geos in which you don't do business. I for instance found a ton of bot scans on some of my sites coming from the near east and africa - places I get no customers from.

If we're talking strictly about uptime monitoring, getting informed if your SSL certificate expires etc then freshping or pingdom are great for that and cheap/free. You can also set them up to require a specific output for an http request. Sure, you can roll your own with a cron but that requires coding which if he can't do himself will cost him more then probably several years of either of those services, especially if he wants to implement an SMS gateway to get notifications that way. I personally have email notifications on my phone turned off for instance because it was turning my phone into a leash.

Suplemental idea not the main solution But, yes, there is a point regarding operator's competence.

I dis-agree. It's better to do in-house shell scripting and deploy it thru cron then to use other solution.

Well, blocking geo attributes may be ad hoc and urgent solution. But, they are a technique and specific vector - we need to analyze and abstract the threats.

You haven't specified what you meant under blocking. But, I would always reroute the traffic and trigger a script that performs automatic intelligence gathering of the attacking host - we would acquire significant threat intelligence thatway.

 

[chermo]

@EliasIT ping me I will provide information free of charge and even point options for free and paid software. I am in the Cloud and Security Business.