PCI – Going Beyond the Standard: Part 20, Incident Response (IR)

First, you may be asking why this blog does not include Disaster Recovery (DR) and Business Continuity Management (BCM, which governs the entire IR / DR process). Because the PCI DSS section 12.10.x is almost entirely related to IR (with the exception of a VERY brief nod to DR / BCP, below in red), I will handle DR / BCP separately in the series (post 23 in fact).

“12.10.1 – Create the incident response plan to be implemented in the event of system breach. Ensure the plan addresses the following, at a minimum:

    • Roles, responsibilities, and communication and contact strategies in the event of a compromise including notification of the payment brands, at a minimum
    • Specific incident response procedures
    • Business recovery and continuity procedures [This is the only requirement in the DSS that goes beyond the protection of CHD.]
    • Data backup processes
    • Analysis of legal requirements for reporting compromises * Coverage and responses of all critical system components
    • Reference or inclusion of incident response procedures from the payment brands.

With regard Incident Response, I put it this way; “What’s the point of being in business, if you don’t intend staying in business?”, and; “Good incident response is what prevents a security event from becoming a business crippling disaster.”

It makes absolutely no sense to me that organisations who basically depend on IT for significant chunks of income (which is most of them), have very little idea how to stop bad things from happening in the first place, let alone fix things when they go wrong. Of course, no incident response is going to predict an earthquake at the datacenter, but the organisations I’ve seen don’t even perform log monitoring properly, let alone consider the impact of acts of nature.

The development of a good incident response plan start with? Yep, a good policy, from there you agree on an appropriate Risk Assessment / Business Impact Analysis process, which in turn provides you everything you need to not only determine if you have any control gaps (after a gap analysis), but – if you’ve done it properly – a good indication of what your incident response and disaster recovery plans should entail.

There is no appropriate IR without an understanding of the business goals. If you have a 4 hour Recovery Time Objective (RTO), your IR will be significantly more robust than one where you can take a week to be back online. Yes, I know that RTOs (and RPOs (Recovery Point Objective for that matter) are DR terms, but if your incident response cannot detect a business crippling event in good time, then neither of those DR goals is an option for you.

When setting up your IR program, the most important word to keep in mind is ‘baseline’. Without a baseline, you don’t have much of a concept of what constitutes an incident in the first place. Only a baseline can give you both context and relevance.

From your baselined system configuration standards (DSS 2.x), to AV (DSS 5.x), to logging (DSS 10.x), to scanning (DSS 11.1.x, and 11.2.x), to FIM (DSS 11.5.x), you have many available inputs into your IR program, none of which will be of the slightest help if you don’t know what they SHOULD look like.

That’s all IR is;, a process whereby an exception to the norm is investigated, and appropriate action taken.

In each of my individual going-beyond-the-standard blogs related to the above DSS requirements, I have stressed the importance of baselining (well, except AV perhaps). The reason I did so was because they all lead up to this. I don’t care how well you have done ANY of the previous requirements, unless you can bring the outputs all together into a comprehensive process of taking action, all you have is a bunch of data to give to your forensics investigator.

You’ll notice though that I did not say a CENTRAL process, because while having a 24X7 Security Operations Centre t manage all of this, it’s rarely practical, even if it involves a outsourced managed service provider (MSP). However, having the correct assignments and procedures to MANAGE the response is of utmost importance, and the details of this plan will vary considerably from company to company.

No IR is not easy, but there is simply too much information and help out there for this difficulty to be any sort of excuse. And no, there is not much in this blog that actually provides guidance, but if this makes SENSE, then you at have at least got enough to begin to ask the right questions.

PCI – Going Beyond the Standard: Part 19, Security Awareness Training (SAT)

I really should give up being surprised when the most basic of information security fundamentals are performed poorly, but this one constantly amazes me. I guess it’s no different than a doctor being surprised at smokers, or the police surprised at repeat offenders, we can accept as common sense what others perceive as new concepts.

Education and Training is so important that I have listed it as one of The 4 Foundations of Security, along with Management Buy-In, Policies and Procedures, and Governance. The fact is that education is the best and cheapest way for an organisation to implement the desired organisational culture, and distribute the policies and procedures in a manner where they actually understood and followed.

The intent of PCI DSS Requirement 12.6.x is to ensure all employees are trained in their security responsibilities as they relate to the protection of cardholder data. That’s it, just cardholder data, so you can obviously ignore every other form of sensitive data in you environment, right? What about your financial data, or intellectual property, or personal data? Unfortunately you cannot go above and beyond in PCI unless it relates to the protection of cardholder data, so with the exception of perhaps frequency of training, there’s not a lot you can do here.

That’s for PCI though, for your BUSINESS it’s a very different matter, and there is a lot you can do to add true benefit across the organisation. Not just in terms of security either.

The mistake most organisations make is the assumption that security education and training only refers to things like keeping your passwords secret, or not lending out your swipe cards. Yes, training includes these things, but it starts with a thorough coverage of all relevant policies and procedures. I say relevant, because you’re not – for example – going to train your sale team on the proper implementation of firewall configuration standards.

Training is not just some paperwork exercise during on-boarding, then an annual obligation thereafter, it’s the way you bring someone into your organisation and have them up to speed and productive in the fastest time possible. It’s also how you begin to instil the corporate culture (i.e. your policies), and how you ensure that they are performing their duties in-line with standard practices (i.e. your procedures).

Once they have the basics, you can move on to role specific training, and then, if you’re REALLY doing this properly, you will have the individual job specifications detailed to the point where anyone being on-boarded can step straight into the leavers’ shoes with barely a backwards step.

That’s really the whole point; security awareness training is NOT just a compliance obligation, it’s an integral part of your business continuity and knowledge management processes. It can be the difference between a constant reinvention of the wheel every time you have a mover or leaver, and uninterrupted growth. You may argue that this is more than just security awareness education and training, but I will counter that without proper knowledge, there IS no security.

While I agree that every time there is a staff change, the training itself should be reviewed and revamped as appropriate (preferably by the person bringing the new pair of eyes to it), NO-ONE who is just starting should have to work out anything for themselves on how to perform the function to which they have been assigned. At least to a minimum standard. Unless of course it’s a brand new role, in which case they will be responsible to develop and document everything necessary to replace themselves in time.

Too often this is seen as making yourself replaceable, but if you can’t be replaced, how can you move up, or even across?

To perform security awareness and training properly, follow these steps:

1. Like access control, the best way to begin developing a good training program is to properly define the requirements, first at a ‘corporate’ level (everyone), then at a more granular ‘role’ level (sales, systems admins. etc.), and finally at an ‘individual’ level.

2. Once this matrix is complete, combine this ‘paperwork’ into an online delivery mechanism which is a combination Document Management System (DMS) and distribution method. That’s really all online training software is; content management.

3. Run everyone through the program, regardless of tenure, and regardless of when they last took it. Track all ‘signatures’ (an online ‘I Accept’ will suffice).

4. Run training again at a minimum annually, but preferably every 6 months. A good balance is full course annually, and Top 10 Things to Remember at the 6 month mark.

5. Throughout the year, use this distribution method to announce major changes to policies and procedures, as well as ‘zero day’ threats (new phishing techniques for example), for significant changes to relevant compliance regulations or laws, and any ad hoc matter for which you require – for liability purposes – a written confirmation of acceptance.

 6. Provide a robust feedback loop and standardised forms for all personnel to request policy / procedures changes, or to create new ones.

I’ve not touched here on the actual content of the security training, it’s too organisation / sector specific, but there are certainly some basics (101 stuff as the Americans would say). However, the development of a comprehensive and sustainable training program requires specialist skills and experience, so make the effort and expense, there’s not one investment you can make that has a greater ROI.

PCI – Going Beyond the Standard: Part 18, File Integrity Monitoring (FIM)

With the exception of Role Based Access Control (RBAC), File Integrity Monitoring (FIM) is the only PCI requirement that achieves security in its purest form; prevention of, or alerts on, deviation from a known-good baseline.

Firewalls/Routers (DSS Req. 1.x) is almost there, configuration standards is even closer (DSS Req. 2.x), anti-virus (DSS Req. 5.x) is basically pointless, and logging (DSS Req. 10.x) could not be further off the mark. But FIM, assuming you have interpreted the requirements correctly, has real benefit when combined with the other requirements done well (especially configurations).

Unfortunately, even to this day, everyone associates Tripwire with the FIM requirement. If you can believe it, their NAME was included in version 1.0 of the PCI DSS! Of course, Tripwire licensing fees went through the roof as a result, and I’ve pretty much hated them for it ever since. Now they’re jumping on the Security Incident and Event Management (SIEM) bandwagon and doing as bad a job of it as everyone else.

PCI DSS v1.0 – “10.5.5 Use file integrity monitoring/change detection software (such a Tripwire) on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

Couldn’t even spell “as” correctly, but I digress.

As in all DSS requirements, the first question you must ask yourself is; “What is the intent of…?” In this case, the intent of FIM is to ensure all of your critical files (both operating system and application) do not change without authorisation (i.e. outside of a known change control scenario). Basically it’s seen by the SSC as a back-up for anti-virus (malware being the primary cause of unauthorised file changes), but in my opinion, the correct implementation of FIM, configuration standards, and baselined logging more of less negates AV altogether (see Annual Validation is Dead, it’s Time for Continuous ComplianceContinuous Compliance Validation: Why The PCI DSS Will Always Fall Short, and PCI – Going Beyond the Standard: Part 10, Anti-Virus for a little more background).

PCI DSS Requirement 11.5 states; “11.5 Deploy a change-detection mechanism (for example, file-integrity monitoring tools) to alert personnel to unauthorised modification of critical system files, configuration files, or content files; and configure the software to perform critical file comparisons at least weekly.” so it should be clear already how to go above and beyond; perform the checks more frequently that weekly. For a start, weekly is ridiculous, a lot can happen in a week, but the requirements’ very limited benefit is compounded by the fact that there is no guidance in WHAT changes you should be looking for.

FIMs can usually detect everything from file existence, size, permissions, hash values and so on, but these are only making checks against themselves from a previous ‘run’. Therefore one of the best ways to go WAY above PCI minimums is to compare the files to central database of known good configs directly from the operating system vendor themselves. Microsoft has a database of the latest and greatest system files (DLLS, EXEs etc.) against which you can run comparisons, and it should be relatively simple to add the baselines from each application you install on top.

Now let’s get REALLY crazy; What if you could then compare what files SHOULD be there as a result of a comprehensive configuration standard / hardening guide, and ensure that everything is as it should be?  Against CIS Security Benchmarks for example? Some FIMs (or similar agents) can also check Windows registry and GPO settings, so you can not only make sure everything is configured correctly per your approved standard(s), you can also automatically report against a significant number of other validation requirements (password complexity, access groups, log settings, time synch settings and so on).

And of course, the best way to blow PCI minimums out of the water is to compare a system against baselines stored centrally against each asset. These are system x’s available services, listening ports, permitted established connections and so on. Now you not only have configuration management, you have both policy and compliance validation built in automatically. Not once a year, but all day every day.

OK, I went WAY too far there, but hopefully you get the point. FIM is not just something you throw on a system because PCI demands it, you do it because it’s integral to how you do real security properly.

Yes Tripwire can do more than PCI asks for, but now you have just another management station to configure, maintain and monitor. FIM done well must integrate with all your other systems to have the necessary context, and FIM is only relevant if you have configured your systems correctly in the first place.

Finally, FIM should NEVER be seen as a stand-alone, end-point product, it can and should be a lot more than that.

PCI – Going Beyond the Standard: Part 17, Vulnerability Scanning & Penetration Testing

Far too often, security is seen as a project, especially if PCI compliance is the goal. The requirements for vulnerability scanning and penetration testing are therefore seen as just another tick-in-a-box and their significant benefits lost.

External vulnerability scanning is the only requirement which must be outsourced and run by an approved scanning vendor (ASV, list here), the other requirements; internal vulnerability scanning, external penetration testing and internal penetration testing can by run by internal resources IF, and ONLY if, you can adequately demonstrate the requisite skill-sets in-house.

Of course, in order to save money, it is very tempting to skate by on the bare minimum, and unfortunately some security vendors (including QSAs) will allow you to do just that. Which is a shame, almost to the point of being irresponsible, as no other requirements give you a truer indication of your actual security posture than these.

Think of it this way; the bad guys use the EXACT same techniques to break into your systems that the good guys use to tell you what’s wrong. The ONLY differences between a hacker and an ethical hacker are intent and moral code, the skill-sets and mind-sets are the same.

Between vulnerability scanning and penetration testing, you have roughly 50% of your vulnerability management program sown up. Patch management management, risk management etc. make up the rest. However, the trick that’s almost always done poorly – if at all – is the integration of vulnerability management with asset management and change control. Any change to your environment should have appropriate vulnerability management processes around them, from a quick directed scan to a full blown credentialed penetration test, and all should be in-line with agreed configuration standards (as defined against each asset).

Going above and beyond PCI in scanning and pen. testing is relatively simple, but it’s not cheap in terms of resource cost. It also demands a maturity of process and a significant shift in culture to accept the ‘overhead’, but it’s more than worth it:

1. External Vulnerability Scanning – No choice but to use an ASV, but you should choose a vendor that provides 2 things at either no, or little, extra cost; Monthly scans (PCI requires quarterly), and unlimited directed scans (against single IPs, or subnets). Performed correctly, monthly scans and directed scans initiated by change control processes go significantly above and beyond. Note: For PCI do NOT open your external firewall/routing devices to your ASV’s IP addresses. Why would you decrease your security posture to test your security posture? Just run one scan for PCI, and THEN open your firewalls so that scanners can do a more thorough job. Keep these profiles separate, one for PCI only, one for your entire business.

2. Internal Vulnerability Scanning – You can do this yourself, and I’ve lost count of the number of clients running basic installations of Nessus, but unless you have significant expertise in how to configure it AND understand the results, don’t do it. For a start, any good QSA will fail you for lack of expertise, but do you really have the time to keep it up to date? Again, running internal scans monthly and as directed by change control goes above and beyond. Having two scan profiles is also a nice feature, but if the scan engine is capable of doing more than just rattle the windows (in the ubiquitous house analogy) and can actually perform a deeper scan / reconnaissance, then you have knocked this one out the park. PCI compliance is never security, do internal scanning as far above PCI minimums as you can afford.

3. External Penetration Testing – PCI requires that you attempt to break in (without breaking) via your Internet-facing presence, but poor guidance on what the test should consist of, combined with an enormous price-compression of pen. testing services means that this effort is usually more automated than I would consider appropriate. A pen. test is supposed to be a person with the necessary skills trying for days on end to discover ways into your systems. This is rarely the case now, but is EXACTLY what your should be doing. The Internet is where most breaches originate (used to be internal), so having a VERY robust security posture from the-outside-in is of paramount importance. Do NOT skimp on this one.

PCI calls for annual pen. tests, and to go above and beyond you need to perform these more frequently. This should not be an enormous cost, and most pen. test vendors can provide an infinitely scalable service based on scope and call-off days.

4. Internal Penetration Testing – Same premise as the external pen. test, but this time from the inside. PCI requires that this test simulate an attacker ‘plugging in’ where the admins sit and seeing what they can do from scratch. Above and beyond is therefore very simple; give the pen. tester FULL access to the environment, as well as credentials to go even further where appropriate. Like scanning, you have one test for PCI, then another test for your business.

There will be times when a simple vuln. scan of a system that has undergone change is not sufficient, so having a directed pen. test process available for critical business changes is very important.

None of the above processes should be stand-alone concepts, and should be very tightly integrated with risk assessment, change control and asset management processes to be truly effective. Vulnerability Management represents the end to each cycle of your security program (Plan > Do > Check > Act > Repeat), and ensures that your security posture always remain in-line with your business goals.

It bears repeating, do NOT skimp on this requirement, you will pay far more when you have to clean up the mess after a breach.

PCI – Going Beyond the Standard: Part 16, Logging & Monitoring

From everything I have seen in my many years performing PCI assessments, logging is not only one of the least understood of the requirements, it is the most under-utilised, and the one that gives/gave my clients the most pain.

Logging is the most important detective security control you have, bar none, and done correctly, logging is the foundation of your incident response program. Notice I didn’t say ‘disaster recovery’ as well, because if your incident response was where it should be you should not HAVE to recover from a disaster.

The confusion stems mostly from a lack of understanding of logging mechanisms themselves, even for Windows (for which PCI was clearly written). For example, do you think that Windows logs to the PCI requirements out of the box? I did too, but have been assured that it does not. Do you know HOW to get it to log appropriately? No, me either.

I have been further assured this if you WERE to turn logging on to cover the 10.2.X requirements, the logging would be so verbose as to render the device that’s doing the logging useless. Is that really the INTENT of logging? Of course not.

Also, can syslog EVER record the events required in 10.2.x, or even the event content as required in 10.3.x? Once again, I have been told no, but I am no expert.

Yes, you SHOULD have people who DO know this stuff, but how many organisations out there can truly afford that kind of deep expertise in-house? Yes you can outsource, but where is the guidance on EXACTLY how to configure operating systems to log to the PCI requirements? It probably exists, but in 10 years of doing PCI I have not found it, and I’ve even asked ‘experts’ in the field; Security Incident and Event Monitoring (SIEM) vendors. On that note, I have yet to see a SIEM vendor also be an expert in PCI, which to me is an absolute joke if that’s why they are selling it to their clients.

We have free hardening guides for Windows (CIS Security Benchmarks for example), but where is the guide that breaks down the Windows operating system into a mapping between the registry settings for logging and the PCI DSS? Or *nix flavours, or Cisco, or AS400, or Power Series? If you have them, please share?!

So let’s, for the sake of argument, assume that you cannot reasonably log to the letter of PCI, or more to the point, you do not WANT to for usability issues. Are you non-compliant? Let me answer that with another question; What’s more important, configuring your logging to record events for a forensics investigation, or configure your logging to help prevent a breach in the first place?

If you chose the second one, you are correct, and if you also choose to maximise your logging mechanisms, you will not only be PCI compliant to its intent, you will also be doing security properly.

First, logging is not about crunching masses of data through a correlation engine, its about the RIGHT data put into a base-lined context. There is no such thing as log event correlation without a deep understanding of what the end systems SHOULD look like, AND what your normal business processes are from start to finish. In other words, tell any vendor trying to sell you compliance though their SIEM, to put it where the sun don’t shine.

While we’re on the subject of SIEMs, how many of them do you think can accept Windows logs natively, or have to convert the logs to syslog via an agent (e.g. Snare)? Very few. What’s the point of buying a log mechanism that cannot even read WINDOWS events without butchering them DOWN to syslog?! You MUST ask the right questions before buying ANYTHING, especially a SIEM.

OK, so how DO you go above and beyond? Simple, in one way;

Do NOT perform your log reviews daily (10.6), because that’s just plain stupid, not to mention impossible to do adequately. Perform log reviews in real-time via some form of automation.

The automation you need is threefold;

  1. Events you should NEVER see: Each system admin, from OS, to network device to application SHOULD know which they events should never be seen under normal operating conditions. Look for these ‘strings’ and alert immediately.
  2. Events you should not see in a certain quantity and velocity (i.e. thresholds): I don’t care if I see an admin fail to log in once, I do care if s/he fails 10 times in 2 seconds (for example).
  3. Quantity of events over the course of time (i.e. trending): You have to save logs for 1 year (DSS 10.7), so why not put them to good use by trending events over time? Even if it’s just quantity of event (as opposed to quantity of type of event per device), the information you get can be extremely useful.

Perform all three of these things, and you have not only covered the ridiculous ‘daily reviews’ automatically, you now have input into your incident response mechanism that gives you real security.

Choosing the right centralised logging mechanism for your business is one of the most important decisions you can make, and it cannot be done ONLY for PCI. You must buy a system that can cover you enterprise-wide, and unless you have significant in-house expertise, you must build into the RFP the requirement for consulting support, and potentially some form of on-going managed service. Nothing stays the same, so your future state / needs will also need to be taken into account.

Do NOT penny-pinch here, but don’t buy anything that’s not appropriate. Your risk assessment process should tell you exactly what you need, and if you’ve not done one, start there.