Fed up with false positives when trying to detect malicious users with network intrusion detection systems (IDS)? Application intrusion detection is the way to go.

Like an advanced robot, applications can build in security protection, detection and response.

Next Thursday 15th July 2010, I will be presenting "Real Time Application Attack Detection and Response" at the next OWASP meeting in London. Like all OWASP chapter meetings, the event is free but prior registration is required.

I will talk about how advanced attackers probe and try to exploit applications, how some common defences against these attacks are of no use, and why we need to use protection that:

• understands the application
• understands normal vs. suspicious use
• can identify and shut down attackers in real time.

Is this possible? Yes. AppSensor specifies how application-based detection points can be used to stop attackers. I will also describe how project leader Michael Coates has demonstrated how real web sites can deploy such measures in practice to protect an application against automated scanners, advanced attackers and build in protection against application worms.

Arrive from 17:30 hrs since the talks start promptly at 18:00. Hope to see you there.

Sometimes when I'm out socially and people ask what I do, the conversation progresses to concerns about their own web site. They may have a hobby site, run a micro-business or be a manager or director of a small and medium-sized enterprise (SME)—there's all sorts of great entrepreneurial activity going on.

It is very common for SMEs not to have much time or budget for information security, and the available information can be poor or inappropriate (ISSA-UK, under the guidance of their Director of Research David Lacey, is trying to improve this). But what can SMEs do about their web presence—and it is very unusual not to have a web site, whatever the size of business.

Last week I was asked "Is using <company> okay for taking online payments?" and then "what else should I be doing?". Remember we are discussing protection of the SME's own web site, not protecting its employees from using other sites. If I had no information about the business or any existing web security issues, this is what I recommend checking and doing before anything else:

• Obtain regular backup copies of all data that changes (e.g. databases, logs, uploaded files) and store these securely somewhere other than the host servers. This may typically be daily, but the frequency should be selected based on how often data changes and how much data the SME might be prepared to lose in the event of total server failure.
  • check backup data can read and restored periodically
  • don't forget to securely delete data from old backups when they are no longer required
• Use a network firewall in front of the web site to limit public (unauthenticated user) access to those ports necessary to access the web site. If other services are required remotely, use the firewall to limit from where (e.g. IP addresses) these can be used.
  • keep a record of the firewall configuration up-to-date
  • limit who can make changes to the firewall
• Ensure the host servers are fully patched (e.g. operating system, services, applications and supporting code), check all providers for software updates regularly and allow time for installing these.
  • remove or disable all unnecessary services and other software
  • delete old, unused and backup files from the host servers
• Identify all accounts (log in credentials) that provide server access (not just normal web page access), such as used for transferring files, accessing administrative interfaces (e.g. CMS admin, database and server management/configuration control panels) and using remote desktop. Change the passwords. Keep a record of who has access and remove accounts that are no longer required and enable logging for all access using these accounts.
  • restrict what each account can do as much as possible
  • add restrictions to the use of these accounts (e.g. limit access by IP address, require written approval for use, keep account disabled by default)
• Check that every agreement with third parties that are required to operate the web site are in the organisation's own name. These may include the registration of domain names, SSL certificates, hosting contracts, monitoring services, data feeds, affiliate marketing agreements and service providers such as for address look-up, credit checks and making online payments.
  • ensure the third parties have the organisation's official contact details, and not those of an employee or of the site's developers
  • make note of any renewal dates
• Obtain a copy of everything required for the web site including scripts, static files, configuration settings, source code, account details and encryption keys. Keep this updated with changes as they are made.
  • verify who legally owns the source code, designs, database, photographs, etc.
  • check what other licences affect the web site (e.g. use of open source and proprietary software libraries, database use limitations).

Do what you can, when you can. Once those are done, then:

• Verify the web site and all its components (e.g. web widgets and other third party code/content) does not include common web application vulnerabilities that can be exploited by attackers (e.g. SQL injection, cross-site scripting).
  • use resources such as the OWASP Top Ten and WASC Threat Classification
  • build security requirements into future developments
• Check what obligations the organisation is under to protect business and other people's data such as the Data Protection Act, guidance from regulators, trade organisation rules, agreements with customers and other contracts (e.g. PCI DSS via the acquiring bank).
  • impose security standards and obligations on suppliers and partner organisations
  • keep an eye open for changes to business processes that affect data
• Document (even just some short notes) the steps to rebuild the web site somewhere else, and to transfer all the data and business processes to the new site.
  • include configuration details and information about third-party services required
  • think about what else will need to be done if the web site is unavailable (does it matter, if so what exactly is important?)
• Provide information to the web site's users how to help protect themselves and their data.
  • point them to relevant help such as from GetSafeOnline, CardWatch and Think U Know
  • provide easy methods for them to contact the organisation if they think there is a security or privacy problem
• Monitor web site usage behaviour (e.g. click-through rate, session duration, shopping cart abandonment rate, conversion rate), performance (e.g. uptime, response times) and reputation (e.g. malware, phishing, suspicious applications, malicious links) to gather trend data and identify unusual activity.
  • web server logs are a start, but customised logging is better
  • use reputable online tools (some of which are free) to help.

That's just the basics. So, what would be next for an SME? If the web site is a significant sales/engagement channel, the organisation has multiple web sites, is in a more regulated sector or one that is targetted particularly by criminals (e.g. gaming, betting and financial), takes payments or does other electronic commerce, allows users to add their own content or processes data for someone else, the above is just the start. Those SMEs probably need to be more proactive.

This helps to protect the SME's business information, but also helps to protect the web site users and their information. After all, the users are existing and potential customers, clients and citizens.

Oh, the best response I had to someone when I was explaining my work: "You're an anti-hacker than?". Well, I suppose so, but it's not quite how I'd describe it.

Any comments or suggestions?

Security and audit logging should be defined, implemented and tested for every web application. But what about log management and analysis?

This week Raffael Marty posted an updated item to his blog about a Maturity Scale for Log Management and Analysis. It is an excellent review.

Whilst much of this management and analysis is intended to be external to an application, we need to remember each application needs to record adequate information to feed into these analysis and reporting tools. And why do that? Read the bullet points under return on investment (ROI) at the end of the article. What else? Well perhaps also:

• feedback into the development lifecycle (to improve subsequent patches, versions and other projects)
• greater trust by users
• brand protection
• protection of information assets (not just preventing leaks, but ensuring accuracy and integrity).

Therefore, build adequate logging in from the start. Web server logs are not enough!

Almost every sale, citizen enquiry and support request now seems to lead to being asked to complete an online customer survey. Almost without exception, the user experience, privacy and security of these online customer surveys are worse than the service being asked to comment on. Here are a couple of customer surveys I was asked to complete last week.

Problems with using SSL, such as shown above, do occur but more often than not people are asked to submit personal identifiable information and other forms of personal data without the use of SSL. Bad layout, poorly designed questions, missing privacy notices and improper validation are extremely common. Many forms have mis-configured web sites that give away sensitive information about how the site and server are set up when they don't work:

And on Saturday I was given a coupon given at a shop's physical checkout to provide feedback on how they did with the chance of winning an iPod or cash for doing so. Yesterday I typed in the URL from the coupon, entered the required store code, and... that was the end of that marketing exercise:

I didn't time out as the message suggested, unless you have less than 5s to answer one question. Perhaps there is only one custom error page for all server-side errors, or the wrong error page is assigned? Points for hiding internal error messages, but still a failure.

Is 3/3 customer surveys tried in the last fortnight broken just bad luck? Or does it indicate a poor standard of such efforts? One of these is an international consumer brand, another a major UK High Street retailer and the other, a medium-sized business services company. I can't quite remember the the previous customer survey to these three, but I think it may have been a salary/skill survey. That had poorly thought-out questions and although it didn't obviously fail, it did ask me to log in on submission. So I'm not sure if that meant my efforts had been saved or not.

Do all these problems and errors mean the data from other people's forms that were successfully submitted (if any) are less valid? I can imagine management decisions are being made as a result of the survey feedback (if not, why waste everyone's time?). What is the effect on data quality? It could be that some forms fail when a particular answer is selected or left blank—this could be important marketing knowledge, and if no responses include the particular option it may be incorrectly assumed no-one selected it. The management decisions will be being based on poor data.

Perhaps part of the problem is that customer surveys are often managed, operated and hosted by third parties due to the ease of implementation. But "easy" doesn't mean it meets your own organisation's standards, or general good practice. You are still accountable for the web issues and it's your organisation's reputation that will be affected detrimentally.

Good design, privacy and security impact assessments, thorough testing and verification are required like any other other addition to a web site. Analytics should be used to track survey users through forms and this data combined with server logs of access and errors generated by the web server. Prove your marketing data are valid before you use it in business decisions.

It's encouraging to see commercial information security organisations sharing their experience, knowledge and data, such as in the OWASP Security Spending Benchmarks Project. Last week Verizon also published details of its Incident Sharing Framework.

The framework (beta, 1st March 2010) is used in Verizon's internal security metrics gathering processes and to produce its public data breach investigation reports. The framework provides details of how various security incidents should be classified and recorded, including what is done to remedy the situation and further actions taken, such as education. By publishing the framework, Verizon hope that other organisations might collect similar data and ultimately share it to improve common knowledge.

Some other related frameworks and initiatives are listed at:

• Security Content Automation Protocol (SCAP), NIST, specifications and emerging specifications
• Consensus Information Security Metrics, CIS (see previous post on Web Application Security Metrics

These frameworks may be too complex to consider for some organisations, but even so, they provide a good guide to the kind of things you should be considering in security and privacy incident management policies and procedures. They are also useful standalone references for classifying various types of attacks and accidents.

Last month I described the ability to track users sessions with browser data. A recent posting on IT Law in Ireland highlighted a series of blog posts elsewhere that give further insight into what is possible.

Well, I just got round to reading them properly. The posts on Freedom to Tinker by David Robinson and Harlan Yu are:

• Identifying John Doe: It might be easier than you think
• What Third Parties Know About John Doe
• The Traceability of an Anonymous Online Comment

The conclusion? It is possible to trace and identify individuals easier than you may think. We are dropping evidence like dead skin cells as we traverse the internet. Fact or fiction? Well the US Defense Advanced Research Projects Agency (DARPA) are taking it seriously with a recent call for research into cyber genetics, cyber anthropology and cyber physiology in its Cyber Genome Program. DARPA hopes to develop advanced methods to fingerprint or identify the origins of a cyber attacks by examining digital artifacts, and presumably other criminal activities utilising computer technology.

Getting a bit more down to earth, web site owners need to consider what information is being gathered and why, ensure this is legal, check that consent is implied or has been explicitly given for the purposes and what monitoring and analysis is performed on the data. It could be easy for system developers to carried away with tracking and tagging. Contracts with third parties should state clearly what the expectations are about the security and privacy of information, to protect web site users (employees, customers, clients, citizens) and the business.

This weekend the clocks change as we revert from British Summer Time (BST) to Greenwich Mean Time (GMT) at 02:00 BST on Sunday 25 October 2009 and the clocks go back, giving an extra hour.

What does this mean for web site security? Does running 01:00 to 02:00 twice matter? Well some brave web application owners will be disabling their systems like this online bank:

And, I don't think it's just being done as a finale to the current Energy Saving Week. Most people, quite rightly, won't be taking this rather severe step. Another millennium bug anyone? The date/time should be considered rather like other untrusted user input. Most problems will probably fall into the "business logic" category such as:

• Failure of time-based logic where dates are being compared.
• Assumptions of uniqueness in time-stamped output (e.g. by a single-threaded process).
• Running tasks again leading to possible:
  • loss of data due to overwriting
  • duplication of exports or emails
  • creation of inaccuracies in management information.
• Chronological ordering anomalies leading to other faults.

It's not just banks and other financial organisations that may have difficulties.

The time change may expose some other vulnerabilities that only exist at changeover time and/or during the next overlap hour.

• Circumvention of brute force attacks on user authentication mechanisms.
• Increased risk due to extension of a session's validity where local time is recorded.
• Failure in data validation routines for time-related comparisons.
• Incubated vulnerabilities where a time-related aspect causes the attack to be possible.
• Denial of service due to extension of account lock-out.
• Using time as a loop counter.
• Additional errors caused by any of the above leading to information leakage.

Recording the offset of local time to GMT/UTC and synchronisation should certainly be done, but may not resolve the time overlap issues. The effects on long-running "saga" requests might be especially difficult to determine. Time dependencies need to be specified and considered through the development lifecycle. Perhaps the bank is right after all?

Earlier this year, the Center for Internet Security (CIS) published Consensus Security Metrics to allow organisations to collect, analyse and share data on security performance and outcomes. These are based on the consensus viewpoint of 100 experts.

I've just had a chance to read the whole document and I'm impressed. The document includes twenty consensus metrics definitions for six business functions:

• Incident management
• Vulnerability management
• Patch management
• Application security
• Configuration management
• Financial metrics

Additional metrics for these and other business functions are in development.

The metrics are an excellent reference document and are carefully explained, referenced and excellently presented. These should be of interest to people owning, operating or developing web applications and looking for measures to examine performance, regardless of their role or experience. If you are looking for some metrics, don't re-invent the wheel, read this document first.

It's common for access to some web sites to be restricted to users from particular Internet Protocol (IP) addresses. This is usually in addition to some other identification and authentication method. But other IP addresses are often added to this "allow list" and these should not necessarily be trusted in the same way.

In a typical scenario, a web site hosted on the internet that is used to administer another web application might be restricted to the company's own IP addresses. Then the developers say they need to check something on the live site, or another server needs to index the content, or someone wants to work from home for a while, or the site needs to be demonstrated at a client's location. All these additional IP addresses are added to the "allow list". These restrictions may be being applied at a network firewall, traffic management system, at the web server, in the application itself, in intrusion detection systems or in log analytical software, or in many of these. These are difficult to manage and in time there will be many IP addresses that no-one knows why they are allowed unless they are carefully documented, and subject to a fixed time limit when they are confirmed again by an appropriate person or removed. These extra addresses are quite often hard for someone else to guess.

However, there is another area where IP addresses are added to "allow lists", and this is for remote monitoring and testing services. These might be checking uptime, response times, content changes, HTML validation or security testing. The service providers publish the IP addresses of the source systems so that companies can specifically allow access to their web sites. Since the number of these services is relatively small, it's not too difficult to find which one might give access to areas of a web site or web application that the public (and malicious people) should not be able to get to. The particular danger here is that the IP addresses might be excluded from monitoring and logging, and therefore even a diligent web site manager might not realise for example the uptime monitoring service is making unusual, or excessive, requests.

Although it is not likely a malicious person is using this "trusted" address unless routing has been compromised as well, problems can go undetected, from what might seem to be a legitimate source. The IP address may have been typed incorrectly, or worse, the restrictions/exceptions may not have been implemented correctly allowing more addresses to have the privileged access than intended. Not logging a user's session is privileged access.

Allow traffic through, but be very specific what is allowed and monitor what's going on. Review all the exceptions periodically. Be especially careful about anything that bypasses authentication (such as allowing a search engine to crawl restricted-access content) on an otherwise public site.

A recent proposed revision of the policy on web tracking technologies for US federal web sites by the Office of Management and Budget set out four principles regarding user analytics and tracking.

• Adhere to all existing laws and policies (including those designed to protect privacy) governing the collection, use, retention, and safeguarding of any data gathered from users.
• Post clear and conspicuous notice on the website of the use of web tracking technologies.
• Provide a clear and understandable means for a user to opt-out of being tracked.
• Not discriminate against those users who decide to opt-out, in terms of their access to information.

The document recommends avoiding outsourced tracking and outsourced data analysis—issues not thought about by many organisations. Just because a third-party service is cheap, doesn't necessarily mean it's the appropriate method to use. I'm less convinced about the example of using cookies to record opt-outs.

The proposed revision attracted a well-considered joint response from the Center for Democracy & Technology and the Electronic Frontier Foundation. They suggested three additional principles.

• Limit use of tracking data.
• Limit retention of tracking data.
• Obtain third-party verification.

The response also referenced their May 2009 Open Recommendations for the Use of Web Measurement Tools on Federal Government Web Sites which recommended the following:

• Use data only for measurement.
• Prominently disclose.
• Offer choice.
• Limit data retention.
• Limit cross-session measurement.
• Obtain third-party verification.

Whilst none of the final guidelines will be mandatory outside the US federal sector, the issues raised are worth consideration by all commercial and non-commercial web sites. For example, the recommendations and principles above could be used to help guide a privacy impact assessment of an organisation's own use of web analytics and tracking technologies.