I have been exploring further the possible response actions an application might make once it has detected a suspected or actual attack, as a contribution to the OWASP AppSensor project. There is now a draft document describing response actions, discussed and announced last week.

The draft document AppSensor - Response Actions describes thirteen response actions, provides examples of each, and discusses how they might be categorised in order to help with selection of appropriate responses.

It is still a working document. If you have any suggestions or comments on the draft document, please send them to the AppSensor project's mailing list, or perhaps add them below. In particular, I'd like to discuss whether there are any other responses which aren't covered by the ones already included.

There is additional background information and links relating to web application intrusion detection and the OWASP AppSensor project in my posts about presentations in Newcastle and London, but I hope to present again later in the year.

Software licensing may not be high on your agenda once a web site is operational. But software licences are an important part of ensuring your web site does not infringe any laws, regulations and contracts.

From 15:30 hrs today, train services from a number of operating companies including East Coast, First Capital Connect and Grand Central are being affected by a line-side fire involving acetylene cylinders near Grantham. This has led to cancellations and delays. But curiously an hour ago, the East Coast web site was showing something a little unexpected—only the words "LicenseException: License has expired." were being displayed:

Ooops. It is slightly odd that the web site issue is occurring at the same time as the fire—I wonder if it is due to a licence limit being reached caused by high demand from customers checking the status of their trains, or trying to make alternative arrangements. The wording "expiry" suggests it is simply time related, but it does seem a bit of a coincidence.

Doing a quick search for this error message suggests many other web sites have sent this response in the content whilst being indexed:

So that seems unexpectedly common. Interestingly, some of the sites seem to be development or staging sites (e.g. using just an IP address, or using a "staging." sub-domain). These might well have been using temporary licences, but why are search engines allowed access at all, and even if they are, why isn't the robots exclusion standard for compliant crawlers being used?

Apart from the legal aspects, commercial software licences need to be acquired to allow for the total number of installations, processors, usage (e.g. bandwidth) and concurrent users (however the licence is defined) for:

• peak stress loads allowed to reach the web, application and database servers
• supporting systems
• development, testing, staging and production environments
• clusterering, failover and disaster recovery.

Licensing of all components and third-party services (e.g. data providers, hosting) also need to be considered. Don't just cross your fingers and hope for the best! All types of licence, commercial or otherwise, need to comply fully with their terms (e.g. non-commercial use, one licence per server). Check what happens when licences expire or if limits are exceeded. The situation might occur when most eyes are looking at your organisation.

A lesser related issue is that your own site may be masking the server type quite well, but an error message like this can give the game away. Even if the message doesn't state the type of web server and operating system, another web site with the same message may provide the answer. This can help a malicious user who is probing the site for vulnerabilities.

Shortly afterwards, the normal East Coast Trains web site had returned; much sooner than you would expect if it needed a new licence agreed, purchased and installed. I'm still wondering if it was too many simultaneous users.

I'm hoping the fire is sorted soon so I can travel tomorrow morning, instead of this evening as originally planned.

The UK Centre for the Protection of National Infrastructure (CPNI) have published new guidance on understanding and managing the risks from phishing and pharming.

Whilst most readers of this blog won't work on projects considered part of the national infrastructure, that doesn't mean you should ignore good, free advice.

The CPNI document discusses the threats and impacts (on employees, customers, clients and citizens), the modes of attack and possible countermeasures. I'm pleased to see that countermeasures to reduce the likelihood of successful attacks include both technical and cultural measures. Measures to mitigate the effects of successful attacks are also discussed.

Although some of the document is necessarily technical in places, the case studies in Appendix C should make sense to everyone. Remember, this is about business risk, not technical risk. The "I don't understand technical things" argument does not stand up.

Of course, assessing and implementing information security policies and controls is hardly ever simple or quick. But with the government's aim to reduce the number of different web sites this process may be a little easier. It's good to see such guidance, especially when the Central Office of Information (COI) has to date avoided the subject of security in its own web standards and guidelines. In view of the perception that the government isn't keeping up with threats (for example see the response to the petition to upgrade away from Internet Explorer 6), how are the CPNI phishing and pharming countermeasures being implemented by the government?

Knowledge about the degree to which the cultural countermeasures have been adopted within the government sector cannot be adequately measured from outside, and it would be good to see these included in work performed by the National Audit Office. Similarly most of the technical countermeasures would require privileged access to government networks (and permission!). However "use of SSL and TLS" and "signing of digital communications" should be easily observable, without doing any testing, from the outside world.

These two measures have security benefits beyond protection against phishing and pharming. They can assist citizens wanting to verify the identity of, and rely on the integrity of the information they see on what looks like a government web site, or receive in an official-looking email or other form of correspondence, perhaps during a national emergency. These types of event can attract themed phishing attacks for example. I haven't received any official government electronic communications recently apart from reminders from HMRC about tax deadlines and the like, so can't comment on how the sender and data integrity is verified. The tax reminders don't contain any sensitive data, and occur when there are known forthcoming business events or relate to actions undertaken by myself, so correctly don't need the same degree of verification.

But anyone can visit a web site, so what about those? Well, the CPNI web site appears to also be available over SSL/TLS as we'd expect. But, looking at https://www.direct.gov.uk using SSL (now more correctly called transport layer security, TLS) in the Chrome web browser, I was a bit surprised to see:

and this is the same for the prime minister's web site at https://www.number10.gov.uk/. Another possible primary governmental address is https://www.hmg.gov.uk which gives:

Maybe these have been deemed to be acceptable risks. But let's hope the other recommended countermeasures have been implemented.

Two great papers on web site password security were published this month. We are swamped with passwords and every daily activity is increasingly linked with an online version, which requires users to register to obtain some additional benefits. Every organisation, resource, activity and event encourages us to visit their own website and sign-up.

Firstly, in Where Do [Password] Security Policies Come From?, Dinei Florêncio and Cormac Herley of Microsoft Research discuss the password policies of 75 different web sites, in an effort to determine password strength requirements with other aspects such as size of site, assets protected, number of users and frequency of attacks.

The authors' findings suggest that none of these are the key factors, and in fact some of the largest sites, most attacked and with higher-value assets have the weakest password policies. The authors suggest stronger policies exist where organisations are more insulated from the consequences of poor usability, whereas online retailers and sites that rely on advertising revenues have to compete rigorously for users and traffic. The paper also discusses how strong passwords need to be, and how this is affected also by what attack methods you are considering (e.g. online vs. offline brute-force), and whether other security controls are implemented (e.g. account lock-out).

This idea of considering the whole password environment is taken further in The Password Thicket: Technical and Market Failures in Human Authentication on the Web by Joseph Bonneau and Sören Preibusch at the Cambridge University Computing Laboratory, and presented at this year's Economics of Information Security (WEIS 2010). Their study included 150 web sites looking at password implementations. the study looked more broadly at the protective measures used— not just complexity requirements—but whether these were applied consistently across the site's functionality (e.g. registration/enrolment, log-in/authentication, password change, password reset/recovery, log-out), encryption during transmission, storage of passwords in clear text, inclusion of passwords in emails, as well as protection from brute-force attacks.

The authors found that stricter security in one area was often undermined by weaknesses in another, suggesting that a lack of standards is harming security. The paper also discusses economic interpretations, such as how deploying passwords might be being used to justify collection of marketing data, and how password insecurity can be a negative externality.

Knowledge of application context is used routinely in mobile applications—for example sensing a user's context (e.g. location and physical actions, time, etc), reducing network usage during periods of inactivity and designing for users. But how does this idea transfer to the server?

I almost called this environmental awareness, but didn't want to cause confusion with discussions about network/server environments. By 'situational awareness' I mean awareness of factors external to the application that might be used to affect its behaviour. In my talk this week about application intrusion detection, I will be discussing how an aspect such as the general risk level to an organisation/application might be used to alter an application's actions (e.g. amount of logging, attack detection thresholds). But this awareness, can be used beyond attacker detection and response.

Information is knowledge and additional awareness of external factors can be used to control changes to the application. An adaptive application can learn change in response to outside factors. And no, I don't mean displaying an intrusive and annoying paperclip that says "It looks like you're writing a letter". Apart from standard functionality the user sees, some ways your application may already be doing this are:

• customising content based on:
  • geo-location information
  • user preferences
  • device type (e.g. mobile), browser and screen resolution
  • typical user behaviour
• implementation of additional delays for failed attempts at authentication
• use of reputation-based systems
• displaying the number/identities of active/logged-in users
• detecting usage of the application by users from a different location than they had used previously (e.g. IP address)
• showing advertising based on users' behavioural characteristics.

But what else can be done? I remember chatting with someone during an unexpected period of severe weather which had disrupted travel in south-east England one morning. They had explained that in situations like this when their call centre was under staffed, they had procedures in place to reduce the length of each customer call, by shortening their own scripts taking out offers for helping with anything else and cross-selling/up-selling. The dialogue script was adapted to the situation. A web application could respond in a similar way during increasing, and higher periods of demand, to increase availability:

• switch to more static content (e.g. change the home page to static HTML rather than a scripted dynamic page)
• swap to lower bandwidth assets (e.g. display photographs instead of videos, use lower resolution photos)
• use third-party servers for some content (e.g. video on YouTube)
• reduce the size of pages and number of page elements by dropping out non-core material (e.g. promotional items, banners)
• increase caching
• delay non-core server intensive activities (e.g. management report generation)
• provide links to printable forms to divert some or all users of a particular online service.

Similarly, if a local (e.g. dynamic PDF creation or chart generation), back-office (e.g. data archive) or third-party service (e.g. payment authorisation, address look-up) is detected as running slowly or has become unavailable, some of the following may be possible:

• switch to cached data
• add a queue to access the function
• slow down the speed at which users can undertake the function
• offer alternative (quicker) ways to complete the transaction
• take the service offline, but offer to email users back when it is available again.

Similar changes could occur in advance of, or during, known scheduled application maintenance periods:

• advanced warning notices to users
• timed count-down to function or application shutdown
• preventing users beginning new tasks which might not be able to be completed before the shutdown
• ability for users to request notification that the service is back up.

The important thing (remember "clippy") is not to change the user experience too noticeably, and where there is a significant change (e.g. download the form instead of doing it online), provide a time-stamped explanation of the change and reasons.

These measures all bring complexity, and it is important they do not introduce additional vulnerabilities to the application. The problems are quite likely to be in authentication, authorisation and session management and need to be identified during security specification and verification processes. The effect on data integrity, including accuracy, also needs to be considered. But the measures are worth considering where the alternative is additional standby staff and increased usage of other channels.

Fed up with false positives when trying to detect malicious users with network intrusion detection systems (IDS)? Application intrusion detection is the way to go.

Like an advanced robot, applications can build in security protection, detection and response.

Next Thursday 15th July 2010, I will be presenting "Real Time Application Attack Detection and Response" at the next OWASP meeting in London. Like all OWASP chapter meetings, the event is free but prior registration is required.

I will talk about how advanced attackers probe and try to exploit applications, how some common defences against these attacks are of no use, and why we need to use protection that:

• understands the application
• understands normal vs. suspicious use
• can identify and shut down attackers in real time.

Is this possible? Yes. AppSensor specifies how application-based detection points can be used to stop attackers. I will also describe how project leader Michael Coates has demonstrated how real web sites can deploy such measures in practice to protect an application against automated scanners, advanced attackers and build in protection against application worms.

Arrive from 17:30 hrs since the talks start promptly at 18:00. Hope to see you there.

Yesterday, the UK Information Commissioner's Office (ICO) launched their Personal Information Online Code of Practice.

The new code is available online as an eBook together with associated guidance for individuals Protecting Your Personal Information Online. Hopefully the code will also be available as a standalone PDF for offline use and in print.

The Personal Information Online Code of Practice has been improved substantially since the draft for consultation was issued in December. The code describes the benefits of protecting personal information including increased trust, reduced reputational risk, better take-up of services, reduced risk of data breaches and associated enforcement action, improved competitive advantage, increased quality of data and decreased customer/client/citizen support costs.

I am pleased to see so many practical tips tied to real-world examples such as whether IP addresses are personal data (answer: probably). It is difficult to get the balance of detail and readability correct, but I think this document will hit the mark for many busy web site owners.

The code points to other matters that should be considered (e.g. risk assessments), but correctly doesn't details precisely how these are undertaken.

Update 9th July 2010: The Personal Information Online Code of Practice is now available both as a PDF and in print on request.

The US trade organisation Interactive Advertising Bureau (IAB) has released guidance for advert networks and advert exchanges to standardise methods to make buying easier and to give increased control over where adverts are placed.

Networks & Exchanges Quality Assurance Guidelines includes a detailed glossary of online advertising terms which provides a common vocabulary for advertisement targetting and data collection. The document provides detailed guidance on:

• transparency of inventory sources, publisher relationships, content types and placement details
• defined content categorisation based on 23 main "tier 1" taxonomy tiers
• vetting of the inventory based on a rating system and description of web page content
• data disclosure terms for off-site behavioral targeting and third-party data.

US advert networks and exchanges can voluntarily agree to be certified against these guidelines.

In case you missed it, the IAB and Network Advertising Initiative (NAI) jointly published the CLEAR Ad Notice Technical Specification which defines how to implement the cross-industry Self-Regulatory Principles for Online Behavioral Advertising. This provides a method for advertisers to provide additional information (meta data) with an advert which users can read, and choose whether to opt out. It will be interesting to see how the guidance is implemented in practice—there is an example demonstration advert on the Yahoo! Green web site.

The equivalent UK organisation in the IAB UK.

Sometimes when I'm out socially and people ask what I do, the conversation progresses to concerns about their own web site. They may have a hobby site, run a micro-business or be a manager or director of a small and medium-sized enterprise (SME)—there's all sorts of great entrepreneurial activity going on.

It is very common for SMEs not to have much time or budget for information security, and the available information can be poor or inappropriate (ISSA-UK, under the guidance of their Director of Research David Lacey, is trying to improve this). But what can SMEs do about their web presence—and it is very unusual not to have a web site, whatever the size of business.

Last week I was asked "Is using <company> okay for taking online payments?" and then "what else should I be doing?". Remember we are discussing protection of the SME's own web site, not protecting its employees from using other sites. If I had no information about the business or any existing web security issues, this is what I recommend checking and doing before anything else:

• Obtain regular backup copies of all data that changes (e.g. databases, logs, uploaded files) and store these securely somewhere other than the host servers. This may typically be daily, but the frequency should be selected based on how often data changes and how much data the SME might be prepared to lose in the event of total server failure.
  • check backup data can read and restored periodically
  • don't forget to securely delete data from old backups when they are no longer required
• Use a network firewall in front of the web site to limit public (unauthenticated user) access to those ports necessary to access the web site. If other services are required remotely, use the firewall to limit from where (e.g. IP addresses) these can be used.
  • keep a record of the firewall configuration up-to-date
  • limit who can make changes to the firewall
• Ensure the host servers are fully patched (e.g. operating system, services, applications and supporting code), check all providers for software updates regularly and allow time for installing these.
  • remove or disable all unnecessary services and other software
  • delete old, unused and backup files from the host servers
• Identify all accounts (log in credentials) that provide server access (not just normal web page access), such as used for transferring files, accessing administrative interfaces (e.g. CMS admin, database and server management/configuration control panels) and using remote desktop. Change the passwords. Keep a record of who has access and remove accounts that are no longer required and enable logging for all access using these accounts.
  • restrict what each account can do as much as possible
  • add restrictions to the use of these accounts (e.g. limit access by IP address, require written approval for use, keep account disabled by default)
• Check that every agreement with third parties that are required to operate the web site are in the organisation's own name. These may include the registration of domain names, SSL certificates, hosting contracts, monitoring services, data feeds, affiliate marketing agreements and service providers such as for address look-up, credit checks and making online payments.
  • ensure the third parties have the organisation's official contact details, and not those of an employee or of the site's developers
  • make note of any renewal dates
• Obtain a copy of everything required for the web site including scripts, static files, configuration settings, source code, account details and encryption keys. Keep this updated with changes as they are made.
  • verify who legally owns the source code, designs, database, photographs, etc.
  • check what other licences affect the web site (e.g. use of open source and proprietary software libraries, database use limitations).

Do what you can, when you can. Once those are done, then:

• Verify the web site and all its components (e.g. web widgets and other third party code/content) does not include common web application vulnerabilities that can be exploited by attackers (e.g. SQL injection, cross-site scripting).
  • use resources such as the OWASP Top Ten and WASC Threat Classification
  • build security requirements into future developments
• Check what obligations the organisation is under to protect business and other people's data such as the Data Protection Act, guidance from regulators, trade organisation rules, agreements with customers and other contracts (e.g. PCI DSS via the acquiring bank).
  • impose security standards and obligations on suppliers and partner organisations
  • keep an eye open for changes to business processes that affect data
• Document (even just some short notes) the steps to rebuild the web site somewhere else, and to transfer all the data and business processes to the new site.
  • include configuration details and information about third-party services required
  • think about what else will need to be done if the web site is unavailable (does it matter, if so what exactly is important?)
• Provide information to the web site's users how to help protect themselves and their data.
  • point them to relevant help such as from GetSafeOnline, CardWatch and Think U Know
  • provide easy methods for them to contact the organisation if they think there is a security or privacy problem
• Monitor web site usage behaviour (e.g. click-through rate, session duration, shopping cart abandonment rate, conversion rate), performance (e.g. uptime, response times) and reputation (e.g. malware, phishing, suspicious applications, malicious links) to gather trend data and identify unusual activity.
  • web server logs are a start, but customised logging is better
  • use reputable online tools (some of which are free) to help.

That's just the basics. So, what would be next for an SME? If the web site is a significant sales/engagement channel, the organisation has multiple web sites, is in a more regulated sector or one that is targetted particularly by criminals (e.g. gaming, betting and financial), takes payments or does other electronic commerce, allows users to add their own content or processes data for someone else, the above is just the start. Those SMEs probably need to be more proactive.

This helps to protect the SME's business information, but also helps to protect the web site users and their information. After all, the users are existing and potential customers, clients and citizens.

Oh, the best response I had to someone when I was explaining my work: "You're an anti-hacker than?". Well, I suppose so, but it's not quite how I'd describe it.

Any comments or suggestions?

Last night, after the first day of the OWASP AppSec Research 2010 conference, we had the pleasure of attending the conference gala dinner at the lavishly decorated Stockholm City Hall, also used for the annual Nobel Prize award ceremony.

Steve Lipner (Microsoft) gave the keynote speech today. He described the early step, creation and evolution of Microsoft's Security Development Lifecycle (SDL). This began in early 2002 which included team-wide security training, the introduction of early threat modelling, code review, use of some tools, undertaking security testing and modifying software defaults to make them more secure. These were seen as quick wins but were immature and ad-hoc processes. They then worked on the security "science" and "security audit" to build a more robust and repeatable program leading to the first edition of the SDL in 2004. It is regularly reviewed and updated and version 5.0 was released this year and 5.1 is due in October 2010. Whilst the SDL is based on Microsoft's own experiences and culture, he said it can be applied to non-Windows development, it does not rely on Windows tools and is not just for shrink-wrapped software development. Neither is it only suitable for waterfall or spiral development methodologies; the application of SDL to agile processes has been described recently. But the most important point he made is that SDL at Microsoft is not necessarily what will work in other software development teams—it is a very helpful starting point, but requires commitment and time to create processes and apply these consistently.

Immediately following the keynote speech, Pravir Chandra (Fortify and OWASP SAMM Project Leader) outlined the Software Assurance Maturity Model (SAMM) and lessons learned in its application to real software development programs. He emphasised the need to identify and classify all applications by risk, to determine what security activities are undertaken. He described that the argument for secure software development must be a business argument based on risk, that it has a real return on investment (ROI), and starting with a single development process and enhancing that can be a good way to introduce secure development practices. The activities undertaken need to be mapped to preventative, detective and corrective controls, and that the tasks need to specify roles, responsibilities and mappings to process flows. Also, he said that security knowledge needs to be spread widely with champions and experts, not just kept by a single specialist or group. He believes SAMM has a large proportion of overlap with Microsoft SDL and BSIMM, and is in the process of mapping SAMM's activities to the latter.

David Rajchenbach-Teller (MLState) described a new programming language for web applications called OPA. It has been designed from a clean start to avoid legacy concepts from the 1970s and 80s and is based on formal methods, is safe from the bottom up, using a single language for the whole application and is based on the distributed system model where not all principals are trusted, communications use web standards and security is mostly automatic. He showed some example code and described real applications in use today. He then described how it prevents a number of issues in the OWASP Top Ten 2010 but that is still under development, and for example, they are working on cross-site request forgery (CSRF) prevention mechanisms and extending the security policy feature set.

Cassio Goldschmidt (Symantec and SAFECode) presented an engaging explanation of how we are all responsible to a certain extent for the creation of software flaws. Whilst software manufacturers may be increasingly applying secure development practices, software is very complex, there are multiple layers of software on top of software and there is no effective way to prove software correctness. Adopters (e.g. home and corporate users) desire feature-rich software and security is not always visible. The environment affects purchasing decisions and home users in particular may not keep software patched. He said purchasing decisions in corporate entities may be made by different people than the users leading to a disconnect, and even patching can be delayed due to corporate cycles. Security researchers also have a part to play where the motivation and consequences of actions are not always transparent. Similarly governments find it difficult to make good law and the timescales cannot keep up with the fast pace of developments. They may provide incentives or require higher standards, but these can be blunt instruments. In summary he proposed that economics plays a larger part than technical solutions to the risks and impacts, even thought industry is moving in the right direction.

During and after lunch, OWASP board members and leaders discussed opportunities, issues and proposals to assist end-users find organisations who are providing products and services based on OWASP's knowledgebase.

Nick Nikiforakis (KU Leuven) described their analysis of eight file sharing services that are cloud-based, provide "one-click hosting" and are mostly anonymous. They found that although the services tended to offer both private distribution (e.g. by email link or instant messaging) and public distribution (e.g. links added to forums, blogs, etc) most of the services were relying on obscurity through obscurity. In many cases the URL token was predicable and even if the source filename was included, this was often not required. Given the predictability of tokens, they were able to obtain details of many different files on the file sharing systems, and tried to identify which were of the private or public type by an examination of whether the source filename could be found elsewhere using Yahoo. The remaining non-binary types were downloaded and examined to find a wide variety of data including bank statements, company budgets & salaries, personal data, documents with admin credentials, doctors notes and even a death certificate. Their advice, choose file sharing systems that have unpredictable tokens, encrypt the files and remove from the store as soon as possible.

The conference closed with thanks being given to the organisers, Kate Hartmann (OWASP Operations Director), OWASP board, helpers from the university, the sponsors, the sound and video teams, the caterers and the attendees. Prizes from various sponsor competitions and the capture the flag event were given. John Wilander reminded attendees about the upcoming AppSec US 2010 in September and announced that next year's AppSec EU would be help in Trinity College, Dublin, Ireland, and in Athens the year after.

Congratulations to the team from Sweden, Norway and Denmark for such a well-organised, and excellent appsec conference!