On the morning of Thursday 29th August it became apparent that our WIKI was empty and missing all of its pages. We took it offline shortly afterwards and started to investigate why the data was missing and how we might restore it.
The previous day a team member came over to ask why they didn’t have permissions to edit the attachments they had uploaded so they could replace it with a different one. We are using Roadkill Wiki which unfortunately does not support granular permissions, you are either an editor or an admin and only admins can edit uploaded files. To unblock the user we promoted all domain users to admins which at the time was sub-optimal but didn’t seem totally unreasonable.
What happened to the data?
The data was intact and accessible that evening but missing the following morning, after checking the event logs we noticed a call to a page on the wiki that users would not usually ask for.
Request URL: http://tfs-wiki/pages/version/b81d890b-b3ae-48cd-864b-06550aee8490
Request path: /pages/version/b81d890b-b3ae-48cd-864b-06550aee8490
User host address: 10.200.95.135
Is authenticated: True
Authentication Type: Negotiate
Thread account name: NT AUTHORITY\NETWORK SERVICE
We checked with infrastructure about the SCOM account and it seemed likely to be related to the Search Server 2010 indexer. This would normally be fine and has been running for a while, however with the elevated permissions this crawler would have access to links on the wiki which it lacked before.
On the allpages page there is a full list of pages and their corresponding delete links. Unfortunately the delete links are GET requests which look like normal links.
<a class="btn btn-mini confirm" href="/pages/delete/134">Delete</a>
So there was a good chance that all these delete links were hit by the indexer as it crawled our wiki. We were able to verify this in the search indexers database. A count of 116 pages (which is all of them) were removed.
A better solution?
Promoting all domain users to administrators to get around the limitations of the permission model was a grave error, which was unfortunately compounded by the URL design on the wiki engine. Considering a request to delete an object changes server state, it should have been a POST or DELETE. This is mentioned in the HTTP protocol section 9.1.1.
In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.
In the future we will definitely be more careful when making permission changes, and I will personally be even more mindful of HTTP guidelines when designing web applications and web services. Time permitting I hope to send a patch to the Roadkill Wiki contributors.