Submitted by : simon at: 2006-08-09T22:46:05+00:00 (11 years ago)
Name :
Category : Severity : Status :
Optional subject :  
Optional comment :

Junk edits on issue pages was a frequent problem on this and other zwikis due to some of the issue forms being vulnerable to spambots and spiders in older ZWiki releases. Still to do: clean up the renamed issue pages on zwiki.org, and encourage all other public zwikis to upgrade.

issue repair

On zwiki.org: we are done with all listed issues!

needed

done
#12 #39 #47 #54 #64 #82 #121 #172 #233 #237 #300 #342 #347 #385 #396 #463 #512 #550 #557 #575 #591 #603 #604 #607 #608 #648 #651 #695 #722 #761 #765 #776 #791 #802 #809 #813 #814 #855 #876 #877 #878 #887 #902 #904 #925 #926 #929 #936 #938 #950 #952 #964 #966 #1064 #1065 #1068 #1080 #1092 #1093 #1097 #1107 #1132 #1141 #1143 #1146 #1149 #1161 #1162 #1163 #1166 #1172 #1174 #1187 #1191 #1193 #1213 #1214 #1221 #1223 #1226 #1236 #1237 #1239 #1240 #1243 #1244 #1246 #1247 #1248 #1249 #1264 #1265 #1266

Elsewhere: list any other public zwiki trackers that might need help..

... --EmmaLaurijssens, Thu, 10 Aug 2006 01:39:55 -0700 reply

I don't recall what checking is being performed when creating a new page through mail-in. Would restricting this to FrontPage subscribers help?

... --Simon Michael, Thu, 10 Aug 2006 01:50:46 -0700 reply

Only senders who are subscribed somewhere in the wiki can send mail in. I don't think these issue pages are being created by mail-in, I think it's a web script.

... --Simon Michael, Thu, 10 Aug 2006 01:51:43 -0700 reply

PS I have been setting up awstats to help with investigations. http://zwiki.org/cgi-bin/awstats.pl (and I hope it's safe to let that link be crawled..)

... --EmmaLaurijssens, Thu, 10 Aug 2006 02:30:45 -0700 reply

I think you need to look at Z2.log or your Apache log directly to find that out.

found it --simon, Thu, 10 Aug 2006 03:46:19 -0700 reply

Yes; for example, here's that recent junk issue #1279 fL03uTJxt4 being created:

62-101-75-14.ip.fastwebnet.it - - [08/Aug/2006:23:29:59 -0700] "POST /IssueTracker HTTP/1.1" 302 65084 "-" "Mozilla/5.0" zwiki.org -

I'd guess it's a script that scans pages for forms and submits them. It finds the add issue form on IssueTracker, and (aha!) this form posts to createNextIssue and createIssue which don't do any kind of permissions check.

I've now made them respect the 'edits_need_username' option like changeIssueProperties. This hopefully will stop the junk issues.

Basically you can't have any anonymously-postable form that changes state, these days.

found it --Simon Michael, Thu, 10 Aug 2006 04:08:22 -0700 reply

simon wrote: > createNextIssue and createIssue which don't do any kind of permissions check.

Correcting myself: they did check the Zwiki: add pages permission, what I meant was they didn't check the edits_need_username property which is the extra authentication trick we depend on at zwiki.org.

I've checked in the fix. Does it work ? Course not, it's too late. MaƱana.

puzzled --simon, Thu, 10 Aug 2006 13:07:30 -0700 reply

  • User or robot submits the add issue form
  • which posts back to the IssueTracker page
  • the DTML form handler at the end of the page handles this
  • by calling dtml-call "createNextIssue(...)"
  • but in that method, self does not have the proper acquisition context and can't acquire the edits_need_username property from the folder
  • so the checkSufficientId() call is neutralised, and unidentified users can still create issues

In fact, in any page DTML's _.this can't acquire things at all, right now. Look:

<dtml-var "_.this.edits_need_username"> <- AttributeError
<dtml-var "edits_need_username">        <- True, works fine

I'm surprised by this. Was it always the case ? I would have thought no. FWIW here's how such DTML code is invoked:

DTMLDocument.__call__(self,client,REQUEST,RESPONSE,**kw)

self and client are both the page object, and both of them can acquire things as you'd expect. I'm pretty sure I'm invoking this correctly and providing all the proper context.

Weird!

slightly more --simon, Fri, 11 Aug 2006 13:03:56 -0700 reply

If I read it right, it has always been this way - in DTML the current object (_.this) can't acquire things from the container. I observe the same thing in a DTML Document on the older zope at zope.org.

On the other hand, it works just fine when Zwiki renders it's built-in issuetracker template (not an in-wiki version). That template invokes the same DTML but somehow the context is different and the acquisition works.

slightly more --simon, Fri, 11 Aug 2006 15:54:17 -0700 reply

It works in the latter case because Zwiki's getSkinTemplate carefully does:

# return it with both folder and page in the acquisition context,
# setting container and here
return obj.__of__(self.folder()).__of__(self)

I have changed evaluatePreRenderedAsDtml() to do the same thing for regular embedded DTML, making sure self is acquired in the context of the folder. This seems more correct (and it's a change from regular DTML Document - perhaps a Zope bug report is needed).

Also I've refactored create/createIssue/createNextIssue/IssueTracker.dtml slightly, and now (drum roll) the add issue form respects the edits_need_username property, and with luck we'll see no more junk issues on zwiki.org for a while.

issue repair --simon, Fri, 11 Aug 2006 16:01:13 -0700 reply

Also, I scanned our 1200 issues and listed the 100 or so damaged ones above. I'm repairing these in batches every so often; hopefully the mailouts are not a problem.

These are issues whose properties were changed randomly, and most of them don't show the usual comment describing the change. I'm not sure why. They can't be reverted as I've long since packed the database. Fortunately I have the event.log for the last year, which logs the renames, and I've been able to recover all the original issue descriptions so far (and guessed the other properties).

where is the rename log --betabug, Tue, 20 Feb 2007 09:32:28 +0000 reply

During the latest DB problems the link to the rename log was lost. Could we have that one again?

found it at home on my hard disk --betabug, Tue, 20 Feb 2007 19:16:42 +0000 reply

I've found the rename log again (had it still open in a browser window), saved it to my HD and uploaded it to zwiki.org.

done cleaning up --betabug, Wed, 21 Feb 2007 09:00:48 +0000 reply

Status: open => closed

and I'm assuming that the "close holes" part is done too